Job Description

Job Title: Scala Spark Lead 15+ years of experience

Location: Broadway, New York, NY 10036 (3 days onsite per week)

Duration: 10/08/2024 - 12/31/2024 (with potential extension)

Job Role:

We are seeking a highly skilled Scala Spark Lead with over 10 years of experience to join our dynamic team. In this role, you will lead the design, development, deployment, and optimization of data processing applications using Spark and Scala.

Key Responsibilities:

Design, develop, and maintain data processing applications using Scala and Apache Spark.

Collaborate with cross-functional teams to gather data requirements and design efficient solutions.

Troubleshoot and optimize Spark applications for performance issues, ensuring optimal data processing.

Implement test-driven deployment practices to improve the reliability and quality of applications.

Deploy artifacts to higher environments, ensuring smooth transitions and production releases.

Work in an Agile environment, contributing to sprint planning, development, and timely delivery of high-quality solutions.

Provide essential support for production batches, troubleshooting issues to meet business-critical needs.

Mandatory Skills & Experience:

Big Data: 7+ years

Scala: 8+ years

Apache Spark: 7+ years

Core Java: 5+ years

10+ years of experience in software engineering, focusing on Big Data, Scala, and Apache Spark.

Strong understanding of design patterns and data structures for efficient data processing.

Familiarity with database concepts and SQL.

Proficiency in troubleshooting and debugging Spark applications for optimal performance.

Good to Have Skills:

Knowledge of Java and Snowflake.

Familiarity with Python.

Understanding of Databricks platform.

Experience with test-driven deployment practices.

Knowledge of DevOps practices.