Senior Data Engineer - Spark - Python

The available role is a senior data engineer on a team responsible for designing, implementing and supporting data pipelines on a strategic data platform. Delivery will be through close partnerships with other data engineers, data scientists and key business stakeholders to ensure excellent business outcomes.

Besides delivering business value through data pipelines, this role will also include opportunities to:

• Explore DevOps-related work to enhance our existing CI/CD environment
• Contribute to the cloud migration strategy of central Data & Analytics Platform

Role Requirements

This is a senior role so experience across technical design, implementation and support is necessary as well as being able to manage senior stakeholders effectively.

You would need to have a deep understanding of all elements of the software development life cycle as well as good development practices and principles.

From a technical perspective, we build our pipelines by making use of an open-source orchestration tool to wrap around PySpark code, so it is essential to have extensive experience of using PySpark.

Exposure to Hive is necessary, alongside standard methodology in how to craft data pipelines and store data in an efficient and performant manner.

Experience of on-premise data platforms (HDP/CDP) is required, but a broader interest and knowledge of cloud data platforms would be advantageous.

Apart from technical capabilities, this role also entails the following:

• Ability to work directly with users to understand requirements and translate them into technical design and high standard implementation
• Technically manage/mentor other members of team using management skills and technical maturity in data engineering and feedback to team leader
• Be an excellent and helpful teammate with a keen interest in problem-solving to enable you to help the team solve complex challenges

Minimum Essential Criteria

• Experience managing development teams from a technical perspective
• Ability to design pipelines/solutions while incorporating security and service considerations
• Extensive experience of delivering data pipelines on Hortonworks/Cloudera on-prem installations
• Work with testing team to devise relevant and effective testing strategy
• Extensive experience in Python and Spark
• Experience of Datamodelling
• Linux or Unix navigation skills
• Extensive experience of distributed computing
• Strong understanding of standard methodologies for development and use of source control, preferably with experience with Git
• Experience with implementing and enhancing CI/CD environments using industry standard tools, preferably Jenkins
• A passion for learning new technologies and skills

Desirable Criteria

Experience with the following:

• one or more of the main cloud providers for delivery of Data Platforms or Pipelines
• Apache Airflow and Atlas
• Ambari/Cloudera Manager
• Using R
• Using industry-wide analytical and visualisation tools (for example Tableau Server)
• Any line management experience