Python,PySpark Engineer (Azure Databricks)
Posted on Dec 27, 2020 by Connect 44 AG
Python,PySpark Engineer (Azure Databricks) -English speaking role
- You will join a large project, which is transforming our Client's Group Finance department to be cloud native. The business area handles large amounts of financial data.
- Our client's pipeline needs to support high concurrency, accuracy and low latency when preparing and serving the data to consumers.
- Our client is targeting the Microsoft Azure cloud, where they prefer Platform as a Service (PaaS). A key platform for our client is Azure Databricks, which will form the core of our data processing pipeline.
- Work closely with Product Owners and Architects to understand requirements, formulate solutions and evaluate the implementation effort
- Collaborate within a global development team to design and deliver solutions
- Assist stakeholders with data-related functional and technical issues
- Design, develop and maintain scalable data transformation pipelines
Skills and experience required:
- Strong experience Python engineer who has demonstrable experience implementing robust, reliable and successful data processing solutions using PySpark.
- Bachelor's degree level or equivalent in Computer Science or similar discipline
- At least 5 years of experience working with large scale software systems
- Proficiency in Python/PySpark (batch and streaming, Scala is a plus)
- Proficiency in SQL (Synapse/Azure DWH experience is a plus)
- Practical experience with scalable 'big data' data stores (Azure Data lake preferably)
- Practical experience working with (and securing) large data sets on enterprise data platforms and distributed computing (Spark)
- Practical experience applying automated testing and modern deployment practices on the for mentioned platforms, including Azure DevOps
- Practical experience with Scrum/Agile development methodologies
- Experience in the finance sector
- Fluent in English