Python,PySpark Engineer (Azure Databricks)

Python,PySpark Engineer (Azure Databricks) -English speaking role

Job Description

• You will join a large project, which is transforming our Client's Group Finance department to be cloud native. The business area handles large amounts of financial data.


• Our client's pipeline needs to support high concurrency, accuracy and low latency when preparing and serving the data to consumers.


• Our client is targeting the Microsoft Azure cloud, where they prefer Platform as a Service (PaaS). A key platform for our client is Azure Databricks, which will form the core of our data processing pipeline.


• Work closely with Product Owners and Architects to understand requirements, formulate solutions and evaluate the implementation effort


• Collaborate within a global development team to design and deliver solutions


• Assist stakeholders with data-related functional and technical issues


• Design, develop and maintain scalable data transformation pipelines

Skills and experience required:

• Strong experience Python engineer who has demonstrable experience implementing robust, reliable and successful data processing solutions using PySpark.


• Bachelor's degree level or equivalent in Computer Science or similar discipline


• At least 5 years of experience working with large scale software systems


• Proficiency in Python/PySpark (batch and streaming, Scala is a plus)


• Proficiency in SQL (Synapse/Azure DWH experience is a plus)


• Practical experience with scalable 'big data' data stores (Azure Data lake preferably)


• Practical experience working with (and securing) large data sets on enterprise data platforms and distributed computing (Spark)


• Practical experience applying automated testing and modern deployment practices on the for mentioned platforms, including Azure DevOps


• Practical experience with Scrum/Agile development methodologies


• Experience in the finance sector


• Fluent in English