Lead Data Engineer (Databricks & Cloud Migration)

£650 per day (Inside IR35)
Initial 6-Month Contract
London - Hybrid Working (3 Days per Week Onsite)

We are partnering with a leading international bank on a major Databricks-led cloud transformation programme. We are looking for a Lead Data Engineer with deep expertise in Databricks, modern data architectures and large-scale cloud migrations to help deliver a next-generation data platform. You'll play a pivotal role in designing and building Data Marts, developing high-performance data pipelines, and migrating critical data assets from Legacy Oracle environments into a scalable Databricks Lakehouse architecture that underpins finance, regulatory reporting and advanced analytics across the organisation.

Key ResponsibilitiesDatabricks & Modern Data Platform Engineering

• Lead the design and development of scalable Data Marts using Databricks.
• Build and optimise cloud-native data pipelines using SQL and Python.
• Drive engineering best practices across CI/CD, version control, orchestration and automated deployment.
• Ensure platform performance, scalability, governance and operational excellence.
• Champion modern data engineering standards and architectural best practices.

Oracle to Cloud Migration

• Lead the migration of data assets from Legacy Oracle environments to a modern cloud-based platform.
• Define migration strategies, source-to-target mappings and data model redesign activities.
• Oversee the retirement of Legacy ETL processes and implementation of cloud-native alternatives.
• Work closely with architects, DBAs and business stakeholders to ensure data quality, integrity and continuity throughout the migration programme.

Leadership & Stakeholder Engagement

• Provide technical leadership, mentoring and guidance to Data Engineers.
• Collaborate with Architecture, DevOps, Analytics, Finance and Risk teams.
• Communicate complex technical concepts effectively to both technical and non-technical stakeholders.
• Conduct design reviews and ensure delivery aligns with engineering standards and enterprise architecture.

Operational Excellence

• Support production releases and change management activities.
• Drive improvements in automation, monitoring, observability and platform reliability.
• Assist with incident resolution and critical release support when required.
• Ensure documentation, audit controls and governance requirements are maintained.

Required Experience

• Proven experience delivering enterprise-scale Data Engineering solutions using Databricks or similar cloud data platforms.
• Strong experience designing and developing modern data architectures and Data Marts.
• Demonstrable experience migrating Oracle or other Legacy data platforms into cloud environments.
• Deep understanding of data modelling, ELT/ETL design patterns and data integration principles.
• Advanced SQL development skills.
• Strong Python development experience.
• Experience with CI/CD pipelines, Git-based workflows and DevOps practices.
• Experience leading engineering teams and providing technical mentorship.
• Strong stakeholder management and communication skills.

Desirable Experience

• Banking or financial services experience.
• Experience within highly regulated environments.
• Knowledge of Master Data Management (MDM), metadata management and data governance frameworks.
• Experience with orchestration tools such as Azure Data Factory or Airflow.
• Azure and/or Databricks certifications.

What's on Offer

• £650 per day Inside IR35.
• Initial 6-month contract with strong potential for extension.
• Opportunity to work on a high-profile Databricks and cloud transformation programme.
• Hybrid working model with 3 days per week onsite in London.
• Collaborative environment working alongside senior technology and business stakeholders.