Lead Data Engineer

Employment Type: Permanent

Location: London - Hybrid (3 days in the office)

Salary: £95,000 - £110,000 + benefits & bonus

About the Client

Our client is a leading specialist insurance provider, recognised for its innovative approach to technology and data. Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics and business decision-making.

Role Overview

As a Lead Data Engineer, you'll play a key role in designing and evolving a modern cloud-based data platform built on Databricks and Azure. You'll lead the development of scalable data pipelines using Python and PySpark, enabling high-quality data to power underwriting, analytics and business-critical decision-making across the organisation.

Working closely with Product Owners, Architects and business stakeholders, you'll provide technical leadership throughout the full data life cycle. Alongside remaining hands-on, you'll mentor a team of Data Engineers, champion engineering best practices and help shape the future of the organisation's cloud data platform.

Key Responsibilities:

• Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.
• Design, develop and optimise scalable data pipelines using Python, PySpark and Databricks.
• Build and maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions and related services.
• Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
• Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
• Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
• Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.

Key Requirements:

• Strong commercial experience building data engineering solutions using Python, PySpark and Databricks.
• Proven experience designing and optimising complex SQL queries and working with large-scale datasets.
• Strong experience across the Azure technology stack, including Azure Functions and cloud-native data services.
• Previous experience leading or mentoring Data Engineers within an Agile environment.
• Strong understanding of distributed data processing, modern data engineering practices and CI/CD.
• Experience building scalable, reliable and high-performing data platforms.
• Experience within Financial Services or Insurance is highly advantageous.

Why Join:

• Opportunity to shape the future of a modern, cloud-native data platform built on Databricks and Azure.
• Hands-on leadership role combining technical delivery with engineering strategy.
• Work with modern technologies including Python, PySpark, Databricks and the wider Azure ecosystem.
• Collaborative engineering culture focused on innovation, quality and continuous improvement.
• Competitive salary, bonus and comprehensive benefits package.
• Hybrid working from a modern London office.

Data & Application Notice

We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision-making in our selection process.
Your application will be assessed against the requirements of this role.

By applying to this advert, you consent to receiving future relevant roles and industry news and insights from us. To opt out of this, please contact: (see below)

For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy on our website