Job Summary

We are seeking an experienced Senior Azure Databricks Data Engineer to join our team and play a key role in the development and enhancement of a modern Azure-based data platform supporting economic analysis, forecasting, modelling, and Business Intelligence initiatives.

The successful candidate will be responsible for designing, developing, and optimising scalable data pipelines, implementing complex data transformations, and ensuring high standards of data quality, governance, and performance across the Azure ecosystem.

This is an excellent opportunity for a highly skilled data engineering professional with strong Azure Databricks expertise and a passion for delivering reliable, scalable, and governed data solutions.

Note: Must have active SC clearance

Key Responsibilities

Data Engineering & Pipeline Development

• Design, build, and maintain scalable ETL/ELT data pipelines using Azure Databricks and Azure Data Factory.
• Develop robust data ingestion frameworks for structured, semi-structured, and streaming data sources.
• Optimise data processing workflows for performance, scalability, and cost efficiency.
• Implement data validation, monitoring, and quality assurance processes.

Azure Databricks Development

• Develop data transformation logic using PySpark, Spark SQL, SQL, and related technologies.
• Work extensively with Delta Lake, Unity Catalog, and Azure Databricks best practices.
• Implement and optimise batch and streaming data processing solutions.
• Monitor and tune Databricks workloads to ensure high performance and reliability.

Data Integration & Governance

• Integrate data from databases, APIs, cloud storage platforms, and external data providers.
• Implement data lineage, metadata management, and governance processes using Azure Purview.
• Ensure compliance with data governance, security, and regulatory standards.
• Support data quality monitoring and remediation activities.

DevOps & Automation

• Develop and maintain CI/CD pipelines using Azure DevOps.
• Automate deployment, monitoring, and operational processes.
• Promote DevOps best practices across data engineering activities.

Stakeholder Collaboration

• Collaborate with business stakeholders, data scientists, analysts, and technical teams to deliver data-driven solutions.
• Participate in design discussions, code reviews, and knowledge-sharing initiatives.
• Translate business requirements into scalable technical solutions.

Essential Skills & Experience

• 10+ years of experience in Data Engineering.
• Minimum 3+ years of hands-on Azure Databricks experience.
• Strong expertise in Python, PySpark, Spark SQL, and SQL.
• Extensive experience with Azure Data Factory, Azure Databricks, Azure Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.
• Strong understanding of Delta Lake architecture and Unity Catalog.
• Experience building and optimising large-scale ETL/ELT pipelines.
• Hands-on experience with Azure Purview, data governance, metadata management, and lineage.
• Experience with Azure DevOps, Git, and CI/CD implementation.
• Strong understanding of data warehousing, dimensional modelling, and data integration patterns.
• Experience with SQL and NoSQL databases.
• Strong analytical, problem-solving, and communication skills.
• Experience working within Agile delivery environments.

Desirable Skills

• Experience with streaming technologies such as Azure Event Hubs or Kafka.
• Experience with Power BI, Tableau, or other data visualisation tools.
• Knowledge of Docker, Kubernetes, Terraform, or related DevOps technologies.
• Experience within Financial Services, Banking, Economic Data, or Regulatory environments.
• Azure Data Engineer certifications (DP-203 or equivalent).