Senior Data Engineer
Posted on Jul 25, 2024 by CV-Library
Greenford Green, Greater London, United Kingdom
IT
Immediate Start
£70k - £80k Annual
Full-Time
Data Engineer
Role Overview:
As a vital member of the team, the Data Engineer will be responsible for managing and optimising diverse data sets from various systems, extracting valuable business insights. This role involves developing raw data pipelines and using Azure Cloud tools to ensure seamless integration and optimisation. The candidate will also play a key role in migrating on-premises databases to the Azure Cloud, utilising their expertise in cloud architecture, database management, and change management to ensure a smooth transition for stakeholders.
Responsibilities:
Data Architecture: Develop, build, test, and maintain data architectures within large-scale data processing systems.
Cloud Migration: Migrate existing on-premises databases to the Azure Cloud.
Pipeline Management: Develop and manage data pipelines using Azure Data Factory, Delta Lake, and Spark, ensuring secure, reliable, and accessible data sets.
Scalable Solutions: Design and implement scalable data solutions leveraging Azure Cloud architecture.
Data Manipulation: Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain insights into dataset characteristics.
API Interaction: Interact with API systems to query and retrieve data for analysis.
Stakeholder Collaboration: Work closely with Business Analysts, IT Ops, and other stakeholders to understand and meet data needs.
Governance and Quality: Ensure compliance with data governance and data quality principles.
Data Modelling: Design robust data models to enhance data accessibility and facilitate deeper analysis.
Documentation: Maintain technical documentation for the entire codebase.
Lifecycle Ownership: Take end-to-end ownership of the Data Engineering Lifecycle.
ETL Processes: Implement and manage Fivetran for efficient and reliable ETL processes.
Desired Experience:
Bachelor's degree in computer science, engineering, or equivalent experience.
Extensive experience as a Senior Data Engineer/Cloud Data Architect, or similar role.
Deep knowledge of Azure Cloud architecture, Azure Databricks, DevOps, and CI/CD.
Extensive experience migrating on-premises data warehouses to the cloud.
Proficiency with Spark, SQL, Python, R, and other data engineering development tools.
Experience with metadata-driven pipelines and SQL serverless data warehouses.
Extensive knowledge of querying API systems.
Extensive experience building and optimising ETL pipelines using Databricks.
Understanding of data governance and data quality principles.
Experience implementing and managing Unity Catalogue for data governance.
Familiarity with Fivetran ETL tool for seamless data integration
Role Overview:
As a vital member of the team, the Data Engineer will be responsible for managing and optimising diverse data sets from various systems, extracting valuable business insights. This role involves developing raw data pipelines and using Azure Cloud tools to ensure seamless integration and optimisation. The candidate will also play a key role in migrating on-premises databases to the Azure Cloud, utilising their expertise in cloud architecture, database management, and change management to ensure a smooth transition for stakeholders.
Responsibilities:
Data Architecture: Develop, build, test, and maintain data architectures within large-scale data processing systems.
Cloud Migration: Migrate existing on-premises databases to the Azure Cloud.
Pipeline Management: Develop and manage data pipelines using Azure Data Factory, Delta Lake, and Spark, ensuring secure, reliable, and accessible data sets.
Scalable Solutions: Design and implement scalable data solutions leveraging Azure Cloud architecture.
Data Manipulation: Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain insights into dataset characteristics.
API Interaction: Interact with API systems to query and retrieve data for analysis.
Stakeholder Collaboration: Work closely with Business Analysts, IT Ops, and other stakeholders to understand and meet data needs.
Governance and Quality: Ensure compliance with data governance and data quality principles.
Data Modelling: Design robust data models to enhance data accessibility and facilitate deeper analysis.
Documentation: Maintain technical documentation for the entire codebase.
Lifecycle Ownership: Take end-to-end ownership of the Data Engineering Lifecycle.
ETL Processes: Implement and manage Fivetran for efficient and reliable ETL processes.
Desired Experience:
Bachelor's degree in computer science, engineering, or equivalent experience.
Extensive experience as a Senior Data Engineer/Cloud Data Architect, or similar role.
Deep knowledge of Azure Cloud architecture, Azure Databricks, DevOps, and CI/CD.
Extensive experience migrating on-premises data warehouses to the cloud.
Proficiency with Spark, SQL, Python, R, and other data engineering development tools.
Experience with metadata-driven pipelines and SQL serverless data warehouses.
Extensive knowledge of querying API systems.
Extensive experience building and optimising ETL pipelines using Databricks.
Understanding of data governance and data quality principles.
Experience implementing and managing Unity Catalogue for data governance.
Familiarity with Fivetran ETL tool for seamless data integration
Reference: 222059875
https://jobs.careeraddict.com/post/93142758
Senior Data Engineer
Posted on Jul 25, 2024 by CV-Library
Greenford Green, Greater London, United Kingdom
IT
Immediate Start
£70k - £80k Annual
Full-Time
Data Engineer
Role Overview:
As a vital member of the team, the Data Engineer will be responsible for managing and optimising diverse data sets from various systems, extracting valuable business insights. This role involves developing raw data pipelines and using Azure Cloud tools to ensure seamless integration and optimisation. The candidate will also play a key role in migrating on-premises databases to the Azure Cloud, utilising their expertise in cloud architecture, database management, and change management to ensure a smooth transition for stakeholders.
Responsibilities:
Data Architecture: Develop, build, test, and maintain data architectures within large-scale data processing systems.
Cloud Migration: Migrate existing on-premises databases to the Azure Cloud.
Pipeline Management: Develop and manage data pipelines using Azure Data Factory, Delta Lake, and Spark, ensuring secure, reliable, and accessible data sets.
Scalable Solutions: Design and implement scalable data solutions leveraging Azure Cloud architecture.
Data Manipulation: Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain insights into dataset characteristics.
API Interaction: Interact with API systems to query and retrieve data for analysis.
Stakeholder Collaboration: Work closely with Business Analysts, IT Ops, and other stakeholders to understand and meet data needs.
Governance and Quality: Ensure compliance with data governance and data quality principles.
Data Modelling: Design robust data models to enhance data accessibility and facilitate deeper analysis.
Documentation: Maintain technical documentation for the entire codebase.
Lifecycle Ownership: Take end-to-end ownership of the Data Engineering Lifecycle.
ETL Processes: Implement and manage Fivetran for efficient and reliable ETL processes.
Desired Experience:
Bachelor's degree in computer science, engineering, or equivalent experience.
Extensive experience as a Senior Data Engineer/Cloud Data Architect, or similar role.
Deep knowledge of Azure Cloud architecture, Azure Databricks, DevOps, and CI/CD.
Extensive experience migrating on-premises data warehouses to the cloud.
Proficiency with Spark, SQL, Python, R, and other data engineering development tools.
Experience with metadata-driven pipelines and SQL serverless data warehouses.
Extensive knowledge of querying API systems.
Extensive experience building and optimising ETL pipelines using Databricks.
Understanding of data governance and data quality principles.
Experience implementing and managing Unity Catalogue for data governance.
Familiarity with Fivetran ETL tool for seamless data integration
Role Overview:
As a vital member of the team, the Data Engineer will be responsible for managing and optimising diverse data sets from various systems, extracting valuable business insights. This role involves developing raw data pipelines and using Azure Cloud tools to ensure seamless integration and optimisation. The candidate will also play a key role in migrating on-premises databases to the Azure Cloud, utilising their expertise in cloud architecture, database management, and change management to ensure a smooth transition for stakeholders.
Responsibilities:
Data Architecture: Develop, build, test, and maintain data architectures within large-scale data processing systems.
Cloud Migration: Migrate existing on-premises databases to the Azure Cloud.
Pipeline Management: Develop and manage data pipelines using Azure Data Factory, Delta Lake, and Spark, ensuring secure, reliable, and accessible data sets.
Scalable Solutions: Design and implement scalable data solutions leveraging Azure Cloud architecture.
Data Manipulation: Utilise Spark, SQL, Python, R, and other data frameworks to manipulate data and gain insights into dataset characteristics.
API Interaction: Interact with API systems to query and retrieve data for analysis.
Stakeholder Collaboration: Work closely with Business Analysts, IT Ops, and other stakeholders to understand and meet data needs.
Governance and Quality: Ensure compliance with data governance and data quality principles.
Data Modelling: Design robust data models to enhance data accessibility and facilitate deeper analysis.
Documentation: Maintain technical documentation for the entire codebase.
Lifecycle Ownership: Take end-to-end ownership of the Data Engineering Lifecycle.
ETL Processes: Implement and manage Fivetran for efficient and reliable ETL processes.
Desired Experience:
Bachelor's degree in computer science, engineering, or equivalent experience.
Extensive experience as a Senior Data Engineer/Cloud Data Architect, or similar role.
Deep knowledge of Azure Cloud architecture, Azure Databricks, DevOps, and CI/CD.
Extensive experience migrating on-premises data warehouses to the cloud.
Proficiency with Spark, SQL, Python, R, and other data engineering development tools.
Experience with metadata-driven pipelines and SQL serverless data warehouses.
Extensive knowledge of querying API systems.
Extensive experience building and optimising ETL pipelines using Databricks.
Understanding of data governance and data quality principles.
Experience implementing and managing Unity Catalogue for data governance.
Familiarity with Fivetran ETL tool for seamless data integration
Reference: 222059875
![CareerAddict](/assets/job-alerts-704d6c2677939a4f022b5e6bdb810ad68a0fb447f53071d9c5753bc54be234b4.png)
Alert me to jobs like this:
Amplify your job search:
Expert career advice
Increase interview chances with our downloads and specialist services.
Visit Blog