This Job Vacancy has Expired!

Data Engineer

CV-Library

Posted on Nov 25, 2021 by CV-Library

Fareham, Hampshire, United Kingdom
IT
Immediate Start
Annual Salary
Contract/Project
An exciting opportunity is available to work at the forefront of an industry-academic collaboration, by providing a combination of data engineering expertise and Azure Cloud administrator support and to a major research project working on Data Science, the creation of a Digital Twin and Machine Learning. The project is in its early stages and will require the design and set up of data pipelines in Azure from multiple sources, with several data processing steps, the creation of a data lake and data warehouse, and integration with on-site systems.

We are looking for an experienced and highly capable data engineer with AZURE experience to join our accomplished team, who can help us to design and implement this critical infrastructure to meet the needs of this large and exciting research collaboration. The candidate will be working with the Azure Cloud Architect to meet the needs of the research engineering community. This includes the creation of data pipelines to transform raw data to curated datasets and will require proven programming and data engineering experience. A good knowledge of cyber security issues, vulnerabilities, and ways to manage and address them.

Follow the below link for full details on the project-

https://(url removed)/research/research-projects/project-bluebird-ai-system-air-traffic-control

Essential Skills - "Azure Data Engineer"

A minimum of 3 years' experience designing successful solutions to the Cloud
Data Engineering & pipeline design for data science applications
Proven understanding of Data Science and Machine Learning concepts
Minimum of 3 years hands on Data Engineering and Software experience
Proven experience in data processing with Python, including a good working knowledge of PySpark, Sklearn, Pandas, Numpy, SciPy
In depth understanding of available Azure services including:
Data Lake, Data Warehouse, DataFactory, Databricks
Structured and unstructured data storage
Notebooks
Active Directory
Database options: SQL, NoSQL
Virtual Machines
DevOps
Azure Container Instances, Docker
Azure functions and Logic Apps
Working with unstructured data in Azure data pipelines
Workflows involving the use of DevOps and source control
Evidence of developing sound, explainable design decisions and judgement
Building ETL and ELT pipelines

"Desirable Skills - Azure Data Engineer"

Experience working in research environments in both academia and industry
Building Continuous Integration Pipelines
Good understanding of cyber security & implementation for sensitive datasets with experience of working with sensitive datasets
Experience using IaC technologies, especially Terraform and ARM templates
Implementation of data pipelines using PySpark within Databricks
Building complex SQL queries
Proven experience of database setup and implementation for both SQL and NoSQL databases
Microsoft Azure Certifications
AZ-300 and AZ-301 (Architect Technology)
DP-200 and DP-201 (Data Engineering)
Experience of Hybrid systems with cloud and on-prem elements

Qualifications: PhD in Computer Science, Mathematics, Engineering, Physics or equivalent level of qualification and/or in related fields

Reference: 215254126

Set up alerts to get notified of new vacancies.

Similar Jobs

Data Engineer

Leeds, West Yorkshire, United Kingdom

£1 - £100k Annual

Data Engineer

Edinburgh, City of Edinburgh, United Kingdom

£45k - £60k Annual

Data Engineer

Edinburgh, City of Edinburgh, United Kingdom

£480 - £525 Daily

Data Engineer

Lancashire, United Kingdom

£35k - £55k Annual

Data Engineer

Bristol, Bristol (County), United Kingdom

£38k - £42k Annual

Data Engineer

London, Greater London, United Kingdom

Annual Salary

Data Engineer

Bristol, Bristol (County), United Kingdom

£38k - £42k Annual

Data Engineer

London, United Kingdom

Annual Salary

Data Engineer

London, United Kingdom

£80k - £100k Annual

Data Engineer

Ipswich, Suffolk, United Kingdom

£50k - £65k Annual