Data Engineer Azure
Posted on Jun 6, 2021 by Darwin Recruitment
The responsibility of the Data Engineer is to build a lakehouse setup on Azure for two strategic projects in Finance IT and Risk Management IT. Hands-on experience in building a lakehouse using Datalake, Databricks and Synapse on Azure is required.
* Review existing solutions and their stages in the different data zones (bronze, silver, gold)
* Suggest the architecture and implement the lakehouse according to the requirements of the two projects
* Benchmark which load to do with either Databricks or Synapse
* Evaluate optimal VM infrastructure for the respective Databricks workload
* Implement productive data pipelines with the logic of existing Databricks notebooks
* Implement optimal setup and tuning of Delta files and its handling using Python and SQL
* Define and implement the interplay of ADF, Datalake, Databricks, Synapse up to Power BI
* Ability to work as part of a team, as well as work independently or with minimal direction.
* Excellent written, presentation, and verbal communication skills.
* Collaborate with data architects, modelers and IT team members on project goals.
* Strong knowledge of Lakehouse components such as ADF, Datalake, Databricks, Synapse
* Good knowledge of Power BI, SQL, Python
Darwin Recruitment AG is a Zurich based, SECO licensed, privately owned subsidiary of Darwin Professional Staffing Group Ltd (a Global IT Recruitment Consultancy).
Darwin Recruitment AG manages client relationships whilst also utilising Darwin Professional Staffing Group databases and networks to source Candidates and fulfil client requests.
We do not ask for a placement fee from Candidates/Employees.