Big Data DevOps Engineer
Posted on Jul 16, 2021 by Darwin Recruitment
Tasks and responsibilities
- Proficient in Python, experience in JVM languages desirable (eg Java/Scala)
- Take on initiative to set up and administer Azure Cloud infrastructure for building out the data ingestion and transformation pipeline with Azure Data Factory, Azure Databricks and Azure Data Lake Storage Gen2
- Support the development team with test and deployment automation tasks
- Extensive experience in DevOps Cloud automation, application monitoring and building CI/CD pipelines using Azure DevOps, Azure Monitor and Azure Log Analytics
- Good working knowledge of Big Data Engineering frameworks, such as Spark, Databricks and Delta Lake with experience of configuring clusters and optimizing performance of Spark jobs
- Hands-on working experience with Git, Terraform, ARM templates, SonarQube, test automation frameworks, CI/CD pipeline tools and applying in big data/data engineering projects
- Knowledge of key security concepts in Azure relating to permissions, network configuration
- Excellent analytical and conceptual skills required to integrate diverse data tech stacks and understand their dependencies
- Drive the development of an automated continuous delivery platform using Azure DevOps, ranging from development infrastructure tooling to reduce operational tasks to automated testing capabilities to reliable release and deployment strategies for Azure Data Factory and Azure Databricks
- Ensure development and infrastructure backlogs, standards, artefacts and guidelines include DevOps principles, as well as driving ideas and solutions that promote better ways of working
- End-to-end engagement, from inception to production, in designing, deploying, operating and continuously improving high-availability services in Azure
- Deliver high-quality solutions and deployment strategies, focusing on reliability, automation, maintainability and security
Darwin Recruitment AG is a Zurich based, SECO licensed, privately owned subsidiary of Darwin Professional Staffing Group Ltd (a Global IT Recruitment Consultancy).
Darwin Recruitment AG manages client relationships whilst also utilising Darwin Professional Staffing Group databases and networks to source Candidates and fulfil client requests.
We do not ask for a placement fee from Candidates/Employees.