GCP Data Engineer
Posted on Jul 5, 2021 by Darwin Recruitment
We are looking for a Data Engineer to work on the implementation of a whole new BI architecture.
You'll be part of the Data & analytics centre of excellence in the global Digital Factory. It's a role in a highly dynamic international environment. Lead our operating companies implementing a fully cloud based BI architecture that serves as the blueprint for the entire company, which consists of 38 countries.
As a Data Engineer you have an overview of the data landscape. You build data pipelines that extract data from internal- and external sources and make this available in the country specific data lakes for various data consumers (BI tool, AI projects, Data scientist etc). To ingest and transform data we have an inhouse developed framework based on the latest cloud technologies. Our tech stack is GCP (Google Cloud Platform) BigQuery with the extension of tools like Composer/Airflow, Dataflow/Apache Beam, Gitlab, Spinnaker, SonarCube a.o.
Ingesting data from sources into the data lake is only part of your job. Together with your team you are also responsible for transforming the data through different stages (ie from landing zone to data warehouse and data mart). This means that designing, standardizing and reviewing data(warehouse) models, both technical and logical, is also part of your job. Because of the central position of the data engineer you will play a crucial role in translating business requirements into data products like pipelines and data marts. Additionally, you also manage the required data quality standards (availability, integrity, confidentiality, actuality, frequency and completeness). We process a lot of personal data so we expect our data engineers to act as a custodian. Designing solutions with the highest security standards in mind.
- Develop next-generation scalable, flexible, and high-performance data pipelines
- Optimize and expand data warehouses, through integration of new data
- Build data marts and data models to support data science and other internal customers
- Act as an intermediary for problems, with both technical and non-technical audiences
- Help define quality standards for the team and share your knowledge
- A close collaboration with our local data teams in continuing to shape the technical vision and recast problems and challenges into innovative solutions
- Work closely with the business functions to identify the problem statement and work towards answering those from a data perspective
- Give advice on potential areas where data streams can be optimised
- Experiment with available tools and advice on new tools in order to determine optimal solutions given the requirements
- Has a bachelor or master degree in informatics/computer science
- You will have at least 2/3 years of working experience with ETL (preferably ELT)
- You will have at least 2/3 years of working experience with database design, Datamodelling and normalisation techniques
- Expert knowledge of SQL
- Experience in cloud infrastructure and cloud based way of working, knowledge of gcp and technologies like BigQuery, dataflow/Apache Beam and composer/Airflow is a (big)plus
- Have some experience in architecture in complex it landscapes
- Experience with multiple program coding languages
- Experience with scalable data platforms, security, authorisation and authentication.
- Experience with common devops and ci/cd practices to guarantee the quality of our products
Darwin Recruitment is acting as an Employment Business in relation to this vacancy.