Data Engineer (Python/PySpark)
Posted on Jul 12, 2021 by Gazelle Global Consulting
One of our banking clients is seeking for Data Engineer for a 6-12 month contract in Amsterdam, Netherlands. Start Date: ASAP
- The conversion of 7 existing data pipelines currently written in Python/PySpark into a new PySpark based feature framework, including the corresponding parallel tests and unit tests.
- The conversion first requires a breakdown of the current pipeline into smaller logical steps. Key in this beak down is to ensure an optimal re-use of intermediate steps for the various features.
- The individual steps are written in Python into a method with a predefined method signature.
- Additionally, the new pipeline uses a different input source and as result the new input source needs additional pre-processing step and parallel tests .
Relevant knowledge, skills, competences & desired education level
- Good working knowledge of Python. Preferably at least 3 years Python experience and 5 years programming experience in total.
- Preferably experience with PySpark and/or big data.
- Preferably experience with data engineering.
- Can work independently.
Please apply if you are interested and available