Essential requirements

Must be able to:

• implement data flows to connect operational systems, data for analytics and Business Intelligence (BI) systems

• document source-to-target mappings

• re-engineer manual data flows to enable scaling and repeatable use

• support the build of data streaming systems

• write ETL (extract, transform, load) scripts and code to ensure the ETL process performs optimally

• develop Business Intelligence reports that can be reused

• build accessible data for analysis

• Hands-on coding experience in R or Python. Clarity of programming paradigms and concepts needed in Data Engineering. Extract/Transpose/Load of data. Comfortable with numpy, scipy, pandas, and data visualisation in python or equivalent packages in R

• Strong knowledge of SQL and experience in working with traditional RDBMS as well as distributed datasets. Deep knowledge of SQL, Cloud based data pipelines, architectures, and data sets. Have experience in writing complex queries against relational and non-relational data stores.

• Good understanding of Data Warehousing and Data Lakehouse concepts

• Experience working with large data sets, data pipeline, cloud services

• Experience in building reliable, efficient data applications systems, services and platforms.

• Work with Data Scientists to build analytics tools that utilize the data pipeline to provide actionable insights for management, operational efficiency, and other key business performance metrics

• Degree or equivalent in a relevant subject eg Computer Science, Information Systems, or related technical discipline

• Experience working in an agile environment or organizations with an agile culture

• Experience building Data Pipelines preferably through Azure or AWS would be advantageous

• Experience working with big data tools such as Hadoop, Spark is required

• A professional attitude and an ability to think outside of the box to solve complex challenges.

• Background in programming using Open Source technologies along with Python, Java or C++/.Net would be beneficial

• Good written and verbal communication skills along with strong desire to work in cross-functional teams.

• Must be aware of data security best practices and GDPR compliance toward personal data

Nice to have skills

• Knowledge of Machine Learning and traditional Data Science concepts beneficial but not mandatory for Data Engineer

• Have a good understanding of design choices for data storage and data processing, with a particular focus on cloud data services.

• Understanding of Apache Spark or Airflow or similar technologies beneficial but not mandatory

• Have experience in using parallel computing to process large datasets and to optimise computationally intensive tasks. Not mandatory.