Lead Data Engineer

Data Engineer Lead

Perm - Fully remote

On behalf of a key client we are resourcing for a fully remote Lead data engineer, this is a permanent fully remote role

Role summary

The Data Engineering Lead will be responsible for the design, development and maintenance of robust scalable and high-performance data platform with Snowflake and Fivetran including ETL/ELT

Responsibilities

• Demonstrate experience architecting and implementing solutions in cloud environments under an agile framework. Being comfortable defining and maintaining a high standard for development in close collaboration with data analysts, data scientists, product owners, DBAs and external stakeholders of varied seniority. Use your skills as a critical thinker who is able to anticipate and prevent challenges before they occur with solid design and a careful attention to detail and will be commercial enough to understand the purpose of your output - being able to competently challenge assumptions, business cases and approaches if there could be a better option.
• Responsible for the analysis, design, development, testing and maintenance of ELT solutions, predictive models, tabular models and data marts.
• Ownership for cloud-based data lake, using serverless architectures wherever possible to absolutely minimise the need for platform maintenance and permit exponential growth with diminishing effort.
• Delivery of data governance which informs a single source of truth in respect to information.
• To highly automate the entire SDLC according to best practice patterns and practices which result in a high velocity data engineering service to the business.
• To work according to best practices and patterns which enable us to move at the highest velocity in managing change.
• Helping users adhere to agile practices utilising a toolset that includes Clickup, AzureDevOps and continuous integration build systems.
• Establish and maintain a best-in-class approach to testing, leaning heavily towards full automation of unit test, regression test and integration test and leading by example in respect to testing and data quality processes.
• Own and continuously improve development standards for the data engineering team. This includes coding standards, version control practices, peer reviews, and the creation and versioning of ETL documentation artefacts.
• Perform root cause analysis for troubleshooting production run-time issues.
• To rigorously investigate failed releases, identifying, prioritising and eliminating root causes as a process for continuous improvement.
• Work with business SMEs and Business Analysts to convert business needs into technical requirements.
• Ownership of a comprehensive data catalog for all metadata, business rules and logic.
• Optimize SQL, Python, R code performance and processing methods.
• Perform design validation, reconciliation and error handling across the entire ELT process (initial, incremental).

Required Skills

• Strong programming knowledge and skills with Snowflake
• Strong programming using Python and R.
• Solid experience working in an environment where analytics for data surfaces in PowerBI and Excel.
• Strong background Microsoft SQL 2019- our application database which is ingested into Snowflake.
• Demonstrated skill and ability in the development of data warehouse using best practices, including star and snowflake schemas, FACT tables, cubes and tabular models and associated languages such as TSQL.
• Good understanding of OLTP and OLAP databases
• Demonstrably excellent understanding of Kimball Data Warehouse concepts in theory and practice
• Excellent on building and tuning queries in TSQL
• Evidence of heavy involvement in creation of data warehouse from inception to delivery.
• Experience with Agile development practices
• Experience with continuous integration and delivery in theory and practice

Desirable skills

• Databricks, Apache Spark, Hadoop and Hive or another similar platform utilising map-reduce functionality such as EMR.
• Working with Supply chain
• A genuine interest in sustainability and a desire to save the planet using technology.
• Presenting data engineering status/projects to senior management