Lead Data Engineer
Posted on Jan 4, 2023 by Tank Recruitment Limited
Data Engineer Lead
Perm - Fully remote
On behalf of a key client we are resourcing for a fully remote Lead data engineer, this is a permanent fully remote role
The Data Engineering Lead will be responsible for the design, development and maintenance of robust scalable and high-performance data platform with Snowflake and Fivetran including ETL/ELT
- Demonstrate experience architecting and implementing solutions in cloud environments under an agile framework. Being comfortable defining and maintaining a high standard for development in close collaboration with data analysts, data scientists, product owners, DBAs and external stakeholders of varied seniority. Use your skills as a critical thinker who is able to anticipate and prevent challenges before they occur with solid design and a careful attention to detail and will be commercial enough to understand the purpose of your output - being able to competently challenge assumptions, business cases and approaches if there could be a better option.
- Responsible for the analysis, design, development, testing and maintenance of ELT solutions, predictive models, tabular models and data marts.
- Ownership for cloud-based data lake, using serverless architectures wherever possible to absolutely minimise the need for platform maintenance and permit exponential growth with diminishing effort.
- Delivery of data governance which informs a single source of truth in respect to information.
- To highly automate the entire SDLC according to best practice patterns and practices which result in a high velocity data engineering service to the business.
- To work according to best practices and patterns which enable us to move at the highest velocity in managing change.
- Helping users adhere to agile practices utilising a toolset that includes Clickup, AzureDevOps and continuous integration build systems.
- Establish and maintain a best-in-class approach to testing, leaning heavily towards full automation of unit test, regression test and integration test and leading by example in respect to testing and data quality processes.
- Own and continuously improve development standards for the data engineering team. This includes coding standards, version control practices, peer reviews, and the creation and versioning of ETL documentation artefacts.
- Perform root cause analysis for troubleshooting production run-time issues.
- To rigorously investigate failed releases, identifying, prioritising and eliminating root causes as a process for continuous improvement.
- Work with business SMEs and Business Analysts to convert business needs into technical requirements.
- Ownership of a comprehensive data catalog for all metadata, business rules and logic.
- Optimize SQL, Python, R code performance and processing methods.
- Perform design validation, reconciliation and error handling across the entire ELT process (initial, incremental).
- Strong programming knowledge and skills with Snowflake
- Strong programming using Python and R.
- Solid experience working in an environment where analytics for data surfaces in PowerBI and Excel.
- Strong background Microsoft SQL 2019- our application database which is ingested into Snowflake.
- Demonstrated skill and ability in the development of data warehouse using best practices, including star and snowflake schemas, FACT tables, cubes and tabular models and associated languages such as TSQL.
- Good understanding of OLTP and OLAP databases
- Demonstrably excellent understanding of Kimball Data Warehouse concepts in theory and practice
- Excellent on building and tuning queries in TSQL
- Evidence of heavy involvement in creation of data warehouse from inception to delivery.
- Experience with Agile development practices
- Experience with continuous integration and delivery in theory and practice
- Databricks, Apache Spark, Hadoop and Hive or another similar platform utilising map-reduce functionality such as EMR.
- Working with Supply chain
- A genuine interest in sustainability and a desire to save the planet using technology.
- Presenting data engineering status/projects to senior management
Amplify your job search: