Data Engineer/Lead Data Engineer- Permanent-Hybrid
Posted on Apr 7, 2022 by McCabe & Barton
London, United Kingdom
£55k - £99k Annual
- Design, develop, test, and deploy data integration processes (batch or Real Time) using tools such as Microsoft SSIS, Azure Data Factory, Databricks, Matillion, Airflow, Sqoop, etc.
- Create functional & technical documentation - eg ETL architecture documentation, unit testing plans and results, data integration specifications, data testing plans, etc.
- Provide a consultative approach with business users, asking questions to understand the business need and deriving the data flow, conceptual, logical, and physical data models based on those needs. Perform data analysis to validate data models and to confirm ability to meet business needs.
- May serve as project or DI lead, overseeing multiple consultants from various competencies
- Stays current with emerging and changing technologies to best recommend and implement beneficial technologies and approaches for Data Integration
- Ensures proper execution/creation of methodology, training, templates, resource plans and engagement review processes
- Coach team members to ensure understanding on projects and tasks, providing effective feedback (critical and positive) and promoting growth opportunities when appropriate.
- Coordinate and consult with the project manager, client business staff, client technical staff and project developers in data architecture best practices and anything else that is data related at the project or business unit levels
- Architect, design, develop and set direction for enterprise self-service analytic solutions, Business Intelligence reports, visualisations and best practice standards. Toolsets include but not limited to: SQL Server Analysis and Reporting Services, Microsoft Power BI, Tableau and Qlik.
- Work with report team to identify, design and implement a reporting user experience that is consistent and intuitive across environments, across report methods, defines security and meets usability and scalability best practices.
Education & Experience
- 5-10 Years industry implementation experience with data integration tools such as Microsoft SSIS, Azure Data Factory, Databricks, Glue, Step Functions, Airflow, Apache Flume/Sqoop/Pig, etc.
- 3-5 years of management experience required (lead engineer)
- 1-5 years consulting experience preferred
- Bachelor's degree or equivalent experience, Master's Degree Preferred
- Strong data warehousing, OLTP systems, data integration and SDLC
- Strong experience in big data frameworks & working experience in Spark or Hadoop or Hive (incl. derivatives like pySpark (preferred), SparkScala or SparkSQL) or Similar, along with experience in libraries/frameworks to accelerate code development
- Experience using major Datamodelling tools (examples: ERwin, ER/Studio, PowerDesigner, etc.)
- Experience with major database platforms (eg SQL Server, Oracle, Azure Data Lake, Hadoop, Azure Synapse/SQL Data Warehouse, Snowflake, Redshift etc.)
- Strong experience in orchestration & working experience in either Data Factory or HDInsight or Data Pipeline or Cloud composer or Similar
- Understanding and experience with major Data Architecture philosophies (Dimensional, ODS, Data Vault, etc.)
- Understanding of modern data warehouse capabilities and technologies such as Real Time, cloud, Big Data.
- Understanding of on premises and cloud infrastructure architectures (eg Azure, AWS, GCP)
- Strong experience in Agile Process (Scrum cadences, Roles, deliverables) & working experience in either Azure DevOps, JIRA or Similar with Experience in CI/CD using one or more code management platforms