*fully onsite*

On-site: near Worcester/Great Malvern/Gloucester

Daily rate: up to £700 outside IR35

Duration: 6months with possible extension

Clearance: Must be SC cleared; DV preferred

Note: this position can be contract or permanent(up to £90k)

Methods Analytics are seeking a seasoned Senior Data Engineer (Infrastructure) to join our team. This role is essential for designing, building, and maintaining sophisticated data infrastructure systems that operate across both on-premises and Azure cloud environments. The position involves deploying and managing scalable data operations that support advanced analytics and data-driven decision-making, crucial for our organisational growth and innovation.

Requirements

• Develop and Manage Data Pipelines: You will design, construct, and maintain efficient and reliable data pipelines using Python, capable of supporting both streaming and batch data processing across structured, semi-structured, and unstructured data in on-premises and Azure environments.
• Hybrid Cloud and Data Storage Solutions: Implement and manage data storage solutions leveraging both on-premises infrastructure and Azure, ensuring seamless data integration and accessibility across platforms.
• Containerisation and Orchestration: Utilise Docker for containerisation and Kubernetes for orchestration, ensuring scalable and efficient deployment of applications across both cloud-based and on-premises environments.
• Workflow Automation: Employ tools such as Apache NiFi and Apache Airflow to automate data flows and manage complex workflows within hybrid environments.
• Event Streaming Experience: Utilise event-driven technologies such as Kafka, Apache NiFi, and Apache Flink to handle Real Time data streams effectively.
• Security and Compliance: Manage security setups and access controls, incorporating tools like Keycloak to protect data integrity and comply with legal standards across all data platforms.
• Data Search and Analytics: Oversee and enhance Elasticsearch setups for robust data searching and analytics capabilities in mixed infrastructure settings.
• Database Management: Administer and optimise PostgreSQL databases, ensuring high performance and availability across diverse deployment scenarios.

Essential Skills and Experience

• Strong Python Skills: Expertise in Python for Scripting and automating data processes across varied environments.
• Experience with ETL/ELT: Demonstrable experience in developing and optimising ETL or ELT workflows, particularly in hybrid (on-premises and Azure) environments.
• Expertise in Hybrid Cloud Data Architecture: Profound knowledge of integrating on-premises infrastructure with Azure cloud services.
• Containerisation and Orchestration Expertise: Solid experience with Docker and Kubernetes in managing applications across both on-premises and cloud platforms.
• Proficiency in Workflow Automation Tools: Practical experience with Apache NiFi and Apache Airflow in hybrid data environments.
• Experience in Event Streaming: Proven ability in managing and deploying event streaming platforms like Kafka, Apache NiFi, and Apache Flink.
• Data Security Knowledge: Experience with implementing security practices and tools, including Keycloak, across multiple platforms.
• Search and Database Management Skills: Strong background in managing Elasticsearch and PostgreSQL in environments that span on-premises and cloud infrastructures.

Your Impact

In this role, you will empower business leaders to make informed decisions by delivering timely, accurate, and actionable data insights from a robust, hybrid infrastructure. Your expertise will drive the seamless integration of on-premises and cloud-based data solutions, enhancing both the flexibility and scalability of our data operations. You will champion the adoption of modern data architectures and tooling, and play a pivotal role in cultivating a data-driven culture within the organisation, mentoring team members, and advancing our engineering practices.

Desirable Skills and Experience

• Certifications in Azure and Other Relevant Technologies: Certifications in cloud and on-premises technologies are highly beneficial and will strengthen your application.
• Experience in Data Engineering: A minimum of 5 years of experience in data engineering, with significant exposure to managing infrastructure in both on-premises and cloud settings.