Job Title: Data Warehouse Consultant

Location: Luxembourg (4 days on-site, 1 day remote per week)

Contract Type: Freelance Duration: 4 months initially

About the Role:

As part of our data warehouse deployment, we are looking for an consultant with advanced expertise in PySpark to accelerate our progress and ensure an efficient production rollout.

Our key challenge is optimizing our Real Time data ingestion and processing pipeline, which involves:

• Apache PySpark for large-scale data transformation and processing
• Apache Kafka for streaming data management
• Debezium for handling Change Data Capture (CDC) logs from PostgreSQL

Thus, we are seeking an experienced PySpark expert to support the implementation phase, with additional knowledge in Kafka and Debezium to help optimize our Real Time data processing capabilities.
We are looking for a consultant who can provide immediate expertise with a strong collaborative mindset to help our team gradually upskill while delivering a high-performance Real Time data warehouse.

• Support the design and implementation of Spark-based data processing workflows
• Optimize performance and reliability of Real Time data ingestion pipelines
• Implement best practices to ensure scalability and maintainability of the data warehouse
• Assist in integrating and fine-tuning Kafka and Debezium for efficient CDC log processing
• Train and support the internal team to ensure a smooth transition after the mission

• Advanced expertise in PySpark (optimization, large-scale data processing, structured job management)
• Strong experience in Real Time data ingestion and transformation
• Knowledge of best practices in data engineering and data warehouse modelling
• Proficiency in PostgreSQL and TimescaleDB is a plus
• Experience with Kafka and Debezium for CDC log processing
• Understanding of Real Time data pipeline architectures
• Understanding of version control systems (eg, Git) and CI/CD pipelines
• Fluency in English (spoken and written) is required