Job Description

Key Responsibilities and Deliverables:

Data Warehouse Operations:Oversee the operations of the data warehouse on Databricks within AWS, ensuring optimal performance and reliability.

Continuously optimize data warehouse performance, including query performance, storage, and data processing efficiency.

ETL Development and Maintenance:Design, develop, and maintain ETL (Extract, Transform, Load) processes to integrate data from various sources into the data warehouse.

Manage and optimize data pipelines to ensure timely and accurate data processing and delivery.

Data Quality and Monitoring:Implement and monitor data quality checks to ensure accuracy, completeness, and consistency of data within the warehouse.

Set up and manage monitoring systems to detect and address data issues, performance bottlenecks, and system anomalies.

Collaboration and Support:Collaborate with cross-functional teams, including software engineers, data scientists, and product managers, to design and architect scalable, high-performance data solutions that align with business objectives and best practices

Provide technical support and expertise on data-related issues and queries from other teams.

Best Practices and Documentation:Create and maintain comprehensive documentation for data processes, ETL jobs, and data warehouse configurations.

Implement and promote best practices for data engineering, including coding standards, data governance, and security measures.