Key Responsibilities:

Databricks Platform Support:

• Provide daily support for the Databricks platform running on AWS.
• Ensuring data security and access controls are setup and maintained
• Monitor and troubleshoot clusters, workspaces, jobs, and other Databricks resources
• Familiar with setting up and using medallion architecture, and the unity catalog
• Handle incidents, requests, and changes in Databricks environments.
• Collaborate with engineering and DevOps teams to resolve complex platform issues.
• Familiarity with Databricks ML flows and monitoring

Performance Tuning & Optimization:

• Analyse workloads and optimize Databricks clusters for performance and cost-efficiency.
• Recommend best practices for efficient Spark job execution and data processing.

AWS Infrastructure Management:

• Manage and configure AWS services (eg, S3, EC2, IAM, Lambda) that integrate with Databricks.
• Ensure proper security configurations and compliance with AWS standards.
• Manage networking, security groups, and VPCs related to Databricks setup.

Automation & DevOps:

• Develop and maintain automation scripts for platform maintenance and monitoring.
• Work closely with DevOps teams to automate cluster management, scaling, and infrastructure provisioning.
• Implement CI/CD pipelines for Databricks deployments using tools like Jenkins, Git, or Terraform.

Qualifications:

• Bachelor's degree in computer science, Information Technology, or related field. and 5+ years of experience supporting Databricks or similar data platforms.
• Strong expertise in Databricks on AWS: cluster management, job execution, Spark optimization.
• Hands-on experience with AWS services like EC2, S3, IAM, Lambda, RDS, VPC.
• Proficiency in Python, Scala, or SQL for data engineering tasks.
• Solid understanding of Apache Spark, ETL pipelines, and big data processing.
• Familiarity with DevOps practices, automation tools (Terraform, Ansible), and CI/CD pipelines.
• Experience with monitoring and logging tools (CloudWatch, Datadog, or similar).
• Strong problem-solving and troubleshooting skills.
• Excellent written and verbal communication skills.