Databricks Platform Engineer

Confiz is seeking a Databricks Platform Engineer to build, automate, and operate the foundational services that make our enterprise Databricks platform secure, scalable, governed, and cost-efficient. This role is responsible for developing reusable platform capabilities, governance automation, CI/CD pipelines, monitoring, and operational tooling that enable engineering teams to deliver data and AI solutions efficiently. The ideal candidate has strong software engineering fundamentals, hands-on Databricks experience, and a passion for building highly automated, production-grade cloud data platforms.

Responsibilities

• Design, build, and maintain governance automation for Unity Catalog, including access controls, metadata tagging, policy enforcement, and governance-as-code.
• Develop metadata-driven frameworks for scalable data ingestion, transformation, and publishing using Spark and Delta Lake.
• Build reusable platform services, libraries, templates, and integrations that accelerate development and promote engineering best practices.
• Develop and maintain automated data quality pipelines, validation frameworks, quality scorecards, and issue escalation workflows.
• Own the Databricks CI/CD framework, including repository standards, automated testing, Databricks Asset Bundles, environment promotion, and release automation.
• Implement Infrastructure-as-Code using Terraform and other modern DevOps practices to ensure repeatable, auditable deployments.
• Build and maintain monitoring, logging, alerting, and observability solutions for Databricks workloads and platform operations.
• Manage platform reliability by supporting incident response, operational runbooks, and continuous service improvements.
• Monitor Databricks usage and cloud costs, implementing optimization strategies to improve DBU efficiency and reduce operational expenses.
• Support workspace provisioning, identity integration, and access management while automating provisioning workflows where possible.
• Partner with architects, data engineers, and platform stakeholders to continuously enhance platform capabilities and developer experience.

Requirements

• 5+ years of experience in Data Engineering, Platform Engineering, Cloud Engineering, or a related field.
• 2+ years of hands-on experience supporting Databricks in production environments.
• Strong proficiency in Python and SQL with solid software engineering practices including testing, version control, and code reviews.
• Experience with Databricks Asset Bundles or equivalent deployment tooling.
• Hands-on experience with CI/CD platforms such as GitHub Actions or Azure DevOps.
• Experience building metadata-driven or configuration-based data pipelines using Spark and Delta Lake.
• Working knowledge of Unity Catalog, including governance, permissions, lineage, tagging, and policy automation.
• Experience implementing monitoring, observability, alerting, and production support for cloud data platforms.
• Experience with Infrastructure-as-Code, preferably Terraform.
• Experience with one or more major cloud platforms, including Azure, AWS, or Google Cloud Platform.
• Experience optimizing Databricks workloads, managing DBU consumption, and implementing cloud cost optimization strategies.
• Databricks certifications (preferred).
• Experience with data quality frameworks, identity and access automation, or AI/ML workloads on Databricks is a plus.

We have a global team of amazing individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is ISO 9001:2015 (QMS), ISO 27001:2022 (ISMS), ISO 20000-1:2018 (ITSM) and ISO 14001:2015 (EMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting-edge technologies while contributing success to the company as well as to themselves. 

To know more about Confiz Limited, visit https://www.linkedin.com/company/confiz/