Databricks Infrastructure Engineer

Overview

Zachary Piper is seeking a Databricks Infrastructure Engineer to design, build, and support a secure, scalable cloud data platform that enables enterprise analytics, forecasting, and machine learning workloads. This role focuses on Databricks administration, cloud infrastructure, automation, platform reliability, security, and cost optimization.

Responsibilities

• Design, deploy, and support Databricks platform infrastructure and cloud-based data environments.
• Administer Databricks workspaces, clusters, jobs, workflows, SQL warehouses, Unity Catalog, permissions, and service principals.
• Implement Infrastructure as Code using Terraform and related automation tools.
• Build and maintain CI/CD pipelines for Databricks notebooks, workflows, and data solutions.
• Partner with Data Engineers, Data Scientists, and IT teams to deliver reliable analytics and machine learning platforms.
• Monitor platform performance, reliability, usage, and costs.
• Optimize cluster configurations, workload scheduling, storage, and compute resources.
• Implement security controls including IAM, access governance, auditing, logging, and secrets management.
• Troubleshoot production issues, perform root cause analysis, and support disaster recovery planning.
• Create operational documentation, runbooks, and platform standards.
• Evaluate and implement new cloud and Databricks capabilities to improve performance and scalability.

Required Qualifications

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related field.
• 3+ years of experience supporting or engineering cloud data platforms.
• Hands-on Databricks administration experience.
• Experience with Azure, AWS, or Google Cloud Platform.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience building and supporting CI/CD pipelines.
• Strong Python and SQL skills.
• Knowledge of data engineering concepts including data lakes, batch processing, streaming pipelines, and data quality controls.
• Understanding of cloud networking, identity management, monitoring, logging, and security best practices.
• Strong troubleshooting, communication, and documentation skills.

Preferred Qualifications

• Experience with Unity Catalog, MLflow, Delta Lake, Delta Live Tables, Databricks Asset Bundles, or model serving.
• Experience supporting machine learning, forecasting, analytics, supply chain, or demand planning environments.
• Databricks or cloud platform certifications.
• Experience with cloud cost optimization and FinOps practices.
• Familiarity with ERP, SAP, supply chain, sales, inventory, or related data integrations.

Compensation

·      Salary: $150,000 - $180,000

·      Location: REMOTE

#LI-BR1

#LI-REMOTE

Key Skills

Databricks, Azure, AWS, GCP, Terraform, Infrastructure as Code, CI/CD, Azure DevOps, GitHub Actions, Python, SQL, Unity Catalog, MLflow, Delta Lake, Delta Live Tables, Data Engineering, Cloud Infrastructure, Platform Engineering, IAM, Monitoring, Logging, FinOps, Data Lakes, Machine Learning Infrastructure.