Senior Data Bricks Architect

About the role

We are seeking a hands‑on Senior Data Bricks Architect to design, build, and govern a Lakehouse data platform from the ground up. You will own the end‑to‑end architecture of our data infrastructure, from raw ingestion through Medallion layers to serving, and set engineering standards for the whole data organization.

Key responsibilities

• Design and implement a production‑grade Medallion Architecture (Bronze/Silver/Gold) across all pipelines.
• Establish best practices for Delta Lake table design, partitioning, Z‑ordering and optimization.
• Lead the setup and rollout of Unity Catalog, defining catalog hierarchy, tagging, fine‑grained access controls and audit logging.
• Build and maintain production‑grade pipelines using PySpark, Delta Live Tables and Databricks Workflows, including CDC and incremental ingestion patterns.
• Implement pipeline observability, logging, alerting, lineage tracking and SLA monitoring.
• Integrate CI/CD with Databricks Repos, Git and manage code promotion across environments.
• Optimize Spark execution plans, right‑size cluster configurations and manage cost governance for compute, storage and DBUs.
• Define Python coding standards, notebook best practices and reusable library patterns for the engineering team.

Required profile

• 5+ years of experience with Databricks and Lakehouse architectures.
• Proven hands‑on expertise designing production‑grade Medallion architectures.
• Strong background in data governance, Unity Catalog, and security controls (row‑level, column‑level, data masking).
• Experience building scalable pipelines with PySpark, Delta Live Tables and Databricks Workflows.
• Demonstrated ability to optimize Spark performance and manage cost‑efficient compute resources.

Required skills

• Databricks platform
• Delta Lake & Unity Catalog
• PySpark & Apache Spark
• Python programming
• Delta Live Tables
• Databricks Workflows / Jobs
• Databricks Repos, Git, CI/CD
• Serverless and SQL warehouses
• Data modeling, schema evolution, partitioning, Z‑ordering
• CDC, incremental ingestion, full‑refresh pipelines
• Data governance, lineage, masking, row‑level & column‑level security
• Cost governance and DBU management