Senior Data Engineer (Databricks)

About the role

We are looking for a Senior Data Engineer with extensive production experience on Databricks to design, build, and maintain robust data pipelines. The role is on‑site near FC College Lahore and works closely with a US‑based team.

Key responsibilities

• Develop and optimise Spark SQL, PySpark, Delta Lake, and medallion architecture pipelines on Databricks.
• Implement Structured Streaming or equivalent ingestion (Event Hubs, Kafka, Kinesis) with checkpoint recovery, watermarking and deduplication.
• Debug source‑vs‑warehouse discrepancies and resolve data‑matching issues on messy text data.
• Integrate third‑party REST APIs handling pagination, rate limits, retries and schema‑drift.
• Work with metrics/semantic layers such as Holistics AML/AQL, dbt metrics or LookML.

Required profile

• 5+ years of data engineering experience, strong SQL and Python skills.
• Fluent written and spoken English for asynchronous collaboration with a US team.

Required skills

• Databricks, Spark SQL, PySpark, Delta Lake, DLT
• Structured Streaming, Event Hubs, Kafka, Kinesis
• REST API integration, pagination, rate‑limit handling, schema drift
• Entity resolution / data matching
• Holistics AML/AQL, dbt metrics, LookML
• Azure ADLS, Azure Data Factory, Azure Key Vault