Senior Data Engineer (Databricks)

About the role

The company is looking for a Senior Data Engineer with deep expertise in Databricks to design, build, and maintain production‑grade data pipelines. The role is on‑site near FC College Lahore and works evenings to align with a US‑based team.

Key responsibilities

• Develop and operate end‑to‑end data pipelines using Spark SQL, PySpark, Delta Lake, and Databricks DLT with a medallion architecture.
• Implement structured streaming solutions ingesting data from Event Hubs, Kafka or Kinesis, handling checkpoint recovery, watermarking and deduplication.
• Integrate third‑party REST APIs, managing pagination, rate limits, retries and schema‑drift handling.
• Debug and resolve source‑vs‑warehouse data discrepancies and perform entity‑resolution on messy text data.

Required profile

• 5+ years of data engineering experience with production Databricks workloads.
• Strong command of SQL and Python, able to own pipelines with minimal supervision.
• Excellent written and spoken English for asynchronous collaboration with a US team.

Required skills

• Spark SQL, PySpark, Delta Lake, Databricks DLT, medallion architecture
• Structured Streaming, Event Hubs, Kafka, Kinesis
• REST API integration, pagination, rate‑limit handling, schema drift
• Entity resolution / data matching, metrics or semantic layer (Holistics AML/AQL, dbt metrics, LookML)
• SQL, Python
• Azure ecosystem (ADLS, ADF, Key Vault)