Senior Data Engineer

About the role

Shadow is the AI coordination layer that powers marketing teams of fast‑growing consumer brands. As a Senior Data Engineer you will own the pipelines that ingest, normalize, and keep reliable the world’s marketing data, enabling AI agents to make informed decisions.

Key responsibilities

• Build and scale ingestion layers for third‑party marketing APIs (Meta, Google, TikTok, GA4, Shopify, Klaviyo, etc.), handling authentication, rate limits, backfills and incremental syncs.
• Design transformation pipelines that map messy platform‑specific data into shared, queryable schemas (e.g., unified campaign models).
• Ensure data reliability at scale – monitor sync accuracy, freshness, coverage and build observability to detect broken connections or anomalies before users notice.
• Engineer multi‑tenant, cost‑efficient storage and pipelines that maintain strict data isolation, privacy and compliance for 1,000+ users.
• Partner with AI and data‑science teams to expose clean, well‑modeled data for agent retrieval and reasoning.

Required profile

• Proven experience building and operating large‑scale enterprise data pipelines serving 1,000+ users or equivalent data volume.
• Strong SQL and Python skills with production experience in modern data warehouses (BigQuery, Snowflake, Redshift, etc.).
• Deep familiarity with ETL/ELT patterns, incremental syncs, schema design and analytics data modeling.
• Hands‑on experience integrating third‑party APIs, managing OAuth flows, pagination, rate limits and schema drift.
• Bias toward observability and data quality – you instrument pipelines and never ship untrusted data.
• Experience with SOC 2‑compliant, enterprise‑grade security and privacy requirements.

Required skills

• SQL
• Python
• BigQuery
• Snowflake
• Redshift
• ETL/ELT
• OAuth API integration
• Observability tooling
• SOC 2 compliance
• GCP (BigQuery, Cloud Run)
• PostgreSQL & pgvector
• dbt
• Airflow or Dagster
• Langfuse (AI/LLM observability)

What we offer

• Unlimited PTO plus local holidays.
• Remote‑first culture with optional hybrid weeks for New York or Lisbon hubs.
• Flexible parental leave.
• Interdisciplinary growth opportunities across the company.