Description

You will build and maintain scalable, distributed, fault-tolerant data pipelines on GCP, including BigQuery-based lakehouse layers and Dataproc-driven Delta Lake workflows.

This role is on-site.

Responsibilities

• Design and implement bitemporal data models on BigQuery to support certified, regulatory-grade time-series datasets.
• Build, schedule, and maintain pipelines using Cloud Composer (Apache Airflow), handling DAG authoring, SLA alerting, and dependency management.
• Develop and optimize data transformation logic using Python and SQL, leveraging BigQuery partitioning, clustering, and materialized views.
• Integrate AI-assisted ingestion, anomaly detection, and semantic search over the lakehouse using Vertex AI.
• Take complete ownership of solutions, including ingestion pipelines, QA workflows, correction management, and audit trail implementation.

Required Skills

• 6-8 years of experience in data engineering.
• Proficient in Python for data pipeline development, transformation logic, and automation scripts.
• Strong hands-on experience in BigQuery, including time-series query patterns at scale.
• Working knowledge of Dataproc (Apache Spark) for batch ingestion and Delta Lake merge operations.
• Experience with Cloud Composer (Apache Airflow) for DAG authoring and pipeline scheduling.
• Familiarity with GCP technologies: Cloud Storage, Pub/Sub, Datastream, IAM, and VPC Service Controls.
• Proficient in AI-assisted development tools such as GitHub Copilot or Cursor.
• Code versioning and collaboration using Git, including branching strategies and pipeline-as-code practices.
• Familiarity with REST APIs for consuming external data vendor APIs.

Preferred Skills

• Understanding of bitemporal data modeling concepts and challenges within BigQuery's append-optimized design.
• Exposure to CI/CD pipelines and infrastructure-as-code using Terraform for data platform deployments.
• Understanding of financial reference data, including equities, fixed income identifiers, and corporate actions.

Key Skills

Education

Any Gradute