Description

Build and maintain scalable, distributed data pipelines on GCP, focusing on BigQuery lakehouse layers and Dataproc-driven Delta Lake workflows for financial data.

This role is on-site.

Responsibilities

• Design and implement bitemporal data models on BigQuery to support regulatory-grade time-series datasets.
• Build, schedule, and maintain pipelines using Cloud Composer (Apache Airflow) and Dataproc (Apache Spark) for batch ingestion and Delta Lake operations.
• Develop Python and SQL solutions for data acquisition, normalisation, and transformation within the OMDP data factory.
• Implement software testing frameworks (unit, non-regression, UAT) and manage QA workflows, correction management, and audit trails.
• Integrate AI solutions using Vertex AI for anomaly detection, semantic search, and AI-assisted ingestion.

Required Skills

• 6-8 years of experience in data engineering.
• Proficient in Python for pipeline development, transformation logic, and automation.
• Strong hands-on experience with BigQuery, including partitioning, clustering, materialised views, and time-series query patterns.
• Experience with Cloud Composer (Apache Airflow) for DAG authoring, SLA alerting, and dependency management.
• Working knowledge of Dataproc (Apache Spark) for batch ingestion, Delta Lake merge operations, and incremental processing.
• Familiarity with GCP technologies: Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, and VPC Service Controls.
• Proficient in SQL with strong analytical capabilities.
• Experience with CI/CD practices, Git workflows, and infrastructure-as-code (Terraform).
• Familiarity with AI-assisted development tools like GitHub Copilot or Cursor.

Preferred Skills

• Understanding of financial reference data (equities, fixed income, corporate actions) and bitemporal data modeling.
• Experience with Pandas, PySpark, or columnar storage systems like ClickHouse.
• Familiarity with Dataplex for data discovery, lineage, and policy tagging.

Key Skills

Education

Any Gradute