Description

You will build and maintain scalable, distributed data pipelines on GCP, focusing on BigQuery lakehouse layers and Dataproc-driven Delta Lake workflows. You own the full lifecycle of data ingestion, transformation, and release for large volumes of financial data through the OMDP data factory. You design bitemporal data models on BigQuery to support regulatory-grade time-series datasets and implement comprehensive testing frameworks for pipeline logic. You collaborate with global stakeholders to integrate AI solutions, including anomaly detection and semantic search, into the data platform.

This role is on-site.

Responsibilities

• Build and maintain fault-tolerant data pipelines using Cloud Composer, Dataproc, and BigQuery.
• Design and implement bitemporal data models and certified time-series datasets.
• Develop unit, non-regression, and user acceptance testing frameworks for data transformations.
• Integrate AI-assisted tools and services, such as Vertex AI, for ingestion and anomaly detection.
• Manage CI/CD practices, version control, and audit trails for data solutions.

Required Skills

• 6-8 years of experience in data engineering.
• Proficient in Python for pipeline development, transformation logic, and automation.
• Strong hands-on experience with BigQuery, including partitioning, clustering, materialized views, and time-series patterns.
• Experience building and scheduling pipelines with Cloud Composer (Apache Airflow), including DAG authoring and SLA alerting.
• Working knowledge of Dataproc (Apache Spark) for batch ingestion and Delta Lake operations.
• Familiarity with GCP technologies: Cloud Storage, Pub/Sub, Datastream, Cloud Monitoring, IAM, and VPC Service Controls.
• Proficient in SQL and experience with Git branching strategies and pull request workflows.
• Experience consuming REST APIs and integrating external data vendor services.
• Familiarity with AI-assisted development tools like GitHub Copilot or Cursor.

Key Skills

Education

Bachelor's degree