Description

Role Description & Requirements:
● Lead technical architecture and solution design for a large FSI customer Transactions Ingestion Data Platform Real Time and Batch
● Interact with customer’s data engineering team to provide opinionated guidance on Transactions Ingestion Data platform architecture, best practices, schema designs (Spanner/BigQuery), and solution development
● Independently develop and unit test ingestion and processing pipelines (e.g., Kafka to Spanner/BigQuery via Dataflow, GCS to Spanner/BigQuery), and handover developed solutions to customer’s technical team
● Perform pair programming with customer’s data engineers
● Project manage sprint-based engagements, leading sprint planning, backlog management, scope definition, risks/issues, and timelines for successful MVP delivery
● Collaborate with cross-disciplinary project team, play the role of workstream or overall project tech lead depending on size and scale of project / program

Minimum Qualifications
● Expertise Required
○ In-depth knowledge and experience of GCP data and analytics technologies
○ Designing Large scale Transaction Data Platform architecture for
(a) migrating open source or public cloud data platforms to GCP cloud native services (b)designing greenfield data platforms on GCP
○ Define solution architecture and detailed design, and perform hands-on implementation for
■ data pipelines for batch and event driven ingestion and processing of data from a variety of sources such as on-prem files, on-prem databases, APIs, etc.
■ real-time data ingestion and event-driven processing pipelines from Confluent Kafka and GCS to Spanner and BigQuery
○ Automatic transpilation of legacy code (HIVE, Teradata, python logic etc.) to BQ SQL
○ BQ query performance optimisation
○ CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, Terraform
○ Orchestration setup and Cloud Composer DAG development for data pipeline workflows
○ Data governance solutioning using GCP governance tooling (Dataplex, Data Catalog)
○ Experience applying Generative AI technologies and integrations in enterprise environments.

Tools and languages experience Required
Must have
● GCP Dataflow, Cloud Spanner, BigQuery, Pub/Sub, Cloud Composer, Cloud Workflows, Confluent Kafka, Cloud Run, Cloud Build, Terraform
● Must have: programming knowledge and willingness to be hands-on - Python, Java
Good to Have
● Experience with dbt or Dataform, Terraform in context of data pipelines
● Experience with open source ecosystem and distributions such as Hadoop, Spark, Cloudera/Hortonworks and frameworks and tech such as SPARK, Oozie, Kafka,
HBASE
● Understanding and experience with NoSQL databases such as HBASE, MongoDB
● Knowledge of cloud databases such as Spanner, BigTable, Cloud SQL, DB migrations
Certification
● Good to have GCP Data engineering

Key Skills

Education

Any Graduate