Description

Design and build production data pipelines within a hybrid big data architecture.

Responsibilities

• Build production data pipelines from ingestion to consumption.
• Design and implement ETL processes for data lakes and data warehouses.
• Execute data migrations from legacy systems including Hadoop, Exadata, Oracle Teradata, or Netezza.
• Develop streaming and batch patterns using PySpark and structured streaming.
• Manage data workflows using GCP tools like Dataproc, Composer, and Airflow.

Required Skills

• 12+ years of experience in big data architecture and pipeline design.
• Expertise in Python, Scala, or Java.
• Strong proficiency in PySpark and structured streaming.
• Deep experience with GCP services including BigQuery, Dataproc, Composer, and Airflow.
• Hands-on experience with Hadoop, Hive, HDFS, and HBase.
• Knowledge of GCP CI/CD and SDKs.
• Experience migrating data from Exadata, Oracle Teradata, or Netezza.

Key Skills

Education

Any Graduate