Description

You will lead the design and optimization of large-scale ETL pipelines within a Google Cloud environment.

This role is on-site.

Responsibilities

• Design, develop, and optimize Spark-based ETL processes using PySpark on GCP.
• Automate data workflows and orchestration using Cloud Composer (Apache Airflow).
• Implement data governance, security, and monitoring practices across the cloud environment.
• Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution.
• Ensure data quality, validation, and consistency throughout all data pipelines.

Required Skills

• 6+ years of professional experience in ETL development.
• Advanced proficiency in Python and PySpark for big data processing.
• Hands-on experience with Google Cloud Platform services: BigQuery, Cloud Dataflow, Cloud Composer, and Cloud Storage.
• Strong command of SQL for complex data querying and transformation.
• Experience implementing CI/CD pipelines for data automation.
• Practical knowledge of Terraform for GCP infrastructure automation.
• Experience with containerization using Docker and Kubernetes.
• Understanding of data lake architectures and data warehousing concepts.

Preferred Skills

• Experience with Kafka or Pub/Sub for real-time data processing.
• Familiarity with DevOps and monitoring tools such as Prometheus, Stackdriver, or Datadog.

Key Skills

Education

Any Graduate