Description

You will design, build, and optimize data pipelines on Google Cloud Platform using PySpark and Spark SQL. You are responsible for managing ETL workflows, ensuring data integrity through validation, and tuning cluster performance for efficiency. You will collaborate with clients to deliver data-driven solutions while maintaining strong Unix and SQL fundamentals.

Responsibilities

• Build and manage ETL pipelines using GCP tools and existing platforms.
• Develop and optimize Spark SQL queries and PySpark scripts for data processing and validation.
• Tune Spark clusters to improve data processing performance and resource utilization.
• Write complex SQL queries for data analysis and ensure data quality.
• Support client-facing delivery by providing robust data-driven solutions.

Required Skills

• 8+ years of experience in data engineering or analytics.
• Strong proficiency in SQL for complex querying and analysis.
• Hands-on experience with Google Cloud Platform (GCP) environments.
• Experience with Unix/Linux command line tools and scripting.
• Proficiency in PySpark for data processing tasks.
• Experience with Spark SQL and cluster optimization techniques.
• Understanding of data engineering and analytics concepts.

Preferred Skills

• Experience working in client-facing delivery environments.

Key Skills

Education

Any Gradute