Description

You will design and maintain scalable ETL workflows and data pipelines within Google Cloud Platform (GCP).

Responsibilities

• Design and develop data pipelines for structured and unstructured data using Hive, Spark, and Airflow.
• Implement data storage and processing solutions leveraging Apache Hudi and BigQuery.
• Write clean, maintainable code in Scala and Python to transform large datasets.
• Ensure data quality and integrity through validation and monitoring techniques.
• Troubleshoot performance and scalability issues in distributed data processing systems.

Required Skills

• 3 years of experience in a Data Engineering role, ideally in big data environments.
• Expertise in Hive, Spark, and Apache Hudi.
• Hands-on experience with Google Cloud Platform (GCP), including BigQuery, GCS, Dataflow, and Pub/Sub.
• Proficiency in Scala and Python for building ETL processes.
• Experience with workflow orchestration using Apache Airflow.
• Strong knowledge of data warehousing, data modeling, and schema design.
• Understanding of distributed systems and parallel processing.
• Programming proficiency in Java and Perl.

Preferred Skills

• Bachelor of Engineering or Master of Computer Application degree.

Key Skills

Education

Bachelor of Engineering, Master of Computer Application