Description

You will build and optimize large-scale data processing pipelines and manage big data workloads.

Responsibilities

• Run Spark and Hadoop workloads using Dataproc, Dataflow, or Cloud Composer.
• Write complex, highly optimized SQL queries across large datasets.
• Manage data workflows using tools like Airflow and Jenkins.
• Deploy and orchestrate services using Docker and Kubernetes.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Spark, PySpark, and Hive.
• Proficiency in Scala and SQL.
• Experience with GCP, Dataproc, and Hadoop.
• Hands-on work with Airflow and Jenkins.
• Build tool experience with SBT and Maven.
• Containerization experience using Docker and Kubernetes.
• Knowledge of Kafka, Storm, Druid, and Presto.

Preferred Skills

• Experience with EMR or HD Insights.

Key Skills

Education

Any Graduate