Description

You will build and maintain distributed data processing pipelines to manage large-scale datasets.

Responsibilities

• Build and maintain distributed data processing pipelines using Apache Spark.
• Write efficient SQL queries to extract, transform, and analyze large datasets.
• Optimize Spark jobs by tuning configurations and improving query efficiency.
• Develop and manage ETL workflows to move data from various sources into data lakes or warehouses.
• Monitor Spark jobs and cluster performance to address bottlenecks and failures.

Required Skills

• 5+ years of experience in data engineering.
• Strong hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and PySpark.
• Proficiency in Python, Scala, or Java for Spark application development.
• Advanced SQL skills, including complex joins, aggregations, and window functions.
• Deep understanding of distributed computing concepts and Spark architecture (RDDs, DAGs, partitions).
• Experience working with large datasets, data lakes, and data warehouses.
• Knowledge of file formats such as Parquet, Avro, and ORC.
• Ability to optimize Spark jobs and SQL queries for scalability.
• Experience with NoSQL databases.

Key Skills

Education

Any Graduate