Description

You will build and manage large-scale data processing pipelines using Big Data frameworks and cloud infrastructure.

This role is hybrid.

Responsibilities

• Develop data processing workflows using PySpark, Spark-Scala, and Hadoop.
• Orchestrate data pipelines using AWS MWAA (Airflow).
• Manage data storage and querying across Hive, HBase, and relational databases.
• Maintain data integration across AWS services like S3, EMR, and RDS.
• Process and transform diverse file formats including Parquet, Avro, JSON, and CSV.

Required Skills

• 5+ years of experience in Big Data development.
• Extensive experience with PySpark, Spark-Scala, Hadoop, Hive, and HBase.
• Proficiency in Python, SQL, and Linux shell scripting.
• In-depth experience with AWS S3, EMR, RDS, and MWAA (Airflow).
• Strong knowledge of Oracle, Redshift, and NoSQL databases.
• Expertise handling Parquet, Avro, JSON, and CSV file formats.

Preferred Skills

• Familiarity with Scala programming.

Key Skills

Education

Any Gradute