Description

You will build and maintain data pipelines within a Hadoop ecosystem.

Responsibilities

• Develop Scala, Spark, and PySpark jobs for data transformation and aggregation.
• Write unit tests for Spark transformations and helper methods.
• Use Spark-SQL to read Parquet data and create Hive tables via the Scala API.
• Perform code quality reviews and act as a gatekeeper for quality checks through pair programming.
• Collaborate with Business Analysts to review test results and obtain sign-off.
• Prepare design and operations documentation for production deployments and post-deployment verification.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Hadoop, Spark, Scala, and Java.
• Experience with Hive, Impala, and Cloudera Distribution.
• Hands-on expertise with SQL, Shell Scripting, and Ada.
• Solid understanding of data structures, data manipulation, and distributed processing.
• Experience with CI/CD tools including Git and Jenkins.
• Knowledge of Agile methodologies and DevOps practices.
• Familiarity with Oracle, Kafka, Spark streaming, and ML.
• Strong knowledge of data warehousing methodology.

Preferred Skills

• Experience with PySpark for data processing.

Key Skills

Education

Any Graduate