Description

You will design, build, and maintain data pipelines and storage architectures.

Responsibilities

• Develop scripts using Python and shell scripting languages.
• Create Sqoop jobs with incremental loads to store data in HDFS snapshot areas.
• Design Hive schemas and implement performance tuning via partitioning and bucketing.
• Develop complex logic and generic validation code using Spark.
• Ingest data using Avro schema files into Hive NoSQL databases.

Required Skills

• Python and shell scripting.
• Sqoop and HDFS.
• Hive schema design, partitioning, and bucketing.
• Spark for complex data validation logic.
• Avro schema files.
• ETL development for data warehousing.
• Java and C#.
• Experience with AWS, Azure, DevOps, or AngularJS.

Preferred Skills

• Master’s degree in Science, Technology, or Engineering.
• Bachelor’s degree in the above fields with 5 years of related experience.

Key Skills

Education

Any graduate