Description

You design, build, and maintain data pipelines and storage architectures for warehousing.

This role is on-site.

Responsibilities

• Develop complex logic and generic validation code using Spark.
• Create Sqoop jobs with incremental loads to store data in HDFS snapshot areas.
• Design Hive schemas and implement performance tuning via partitioning and bucketing.
• Ingest data using Avro schema files into Hive NoSQL databases.
• Develop scripts using Python and shell scripting languages.

Required Skills

• Python and shell scripting for automation and tooling.
• Spark for complex data validation and processing logic.
• Sqoop and HDFS for data ingestion and storage management.
• Hive schema design, partitioning, and bucketing for performance optimization.
• Avro schema files for structured data ingestion.
• ETL development experience for data warehousing solutions.
• Java and C# programming capabilities.
• Experience with AWS, Azure, DevOps, or AngularJS.

Preferred Skills

• Master’s degree in Science, Technology, or Engineering.
• Bachelor’s degree in the above fields with 5 years of related experience.

Key Skills

Education

Any graduate