Description

You will build and manage data processing pipelines and distributed computing environments on AWS.

Responsibilities

• Develop Python scripts to execute PySpark jobs for large-scale data transformations, including converting TSV files to compressed Parquet formats.
• Implement Apache Presto and Apache Drill setups on AWS EMR clusters to enable cross-database querying between MySQL and Hive.
• Automate infrastructure provisioning using CloudFormation and JSON templates to replace manual service creation.
• Manage data integrity and security by developing shell scripts to collect and store user activity logs in AWS S3.
• Optimize Hive database performance through partitioning and bucketing strategies.

Required Skills

• 5+ years of experience in data development or engineering.
• Proficiency with PySpark and Python for data processing.
• Hands-on experience with AWS services including EMR, S3, RDS, DynamoDB, and CodeCommit.
• Strong knowledge of Apache Hive, including hive-site.xml configuration and metastore migration.
• Experience with SQL databases such as MySQL.
• Ability to work with JSON for infrastructure automation.
• Experience managing distributed query engines like Apache Presto or Apache Drill.
• Knowledge of shell scripting for log management and automation.

Preferred Skills

• Any Graduate

Key Skills

Education

Any Graduate