Description

Design, build, and maintain scalable data pipelines and architectures on AWS. Own the end-to-end lifecycle of batch and real-time data processing.

Responsibilities

• Design and maintain ETL processes and data pipelines on AWS for batch and real-time data.
• Implement data storage solutions, including data lakes and warehouses, using S3, Redshift, DynamoDB, RDS, Glue, EC2, and EMR.
• Optimize and tune data processing and transformation jobs for performance and scalability.
• Implement standardized, governed, and optimized data models in the consumption layer.
• Monitor pipeline health, troubleshoot issues, and implement proactive solutions.

Required Skills

• 5+ years of experience in data engineering with a focus on AWS cloud technologies.
• Proficiency in Python, Scala, or Java.
• Strong experience with Apache Spark and ETL frameworks.
• Hands-on experience with AWS services: S3, Redshift, Glue, EMR, EC2, DynamoDB, RDS, Lambda, and Athena.
• Experience with SQL and NoSQL databases.
• Proficiency in scripting languages including Bash and Shell.
• Knowledge of Airflow, Hadoop, and Kafka.
• Experience with data modeling, governance, security, and compliance practices.
• Familiarity with data formats such as Parquet, ORC, Avro, and JSON.

Preferred Skills

• AWS Certifications, such as AWS Certified Data Engineer.

Key Skills

Education

Any Graduate