Description

You will build, maintain, and optimize large-scale data pipelines and features on AWS using Spark.

Build and optimize data pipelines using Spark to consolidate data assets. Migrate data platforms into AWS environments and implement process automation. Design resilient, secure, and high-performing platforms using AWS services like EC2, S3, RDS, and DynamoDB. Partner with architects and analysts to develop technical architectures for enterprise projects. Own end-to-end platform issues, providing solutions for build and performance optimization.

Required Skills

• 5+ years of experience in a data engineering position.
• Proficiency in Python and SQL.
• Strong experience building data pipelines with Spark and PySpark.
• Hands-on experience building data processing infrastructure in AWS (EMR, Glue, S3, EC2, etc.).
• Experience with relational datastores, NoSQL datastores, and cloud object stores.
• Experience with container orchestration services including Docker and Kubernetes.
• Ability to design and build web environments on AWS including ALB, NLB, and Aurora Postgres.

Preferred Skills

• Experience with infrastructure as code solutions, specifically Terraform.
• Production experience with ACID compliant formats such as Hudi, Iceberg, or Delta Lake.

Key Skills

Education

Any Graduate