Description

You will build, maintain, and optimize data pipelines and features for a large-scale data platform using Spark and AWS.

Responsibilities

• Build and optimize data pipelines using Spark to consolidate data assets.
• Migrate data platforms into AWS environments and implement process automation.
• Design resilient, secure, and high-performing platforms using AWS services like EC2, S3, RDS, and DynamoDB.
• Partner with architects and analysts to develop technical architectures for enterprise projects.
• Own end-to-end platform issues, providing solutions for build and performance optimization.

Required Skills

• 5+ years of experience in a data engineering position.
• Proficiency in Python and SQL.
• Strong experience building data pipelines with Spark and PySpark.
• Hands-on experience building data processing infrastructure in AWS (EMR, Glue, S3, EC2, etc.).
• Experience with relational datastores, NoSQL datastores, and cloud object stores.
• Experience with container orchestration services including Docker and Kubernetes.
• Ability to design and build web environments on AWS including ALB, NLB, and Aurora Postgres.

Preferred Skills

• Experience with infrastructure as code solutions, specifically Terraform.
• Production experience with ACID compliant formats such as Hudi, Iceberg, or Delta Lake.

Key Skills

Education

Any Graduate