Description

Design and implement large-scale data engineering solutions on AWS, building production-grade pipelines using Databricks, Spark, and PySpark.

This role is remote.

Responsibilities

• Build and maintain data pipelines using Spark and PySpark within Databricks.
• Design and deploy scalable data engineering solutions on AWS from MVP to production.
• Manage data governance, compliance, and cataloging using Databricks Unity Catalog.
• Develop data models for 360 customer views and behavior analysis.
• Collaborate with stakeholders to define system specifications and strategic IT solutions.

Required Skills

• 13+ years of professional experience in data engineering or cloud architecture.
• Expertise in Spark and PySpark for pipeline development.
• Hands-on experience with Amazon S3, IAM, EC2, EMR, Kinesis, Lambda, Redshift, CloudWatch, SNS, SQS, Glue, and Athena.
• Deep understanding of Delta Lake, Medallion architecture, and Delta tables.
• Experience handling structured and unstructured data in ORC and Parquet formats.
• Proficiency in data validation, data modeling, and data visualization.
• Knowledge of enterprise systems, networking modules, and software integration.

Preferred Skills

• Ability to manage projects independently from development to production.

Key Skills

Education

Any Graduate