Description

You will build and maintain end-to-end data pipelines and automation logic within an AWS environment.

Responsibilities

• Develop and maintain automation for data validation, pipeline testing, and transformation logic.
• Handle nested JSON structures and large datasets using local data processing.
• Execute ETL/ELT testing and data pipeline validation.
• Code, test, and troubleshoot independently without strict Dev/QA separation.
• Implement data modeling for streaming and batch architectures.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python (Pandas) and PySpark.
• Advanced SQL including Window Functions and CASE statements.
• Experience with AWS Lakehouse components: S3, Glue, Redshift, and Step Functions.
• Hands-on work with AWS Lambda and S3.
• Experience with CI/CD using Terraform and GitHub Actions.
• Knowledge of Shell scripting and Linux environments.
• Ability to perform multi-threaded debugging.
• Experience with healthcare data compliance.

Preferred Skills

• Familiarity with Apache Spark, Kafka, Airflow, and Delta Lake.

Key Skills

Education

Any Gradute