Description

You will lead the design and implementation of complex data products, integrated data quality frameworks, and cloud-based big data platforms.

This role is remote.

Responsibilities

• Build new data pipelines and automated ETL processes to deliver enriched data to applications.
• Lead the transition of on-premise big data platforms to AWS cloud-based environments.
• Design real-time, event-driven data ingestion streams and ensure data security through encryption and access controls.
• Perform hands-on development, peer reviews, and impact analysis for design changes.
• Support data scientists with model testing and validation while ensuring test-driven development.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Extensive experience with AWS services including S3, EMR, Glue, Lambda, Athena, Kinesis, EC2, SNS, SQS, and CloudWatch.
• Proficiency in Python, Java, Scala, and Django.
• Hands-on experience with Spark, PySpark, and SQL.
• Infrastructure as Code expertise using Terraform.
• Experience with databases including Redshift, DocumentDB, DynamoDB, and MongoDB.
• Knowledge of big data technologies such as Hadoop, Hive, HBase, and Druid.
• Experience with DevOps pipelines, CI/CD, Bitbucket, and Concourse.
• Background in Power Systems engineering, DER Dispatch, or automated fault analysis systems.

Preferred Skills

• Experience with Pandas for data manipulation.

Key Skills

Education

Any Graduate