Description

You will design, build, and manage scalable data infrastructure and pipelines using AWS services to support large-scale data requirements.

This role is on-site.

Responsibilities

• Lead the design and optimization of complex data models and ETL processes.
• Architect scalable data infrastructure using AWS services including S3, Redshift, Glue, Kinesis, DynamoDB, and Timestream.
• Design real-time data processing pipelines for IoT and time-series datasets.
• Build and optimize workflows using Apache Airflow and AWS Step Functions.
• Mentor junior engineers and collaborate with data scientists to ensure secure data access.

Required Skills

• 5+ years of experience in data engineering.
• Extensive experience developing PySpark code for AWS Glue jobs and EMR.
• Proficiency in Python and PySpark for building frameworks and large-scale processing workflows.
• Expertise in optimizing Spark SQL and Hive queries.
• Strong experience integrating APIs with Python and handling large datasets using Spark DataFrames and RDDs.
• Deep knowledge of AWS services: S3, Lambda, Glue, Kinesis (Data Streams/Firehose), Redshift, DynamoDB, and Timestream.
• Experience with the Hadoop ecosystem and distributed data systems.

Preferred Skills

• Understanding of DevOps practices and CI/CD pipelines for data solutions.
• Hands-on experience with time-series databases and IoT data handling.
• Knowledge of Scala, Java, or Snowflake.

Key Skills

Education

Not specified