Description

You will build and manage data pipelines, models, and infrastructure to support large-scale data requirements.

Responsibilities

• Lead the design, optimization, and implementation of complex data models and ETL processes.
• Architect and manage scalable data infrastructure using AWS services including S3, Redshift, Glue, Kinesis, DynamoDB, and Timestream.
• Design real-time data processing pipelines for IoT and time-series datasets using Kinesis and AWS IoT Core.
• Build and optimize workflows using orchestration tools such as Apache Airflow and AWS Step Functions.
• Mentor junior engineers and collaborate with data scientists and analysts to ensure secure data access.

Required Skills

• 5+ years of experience in data engineering.
• Extensive experience developing PySpark code for AWS Glue jobs and EMR.
• Proficiency in Python and PySpark for building frameworks and large-scale processing workflows.
• Expertise in optimizing Spark SQL and Hive queries.
• Strong experience integrating APIs with Python and handling large datasets using Spark DataFrames and RDDs.
• Deep knowledge of AWS services: S3, Lambda, Glue, Kinesis (Data Streams/Firehose), Redshift, DynamoDB, and Timestream.
• Experience with the Hadoop ecosystem and distributed data systems.

Preferred Skills

• Understanding of DevOps practices and CI/CD pipelines for data solutions.
• Hands-on experience with time-series databases and IoT data handling.
• Knowledge of Scala, Java, or Snowflake.

Key Skills

Education

Not specified