Description

You design, build, and manage large-scale data processing systems and distributed computing applications.

This role is hybrid.

Responsibilities

• Build and orchestrate complex data pipelines and ETL processes.
• Develop distributed computing applications using PySpark.
• Write, maintain, and execute automated unit tests in Python.
• Develop and maintain APIs for data access.
• Manage data warehousing and storage architectures.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Proficiency in Python programming.
• Strong expertise in SQL, Presto, Hive, and Spark.
• Experience with PySpark for distributed computing.
• Hands-on experience with Pandas, Polars, and NumPy.
• Extensive experience with AWS services including EMR, Lambda, Glue ETL, Step Functions, S3, ECS, Kinesis, IAM, RDS PostgreSQL, DynamoDB, Athena, SNS, SQS, and VPC.
• Knowledge of CloudWatch Events, Event Bridge, and Time-series databases.

Key Skills

Education

Any Graduate