Description

You will design, develop, and maintain data integration pipelines within an Apache Spark-based Enterprise Data Platform.

This role is on-site.

Responsibilities

• Design, test, and deploy data integration solutions to connect enterprise systems.
• Develop and improve batch and stream data processing pipelines.
• Optimize data pipelines for performance and cost within cloud environments.
• Facilitate requirements gathering workshops and author technical design documents.
• Implement standard operating procedures and lead technical review sessions.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency in Python, specifically PySpark, and common Python libraries.
• Advanced SQL skills including complex queries, query optimization, and UDFs.
• Experience with AWS services: EMR, Glue, Kinesis, Lambda, S3, Redshift, and RDS.
• Hands-on work with Apache Spark, Hadoop, Kafka, KStream, and Snowflake.
• Experience with ETL tools such as Informatica.
• Orchestration experience using Airflow, NiFi, or Autosys.
• Proficiency in Shell scripting and Git, Bitbucket, or Jenkins.
• Proven experience with data migration projects from on-premise to cloud.

Preferred Skills

• Experience with Databricks.

Key Skills

Education

Any Graduate