Description

You will design and maintain data pipelines using big data technologies and cloud infrastructure.

Responsibilities

• Build and manage ETL/ELT processes using AWS Glue, Python, and PySpark.
• Develop data models for both structured and unstructured data processing.
• Implement data integration and migration workflows using Hadoop and Kafka.
• Manage real-time streaming and data lake architectures.
• Automate workflows and deployment pipelines using Airflow, Jenkins, and Git.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python, Java, Scala, and SQL.
• Hands-on experience with Spark, PySpark, and Hadoop (HDFS, Hive, Cloudera, Hortonworks).
• Experience with AWS services including Glue, Lambda, Redshift, S3, and CloudWatch.
• Strong understanding of HiveQL and dimensional data modeling.
• Experience with Kafka and real-time streaming architectures.
• Knowledge of CI/CD and automation tools like Jenkins, Git, Autosys, and Airflow.
• Ability to work with Databricks environments.

Preferred Skills

• Experience managing large-scale data migrations.

Key Skills

Education

Any Graduate