Description

You will build scalable data pipelines, develop ML models, and integrate big data systems across structured and unstructured data formats.

Responsibilities

• Build scalable ETL pipelines using Apache Spark, Talend, AWS Glue, Google Dataflow, and Apache NiFi.
• Ingest data from APIs, file systems, and databases.
• Perform data cleaning, transformation, and validation using Pandas, Apache Beam, and Dask.
• Process large datasets with Hadoop, Kafka, Apache Flink, and Apache Hive.
• Optimize system performance through distributed processing, parallelization, and data partitioning.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Apache Spark, AWS Glue, Google Dataflow, and Talend.
• Proficiency with Hadoop, Kafka, Apache Flink, Hive, and Presto.
• Strong experience with MySQL, PostgreSQL, MongoDB, and Cassandra.
• Hands-on experience with TensorFlow, PyTorch, Scikit-learn, and XGBoost.
• Experience managing data in AWS, Azure, GCP, or Databricks.
• Knowledge of S3, BigQuery, Redshift, Synapse Analytics, and HDFS.
• Ability to automate data quality checks using Pytest or Unittest.

Preferred Skills

• Experience managing asynchronous processing with Celery, RQ, RabbitMQ, or Kafka.

Key Skills

Education

Any Graduate