Description

You will design and implement data pipeline architectures moving data from source to target systems.

Responsibilities

• Design data pipeline architecture from source to target.
• Develop Spark applications using PySpark and Spark-SQL for extraction, transformation, and aggregation.
• Build NiFi workflows to ingest data from source systems into Hadoop and Amazon S3.
• Develop Kafka producers and consumers for topic publishing and subscription.
• Analyze datasets to debug data quality, cleanse data, and generate reports.

Required Skills

• 5+ years of experience in big data development.
• Proficiency with PySpark and Spark-SQL.
• Experience with Hadoop ecosystem including Hive, MapReduce, Sqoop, and Oozie.
• Hands-on experience with NiFi and Kafka.
• Working knowledge of Shell scripting.
• Experience with ZooKeeper.
• Practical use of Snowflake and Databricks.
• Bachelor's degree in Computer Science or equivalent.

Key Skills

Education

Bachelors