Description

You will design, build, and optimize data pipeline architectures and data flows to support cross-functional teams.

Responsibilities

• Expand and optimize data pipeline architecture and data collection processes.
• Design efficient, well-structured data schemas and optimize storage in data warehouses.
• Build and maintain ETL processes to ensure seamless data movement and transformation.
• Troubleshoot data-related issues to maintain data quality and consistency.
• Collaborate with developers, architects, and data scientists to ensure optimal data delivery across projects.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Python, including libraries like Pandas and NumPy.
• Hands-on experience with the Hadoop ecosystem, specifically Hive, Sqoop, and Spark.
• Advanced SQL skills for crafting complex queries, transformations, and aggregations.
• Strong understanding of ETL methodologies and data modelling.
• Experience processing and analyzing large-scale datasets.
• Ability to write and maintain shell scripts or automation scripts for data processing.
• Experience with AWS or Azure.
• Any Graduate degree.

Preferred Skills

• Knowledge of Data Warehousing, Data Lakes, and Stream Processing.
• Experience with GCP or other Big Data frameworks.

Key Skills

Education

Any Graduate