Description

Design and optimize data pipeline architectures to support cross-functional teams.

Responsibilities

• Build and expand data pipeline architectures and collection processes.
• Design efficient, well-structured data schemas and models to optimize storage and retrieval.
• Execute ETL processes to ensure seamless data movement and transformation.
• Troubleshoot data-related issues to maintain data quality and consistency.
• Create technical documentation to facilitate knowledge sharing across software developers and data scientists.

Required Skills

• 5-7 years of experience in data engineering.
• Proficiency in Python, including libraries such as Pandas and NumPy.
• Hands-on experience with the Hadoop ecosystem, specifically Hive, Sqoop, and Spark.
• Advanced SQL skills for crafting complex queries, data transformation, and aggregation.
• Strong understanding of ETL methodologies and data modelling.
• Experience processing and analyzing large-scale datasets efficiently.
• Ability to write and maintain shell scripts or automation scripts for data processing.

Preferred Skills

• Experience with Cloud Platforms including AWS, Azure, or GCP.
• Knowledge of Data Warehousing, Data Lakes, and Stream Processing.

Key Skills

Education

Any Graduate