Description

Key Skills: Hadoop, Spark, PySpark, Testing, Automation Testing, Data Engineering, MS SQL

Roles & Responsibilities:

• Design, develop, and maintain scalable big data solutions using Hadoop and Spark.
• Build and optimize data processing pipelines using PySpark.
• Perform testing and automation testing to ensure data quality, accuracy, and integrity.
• Collaborate with data engineering teams to design and optimize data workflows.
• Utilize Unix/Shell scripting for data processing, automation, and job orchestration.
• Apply testing methodologies to validate data warehouse and ETL solutions.
• Monitor and troubleshoot data pipelines to ensure performance and reliability.
• Work closely with stakeholders to gather requirements and deliver data-driven solutions.
• Ensure best practices in data engineering, data governance, and system performance.

Experience Required:

• 5 - 9 years of experience in big data engineering or data engineering roles.
• Strong hands-on experience with Hadoop, Spark, and PySpark.
• Experience in data pipeline development and ETL processes.
• Good understanding of testing and automation testing in data environments.
• Experience with Unix/Linux and Shell scripting.
• Knowledge of data warehousing concepts and tools.
• Familiarity with MS SQL and relational databases is an advantage.
• Strong analytical, problem-solving, and communication skills.

Education: B.Tech

Key Skills

Education

Any Graduate