Description

You will build and maintain data pipelines within Big Data and Microsoft ecosystems.

Responsibilities

• Build PySpark frameworks to ingest data from various sources into Hadoop and SQL Server regions.
• Execute database migrations to MS SQL Server and Hadoop environments.
• Perform database query tuning, performance optimization, and troubleshooting on OLTP/OLAP systems.
• Set up CI/CD pipelines for database changes using GitHub, Jenkins, and Liquibase.
• Guide team members on project requirements and optimized solution design.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency in Spark, Scala, PySpark, and Hadoop (Hortonworks).
• Strong SQL skills with deep knowledge of RDBMS architecture and T-SQL optimization.
• Experience with Cloudera Data Platform (CDP) and Hive.
• Practical experience with Python, Shell Scripting, and Batch Scripting.
• Hands-on work with MS-SQL and RDBMS concepts.
• Experience using Git, SVN, Jira, and Agile/Scrum methodologies.
• Any graduate degree.

Preferred Skills

• Experience migrating data from Hadoop to CDP.

Key Skills

Education

Any gradudate