Description

Manage and optimize data integration and ETL processes to ensure seamless data flow across systems.

Responsibilities

• Design, develop, and implement ETL processes using Pyspark, Hive, Hadoop, HDFS, Impala, Spark, Spark SQL, and Kafka.
• Collaborate with cross-functional teams to translate data requirements into technical specifications.
• Monitor existing ETL workflows, troubleshoot issues, and perform performance optimizations.
• Execute data quality and cleansing initiatives to maintain accurate and reliable datasets.
• Participate in the design and development of data warehouses, datamarts, and other storage solutions.

Required Skills

• 3-5 years of experience designing and implementing automation processes.
• Proficiency with Pyspark, Hive, Hadoop, HDFS, Impala, Spark, Spark SQL, and Kafka.
• Strong SQL skills and experience with relational databases such as Oracle, SQL Server, or MySQL.
• Solid understanding of data warehousing concepts, data modeling, and ETL best practices.
• Experience in performance tuning, troubleshooting, and debugging ETL processes.
• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Ability to communicate technical details to both technical and non-technical stakeholders.

Preferred Skills

• Previous exposure to cloud-based data integration using Informatica Cloud or other cloud ETL tools.
• Knowledge of Data Privacy regulations and best practices for handling sensitive data.
• History of leading ETL projects from initial design through to implementation.

Key Skills

Education

bachelor's degree in Computer Science, Information Technology