Hadoop/Big Data Developer

Key Skills: Hdfs, Debugging, Hive, UNIX, Map Reduce, Hadoop, Spark, Banking Operations, Autosys, Agile Methodology, Kafka

Roles and Responsibilities:

• Design, develop, and maintain scalable data pipelines for extracting, transforming, and loading data from various sources to ensure seamless data flow and accessibility.
• Collaborate with cross-functional teams to integrate data from multiple disparate sources, ensuring consistency, accuracy, and reliability of data.
• Optimize data processing workflows and storage solutions for performance, scalability, and cost-efficiency.
• Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data throughout the data lifecycle.
• Monitor the performance of data pipelines and infrastructure, identifying and resolving issues to maintain system stability and reliability.

Skills Required:

Must-Have:

(Ideally should not be more than 3-5)

• Strong experience in Snowflake, SQL for data manipulation and engineering tasks.
• Strong expertise in SQL and Snowflake architecture.
• Expertise in data architecture, data governance, data quality standards, and data security practices to ensure compliance and protection of sensitive information.
• Experienced in designing, owning, and delivering Snowflake data platforms based on customer requirements.
• Experience in ETL tools - Informatica / DBT.
• Experience in one or more cloud platforms - Preferably Azure / AWS / GCP.
• In-depth knowledge of data warehousing concepts and experience in building and managing data warehouses.
• Knowledge in ETL concepts - needs to manage data migration into Snowflake. This includes extracting data from various sources, transforming it into a usable form, and loading it into the Snowflake platform.

Good-to-Have

(Ideally should not be more than 3-5)

• Strong understanding of data modeling concepts and techniques to create efficient and scalable data models.
• Knowledge of agile based execution.
• Experience with version control systems such as Git for code management and collaboration.

Education: Bachelor's or Master's degree in Engineering