Description

Design and maintain complex data ingestion pipelines and transformations within a centralized enterprise data hub.

Responsibilities

• Build and manage data pipelines using Spark, Hive, Sqoop, and Impala.
• Execute migrations from legacy systems to Snowflake and AWS cloud environments.
• Manage data serialization using Avro, Parquet, JSON, and ORC formats.
• Handle full SDLC phases including system design, development, testing, and production support.
• Implement data governance and management strategies across the big data ecosystem.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency in Python, SQL, and Shell Scripting.
• Deep expertise in HDFS, Hive, Oozie, Sqoop, Impala, and Presto.
• Hands-on experience with AWS services including EMR, S3, Athena, Lambda, CloudWatch, SQS, SNS, Step Functions, and IAM.
• Strong experience with Snowflake and data warehousing.
• Competency with orchestration tools such as Airflow, Oozie, and CTRL-M.
• Experience with CI/CD and build tools like Jenkins, Maven, and GIT.
• Database management experience with MySQL, Teradata, and SQL Server.
• Knowledge of data serialization techniques like Avro, Parquet, and ORC.

Preferred Skills

• Experience with the Kubernetes ecosystem.
• Ability to perform data analysis and visualization using Databricks.

Key Skills

Education

Any Graduate