Description

Manage and optimize big data infrastructure and processing pipelines in Franklin, TN.

Manage and optimize big data infrastructure and processing pipelines in Franklin, TN.

Responsibilities

• Deploy and manage multi-node development and production clusters.
• Build PySpark applications to load streaming data with minimal latency.
• Develop Hive queries and implement query optimization techniques.
• Monitor and schedule workflows using Oozie.
• Import and export data from MySQL and RDBMS into HDFS using Sqoop.

Required Skills

• 5+ years of experience in Hadoop and Big Data technologies.
• Proficiency in PySpark programming and spark-based applications.
• Hands-on experience with AWS cloud data warehouse and AWS S3.
• Expertise in writing complex SQL queries for data integrity and database testing.
• Experience with Hive, Sqoop, and Oozie.
• Working knowledge of MySQL and RDBMS.
• Experience operating within Scrum, Agile, or Waterfall methodologies.

Preferred Skills

• Experience managing large-scale HDFS environments.

Key Skills

Education

Any Graduate