Description

You will own the design and maintenance of scalable ETL pipelines for data ingestion, transformation, and loading.

Responsibilities

• Develop and optimize Spark and Spark SQL jobs for large-scale data processing.
• Implement data warehousing solutions using SQL, PLSQL, and Teradata to ensure data integrity.
• Manage job scheduling via Autosys Scheduler to guarantee timely data workflows.
• Leverage AWS services (S3, EMR, Athena) for efficient cloud data lake processing.
• Utilize UNIX shell scripting and Python for data processing and automation tasks.

Required Skills

• 5+ years of experience in data engineering roles.
• Strong proficiency in Python and UNIX shell scripting.
• Practical experience with Apache Spark and Spark SQL.
• Solid knowledge of SQL, PLSQL, and Teradata database management.
• Hands-on experience designing and managing ETL pipelines.
• Experience with AWS data products including S3, EMR, and Athena.
• Deep understanding of data warehousing concepts and best practices.
• Experience with Autosys Scheduler or similar job scheduling tools.

Preferred Skills

• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree