Description

You will build and optimize data pipelines using PySpark and Python.

Responsibilities

• Develop ETL solutions and ELT data pipelines using PySpark and Python.
• Optimize Spark clusters within containerized environments.
• Integrate security, file transfer, and reporting tools into ETL workflows.
• Manage data movement across S3, NAS, and various database systems.

Required Skills

• 5+ years of experience in ETL development.
• Proficiency with Python and PySpark.
• Hands-on experience with Kubernetes and OpenShift.
• Working knowledge of S3 bucket and NAS storage systems.
• Experience with ETL tools such as Talend, Apache Airflow, or SSIS.
• Experience managing security via CyberArk and file transfers via SFTP.
• Familiarity with MongoDB, Hive, or MSaaS.
• Experience with Tableau for reporting.

Preferred Skills

• Experience with IBM DataStage.

Key Skills

Education

Any Graduate