Description

Design and build large-scale data systems and complex processing pipelines. This role is on-site.

Responsibilities

• Design functional and technical architectures for big data systems.
• Build complex data pipelines using Spark and PySpark.
• Optimize and tune Spark application performance.
• Schedule and run Spark applications within the ecosystem.
• Develop applications on big data platforms using open source programming languages.

Required Skills

• 10+ years of IT experience with in-depth big data technology.
• Extensive programming experience in Python and PySpark.
• Hands-on experience with Spark and PySpark.
• Experience with Hadoop, Hive, and Spark.
• Knowledge of cluster management and storage mechanisms in Big Data Cloud.
• Ability to work closely with Administrators, Architects, and Developers.

Preferred Skills

• Experience with Airflow, Databricks, and Azure.

Key Skills

Education

Any Gradute