Description

You will build and maintain large-scale data processing pipelines and management programs.

Responsibilities

• Develop and execute tests for data processing workflows.
• Collect and analyze data to ensure quality and accuracy.
• Identify, debug, and fix defects within the data stack.
• Integrate data streams using Spark or PySpark with Kafka.

Required Skills

• 10+ years of experience in Big Data stacks and Enterprise Architecture.
• 4+ years of hands-on experience with Spark.
• 3+ years of experience with SQL and NoSQL platforms.
• Proficiency in Python and PySpark.
• Experience with AWS and S3.
• Working knowledge of Snowflake.
• Ability to write Unix shell scripts.

Preferred Skills

• Experience with Big Data technologies and data management.

Key Skills

Education

Any Gradute