Description

You will design and build data pipelines and microservices within a big data ecosystem.

Responsibilities

• Develop data pipelines using PySpark libraries.
• Design and implement unit, component, and integration tests.
• Collaborate with engineers to create common code bases using GitHub SCM.
• Manage non-functional requirements including security, compliance, performance, and maintenance.
• Provision RESTful APIs to enable real-time data consumption.

Required Skills

• 7+ years of industry experience with 3+ years of relevant big data experience.
• Strong programming experience in Python and PySpark.
• Experience with Hadoop and big data frameworks such as Spark, Hive, NiFi, Spark-Streaming, or Flink.
• Proficiency with SQL and SQL analytical functions.
• Experience with NoSQL data stores like HBase or DynamoDB.
• Knowledge of SCM tools including GitHub and Bitbucket.
• Experience building domain-driven microservices.
• Hands-on work with structured and unstructured datasets including CSV, TXT, JSON, and XML.
• Bachelor's degree in computer science, engineering, mathematics, statistics, or a related field.

Preferred Skills

• Master's degree in a quantitative or technical field.
• Experience with big data streaming services such as Kinesis or Kafka.
• Proven skill in using Agile and LEAN methodologies.

Key Skills

Education

Not specified