Description

You will build and maintain data pipelines and infrastructure using PySpark, Databricks, and Python.

This role is remote.

Responsibilities

• Develop data processing jobs using PySpark and Spark SQL.
• Implement data workflows within Databricks, utilizing Unity Catalog.
• Build and consume data services via REST API frameworks.
• Write and maintain data validation logic using Python and the Pandera Framework.

Required Skills

• 5+ years of professional experience in data engineering.
• Expert proficiency in PySpark and Spark SQL.
• Hands-on experience with Databricks and Unity Catalog.
• Strong command of Python programming.
• Experience designing and implementing REST APIs.
• Familiarity with CI/CD concepts.
• Experience with infrastructure as code using Terraform.

Preferred Skills

• Experience with GCP services (Firestore, Google Cloud Storage, Cloud Build, BigQuery).
• Familiarity with data movement tools like Airbyte or Fivetran.
• Experience writing unit tests using the Nutter Framework.

Key Skills

Education

Any Graduate