Description

You will architect and maintain backend systems that ingest, cache, and process terabyte-level weather data.

This role is on-site.

Responsibilities

• Architect AWS and Databricks solutions for existing backend systems handling large-scale weather data.
• Build infrastructure as code using Terraform to improve scalability and optimize cloud costs.
• Create and maintain REST APIs and backend processes using AWS Chalice or Lambda.
• Support Databricks and Spark pipelines for data backfill into S3 using Parquet or Delta Lake.
• Migrate CI/CD pipelines to GitHub Actions and enhance monitoring using Datadog.

Required Skills

• 5+ years of experience with Python and production software deployment.
• Core understanding of AWS services including IAM, Lambda, S3, and RDS.
• Experience with big data processing at terabyte or petabyte scale.
• Proficiency in SQL and PySpark for data analysis and pipeline development.
• Experience building REST APIs and backend services.
• Knowledge of containerization (Docker), caching (Redis), and databases (PostgreSQL/PostGIS).
• Familiarity with Databricks and Terraform.

Key Skills

Education

Bachelor's degree