Description

You will build and maintain threat detection pipelines for the security team using PySpark, SQL, and Databricks. Your primary focus is migrating detection rules and content from a legacy platform into Databricks' native internal detection framework. This involves creating equivalent PySpark log pipelines, writing rule configuration files, and deploying infrastructure via Terraform. You will write unit tests, perform data validation checks, and own the deployment into production. Minimum 5 years of experience is required.

Responsibilities

• Build PySpark log analytics pipelines for threat detection.
• Migrate detection rules and content from legacy systems to Databricks.
• Deploy and manage pipeline infrastructure using Terraform.
• Write unit tests and perform data validation checks.
• Deploy production-ready detection frameworks.

Required Skills

• 5+ years of professional experience in data engineering.
• Advanced proficiency in SQL.
• Strong experience with PySpark and Python.
• Hands-on experience with Databricks platform.
• Proficiency in Terraform for infrastructure deployment.
• Experience with GitHub for version control.

Preferred Skills

• Experience in security or threat detection domains.
• Familiarity with legacy platform migration projects.

Key Skills

Education

Any Gradute