Description

You will build and migrate security analytics pipelines to support threat detection workflows within Databricks.

This role is on-site.

Responsibilities

• Design and build threat detection pipelines using PySpark, SQL, and Databricks.
• Migrate detection rules and content from legacy systems to Databricks native frameworks.
• Develop PySpark log pipelines, rule configuration files, unit tests, and data validation checks.
• Deploy data infrastructure and pipelines using Terraform.
• Optimize data workflows and queries for performance and scalability.

Required Skills

• 10+ years of experience as a PySpark / SQL Engineer.
• Proficiency in PySpark for large-scale data processing and transformation.
• Expertise in SQL and Databricks for building analytics pipelines.
• Hands-on experience with Terraform for infrastructure deployment.
• Experience building threat detection or log analytics pipelines.
• Ability to perform data validation and write unit tests for pipeline stability.
• Strong analytical skills for working with complex datasets.

Preferred Skills

• Familiarity with AWS or Azure cloud platforms.

Key Skills

Education

Any Gradute