Description

You will build and maintain large-scale data processing systems using Databricks and PySpark.

This role is remote.

Responsibilities

• Develop, test, and maintain data processing systems using Spark clusters, SQL, and delta tables.
• Design and implement ETL processes including data cleansing, transformation, and integration.
• Build and manage data lakes following Medallion architecture principles.
• Monitor and optimize data pipelines using CI/CD practices.
• Collaborate with cybersecurity analysts and data architects to ensure data quality and governance.

Required Skills

• 3+ years of experience with Databricks, PySpark, and SQL.
• Hands-on experience with Spark clusters and Jupyter Notebooks.
• Proven experience building data lakes using Medallion architecture.
• Deep understanding of delta tables and the delta file format.
• Strong knowledge of ETL processes, data modeling, and data warehousing.
• Familiarity with CI/CD pipelines and Agile methodologies.
• Bachelor's or Master's degree in Computer Science or a related field.

Preferred Skills

• Experience with Power BI or other data visualization tools.
• Knowledge of cybersecurity data, specifically vulnerability scan data.

Key Skills

Education

Bachelor's or Master's degree in Computer Science