Description

You will build and maintain big data applications using Scala and Apache Spark on the Databricks platform.

This role is remote.

Responsibilities

• Design and develop ETL/ELT pipelines to ingest and transform structured and unstructured data.
• Optimize Spark jobs for performance and cost-effectiveness within Databricks environments.
• Collaborate with data scientists and engineers to translate business requirements into scalable solutions.
• Monitor data pipelines and troubleshoot production issues to ensure data quality.
• Implement unit and integration tests using tools like Delta Lake or Great Expectations.

Required Skills

• 5+ years of professional experience in software development.
• Strong proficiency in Scala and functional programming principles.
• Hands-on experience with Apache Spark and distributed data processing.
• Deep experience with Databricks, including workflows, jobs, cluster management, and Delta Lake.
• Experience building ETL/ELT pipelines.
• Knowledge of cloud platforms, with Azure preferred, or AWS/GCP.
• Familiarity with data formats such as Parquet, Avro, JSON, and CSV.
• Experience with file systems including HDFS, S3, or ADLS.
• Proficiency with Git, CI/CD pipelines, and orchestration tools like Airflow or Azure Data Factory.

Preferred Skills

• Experience with Great Expectations for data validation.

Key Skills

Education

Any Graduate