Description

Job Description

Senior Databricks Developer is responsible for designing, developing, optimizing, and operating data solutions on the Databricks platform. The role contributes to the implementation and evolution of the organization's Data Lakehouse by developing data ingestion, transformation, processing, and delivery processes for reporting, advanced analytics, and AI/ML initiatives. The individual will have end-to-end responsibility for data pipelines and will act as a technical reference within the team.

Responsibilities 

Data Development and Integration

• Develops and maintains data ingestion, transformation, and publishing processes using Databricks and PySpark.
• Builds and optimizes ETL/ELT pipelines for large volumes of data.
• Implements Delta Lake structures and data models for operational and analytical consumption.
• Develops incremental loading, historization, and CDC (Change Data Capture) mechanisms.
• Integrates data from multiple sources, including Oracle, SQL Server, APIs, files, banking applications, and others.

Data Architecture and Modeling

• Participates in defining the Data Lakehouse architecture.
• Proposes scalable and high-performance data models.
• Contributes to the development of coding standards and platform architecture.
• Ensures data traceability and consistency throughout the entire processing flow.

Data Quality and Operations

• Implements automated data quality controls.
• Develops data reconciliation and validation processes.
• Configures process monitoring, alerting, and logging.
• Analyzes and resolves production incidents.
• Manages data reloads, reruns, corrections, and recovery processes.

Optimization and Performance

• Optimizes PySpark and SQL processes for cost efficiency and performance.
• Identifies and eliminates processing bottlenecks.
• Optimizes the use of Databricks resources.
• Contributes to the automation of development and deployment processes.

Job Requirements

• Minimum 5 years of experience in Data Warehouse, Big Data, or enterprise data platform development.
• Minimum 2-3 years of hands-on experience with Databricks.
• Demonstrated experience in complex Data Warehouse, Data Mart, or Data Lake projects.
• Experience developing end-to-end data pipelines.
• Strong knowledge of Data Lakehouse architecture.
• Experience implementing historization, CDC, reconciliation, and Data Quality processes.
• Ability to take full ownership of technical deliveries and provide technical leadership to the team.

Mandatory Tech Skills:

• Azure Databricks/PySpark
• SQL avansat
• Delta Lake
• Git & Jira
• Data Modelling
• ETL / ELT
• Data Quality Frameworks
• Pipeline Orchestration

Nice to have technical skills: 

• Azure Data Factory
• Kafka
• Event Streaming
• Python avansat
• CI/CD
• Unity Catalog
• Data Governance
• Machine Learning pe Databricks
• Qlik Sense

Key Skills

Education

Any Graduate