Description

Key Responsibilities

• Lead the design and development of scalable data engineering solutions using Databricks.
• Build and maintain ETL and ELT pipelines for batch and large-scale data processing.
• Develop data transformation workflows using SQL, Python, PySpark, and Databricks notebooks.
• Design and optimize PostgreSQL databases, schemas, tables, views, functions, and stored procedures.
• Implement database version control and automated schema deployment using Liquibase.
• Develop and maintain Delta Lake tables and lakehouse-based data solutions.
• Integrate data from relational databases, APIs, files, cloud storage, and enterprise applications.
• Optimize Databricks workloads, Spark jobs, SQL queries, and PostgreSQL database performance.
• Establish standards for data modeling, coding, testing, deployment, and monitoring.
• Implement data validation, reconciliation, error handling, logging, and recovery mechanisms.
• Support CI/CD pipelines for data engineering and database deployment processes.
• Conduct code reviews and provide technical guidance to data engineering teams.
• Troubleshoot production issues and perform root-cause analysis.
• Collaborate with data architects, analysts, governance teams, and business stakeholders.
• Ensure data solutions meet security, scalability, reliability, and performance requirements.
• Create technical documentation, data mappings, deployment procedures, and support runbooks.

Required Qualifications

• 7+ years of experience in data engineering, database development, or data platform development.
• Advanced hands-on experience with Databricks.
• Advanced experience with PostgreSQL.
• Strong experience using Liquibase for database change management and schema migrations.
• Strong knowledge of SQL, Python, PySpark, Spark SQL, and distributed data processing.
• Experience building ETL/ELT pipelines and cloud-based data platforms.
• Strong understanding of data warehousing, lakehouse architecture, and data modeling.
• Experience with performance tuning, query optimization, and troubleshooting.
• Experience implementing CI/CD processes for data pipelines and database changes.
• Strong leadership, communication, and problem-solving skills.

Preferred Qualifications

• Experience with Delta Lake and Databricks Workflows.
• Experience with AWS, Azure, or GCP cloud data services

Key Skills

Education

Bachelor's degree