Description

Key Responsibilities
· Design, build, and maintain scalable data pipelines.
· Develop ETL/ELT workflows to ingest data from multiple sources.
· Develop and optimize data pipelines using Databricks, PySpark, and Spark SQL.
· Build and optimize data models for analytics and reporting.
· Integrate APIs, databases, and third-party systems.
· Work with Databricks and Delta Lake for data processing and storage.
· Ensure data quality, governance, security, and reliability.
· Optimize SQL queries and improve data pipeline performance.
· Collaborate with data analysts, BI developers, and application teams.
· Troubleshoot production issues and implement performance improvements.
· Document data architecture, workflows, and technical solutions.

Required Skills
· Strong SQL and database optimization skills.
· Strong Python skills for data engineering and automation.
· Hands-on experience with Azure Databricks.
· Strong knowledge of PySpark / Apache Spark.
· Experience with Delta Lake and Databricks data pipelines.
· ETL tools such as Azure Data Factory, Dataflow Gen2, SSIS, or Informatica.
· Experience with cloud platforms (Azure, AWS, or GCP).
· Data warehousing concepts and technologies such as Snowflake, Azure Synapse, Redshift, or BigQuery.
· Experience with Git and CI/CD practices.
· Familiarity with REST APIs and JSON/XML data formats

Key Skills

Education

Bachelor's degree