Description

Build and optimize data pipelines for ingestion, transformation, and loading across enterprise-level systems.

Responsibilities

• Design, develop, and deploy Spark programs in Databricks to process large-scale data volumes.
• Optimize Databricks jobs for performance and scalability to handle big data workloads.
• Monitor and troubleshoot jobs to identify and resolve bottlenecks, ensuring data quality.
• Perform code reviews to ensure adherence to standards and optimal execution patterns.
• Implement data management, security, and governance best practices within Databricks.

Required Skills

• 5+ years of Python coding experience.
• 5+ years of SQL Server-based development of large datasets.
• 5+ years of experience developing and deploying ETL pipelines using Databricks and PySpark.
• Proficiency in SQL, including stored procedures and complex query writing.
• Experience with Snowflake, Databricks, Synapse, BigQuery, or Redshift.
• Strong knowledge of Data Warehousing concepts, including OLTP, OLAP, Dimensions, Facts, and Dimensional modeling.
• Experience with Delta Lake, data integration, and designing Enterprise Data Warehouse solutions.
• Ability to work with structured, semi-structured, and unstructured data.
• Experience with event-based or streaming technologies for data ingestion.

Preferred Skills

• Experience leading enterprise-wide Cloud Data Platform migrations.
• Background in Banking and Capital Markets, specifically investment banking.

Key Skills

Education

Any Graduate