Description

You will design, build, and maintain scalable data pipelines using SSIS, SQL Server, Databricks, PySpark, and Spark SQL. You will own the development and optimization of batch and near-real-time data ingestion and transformation processes.

Responsibilities

• Develop and optimize Databricks pipelines using PySpark and Spark SQL for large-scale data transformation.
• Design and maintain enterprise data processing workflows using SSIS and SQL Server.
• Integrate legacy ETL processes with modern cloud-based platforms including Azure and AWS Lakehouse architecture.
• Ensure data quality, validation, reconciliation, and governance across all pipelines.
• Implement logging, error handling, and auditing frameworks for pipeline operations.

Required Skills

• 5+ years of hands-on experience with SSIS, SQL Server, and ETL development.
• Proficiency in Databricks, PySpark, and Spark SQL for data transformation.
• Strong SQL skills, including stored procedures and performance tuning.
• Experience with data pipeline architecture and performance optimization.
• Familiarity with Azure and AWS cloud environments.
• Understanding of data governance and audit frameworks.
• Ability to work with Policy, Claims, Billing, and Underwriting data domains.
• Experience with data migration from on-premise to cloud platforms.

Preferred Skills

• Property & Casualty Insurance domain knowledge.
• Experience with Agile methodology and regulatory compliance.

Key Skills

Education

Any Graduate