Description

Lead the design, build, and maintenance of high-performance analytical data pipelines and systems.

Responsibilities

• Own the architecture for data ingestion and near-real-time processing modules, ensuring performance and maintainability.
• Lead a team of data engineers by providing mentorship, setting priorities, and aligning technical work with business goals.
• Architect and build scalable pipelines for large volumes of structured and unstructured data using stream-based and batch processing.
• Create and manage RFCs, ADRs, and technical documentation following established architecture governance processes.
• Implement monitoring and alerting systems to detect and resolve pipeline failures and performance bottlenecks.

Required Skills

• 5+ years of experience in software engineering and data system design.
• Hands-on experience designing and implementing large-scale data pipelines in AWS.
• Expertise in PySpark, Python, and SQL.
• Proven experience owning large, complex system architectures.
• Experience with cloud-based PaaS Big Data technologies such as Databricks, Snowflake, or EMR.
• Proficiency with distributed file systems (S3, HDFS, ADLS) and table formats (HUDI, Iceberg).
• Deep understanding of cloud-based ETL offerings and orchestration technologies.
• Strong database design skills, including normalization, de-normalization, and data warehouse design.
• Knowledge of data security, privacy, and governance practices (SOX, PCI, HIPAA).

Preferred Skills

• Experience in the healthcare industry.
• AWS Certified Data Engineer or AWS Certified Solutions Architect.

Key Skills

Education

Any Graduate