Big Data Scientist

Responsibilities

• Define data requirements; discover, integrate, and wrangle large volumes of structured and semi/unstructured data; validate outcomes using appropriate tooling in our data environment.
• Support standardized datasets and ad hoc analysis needs; build mechanisms to ingest, validate, normalize, cleanse, and enrich data for downstream consumption.
• Contribute to data policies and technical controls where needed (access patterns, retention considerations, and synthesizing/anonymizing sensitive attributes in line with enterprise standards).
• Implement rigorous data quality approaches for new and evolving sources; iterate with analytics partners to improve sourcing, preparation, and trust in datasets used for insights and modeling.
• Champion data engineering best practices (testing, observability, operational readiness) and contribute practical guidance on analytics preparation and visualization-friendly dataset design where applicable.
• Partner closely with data science and business intelligence teams to deliver data models and pipelines that support reporting, research, and machine learning workflows.
• Build pipelines that clean, transform, aggregate, and publish data from disparate systems into curated layers suitable for BI and advanced analytics.
• Use Databricks, SQL, Spark, scripting, and AWS services to integrate systems reliably and efficiently.
• Apply solid knowledge of data architecture principles and help lead initiatives from requirements through implementation—balancing correctness, performance, cost, and maintainability.

Qualifications Required

• • 5+ years of professional experience in data engineering (or equivalent depth), including manipulating, processing, and extracting value from large datasets in production settings.
• Databricks (must have): hands-on delivery on the Databricks platform, including developing and operating Databricks Jobs/Workflows, notebooks, and production pipelines using Apache Spark (Spark SQL and/or PySpark) on Databricks.
• Delta Lake on Databricks (must have): building and maintaining Delta Lake tables (for example: incremental processing, merges/upserts, schema evolution, and performance patterns aligned with Delta best practices).
• Delta Live Tables / DLT (must have): designing, implementing, and operating DLT pipelines (Python or SQL as used in your standards), including expectations around pipeline dependencies, incremental processing, and operational monitoring as supported by DLT.
• Databricks governance (must have): practical experience with Unity Catalog (or equivalent Databricks governance patterns strongly aligned to UC), including secured sharing concepts and catalog/table governance as implemented in your environment.
• 4+ years hands-on Apache Spark engineering for analytics workloads (batch and/or streaming), with ability to implement reliable transformations and troubleshoot performance issues.
• 3+ years working with Apache Kafka (or managed equivalents such as Confluent Kafka) for high-volume event/stream processing, including operational considerations (throughput, lag, scaling, replay scenarios).
• 3+ years building and operating AWS capabilities supporting analytics platforms (for example: S3, IAM-aligned access patterns, integration with compute/storage patterns common to lakehouse architectures; familiarity with services such as Glue/Lambda/MSK as applicable).
• Strong SQL: ability to write and optimize intermediate-to-advanced queries and translate business logic into reliable datasets.
• Demonstrated experience delivering ETL/ELT pipelines in a Databricks lakehouse environment; comfort with incremental loads, schema evolution, and data quality checks.
• Experience working in Agile delivery (Scrum/Kanban/SAFe or similar): backlog refinement, iterative delivery, dependency coordination.

Preferred

• Experience with Snowflake , or other large-scale analytical databases (helpful for hybrid/platform comparisons).
• NoSQL exposure (Cassandra or similar) where operational systems feed analytics platforms.
• Experience with enterprise messaging/integration ecosystems (TIBCO EMS, IBM MQ, etc.) in addition to Kafka—helpful when bridging operational feeds into analytics landing zones