Lead Data Engineer

We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise-scale data ecosystem. In this senior role, you will spearhead the build-out of high‑performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end‑to‑end reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and mission‑critical, data‑driven decision-making within our insurance-focused business domains.

• Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.

• Build and optimize high‑volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.

• Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.

• Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.

• Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.

• Implement robust data quality and validation frameworks using tools like Great Expectations, DLT expectations, or custom PySpark checks.

• Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.

• Architect end‑to‑end data solutions supporting analytics, ML, actuarial models, and business-critical reporting workloads.

• Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.