Senior Databricks Engineer

We are looking for a skilled and passionate Senior Databricks Engineer to design, build, and optimize enterprise-scale data lakehouse solutions on the Databricks platform. The successful candidate will be responsible for creating Databricks pipeline delivering Financial Crime platforms covering Anti-Money Laundering (AML), Know Your Customer (KYC), Customer Risk Assessment (CRA), Sanctions Screening, Transaction Monitoring, Fraud Detection, and Regulatory Reporting

Databricks Platform Engineering

• Design, build, and maintain Databricks workspaces, clusters, and compute pools across dev/test/prod environments.

• Configure and manage Databricks Unity Catalog for data governance, access control, fine-grained permissions, and data lineage.

• Optimize cluster configurations — instance types, auto-scaling policies, spot/preemptible nodes — for cost and performance.

• Implement workspace-level best practices: folder structures, access controls, secret management (Databricks Secrets / Azure Key Vault / AWS Secrets Manager).

• Manage Databricks jobs, workflows, and multi-task job orchestration with dependency management.

Delta Lake & Lakehouse Architecture

• Design and implement Delta Lake tables with appropriate partitioning, Z-ordering, and file compaction (OPTIMIZE / VACUUM).

• Build Medallion Architecture (Bronze / Silver / Gold) layers for structured data lake organization.

• Implement Delta Live Tables (DLT) pipelines for declarative, reliable ETL/ELT with built-in data quality expectations.

• Manage schema evolution, table versioning, time travel, and Change Data Feed (CDF) for incremental processing.