About AbarVaAbarVa is building Nexus, an AI-native enterprise execution platform that helps organizations turn strategic decisions into governed action, measurable outcomes, and sustained business value. We are building Nexus alongside complex enterprise Data & AI transformations, particularly in regulated industries.
The Opportunity:
We are looking for an experienced Databricks Architect and Engineering Lead to lead the architecture and engineering execution of large-scale data modernization and migration programs. This is a hands-on architecture and engineering leadership role. We are looking for someone who has spent significant time personally building modern data platforms, has deep Databricks expertise, and can lead engineering teams through complex enterprise migrations. This is not a strategy-only or architecture-only position. The successful candidate must be able to design, build, review code, troubleshoot complex engineering issues, establish engineering standards, and lead teams through production delivery.
Required Experience:- 10+ years of hands-on data engineering experience 5+ years of hands-on Databricks experience
- Current Databricks certification is mandatory
- Significant experience as a Databricks Architect, Data Engineering Lead, Principal Data Engineer or equivalent
- Deep hands-on expertise with Databricks, Apache Spark, PySpark and SQLStrong experience with Delta Lake, Unity Catalog, Databricks Workflows and modern Databricks ingestion and transformation patterns
- Demonstrated experience designing and implementing enterprise-scale Bronze, Silver and Gold architectures
- Proven experience leading large-scale migrations from legacy data platforms to Databricks
- Experience assessing existing data workloads, ETL pipelines, data models, reports, KPIs and downstream dependencies
- Strong experience designing metadata-driven ingestion and transformation frameworks
- Strong knowledge of data quality, reconciliation, lineage, observability, security and governance
- Strong cloud platform experience; AWS experience is highly desirable
- Proven ability to lead and mentor teams of data engineers while remaining technically hands-onStrong communication skills and experience working directly with senior client and technology stakeholders
What You’ll Do:- Own the architecture and engineering execution of large-scale Databricks modernization and migration programs
- Assess complex legacy data environments and develop practical modernization strategies
- Determine which workloads should be migrated, refactored, redesigned, consolidated or retired
- Design scalable enterprise Bronze / Silver / Gold architectures
- Lead migration of legacy data warehouses, ETL platforms, analytics environments and reporting ecosystems to Databricks
- Architect reusable, metadata-driven ingestion, transformation and orchestration frameworks
- Lead enterprise data modeling and development of reusable data products
- Establish engineering standards for performance, security, governance, lineage, observability and data quality
- Lead reconciliation and parallel-run activities to prove Databricks outputs against trusted legacy platforms
- Define migration waves, cutover plans and production stabilization approaches
- Conduct architecture, design and code reviews
- Troubleshoot complex engineering, scalability and performance issues
- Lead and mentor distributed engineering teams
- Partner directly with business and technology leaders to translate business outcomes into scalable data solutions
- Introduce modern AI-assisted engineering approaches to improve development productivity while maintaining appropriate governance and human review
- Help develop reusable AbarVa engineering methods, accelerators and intellectual property
- Large-Scale Migration Experience
- Candidates should have personally led significant migrations involving platforms such as:
Netezza
TeradataOracle / ExadataSQL ServerHadoop ecosystems
Traditional enterprise data warehouses
Legacy ETL and integration platforms
Externally hosted analytics platforms
Enterprise reporting and BI environments
Candidates should be able to clearly explain how they:
Assessed the existing data landscape
Identified workload and reporting dependencies
Rationalized legacy pipelines and reports
Designed the target Databricks architecture
Sequenced migration waves
Reconciled legacy and Databricks results
Managed parallel operations
Executed production cutover
Stabilized and optimized the new environment
Architecture & Leadership Expectations
This role requires someone who can lead from the front technically.
You will be expected to:
Own critical architecture decisions
Lead engineering teams through complex delivery programs
Establish architecture and engineering standards
Mentor architects and data engineers
Conduct detailed design and code reviews
Make difficult architecture and engineering trade-offs
Work directly with senior client technology and business leaders
Translate complex technical decisions into clear business implications
Take accountability from architecture through successful production implementation
We are not looking for someone who only manages engineers or produces architecture diagrams. You must be comfortable getting deeply hands-on when required.
Industry Experience
Experience in healthcare, financial services, insurance or another highly regulated industry is a significant advantage.
Healthcare experience is particularly valuable, including exposure to:
Health-plan / payer data
ClaimsMembership, enrollment and eligibility
Medicare and Medicaid
Clinical and provider data
Epic / Epic Clarity
Healthcare financial and operational analyticsPHI and other regulated data environments
Deep healthcare expertise is not mandatory, but experience designing platforms where security, privacy, governance, auditability and data quality are critical is strongly preferred.
Work Authorization:
Candidates must be legally authorized to work in the United States on a W-2 basis without current or future visa sponsorship. AbarVa is not able to provide employment visa sponsorship for this role.
What Makes This Role Different:
You will not simply create an architecture and hand it to another team.
You will be expected to lead discovery, architecture, engineering, migration, reconciliation, cutover and production delivery.
We are looking for a Databricks architect, hands-on engineer and engineering leader who has successfully delivered large-scale modernization programs and can build and lead high-performing teams.
