Azure DataBricks Developer
ace stackDallas, TX
Azure Databricks Developer
Location:
Louisville, KY/ Chicago, IL/ Dallas, TX / Arlington, VAFulltime OnlyMust Have Technical/Functional Skills
- Design, develop, and maintain cloud native data engineering solutions using Azure Databricks.
- Build and manage PySpark notebooks to process large scale structured and semi structured datasets.
- Design, create, and maintain Delta Lake tables, ensuring data reliability, ACID transactions, and schema enforcement.
- Develop scalable data workflows and pipelines using Databricks notebooks and orchestration patterns.
- Optimize performance of Spark jobs, including tuning partitions, memory usage, caching strategies, and query execution.
- Work extensively with PySpark and Spark SQL, choosing the appropriate approach based on use case and performance needs.
- Support cloud data migration initiatives, migrating data pipelines from on prem or legacy platforms to Azure Databricks.
- Integrate Databricks with upstream and downstream systems (e.g., data sources, storage layers, reporting tools).
- Ensure data pipelines are robust, reusable, and maintainable, following enterprise data engineering best practices.
- Implement error handling, logging, monitoring, and recovery strategies for production grade data pipelines.
- Collaborate with data architects, analysts, and downstream consumers to understand data requirements.
- Perform debugging and root cause analysis for data quality, performance, or pipeline failures.
- Support testing, validation, and reconciliation of data during development, migration, and production phases.
- Follow security, governance, and compliance standards applicable to cloud data platforms.
- Actively participate in Agile/Scrum delivery, owning data engineering stories from development through deployment.
- Maintain documentation for notebooks, workflows, data models, and migration approaches.
Roles & Responsibilities:
- Develop and maintain data engineering solutions using Azure Databricks and PySpark.
- Create, enhance, and optimize Databricks notebooks for data ingestion, transformation, and aggregation.
- Design and manage Delta Lake tables and pipelines supporting analytics and reporting use cases.
- Support cloud data migrations, including data validation and performance benchmarking.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Collaborate with platform, DevOps, and data governance teams to ensure environment stability.
- Perform data pipeline testing and validation, ensuring correctness and completeness.
- Troubleshoot and resolve issues related to Spark jobs, Delta tables, and workflow execution.
- Participate in code reviews and enforce data engineering best practices.
- Support production deployments and post deployment stabilization.
- Provide inputs to data architecture and platform improvement initiatives.
- Mentor junior data engineers when required.
Matching similar jobs
Level
Senior L5
Location
Dallas, TX
Occupation
Data Warehousing Specialists
Industry
Computer Systems Design Services
Posted
5 days ago