Databricks Python Developer
pb consultingManhattan, NY
Must Have Technical/Functional Skills
- 10+ years of experience in Data Engineering, Data Warehousing, or Big Data development.
- 5+ years of hands-on experience with Databricks, Python, and PySpark.
- Strong hands-on experience in Databricks, Python, and PySpark development.
- Experience building and supporting large-scale ETL/ELT data pipelines.
- Strong knowledge of Spark SQL, Delta Lake, and Data Lakehouse architecture.
- Experience working with structured and unstructured data from multiple sources.
- Strong SQL skills and experience with relational databases.
- Experience with data modeling, data transformation, and data quality processes.
- Knowledge of orchestration tools such as Airflow, ADF, or similar.
- Experience with Git, CI/CD, and Agile development methodologies.
- Strong analytical, debugging, and performance tuning skills.
- Excellent verbal and written communication skills.
Roles Responsibilities
- Design, develop, and maintain data pipelines using Databricks, Python, and PySpark.
- Build scalable data ingestion, transformation, and integration solutions.
- Collaborate with Data Architects, Analysts, and business teams to deliver data solutions.
- Develop and optimize batch and real-time data processing workflows.
- Ensure data quality, integrity, security, and governance standards are met.
- Monitor, troubleshoot, and resolve production data pipeline issues.
- Optimize Spark jobs, SQL queries, and Databricks workloads for performance.
- Participate in data modeling, code reviews, and platform enhancement activities.
- Create technical documentation and support production deployments.
- Participate in Agile ceremonies and contribute to continuous improvement initiatives.
Matching similar jobs
Level
Senior L5
Location
Manhattan, NY
Occupation
Data Warehousing Specialists
Industry
Custom Computer Programming Services
Posted
yesterday