Principal Data Engineer

Description: Provides expertise on the design and participates in building of data infrastructure to optimize data processing from a variety of data sources. Independently designs and implements data governance policies and procedures for data handling to manage data consistency, integrity, accuracy, and reliability. Provides expertise on and participates in the design and implementation of rigorous data validation and integrity checks, proactively mitigating data quality issues that could impact data pipeline and model performance. Leverages advanced knowledge of Extract, Transform, and Load (ETL) processes to design, develop, and optimize automated, scalable, and efficient data pipeline architectures to build reusable data products. Works independently and collaboratively in an agile development environment with other engineers to develop, maintain, and debug advanced data solutions that are scalable, efficient, cost effective, and reliable.

Responsibilities: Key Responsibilities Data Processing & Pipelining – Data Requirements, Collection, and Infrastructure: Mentors less experienced team members to identify data requirements and business objectives of a project or initiative. Provides expertise on the design and participates in building of data infrastructure to optimize data processing from a variety of data sources. Independently analyzes, designs, and troubleshoots data flows based on business needs. Participates and designs architecture, performance, and security reviews of the technical solution. Adjusts data collection processes that involve indexing and query optimizations for optimal performance. Builds Extract, Transform, and Load (ETL) pipelines to support efficient and scalable data collection and extraction. Engages with and holds the upstream and downstream teams accountable for the predefined service level agreements (SLAs). Manages relationships with the data providers. Data Processing & Pipelining – Data Governance: Independently designs and implements data governance policies and procedures for data handling (e.g., data retention) to manage data consistency, integrity, accuracy, and reliability throughout the data lifecycle. Leads the execution of redaction processes for Personally Identifiable Information (PII) and Protected Health Information (PHI) data, ensuring compliance with data privacy and security standards. Ensures minimal data collection and usage in accordance with data minimization principles. Follows data security measures to protect data from unauthorized access, use, disclosure, alteration, or destruction, proactively identifying and escalating potential issues. Ensures data compliance with relevant laws, regulations, and industry standards. Data Processing & Pipelining – Data Validation & Quality Assurance: Provides expertise on and participates in the design and implementation of rigorous data validation and integrity checks, proactively mitigating data quality issues that could impact data pipeline and model performance. Mentors less experienced team members to define data annotation and labeling processes to ensure data quality. Identifies opportunities for automation of data validation and governance, and implements them. Independently corrects deviations and non-conformance when identified. Data Pipeline and Solutions Engineering – Pipeline Design: Leverages advanced knowledge of ETL processes to design, develop, and optimize automated, scalable, and efficient data pipeline architectures to build reusable data products. Implements advanced data storage solutions to store the processed data to be used in a scalable, optimized, and efficient way for access and analysis. Mentors less experienced team members to manage the flow of data pipeline and storage day-to-day operations. Data Pipeline and Solutions Engineering – Data Solutions Engineering: Works independently and collaboratively in an agile development environment with other engineers to develop, maintain, and debug advanced data solutions that are scalable, efficient, cost effective, and reliable. Reviews runnable code and works on testing and debugging of data solutions with less experienced team members. Evaluates new technologies to create more robust data solutions. Enforces and documents code standards and guidance within the team. Creates documentation for design decisions, and obtains feedback from the broader architecture team before implementing. Gathers data and evidence to secure necessary approvals. Core Responsibilities Planning & Execution: Manages and coordinates moderately complex tasks, monitoring timelines and deliverables to ensure timely completion and adherence to requirements for a moderately sized project or initiative. Efficiently delegates, monitors, and prioritizes work across multiple projects, providing technical oversight and adjusting plans to address shifts in resources or timelines. Collaboration & Partnership: Collaborates across the organization to align on expectations and achieve shared objectives. Leverages understanding of business leaders, stakeholders, and/or customers to ensure proposed solutions meet their needs. Supports inclusivity by actively seeking and listening to diverse perspectives, ensuring others feel heard and respected. Problem Solving: Identifies and addresses moderately complex issues by analyzing a wide range of data and/or information to identify solutions in accordance with standard practices. Proactively escalates unresolved or critical issues with a thorough assessment and suggests potential solutions. Reviews, contributes to, and documents problem solving strategies. Continuous Learning: Pursues learning opportunities to expand knowledge and skills and/or tools in new areas and stays abreast of the latest industry trends and best practices. Proactively seeks and leverages ongoing feedback and training to improve skills. Coaches and mentors junior team members, fostering continuous learning and knowledge sharing within and across teams. Continuous Improvement: Develops ideas, recommends updates, and/or collaborates on the implementation of process improvements to increase the efficiency and effectiveness of processes, protocols, and workflows across teams, and evaluates the impact on key stakeholders. Solicits feedback from others on ideas for alternative approaches and methods for continued improvement. Performance and Development: Contributes to the talent development pipeline by participating in candidate interviews, assessing candidates, and providing hiring recommendations. LI-AJP Qualifications Minimum Job Qualifications Education and/or Experience: 11 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Bachelor's Degree in Software Engineering, Data Science, Computer Science, Information Systems, or related field AND 7 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Master's Degree in Software Engineering, Data Engineering, Data Science, Computer Science, Information Systems, or related field AND 5 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Doctorate in Computer Science, Data Science, Software Engineering, Information Systems, or related field AND 3 year of experience in software engineering, data engineering, data science, computer science, or information systems. Job Skills: Same skills as prior level plus; Data Modeling Demonstrated proficiency in designing data models to facilitate accurate and efficient data management. Innovation Demonstrated ability in or knowledge of innovation, including generating or supporting new ideas, technologies, or processes for organizational growth. Prototyping Demonstrated experience creating and refining prototypes for system validation and stakeholder feedback. API Integration Demonstrated ability to build and manage robust API integrations for seamless interoperability between systems. Cloud Computing Demonstrated ability in or knowledge of cloud computing, including deploying, managing, and securing cloud environments and applications. Preferred Job Qualifications Education and/or Experience: 12 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Bachelor's Degree in Software Engineering, Data Science, Computer Science, Information Systems, or related field AND 8 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Master's Degree in Software Engineering, Data Engineering, Data Science, Computer Science, Information Systems, or related field AND 6 years of experience in software engineering, data engineering, data science, computer science, or information systems OR Doctorate in Computer Science, Data Science, Software Engineering, Information Systems, or related field AND 4 years of experience in software engineering, data engineering, data science, computer science, or information systems.5c143e31-5e48-4549-b638-05792d185386