Astronomer Architect
tata consultancy servicesPlano, TX
$100,000-$120,000 a year
APPLY (opens in a new tab)Astronomer Architect
tata consultancy servicesPlano, TXLead L6
$100,000-$120,000 a year
Technical Architect We're seeking a seasoned engineer to design, operate, and scale our workflow orchestration platform with a primary focus on Apache Airflow. You'll own the Airflow control plane and developer experience end-to-end—architecture, auto security, observability, and reliability—while also evaluating and operating complementary schedulers where appropriate. You'll build automation infrastructure and partner across data, trading, and engineering teams to deliver mission-critical pipelines at scale. Architect, deploy, and operate production-grade Airflow on Kubernetes including all components and user application dependencies, with focus on upgrades, capacity planning, HA, security, and performance tuning.
Roles & Responsibilities:
Operate a multi-scheduler ecosystem: determine when to use Airflow, distributed compute schedulers, or lightweight task runners based on workload requirements; provide unified developer experience across schedulers
Build automation infrastructure: Terraform modules and Helm charts with GitOps-driven CI/CD for environment provisioning, upgrades, and zero-downtime rollouts
Standardize the developer experience: DAG repo templates, shared operator libraries, connection and secrets management, dependency packaging, code ownership, linting, unit testing, and pre-commit hooks
Implement comprehensive observability: metrics collection, dashboards, distributed tracing, SLA/latency monitoring, intelligent alerting, and runbook automation
Enable resilient workflow patterns: build idempotency frameworks, retry/backoff strategies, deferrable operators and sensors, dynamic task mapping, and data-aware scheduling
Ensure reliability at enterprise scale: architect and tune resource allocation (pools, queues, concurrency limits) to support high throughput workloads; optimize large-scale backfill strategies; develop comprehensive runbooks and lead incident response/postmortems
Partner with teams across the organization to provide enablement, documentation, and self-service tooling
Mentor engineers, contribute to platform roadmap and technical standards, and drive engineering best practices
Nice To Have:
- Experience leading migrations to/from Airflow
- Open Lineage/Marquez adoption; Great Expectations or other data quality frameworks; data contracts.dbt Core/Cloud orchestration patterns (state management, artifacts, slim Ci).Cost optimization and capacity planning for schedulers and workers; spot instance strategies.
- Multi-region HA/DR for Airflow metadata DB; backup/restore and disaster drills.
- Building internal developer platforms/portals (e.g., Backstage) for self-service pipelines.
Contributions to Apache Airflow or provider packages; familiarity with recent AlPs/Airflow 2.7+ featuresTCS Employee Benefits Summary: Discretionary Annual Incentive. Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans. Family Support: Maternal & Parental Leaves. Insurance Options: Auto & Home Insurance, Identity Theft Protection. Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement. Time Off: Vacation, Time Off, Sick Leave & Holidays. Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
Salary Range: $100,000-$120,000 a year
Location: Plano, TX
Matching similar jobs
Level
Lead L6
Salary
$100,000-$120,000 a year
Location
Plano, TX
Occupation
Computer Systems Engineers/Architects
Industry
Computer Systems Design Services
Posted
yesterday