Site Reliability Engineer, AI & Agentic Systems

Overview

As our SRE charter continues to evolve, this role demands strong hands-on ownership of production reliability and troubleshooting, coupled with advanced capabilities in AI- and agentic-driven automation and performance engineering.

The Site Reliability Engineer will play a critical role in ensuring reliability, scalability, performance, and operational excellence of our platforms. The ideal candidate will leverage Azure-native AI services and agentic systems to reduce toil, improve incident response, and enable intelligent operations—while also driving performance testing practices to validate system resilience under load.

This is a hybrid role, located at our Plano, TX office.  Candidates must be willing and able to work in-office 3 days per week in Plano, TX.

Applicants must be currently authorized to work in the United States on a full-time basis and must not require sponsorship for employment visa status now or in the future

A DAY IN THE LIFE

In this role, you will…

- Own end-to-end reliability of large-scale, Azure-hosted production systems, ensuring high availability, fault tolerance, and graceful degradation

• Lead hands-on incident troubleshooting, root cause analysis (RCA), and post-incident reviews with actionable follow-ups

• Build and operate resilient, scalable services on Microsoft Azure (AKS, App Services, Functions, Event Hubs, etc.)

• Design and maintain comprehensive observability platforms using Prometheus for metrics, Loki for log aggregation, Tempo for distributed tracing, and Grafana for dashboarding and alerting

• Design, develop, and execute performance testing strategies for distributed systems and microservices, including load testing, stress testing, soak testing, and capacity planning

• Integrate AI agents with Azure monitoring stack, CI/CD tooling, and incident management platforms

• Contribute to evolving SRE standards, tooling, operational processes, and knowledge base

 

Responsibilities

Reliability Engineering & Production Ownership

- Own end-to-end reliability of large-scale, Azure-hosted production systems, ensuring high availability, fault tolerance, and graceful degradation

• Lead hands-on incident troubleshooting, root cause analysis (RCA), and post-incident reviews with actionable follow-ups

• Define, measure, and enforce Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets aligned with business outcomes

• Drive proactive reliability improvements based on operational insights, failure mode analysis, and capacity planning

• Participate in on-call rotations and take real-time ownership during production incidents

Platform & Automation Engineering

- Build and operate resilient, scalable services on Microsoft Azure (AKS, App Services, Functions, Event Hubs, etc.)

• Design and maintain comprehensive observability platforms using Prometheus for metrics, Loki for log aggregation, Tempo for distributed tracing, and Grafana for dashboarding and alerting

• Create automation to eliminate manual operational tasks and reduce Mean Time to Recovery (MTTR)

• Implement self-healing mechanisms, automated remediation workflows, and runbook automation

• Manage and optimize API lifecycle and traffic management using Gravitee API Gateway

• Design and implement durable, fault-tolerant workflows and microservice orchestration patterns using Temporal

• Administer and tune PostgreSQL databases for reliability, performance, and high availability

• Partner with application and platform teams to improve service operability, deployment safety, and change management

Performance Testing & Load Engineering

- Design, develop, and execute performance testing strategies for distributed systems and microservices, including load testing, stress testing, soak testing, and capacity planning

• Build and maintain performance test scripts and virtual user scenarios using Micro Focus LoadRunner and VuGen (Virtual User Generator)

• Analyze performance test results to identify bottlenecks, regressions, and scalability limits; produce clear reports with actionable recommendations

• Integrate performance testing into CI/CD pipelines to enable continuous performance validation and shift-left testing practices

• Establish and monitor performance baselines, benchmarks, and SLAs across critical service endpoints and user journeys

• Collaborate with development and architecture teams to resolve performance issues and optimize system throughput, latency, and resource utilization

AI / Agentic Engineering (Azure Focus)

- Design and implement AI-driven and agentic systems to enhance operational workflows and intelligent decision-making

• Build intelligent automation for operational use cases, including:

  • Incident triage, enrichment, and automated escalation

  • Alert correlation, deduplication, and noise reduction

  • Automated diagnosis and remediation of recurring failures

• Leverage Azure AI services (Azure OpenAI, Cognitive Services, Azure ML) for operational intelligence and predictive insights

• Integrate AI agents with Azure monitoring stack, CI/CD tooling, and incident management platforms

• Ensure safe, reliable, and observable operation of AI-powered systems in production, including guardrails, fallback mechanisms, and audit trails

Collaboration & Technical Leadership

- Act as a reliability, performance, and automation champion across engineering teams

• Mentor junior SREs and influence adoption of best practices in reliability, observability, and performance engineering

• Contribute to evolving SRE standards, tooling, operational processes, and knowledge base

• Participate in architecture reviews and provide guidance on non-functional requirements (reliability, scalability, performance)

Qualifications

Core SRE Skills

- 5+ years of experience in Site Reliability Engineering, DevOps, or Production Engineering roles

• Strong hands-on experience in production troubleshooting of distributed systems at scale

• Solid understanding of Linux internals, networking (TCP/IP, DNS, HTTP, TLS), and system performance tuning

• Deep hands-on experience with Microsoft Azure (compute, networking, storage, managed services, AKS)

• Strong knowledge of Kubernetes, container orchestration, Helm charts, and microservices architectures

• Proficiency in or more programming languages: Python, Go, Java, or equivalent

• Experience with CI/CD pipelines (Azure DevOps, GitHub Actions) and Infrastructure as Code (Terraform, ARM Templates, Bicep)

Observability & Monitoring

- Hands-on experience building and operating observability stacks using Prometheus, Grafana, Loki, and Tempo

• Experience with alerting strategies, SLI/SLO-based monitoring, and on-call incident management

Performance Testing & Load Engineering

- Proven experience designing and executing performance and load testing for large-scale distributed applications

• Hands-on proficiency with Micro Focus LoadRunner and VuGen for scripting virtual user scenarios, parameterization, correlation, and result analysis

• Strong understanding of performance testing methodologies: load testing, stress testing, endurance/soak testing, spike testing, and capacity planning

• Ability to analyze performance metrics (throughput, response time, error rate, resource utilization) and translate findings into engineering actions

• Experience integrating performance tests into automated CI/CD pipelines

Platform & Middleware

- Experience with Gravitee or equivalent API gateway platforms for traffic management, rate limiting, and API lifecycle governance

• Hands-on experience with Temporal for workflow orchestration, durable execution, and distributed task management

• Strong PostgreSQL administration skills, including query optimization, replication, backup/recovery, and performance tuning

AI / Agentic Systems**

• Hands-on experience building or integrating AI-powered automation in production environments

• Experience with agent-based systems, LLM-powered workflows, Retrieval-Augmented Generation (RAG), or intelligent assistants

• Familiarity with Azure-based AI and ML services (Azure OpenAI, Cognitive Services, Azure ML)

• Understanding of reliability, safety, observability, and operational challenges of AI systems in production