Software Test Engineer – AI POC (LLM / RAG & Agentic AI)

This role is part of a time-bound AI Proof of Concept (POC) focused on LLM / RAG output evaluation and Agent-based (Agentic) AI workflows. The selected candidate is expected to contribute independently and be productive from Week 2. There is no training or grooming bandwidth available for this role.

Key Responsibilities:

Test and validate LLM-based and RAG-driven AI systems Perform AI output evaluation for relevance, accuracy, consistency, and hallucinations Apply standard NLP / LLM evaluation metrics during testing Test Agent-based (Agentic) AI workflows, including agent-to-agent interactions Write and execute Python-based test scripts for AI validation Perform functional, regression, and performance testing for AI POCs Identify, document, and clearly communicate AI model issues Collaborate closely with AI engineers to understand model behavior Provide concise, outcome-focused test reports for decision-making

Requirements

Mandatory (Non-Negotiable): 2 to 2.5 years of

experience

(maximum 3 years) in AI / LLM / NLP testing Hands-on

experience

testing LLM or RAG-based systems Practical

experience

with NLP / LLM evaluation metrics, including: BLEU ROUGE METEOR BERTScore COMET Strong Python programming

skills

with independent scripting ability Understanding of: RAG pipelines Agentic / Agent-based AI workflows Model context handling and response validation Ability to work independently in a POC / experimental environment Technical (Supporting): Working knowledge of Java (basic to intermediate) Familiarity with AI / ML frameworks such as TensorFlow or PyTorch Exposure to automation tools (Selenium, BDD Cucumber – secondary) Cloud exposure (AWS / Azure / GCP) – good to have Important Exclusions: This is not a traditional manual or automation QA role Not suitable for candidates learning AI fundamentals Not a buffer or training position

Benefits

Opportunity to work on cutting-edge AI / LLM POCs Hands-on exposure to LLM evaluation, RAG pipelines, and Agentic AI High ownership and visibility in a delivery-focused environment Potential continuation beyond POC based on performance and outcomes

Mandatory (Non-Negotiable): 2 to 2.5 years of

experience

(maximum 3 years) in AI / LLM / NLP testing Hands-on

experience

testing LLM or RAG-based systems Practical

experience

with NLP / LLM evaluation metrics, including: BLEU ROUGE METEOR BERTScore COMET Strong Python programming

skills

with independent scripting ability Understanding of: RAG pipelines Agentic / Agent-based AI workflows Model context handling and response validation Ability to work independently in a POC / experimental environment Technical (Supporting): Working knowledge of Java (basic to intermediate) Familiarity with AI / ML frameworks such as TensorFlow or PyTorch Exposure to automation tools (Selenium, BDD Cucumber – secondary) Cloud exposure (AWS / Azure / GCP) – good to have Important Exclusions: This is not a traditional manual or automation QA role Not suitable for candidates learning AI fundamentals Not a buffer or training position

Benefits

Opportunity to work on cutting-edge AI / LLM POCs Hands-on exposure to LLM evaluation, RAG pipelines, and Agentic AI High ownership and visibility in a delivery-focused environment Potential continuation beyond POC based on performance and outcomes