Overview
The Senior AI Test Analyst is responsible for defining, implementing, and executing testing strategies for AI-powered solutions, including AI agents, generative AI applications, machine learning models, APIs, and business process workflows. The role focuses on AI model validation, data quality assessment, performance testing, responsible AI compliance, and the automation of testing processes to ensure the delivery of robust, scalable, secure, and high-quality AI solutions.
As MUFG Retirement Solutions continues to embed AI across its technology landscape, this role plays a key part in driving the adoption of AI-enabled testing practices and automation capabilities. The position is responsible for identifying innovative and cost-effective solutions, leveraging AI to enhance testing efficiency, quality, and coverage while promoting industry best practices.
The role also ensures that the AI testing and automation strategy aligns with business objectives, quality standards, regulatory requirements, and the broader technology vision of MUFG Retirement Solutions.
Key Accountabilities and main responsibilities
Strategic Focus
- Develop and execute testing strategies for AI, Machine Learning (ML), and Generative AI solutions to ensure accuracy, reliability, performance, and business value.
- Define and implement robust AI test frameworks, methodologies, and automation solutions covering functional, integration, API, system, end-to-end, model validation, data validation, prompt validation, and adversarial testing scenarios.
- Establish comprehensive validation processes for AI model outputs against business requirements, acceptance criteria, Responsible AI principles, regulatory standards, and organizational governance policies.
- Drive quality assurance across the complete AI lifecycle, including data ingestion, data preparation, feature engineering, model training, evaluation, deployment, inference, monitoring, retraining, and reporting.
- Assess AI solution performance across dimensions such as accuracy, relevance, consistency, explainability, fairness, robustness, and reliability.
- Integrate AI testing activities and test automation framework & scripts with continuous integration, CI/CD pipelines and tools to support continuous testing and deployment of AI solutions.
- Collaborate with developers, business analysts, product owners, and project managers to define testing requirements, success criteria, and quality measures.
- Manage the execution of automated test suites, monitor the test results and collaborate with teams to troubleshoot and resolve the issues promptly.
- Lead defect triage, root cause analysis, and quality risk management activities, ensuring timely resolution of issues impacting AI model performance and business outcomes.
- Champion the adoption of AI-enabled testing tools, intelligent automation techniques, synthetic test data generation, and predictive quality engineering practices to improve testing efficiency and coverage.
- Drive research and evaluation of emerging AI testing methodologies, LLM evaluation frameworks, model benchmarking techniques, automation technologies, and industry best practices, providing strategic recommendations for continuous improvement.
- Establish and govern enterprise standards, processes, controls, and best practices for AI testing, validation, Responsible AI compliance, and model quality assurance across multiple AI initiatives.
Operational Management
- Design, develop, execute, and maintain test cases and test scripts for AI, ML, and Generative AI applications.
- Validate data quality, data integrity, and data lineage across AI systems and supporting platforms.
- Lead functional, integration, API, regression, user acceptance, and end-to-end testing of AI-enabled solutions.
- Assess AI model outputs for accuracy, consistency, bias, hallucinations, explainability, fairness, and alignment with business requirements and responsible AI standards.
- Execute prompt testing, response validation, and scenario-based testing for Generative AI solutions.
- Execute model validation and inference testing activities, ensuring AI solutions perform consistently across diverse datasets, use cases, and operating conditions.
- Monitor and analyse test execution results, quality metrics, and model performance indicators, producing actionable insights and comprehensive test reports for stakeholders.
- Identify, document, track, and drive resolution of defects, model performance issues, data quality concerns, and AI-specific risks throughout the testing lifecycle.
- Collaborate with business analysts, developers, and other stakeholders to understand requirements and deliver high-quality AI solutions.
- Support test data preparation, environment validation, and model release verification activities.
- Contribute to continuous improvement initiatives by identifying opportunities to enhance AI testing frameworks, processes, and automation capabilities.
- Provide guidance and knowledge sharing to project teams on AI testing methodologies, Responsible AI validation, test automation approaches, and emerging industry practices.
- Mentor junior testers and contribute to building AI testing capabilities across teams through coaching, standards adoption, and continuous learning initiatives.
Governance & Risk
- Identify quality, model, data, security, privacy, ethical, and compliance risks associated with AI solutions and proactively support mitigation activities.
- Validate adherence to Responsible AI principles, regulatory requirements, organizational policies, and governance standards.
- Conduct and support the assessment and reporting of AI-related risks, including model bias, explainability, fairness, and data quality concerns.
- Maintain appropriate test evidence, validation reports, defect records, and audit documentation to support governance and compliance requirements.
- Escalate risks, issues, and control gaps that may impact AI solution quality, regulatory compliance, or business outcomes.
The above list of key accountabilities is not an exhaustive list and may change from time-to-time based on business needs.
Experience & Personal Attributes
Experience
- 5+ years of experience in Quality Engineering, Test Automation, and AI-driven testing, with preference for candidates having 4+ years of experience in Retirement Solutions, Financial Services, or BFSI domains.
- Strong experience in designing, developing, and executing AI-powered test automation frameworks and quality assurance solutions using tools such as Selenium, Appium, UFT/OpenText (at least mandatory), Python, Maven, Postman, REST APIs, SoapUI, Cortex, and Snowflake.
- Hands-on experience leveraging AI-assisted testing tools and technologies for test case generation, test optimization, defect prediction, intelligent test execution, and quality analytics.
- Strong understanding of software testing principles, methodologies, and best practices, including test strategy, test planning, test design, test execution, defect management, reporting, and test governance.
- Experience validating AI/ML-enabled applications, including model testing, data quality validation, prompt testing and optimization, response evaluation, bias and fairness assessment, hallucination detection, adversarial testing, explainability validation, and performance assessment of AI-powered systems.
- Solid understanding of Software Development Life Cycle (SDLC) and Software Quality Assurance (SQA) processes across Agile, Waterfall, and DevOps delivery models.
- Experience working with CI/CD pipelines and DevOps practices using tools such as Jenkins, Azure DevOps, Azure Repos, Bitbucket, Git, Docker, and Kubernetes.
- Strong analytical and problem-solving skills with the ability to define quality standards, establish AI testing governance, track AI model evaluation metrics and quality KPIs, automate validation processes, and ensure reliable, scalable, and secure AI-driven applications.
- Experience with cloud platforms, data validation, API testing, integration testing, and complex enterprise environments is highly desirable.
- Lead AI testing strategy and provide technical leadership for AI testing initiatives.
- Collaborate with Developers, Testers, and Business Analysts to design test cases and develop them using AI and automation testing tools.
- Perform functional, integration, system, regression, and API testing using tools such as Postman and REST Assured.
- Manage defects through JIRA and Azure DevOps and produce comprehensive test evidence documentation.
- Develop and execute end-to-end test automation solutions covering the entire data lifecycle, from raw data through to the reporting layer.
- Mentor team members and support the, training, and upskilling of testers in AI and automation testing tools, frameworks, and best practices.
Personal Attributes
- Strong analytical, problem-solving, and troubleshooting skills.
- Strong communication, collaboration.
- Strong attention to detail and quality orientation.
- Demonstrate ability to solve technical problems and implement innovative solutions.
- Ability to work independently and as part of a team.
- Ability to learn new AT testing tools, test automation tools, technologies, and trends quickly and effectively.
- Ability to handle multiple tasks and projects simultaneously and prioritize them
