# Senior AI QA Automation Engineer

> CodeNinja · Lahore, Pakistan · Full-time · Posted 2026-06-23

**Workplace:** on_site

**Department:** Professional Services

## Description

**About the Company**  
CodeNinja is a global technology and AI firm with 10+ years in the industry, helping enterprises modernize systems and embed AI into their workflows. We work across enterprise software, financial services, defense, health, logistics, and more, with 240+ clients across 15 countries. We partner with organizations to deliver AI infrastructure, dedicated engineering teams, and forward-deployed engineers who work directly inside client environments to own outcomes.

**About the role**

We are hiring an **Senior AI QA Automation Engineer** an experienced quality engineer who owns **feature-level test strategy** and **automation** for **AI-enabled products**. In this role, you will design and maintain **automated tests** and **AI evaluation suites**, integrate quality signals into **CI/CD pipelines**, and partner with **Scrum teams** to prevent defects and deliver reliable AI capabilities.

**Key Responsibilities**

-   Own **test strategy** and execution for assigned AI features across **unit**, **API**, **UI**, and **end-to-end** layers, aligned to **Definition of Done** and **release criteria**.
-   Create and maintain **automated AI evaluation suites** (including **golden datasets**, **scoring**, and **regression baselines**) covering key user journeys and failure modes.
-   Validate **prompt**, **retrieval**, and **orchestration** changes using repeatable test harnesses and defined **acceptance thresholds**.
-   Build and extend **automation frameworks**, reusable utilities, and improve **test stability**, speed, and **signal-to-noise**.
-   Integrate automated tests and AI evaluations into **CI/CD**, triage pipeline failures, and reduce **flaky tests**.
-   Partner with engineers to improve **testability** (contracts, logging, observability) and drive **shift-left quality** practices.
-   Analyze defects and AI output issues, perform **root cause analysis**, and implement regression coverage to prevent recurrence.
-   Provide guidance to junior SDETs on **automation**, **test design**, and **AI validation** patterns; communicate quality risks to stakeholders.
-   Support **production readiness** by validating **guardrails**, monitoring signals, and performance requirements for AI features.

## Requirements

-   Bachelor’s degree in **Computer Science/Engineering** (or equivalent practical experience).
-   4**\+ years** of experience in **SDET / QA automation / quality engineering** with strong hands-on **automation** and **test design** skills.
-   Proficiency in at least one programming language (e.g., **Python**, **C#**, **Java**) and **SQL**, with the ability to debug services and tests.
-   Experience building and maintaining automated tests (API/UI and/or **data/ETL**) and working in **CI/CD pipelines**.
-   Experience testing **AI-enabled systems** or demonstrated applied AI fluency (prompt/RAG changes, evaluation basics, common failure modes).
-   Experience working in **Agile/Scrum** teams and collaborating effectively with Engineering and Product.
-   Preferred: Experience with **LLM evaluation techniques** (golden sets, rubric-based grading, automated scoring, review workflows).
-   Preferred: Familiarity with **AI safety** and **guardrail testing** (grounding, PII handling, injection risks).
-   Preferred: Experience with **cloud tooling**, **containers**, **observability**, and **release automation**.

**Why Join Us**

-   Build quality systems for real-world **AI-enabled products**, including **LLM evaluations** and reliability guardrails.
-   Own feature-level quality end-to-end: from **test strategy** to **CI/CD quality gates** to **production readiness**.
-   Work with strong cross-functional teams across **Engineering**, **Product**, and AI stakeholders in an **Agile** environment.
-   Grow in modern quality engineering practices: **observability**, **automation frameworks**, **shift-left testing**, and AI evaluation patterns.

**Disclaimer:** CodeNinja is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, & performance. Female and minorities are strongly encouraged and preferred to apply for the role

## Benefits

-   **Provident Fund (PF)**

-   **Medical Insurance (Pakistan):** Coverage includes spouse and parents up to the age of 80.
-   **Life Insurance** 

-   **Performance-Based Increment**

-   **32 Paid Leaves (Annual)**

-   **Easy Loan Facility**

-   **Company-Sponsored Trips**

-   **Gym Passport Membership (Access To 250+ Gyms)**

## Apply

[Apply at CodeNinja](https://apply.workable.com/codeninjapk/j/752F109C36/apply)

---
Powered by [Workable](https://www.workable.com)