QA Tester for AI Applications – Quality Engineer – Atlanta (Remote)

Job Description

Job Description:
We’re hiring a hybrid AI QA + Product Analyst to own end-to-end quality for our AI-powered inference system. This role sits at the intersection of LLM inference quality, event-driven backend state-machines, and freight domain logic. You will define what “correct” means, build the quality measurement and regression approach to enforce it, and lead deep-dive investigations when edge cases or customer-specific rules break downstream behavior. The goal is to make our system more accurate, more diagnosable, and more reliable as email volume and customer complexity scales.

What you’ll do:
1. Own end-to-end system quality
2. Develop and maintain a quality rubric for key use cases and exception types. (what “right” looks like, and what failure looks like).
3. Build and curate golden datasets (representative emails + expected structured output + expected final outcome), including customer-specific variations.
4. Own ongoing quality review in dev and production: regularly inspect high-volume outputs, diagnose what’s breaking and why, and convert discoveries into concrete roadmap items and regression coverage.
5. Define and execute regression tests for new model changes, backend logic changes, or customer-specific use cases.
6. Investigate and diagnose issues across the full stack of the product
7. Triage quality incidents and ambiguous failures by tracing through: email ingestion/parsing prompts / model outputs / normalization steps / data contracts intermediate structured representations event streams and state-machine transitions final audit exception generation and downstream reporting
8. Use logs, traces, event histories, and data queries to isolate root cause.
9. Produce high-signal findings reports: minimal reproduction, suspected component, evidence, impact, and recommended fix.
10. Build scalable quality operations
11. Create a repeatable triage playbook and classification system for quality issues
12. Define monitoring & dashboards for quality signals (volume anomalies, exception drift, per-customer error hotspots).
13. Partner with engineering/AI to improve observability (correlation IDs, structured logging, traceability from email → state transitions).
14. Act as a product/domain translator
15. Understand freight billing workflows and how real-world documents and communication map to our system’s model of “truth”.
16. Convert customer-specific requirements into testable rules and expected outcomes.
17. Identify systemic gaps where “reality” doesn’t fit the current schema, and propose product changes.

Required qualifications:
1. Experience in roles that blend quality + investigation + systems thinking (examples: QA engineer in distributed systems, product analyst with deep debugging, LLM quality analyst, solutions engineer owning incident triage).
2. Demonstrated experience evaluating AI/LLM output quality (extraction/classification, structured outputs, tool calling, RAG, prompt-driven pipelines, or similar).
3. Strong technical ability to debug production issues using: log/trace tools (Datadog, ELK, Honeycomb, Open Telemetry/Jaeger, etc.), SQL and/or Python for analysis and repro event-driven architectures and workflows/state machines (or similar distributed workflow systems)
4. Ability to write crisp requirements and acceptance criteria, and translate ambiguity into test cases.
5. Comfort operating in messy, high-volume, edge-case-heavy environments.

Nice-to-have qualifications:
1. Freight/logistics/audit/billing domain experience (carrier invoices, accessorials, detention, lumper, fuel surcharge, tenders, BOLs, rate confirmations, PODs, etc.).
2. Experience designing evaluation metrics (precision/recall, drift detection, per-customer or per-use-case scorecards).
3. Familiarity with workflow engines/state machines and distributed systems failure modes (event ordering, retries, dedupe, idempotency, partial failure).
4. Experience with annotation/labeling workflows, taxonomy design, and building human-in-the-loop QA processes.
Traits that matter in this role:
5. High ownership: you don’t stop at “it’s broken,” you drive it to root cause and resolution.
6. Comfortable with ambiguity and edge cases; systematic in building clarity.
7. Able to communicate across product, engineering, ML, and operations.