Position Summary

We are looking for a Specialist, QA Test Automation to bridge the gap between our strategic AI QA vision and day‑to‑day technical execution. Working closely with the QA Chapter Lead, you will embed within Cogeco’s AI squads to bring our AI QA strategy to life, handling probabilistic outputs, dataset curation, and leveraging advanced AI evaluation tools.

Key Responsibilities

• Probabilistic Testing & Evaluation
  • Implement and refine testing workflows for AI‑powered solutions where outputs are probabilistic.
  • Build guardrails for regression detection, model drift, prompt regression, hallucination risks, and safety violations.
  • Curate and maintain high‑quality golden datasets that serve as the source of truth for benchmarking model changes.
  • Champion programmatic evaluation tools such as Promptfoo, DeepEval, or Ragas to automate prompt variant and model performance assessment.
• ...