Director of AI Quality&Safety, Legal&Regulatory, Alphen aan den Rijn

Overview As Wolters Kluwer Legal&Regulatory executes its North Star to become the Intelligent Orchestration Platform for legal and regulatory work, the quality of AI‑generated outputs – particularly for research, analysis, and reasoning – becomes mission‑critical. As AI systems increasingly surface legal answers, interpretations, and recommendations at the point of work, trust depends on correctness, grounding, traceability, and consistency of those outputs. The Director of AI Quality&Safety ensures that AI‑driven research and decision support remain reliable, auditable, and compliant as agentic automation scales, reinforcing governed execution and making quality and trust durable differentiators at the orchestration layer.

The Director of AI Quality&Safety is accountable for establishing and operationalizing a comprehensive quality and safety framework across AI‑enabled products, content systems, and agentic workflows within WK Legal&Regulatory. The role ensures that AI systems are reliable, auditable, compliant, and aligned with defined quality standards, while reducing production defects and AI‑related risks. This position sits at the intersection of product, engineering, data science, and regulatory compliance, with a mandate to define measurable standards and enforce quality and product standards across the AI lifecycle.

Key Responsibilities

AI Quality&Evaluation Frameworks

Design and implement standardized evaluation frameworks for AI models and agentic systems (e.g., LLMs, RAG pipelines, autonomous agents)

Define, build, and improve evaluation frameworks with SMEs for output correctness and factuality, task completion accuracy, robustness and edge‑case handling

Establish benchmark datasets and continuous evaluation pipelines (offline + online)

Drive adoption of evaluation tooling and methodologies across product teams

Software Quality&Reliability (AI Systems)

Extend the rigor of traditional software QA practices to the outputs of AI‑driven systems (probabilistic outputs, non‑determinism)

Define SLAs/SLOs specific to AI performance (e.g., hallucination rate, response reliability, latency under load)

Partner with Engineering to integrate quality gates into CI/CD pipelines

Coordinate root‑cause analysis for AI‑related production issues and track implementation of systemic fixes

Content Correctness&Validation

Establish frameworks for validating legal and regulatory content generated or transformed by AI systems

Collaborate with editorial and domain experts to define“ground truth” and validation protocols

Implement human‑in‑the‑loop and automated validation mechanisms where appropriate

Ensure traceability between AI outputs and authoritative sources

Safety, Compliance&Governance

Define and enforce AI safety standards aligned with regulatory requirements (e.g., EU AI Act, data protection laws) and internal WK risk and compliance policies

Implement formal standards and controls alongside internal teams for bias detection and mitigation, harmful or unsafe output prevention, and data privacy and secure handling

Ensure auditability of AI systems (logging, explainability, decision traceability)

Act as primary liaison with Risk, Legal, and Compliance on AI‑related matters

Measurement, Benchmarking&Reporting

Define quality and safety standards and KPIs that apply across the AI lifecycle– including model and data selection, prompt and workflow design, and deployment – working in partnership with Product and Engineering

Build dashboards and reporting mechanisms for executive visibility

Track and benchmark performance over time and across product lines

Track development of, and evaluate products against, external industry benchmarks and work with recognized benchmarking bodies to represent WK interests

Drive continuous improvement loops based on measurable outcomes

Drive QA&KPIs awareness in LR businesses and provide comms support with key findings&insights that can be used for external comms&thought leadership

Partner with Sales, Marketing, and Customer Support on external benchmark communication and AI‑related incident response messaging

Governance&Operating Model

Define operating model for AI quality&safety across CPO and DXG

Introduce review boards, approval processes, and escalation mechanisms

Provide guidance and enablement to product teams on quality and safety best practices

Build and lead a small, high‑impact team as the function scales

Success Criteria

Clear, standardized quality and safety metrics are defined and consistently enforced across all AI‑enabled products

AI system behavior is measurable, benchmarked, and governed through repeatable frameworks

Significant reduction in production defects and AI‑related incidents

High confidence in content correctness and traceability for legal/regulatory use cases

Full auditability and compliance alignment for AI systems across jurisdictions

Required Qualifications

10+ years in product quality, AI/ML systems, or related domains, with at least 3–5 years in AI‑focused roles

Demonstrated experience designing evaluation frameworks for AI/ML systems (e.g., LLM evaluation, model validation)

Strong understanding of modern AI architectures (LLMs, RAG, agents), software quality engineering principles, and data and content validation workflows

Experience with regulatory or compliance‑heavy environments (preferred: legal, financial, healthcare)

Proven ability to operate cross‑functionally at senior levels

Preferred Qualifications

Familiarity with emerging AI governance standards and regulations (e.g., EU AI Act)

Experience implementing human‑in‑the‑loop systems at scale

Background in experimentation platforms, benchmarking, or observability for AI systems

Advanced degree in Computer Science, Data Science, Law, or related field

Key Competencies

Systems thinking (ability to unify software, AI, and content quality domains)

Analytical rigor and metric‑driven decision making

Risk awareness and regulatory sensitivity

Influence without authority in a matrixed organization

Pragmatic execution with high standards for quality

#J-18808-Ljbffr