Auditable Clinical AI

Thesis topics

These four MSc topics use data science, knowledge representation and AI to address compliance, architecture and data governance in clinical AI. Each topic can be taken on its own or combined with others.

Radiology is the primary domain, with AI-assisted reading of lung cancer screening and follow-up CT as the running example. Each topic is designed to transfer to other clinical areas where AI software must meet the EU AI Act, the Medical Device Regulation (MDR), the European Health Data Space (EHDS) Regulation and GDPR, such as pathology, cardiology and clinical decision support.

1. Topic 1 turns regulatory obligations into machine-actionable requirements linked to design and test evidence.
2. Topic 2 checks whether a deployment meets the reference architecture, interoperability standards and privacy constraints.
3. Topic 3 records what the AI system and clinician did at run time.
4. Topic 4 controls how health data from multiple sites may be accessed and processed through machine-readable policies. Topics can be combined to build and validate a larger auditable clinical AI system, for example by linking requirements with run-time provenance or architecture checks with data-use policies.

1. Regulatory Traceability for Medical AI

Medical AI software must meet overlapping regulations: the AI Act, MDR, EHDS and GDPR. Their obligations are spread across legal text, guidance documents and harmonised standards. Development teams translate them into requirements by hand and keep the links between obligations, requirements, design decisions and test evidence in spreadsheets or ticket systems. Gaps are hard to find. Every change to the system or guidance requires these links to be checked again.

This thesis will develop an LLM-assisted pipeline that extracts obligations from regulatory texts and represents them in a knowledge graph. The graph links each obligation to clinical use cases, system requirements, architecture decisions and verification evidence. It can answer questions such as “Which AI Act record-keeping obligations have no verification method?” and “Which requirements are affected if this component changes?” AI-assisted radiology reading is the primary case. A second clinical AI application will test how well the ontology and pipeline transfer. The prototype regulatory traceability tool will detect gaps and analyse the impact of changes.

Literature Review

• Survey requirements engineering and traceability for regulated medical software (e.g. IEC 62304, ISO 14971).
• Review legal knowledge representation, compliance ontologies and normative reasoning (e.g. LegalRuleML, deontic logic).
• Study LLM-based extraction of obligations and requirements from legal and normative text.

Regulatory Analysis

• Select relevant provisions from the AI Act (e.g. technical documentation, record-keeping, human oversight), MDR, EHDS and GDPR.
• Define a classification of obligations by type, addressee, lifecycle phase and the evidence each one requires.

Knowledge Graph Design

• Design an ontology linking Obligation, Regulation, Use Case, Requirement, Architecture Decision, Component, Verification Method and Evidence.
• Support versioning so that changes in regulation, guidance or system design can be tracked over time.

Extraction and Linking Pipeline

• Build an LLM pipeline that extracts obligations, with regulatory experts validating the results.
• Link extracted obligations to existing requirements using semantic similarity and graph reasoning.

Prototype Development

• Implement queries for coverage gaps, orphan requirements and change-impact analysis.
• Integrate with an existing requirements management or issue-tracking tool.

Evaluation

• Measure extraction precision and recall against an expert-annotated set of provisions.
• Measure how well the tool detects known traceability gaps and how much time it saves compared with manual traceability.
• Assess transfer to a second clinical AI application.

2. Architecture Modelling and Conformance Checking for Clinical AI

AI tools in radiology are often integrated late, with each integration rebuilt for a specific PACS and hospital IT environment. Reference architectures based on DICOM/DICOMweb, IHE profiles (e.g. AI Results and AI Workflow for Imaging) and HL7 FHIR describe how to invoke AI and return its results. Privacy-by-design patterns, such as site-local anonymisation and controlled re-identification, add further constraints. In most projects, teams check these constraints by reviewing design documents manually. Violations are found during testing or after deployment.

This thesis will model a reference architecture for AI-assisted radiology reading in a machine-readable form and develop automated conformance checks. A deployment means a specific combination of PACS, AI components, orchestration services and data flows at a site. The checks cover interoperability contracts, data-flow constraints (e.g. no identifiable data leaving a site), required logging points and mechanisms for human oversight. Applying the same metamodel to a second clinical domain will test reuse. The prototype checker will evaluate candidate deployments and explain any violations.

Literature Review

• Survey reference architectures for health IT and medical imaging AI, including relevant IHE profiles.
• Review architecture description languages, architecture-as-code and model-based conformance checking.
• Study privacy-by-design patterns and privacy threat modelling (e.g. LINDDUN).

Requirements Analysis

• Collect architectural, interoperability, privacy and auditability constraints from standards, regulation and clinical stakeholders.
• Express the criteria a component must meet before clinical testing as checkable rules.

Architecture Modelling

• Design a metamodel covering components, interfaces, standards profiles, data flows, trust boundaries and deployment sites.
• Model a reference architecture and at least two deployment variants (e.g. a cloud-based test environment and an on-premise PACS integration).

Conformance Checking

• Implement rule-based and graph-based checks for interface conformance, data-flow constraints and required oversight and logging points.
• Explore LLM support for turning natural-language constraints into rules and for explaining violations.

Prototype Development

• Build a tool that reads deployment descriptions and reports conformance results with explanations.

Evaluation

• Measure violation-detection accuracy against expert review on realistic deployment configurations.
• Measure the effort needed to model a new deployment or a second clinical domain.

3. Provenance and Audit Logging for Clinical AI Workflows

The AI Act requires high-risk AI systems to log events automatically and allow effective human oversight. AI-based medical devices that need a notified-body assessment under MDR fall into this category. For an AI-assisted radiology reading workflow, an auditor needs to establish which model version produced a finding, which images and prior examinations it used and what uncertainty it reported. The auditor also needs to know whether a safety check flagged or withheld the finding and whether the radiologist accepted, edited or rejected it. Application logs rarely capture this chain in a form that supports audit, post-market monitoring or validation studies.

This thesis will design a provenance model and logging architecture for AI-assisted clinical workflows, based on standards such as W3C PROV. The model captures AI invocations, inputs, model versions, gating decisions, uncertainty signals and clinician interactions. It links them to DICOM and FHIR result objects and to the requirements they serve. Radiology reading is the primary case. The model will then be adapted to a second workflow, such as AI-supported pathology or clinical decision support. The prototype provenance service will support queries and reports for audit, oversight and comparison across sites.

Literature Review

• Survey provenance models (W3C PROV and its extensions) and provenance tracking for machine learning pipelines.
• Review AI Act logging, human-oversight and post-market monitoring obligations and audit requirements for medical software.
• Study tamper-evident and privacy-preserving logging techniques.

Requirements Analysis

• Identify the audit, oversight and monitoring questions that clinicians, AI oversight staff and regulators need answered.
• Map these questions to workflow events (AI invocation, safety gating, review states, report finalisation).

Provenance Model Design

• Extend W3C PROV with clinical entities and activities (Study, Prior Examination, Finding, Model Version, Gating Decision, Clinician Action).
• Define links to DICOM/FHIR result objects and to requirement identifiers.

Logging Architecture

• Design capture points across the orchestration layer, AI components and the clinical user interface.
• Address pseudonymisation of logged data and integrity of log records.

Prototype Development

• Implement a provenance store and query interface (e.g. a graph database with SPARQL or Cypher).
• Generate audit reports and oversight dashboards from provenance data.

Evaluation

• Test whether the system answers the predefined audit and monitoring questions on simulated or sandbox workflow runs.
• Measure logging overhead, storage cost and query performance.
• Assess completeness against AI Act record-keeping requirements and transfer to the second workflow.

4. Data Governance and Usage Control under EHDS

Developing and validating clinical AI usually requires imaging studies, reports and clinical records from several hospitals, often in different countries. Each source has its own consent conditions, data processing agreements and national rules. The EHDS Regulation adds a framework for secondary use of health data through Health Data Access Bodies and data permits. These conditions are written in legal documents, so data pipelines cannot check or enforce them directly.

This thesis will express data-use conditions for health datasets as machine-readable policies, for example in ODRL. A usage-control component will evaluate the policies when data is accessed or processed. Standard metadata such as HealthDCAT-AP will describe each dataset and link it to its policies. Multi-site medical imaging for AI development and validation is the primary case. The approach also applies to registries, biobanks and other secondary-use settings. The prototype policy and usage-control layer will show which processing steps a policy allows and record each decision for audit.

Literature Review

• Survey policy languages and usage control (ODRL, XACML, UCON) and data space architectures (e.g. Gaia-X, IDSA).
• Review EHDS secondary-use provisions, data permits and HealthDCAT-AP.
• Study GDPR legal bases, consent models and pseudonymisation for medical imaging data.

Requirements Analysis

• Catalogue typical data sources, consent conditions and contractual constraints in multi-site clinical AI studies.
• Identify processing steps that need policy checks (ingest, anonymisation, model training, evaluation, export).

Policy Modelling

• Define an ODRL profile for health data use, covering purpose, duration, location, permitted processing and obligations such as deletion or reporting.
• Describe datasets with HealthDCAT-AP metadata linked to their policies.

Usage Control Design

• Design policy decision and enforcement points integrated with data pipelines.
• Explore LLM support for drafting policies from agreement text, with expert validation.

Prototype Development

• Implement the policy engine and integrate it with at least one realistic data pipeline.
• Record policy decisions so they can be linked to a provenance and audit layer (see Topic 3).

Evaluation

• Test policy correctness on realistic scenarios, including permitted, denied and edge cases.
• Measure enforcement overhead and the effort needed to add a new site or dataset.