Abstract
This paper reports an exploratory feasibility study of applying the EU Assessment List for Trustworthy Artificial Intelligence (ALTAI) to an off-the-shelf AI-powered product recommender integrated into an industrial AI-enabled service. ALTAI is a voluntary self-assessment checklist, developed to guide users through structured questions and recommended practices aligned with the key requirements for trustworthy AI. Three researchers with complementary expertise and the (industry) case provider each conducted independent ALTAI self-assessments. The objective was to evaluate whether ALTAI can be applied for the purpose of trustworthiness assessment of a procured, off-the-shelf AI component under realistic industrial information constraints. The soundness of ALTAI itself is outside the scope. Our experiences are reported on, in terms of both the process and the resulting assessment outputs. The process facilitated a shared understanding of the trustworthiness requirements and helped uncover trustworthiness concerns. The outputs, which include trustworthiness scores and improvement recommendations, revealed substantial variation across the four assessments. The variation was partly caused by differences in expertise, interpretation, assumptions and limited access to detailed technical and organisational documentation. Overall, ALTAI proved feasible for structuring reflection and discussion about the trustworthiness of the case under consideration. The paper shares our experience of applying ALTAI, and derives recommendations both for practitioners using the framework and for further improvement of the tool.