Assessment of evaluation tools for integrated surveillance of antimicrobial use and resistance based on selected case studies

Abstract

Regular evaluation of integrated surveillance for antimicrobial use (AMU) and resistance (AMR) in animals, humans and the environment is needed to ensure system effectiveness, but the question is how. In this study, six different evaluation tools were assessed after being applied to AMU and AMR surveillance in eight countries: 1) ATLASS: the Assessment Tool for Laboratories and AMR Surveillance Systems developed by the Food and Agriculture Organization (FAO) of United Nations, 2) ECoSur: Evaluation of Collaboration for Surveillance tool, 3) ISSEP: Integrated surveillance system evaluation project 4) NEOH: developed by the EU COST Action ‘Network for Evaluation of One Health’ 5) PMP-AMR: The Progressive Management Pathway tool on AMR developed FAO, 5) and 6) SURVTOOLS: developed in the FP7-EU project ‘RISKSUR’. Each tool was scored using i) 11 pre-defined functional aspects (e.g., workability concerning the need for data, time and people), ii) a SWOT-like approach of user experiences (e.g., things that I liked, or that the tool covered well), and iii) eight predefined content themes related to scope (e.g., development purpose, collaboration). PMP-AMR, ATLASS, ECoSur and NEOH are evaluation tools that provide a scoring system to obtain semi-quantitative results, whereas ISSEP and SURVTOOLS will result in a plan for how to conduct evaluation(s). ISSEP, ECoSur, NEOH and SURVTOOLS allow for in-depth analyses and therefore require more complex data, information and specific training of evaluator(s). PMP-AMR, ATLASS and ISSEP were developed specifically for AMR-related activities – only ISSEP included production of a direct measure for “integration” and “impact on decision-making”. NEOH and ISSEP were perceived as the best tools for evaluation of OH aspects, and ECoSur as best for evaluation of the quality of collaboration. PMP-AMR and ATLASS seemed to be the most user-friendly tools, particularly designed for risk managers. ATLASS was the only tool focusing specifically on laboratory activities. Our experience is that adequate resources are needed to perform evaluation(s). In most cases, evaluation would require involvement of several assessors and/or stakeholders, taking from weeks to months to complete. This study can help direct future evaluators of integrated AMU and AMR surveillance towards the most adequate tool for their specific evaluation purpose.