Over the last two decades, different methods for evaluating mainly the reliability of toxicity and ecotoxicity studies for the purpose of hazard and risk assessment have been described (e.g. Durda and Preziosi 2000; Hobbs et al. 2005; Klimisch et al. 1997; Küster et al. 2009; Schneider et al. 2009; Tluczkiewicz et al. 2013). The Klimish method of categorizing studies as 1) “Reliable without restrictions”, 2) “Reliable with restrictions”, 3) “Not reliable” and 4) “Not assignable”, has been commonly promoted for use in the regulatory setting by different agencies and authorities. However, apart from stating that studies have to be conducted according to standardized test guidelines and GLP to be category 1, no detailed criteria and very little guidance for categorization was provided by Klimisch et al.
The methods proposed in SciRAP are different from the previously proposed methods in that they provide structured and detailed criteria for evaluation of both reliability and relevance of (eco)toxicity studies. They also include additional guidance material for judging criteria and describing common difficult situations. The output from the evaluation is intended to serve as a basis for judging whether a study is adequate for chemical risk assessment. As such, the SciRAP methods were developed to be compatible with requirements for risk assessment in current European regulatory systems. However, they are designed to be flexible and adaptable to different regulatory settings. The main aims of SciRAP are to promote transparency and completeness of the data evaluation process, to enable evaluation based on the inherent quality of studies, and to facilitate regulatory use of all available studies of sufficient reliability and relevance.
The criteria for evaluating reliability and relevance of ecotoxicity studies were developed in the CRED project (Criteria for Evaluating and Reporting ecotoxicity Data). The CRED evaluation method was developed from available methods and OECD reporting recommendations for aquatic ecotoxicity studies. In order to assess if the CRED evaluation method would be a suitable replacement of the Klimsich method, a ring test was performed in which the two methods were compared. The ring test concluded that the risk assessors preferred the CRED evaluation method since it was a more transparent and more detailed method. Specific guidance for evaluation of nanoecotoxicity studies has also been developed.
The criteria developed in SciRAP for evaluating reliability and relevance of in vivo toxicity studies were primarily based on recommendations and requirements set out in corresponding OECD test guidelines for such tests. The reason for this being that such guidelines have been internationally validated and accepted. To evaluate their scientific soundness, relevance and user-friendliness, the criteria have also been reviewed by experts within the field of toxicity testing and risk assessment from research institutions in Europe and the US, and authorities such as the Swedish Chemicals Agency, the US EPA and US FDA.
The method was first published by Beronius et al. (2014) but has since been updated. Further evaluation activities are on-going.
Criteria and tools for evaluating reliability and relevance of in vitro studies in the same format as for in vivo studies are currently being developed.
The SciRAP reporting guidelines are intended for researchers and list details of study aim, design, execution and results considered to be critical for judging study reliability and relevance in the regulatory setting. The reporting guidelines were primarily based on the criteria developed for the reliability and relevance evaluation of ecotoxicity and toxicity studies, respectively. In addition, the ARRIVE reporting guidelines and guidelines from the US National Research Council were considered in the development of reporting guidelines for toxicity studies. Check lists for reporting ecotoxicity (including studies on nanomaterials) and in vivo toxicity studies may be downloaded as excel files from the SciRAP website.