Criteria for Reporting and Evaluating Exposure Datasets (CREED) for Use in Environmental Assessments: A Timely Development

Graham Merrington, wca environment; Lisa Nowell, USGS; Charles Peck, U.S. Environmental Protection Agency

Introducing CREED

The evaluation of risk from chemicals in the environment requires the collection and analysis of chemical concentrations in environmental matrices (for example, soil and water) to identify “exposure data,” which are then combined with ecotoxicity data. A scheme to determine the suitability of ecotoxicity data for specific assessment purposes, by evaluating reliability (data quality) and relevance (fitness for purpose), is available. However, as important as exposure data is to environmental assessments, a comprehensive scheme to evaluate reliability and relevance for exposure data is not available, even though some agencies have guidance for data quality.

To address the lack of a comprehensive method for evaluating exposure datasets for use in risk assessments, a group of SETACers have worked diligently for almost two years under the framework of a SETAC Technical Workshop. Workshop participants are poised to publish a set of criteria for doing just that – The Criteria for Reporting and Evaluating Exposure Datasets (CREED). The group, representing several countries and several sectors including academia, government and business, in true SETAC fashion, collaborated to develop practical, systematic criteria for the consistent evaluation of the reliability and relevance of exposure data for use by researchers, practitioners and regulatory scientists.

CREED’s Goals

CREED was initiated as a SETAC-supported activity to develop a framework and criteria for assessing the reliability, relevance and usability of environmental exposure data, aiming to improve the transparency and consistency with which exposure data are evaluated for use in environmental assessments for the benefit of environmental professionals from all sectors. CREED’s goal is to provide a framework through which expert judgement is guided and documented, so that exposure data-use decisions can be made in a systematic and transparent way, and to facilitate consistency from user-to-user.

CREED’s Applicability

The CREED approach can be applied to many different purposes of assessment that use chemical monitoring data to estimate exposure. For data generators, CREED provides guidance on which parameters and metadata should be reported in their monitoring studies, to maximize the utility of their data by outside data users or risk assessors. For data users, CREED provides guidance on which parameters are important to look for when evaluating whether existing data are reliable and relevant for their specific assessment purpose. For data owners, such as database managers, CREED specifies the types of data fields that should be included (preferably on a mandatory basis) in the database, so that data generators have a place to adequately describe their studies and data users can find the supporting information they need to evaluate exposure datasets for use in environmental assessments.

CREED’s Approach

Under the CREED paradigm, scientists evaluate both data reliability and data relevance. Reliability refers to the inherent quality of the dataset and its evaluation focuses on the methods used for sample collection, chemical analysis, and data processing and statistics. Relevance refers to the degree of suitability of an existing environmental monitoring dataset, or one that will be generated, to address the specific purpose as defined by the assessor.

Purpose Statement

To apply CREED, scientists are first directed to prepare a purpose statement. While evaluating data reliability is generally independent of the purpose for which data are being assessed, evaluating relevance is naturally dependent on the purpose of the study in which the data are being considered for use. CREED developers recommend that the purpose statement be unambiguous and specific with respect to the information required for the assessment to increase the objectivity and transparency of the relevance evaluation. Throughout, consistency and transparency are key principles of the CREED approach.

Gateway Criteria

To determine if a dataset contains sufficient information to do a CREED evaluation, the assessor can first ask six “gateway” questions. These criteria identify the minimum information needed (on the sampling medium, analyte, site location, sampling date, units of measurement and citation or source of data) for a detailed evaluation to be worthwhile. These are pass/fail questions, and if any one or more are failed, the dataset does not contain enough information to undergo CREED evaluation, unless the assessor can locate the missing information.

Detailed Reliability and Relevance Criteria

Datasets that pass the gateway stage are then evaluated for both reliability and relevance using more detailed criteria. There are 19 detailed reliability criteria and 11 detailed relevance criteria. For each criterion, the assessor rates a given study as either:

• Fully met: All conditions described by the criterion are satisfied by the study,
• Partly met: Some of the conditions described by the criterion are met for either part or all of the dataset, or all conditions are met for part of the dataset, 
• Not met or inappropriate: The data or approach is flawed or inappropriate for the purpose,
• Not reported: Insufficient information was provided to evaluate the criterion,
• Not applicable: For circumstance-specific criteria only, the circumstances described by the criterion do not apply to the dataset.

Whenever a criterion is rated as partly met, not met or inappropriate, or not reported, the CREED procedure calls for the assessor to record in writing the data limitations, flaws or missing information that triggered this rating. This is a key element of CREED because these provide information on data gaps that may restrict data use or be useful for estimating uncertainty in the risk assessment.

Scoring System

Ultimately, the ratings for all criteria are considered together and the study or dataset is assigned to one of four categories for each of reliability and relevance:

• Reliable or relevant without restrictions
• Reliable or relevant with restrictions
• Not reliable or relevant
• Not assignable

Not assignable indicates that information needed to evaluate the dataset is missing. The dataset reliability and relevance categories are then combined to determine the overall usability of the dataset for the given purpose. The usability categories are:

• Usable without restrictions
• Useable with restrictions
• Not usable

Datasets that are not useable because of missing information (i.e., because they were not assignable for reliability or relevance or both) can be re-evaluated if the assessor decides it is worthwhile and feasible to locate the missing information.

The overall rating of the reliability and relevance criteria is done using a two-level scoring system – at silver and gold levels. This two-level scoring system was developed recognizing that “perfect” (gold standard) datasets are not common, while potentially usable datasets (i.e., those that meet basic criteria) are frequently encountered and should therefore be within the scope of CREED applicability for the framework to be of practical use. If only perfect datasets were allowable, then CREED would not be very useful. Therefore, CREED distinguishes between “required” criteria (which are important for most assessment purposes) and “recommended” criteria (which are considered less critical). At the silver score level, only the required criteria determine the category or score assignment, whereas all criteria (required plus recommended) determine the category and score at the gold level. Thus, the silver level is less ambitious than the gold level, which represents an ideal dataset.

The CREED Tool

To allow users to readily follow the workflow, evaluate, and create a CREED report card for any given dataset, CREED has been implemented in a Microsoft Excel® workbook template. Each step of the CREED approach is represented in the workbook, starting with the assessment purpose definition and evaluation of the gateway criteria, and ending with an overall score of the dataset’s usability for the specified purpose and a description of any identified dataset limitations or potential use restrictions. In the scoring tool workbook, the assessor defines the purpose statement, ideally including specific thresholds to use in rating individual criteria as fully met; answers the gateway criteria questions; rates each individual reliability and relevance criterion in relation to the specified assessment purpose (for datasets that pass the gateway criteria); and records any data limitations that keeps the dataset from fully meeting any criteria. The CREED scoring tool then automatically scores the dataset at both silver and gold levels, and assigns it to categories of reliability, relevance and data useability. The CREED scoring tool offers the assessor the option to extract the full evaluation outcomes as a downloadable report card. This includes the assessment purpose statement, dataset details, the score card information, any dataset limitations, the silver- and gold-level scores of relevance, reliability and overall data usability.

CREED: Best Practice in the Use of Environmental Monitoring Data in Chemicals Assessments

CREED is designed to document the quality of a chemical monitoring dataset and to describe what the data represent (e.g., study design and environmental conditions) in a systematic and transparent way, and to identify the domain of applicability of that dataset for a specified assessment purpose. CREED provides an approach and tools to consistently and transparently evaluate the quality of a chemical dataset and whether it is fit for a specific assessment purpose, as well as to identify limitations of the dataset that may qualify or constrain the use of the data. CREED can provide guidance to data generators on important study characteristics that should be reported to ensure that their data are useful to the widest possible range of assessment types. CREED offers an opportunity for the scientific community to improve the quality and suitability of monitoring data that are used in environmental assessments. CREED not only supports informed decision-making but also can be valuable in identifying data gaps and in planning for future data collection campaigns. In the interest of better science and decision-making, and to promote cost-effective environmental monitoring and assessment, we hope to make future improvements to CREED, as the scientific community gains experience in the application of this approach.

Stay Tuned

Coming in 2024: The workshop participants have drafted a SETAC Technical Issue Paper titled, "Criteria for Reporting and Evaluating Exposure Datasets (CREED)," are planning a webinar for early next year, and have submitted articles describing CREED to Integrated Environmental Assessment and Management.

Authors' contact information: [email protected]; [email protected], [email protected]

Acknowledgements: The SETAC Technical Workshop was sponsored by Concawe, GSK, Metals Environmental Research Association (MERA), SETAC, Syngenta and Unilever.