The Technology Test and Evaluation Division at NIST is launching a new program to provide researchers with a sequestered testbed environment for the evaluation of AI model performance in a variety of meaningful tasks across diverse datasets, modalities, and domains.
The Artificial Intelligence Technology Evaluation (AITE) provides volunteer testing of AI models on blind data and its sequestered environment mitigates the risk of train/test data contamination to ensure rigorous, objective assessment. The three initial tasks focus on image analysis using large vision language models (VLMs) in the context of (1) quantum science, (2) genomics, (3) public safety. Additional tasks will be added over time. The infrastructure provided by NIST will provide common data, metrics and scoring to help developers understand the performance of their models.
AITE will rely on engagement from participants in two different tracks, each offering distinct advantages:
- Data providers submit an original dataset in their domain that is inaccessible to others and a meaningful task to be performed on that dataset. Data providers will receive careful measurements of top models on their data conducting their task.
- Model providers submit AI models to be tested on the datasets and tasks. Model providers will learn how their models perform on an increasing number of datasets and tasks, and how their models perform relative to others on the same data using the same metrics, improving comparability while ensuring the evaluation data is not used for training any model.
Participation is open to all who wish to engage in one of the ways described above, and who can abide by the AITE Participation Agreement and rules.
The original announcement can be found here.
