81.2 F
Washington D.C.
Tuesday, June 6, 2023

IARPA Issues RFI on Evaluation of Neural Text Generation Models and Methods

The Intelligence Advanced Research Projects Agency (IARPA) is seeking information on established techniques, metrics and capabilities related to the evaluation of generated text and the evaluation of human-interpretable explanations for neural language model behavior. 

The request for information (RFI) notes that neural language models (NLMs) have achieved state-of-the-art performance on a wide variety of natural language tasks. In natural language generation in particular, models such as GPT-3 have produced strikingly human-like text. Methods to evaluate and explain these technologies have not however kept pace with the technologies themselves.

Language generation models can be used for a variety of automated tasks involving modification of a pre-existing text, such as paraphrasing, style transfer, summarization, etc. Measuring success on these tasks can be challenging: a modified text must remain faithful to the meaning of the text from which it is derived (i.e., maintaining sense), while also exhibiting human-like fluency (i.e., soundness). Although numerous automated techniques for evaluating sense and soundness have been developed, techniques that require humans to grade generated text (e.g., with Likert scales or ranking) remain the gold standard.

Furthermore, as language generation models increasingly produce human-like content on the internet, there is growing interest from diverse stakeholders in capabilities to flag artificially generated text content, in its many varieties. As is the case in other text classification tasks, NLM classifiers have seen success in identifying machine generated text; however, it is difficult to derive explanations for the predictions of multi-layer neural models, and the human user’s inability to understand and trust the rationale underpinning individual model predictions places limits on a system’s potential use cases.

The purposes of the RFI are the following:

  • Identification of novel human or automatic techniques, metrics and capabilities for evaluating the sense and soundness of machine modified text
  • Identification of novel methods to derive human-interpretable explanations from NLM text classifiers
  • Identification of novel techniques for measuring the quality of local explanations derived from NLMs

IARPA welcomes responses from all capable and qualified sources from within and outside of the U.S. by December 10, 2021.

See the RFI at SAM.gov

Homeland Security Todayhttp://www.hstoday.us
The Government Technology & Services Coalition's Homeland Security Today (HSToday) is the premier news and information resource for the homeland security community, dedicated to elevating the discussions and insights that can support a safe and secure nation. A non-profit magazine and media platform, HSToday provides readers with the whole story, placing facts and comments in context to inform debate and drive realistic solutions to some of the nation’s most vexing security challenges.

Related Articles

- Advertisement -

Latest Articles