# Abstract

Recent developments in large language models (LLMs) have been impressive. However, these models sometimes show inconsistencies and problematic behavior, such as hallucinating facts, generating flawed code, or creating offensive and toxic content. Unlike these models, humans typically utilize external tools to cross-check and refine their initial content, like using a search engine for fact-checking, or a code interpreter for debugging. Inspired by this observation, we introduce a framework called CRITIC that allows LLMs, which are essentially "black boxes" to validate and progressively amend their own outputs in a manner similar to human interaction with tools. More specifically, starting with an initial output, CRITIC interacts with appropriate tools to evaluate certain aspects of the text, and then revises the output based on the feedback obtained during this validation process. Comprehensive evaluations involving free-form question answering, mathematical program synthesis, and toxicity reduction demonstrate that CRITIC consistently enhances the performance of LLMs. Meanwhile, our research highlights the crucial importance of external feedback in promoting the ongoing self-improvement of LLMs.

# Introduction

The remarkable progress of large language models (LLMs), such as ChatGPT, has been amply demonstrated across an array of language tasks [brown2020language; ouyang2022training]. Their potential to augment human intellect continues to burgeon [saunders2022self]. However, these models are not without their shortcomings. They occasionally exhibit undesirable behaviors, such as hallucination (generating inaccurate or non-truthful responses), faulty code, or even toxic content [maynez2020faithfulness; chen2021codex; gehman-etal-2020-realtoxicityprompts]. Such inconsistent behavior hampers the trust in these models and poses hurdles to their real-world applications [openai2023gpt4].

Traditional approaches to mitigate these limitations typically employ additional training, involving behavior cloning, reinforcement learning, and self-training [saunders2022self; stiennon2020; jeon2020reward; bai2022constitutional]. However, these methods are constrained by the requirement of large-scale human annotation or data construction, which is often resource-intensive and challenging to obtain. To address these challenges, we present Self-**C**o**r**recting w**i**th **T**ool-**I**nteractive **C**ritiquing (CRITIC), a unified framework that empowers _black-box_ LLMs to verify and rectify their own output through human-like interaction with external tools. Drawing inspiration from human cognition [greenfield1991language; vaesen2012cognitive] and critical thinking [marcus1988developing; ennis1991critical], CRITIC offers a versatile framework that supports precise, interpretable verification and correction of generated text.

[IMAGE: framework.pdf - The CRITIC framework consists of two steps: (1) verifying the output by interacting with external tools to generate critiques and (2) correcting the output based on the received critiques. We can iterate such verify-then-correct process to enable continuous improvements.]

As depicted in Figure 1, CRITIC interacts with external tools like search engines and code interpreters to verify the desired aspects of an initial output and subsequently amends the output based on the critiques from the verification. This _verify-then-correct_ process can be repeated to ensure constant output enhancement. Contrary to methods that rely on expensive annotations or task-specific training, CRITIC utilizes in-context learning with tool interaction to proficiently identify and rectify unsatisfactory behaviors using the LLM itself. This unique approach makes CRITIC both practical and accessible, requiring only access to text-to-text tool APIs and a few-shot demonstration.

We evaluate our approach on a range of LLMs, including ChatGPT, Text-Davinci-003, and open-source LLaMA-2 variants (7B, 13B, and 70B), spanning three distinct tasks: free-form question answering, mathematical program synthesis, and toxicity reduction. Our findings demonstrate that CRITIC consistently surpasses prior techniques, obviating the need for supplementary data or training. For example, when applied to ChatGPT, CRITIC attains 7.7 F1 enhancements across three QA tasks, 7.0% absolute gains on three mathematical reasoning tasks, and a 79.2% reduction in toxicity probability. Interestingly, our results underscore the _unreliability_ of all tested LLMs, when it comes to validating their own results. We observe that exclusive reliance on self-correction without external feedback may yield modest improvements or even deteriorate performance.

Our primary contributions include: (1) Proposing the unified CRITIC framework by integrating various tools and diverse tasks, with a series of new prompts that enable frozen LLMs to verify and iteratively self-correct their output through interaction with external tools. (2) Conducting comprehensive experiments across distinct tasks that demonstrate significant performance improvements offered by CRITIC across different base LLMs. (3) Highlighting the inadequacy of LLMs in self-verification and self-correction, and emphasizing that feedback from external tool interaction is crucial for consistent self-improvement of LLMs.

# Related Work

### Truthfulness Evaluation

Untruthfulness [evans2021truthful] is a critical issue for LLMs because it may hallucinate incorrect output that is hard to distinguish [lin2022truthfulqa; lee2022factuality], especially when relying on parametric memory [lewis2020retrieval]. A great deal of previous works design methods to detect hallucinated output [evans2021truthful; zhou2021detecting] of language models for different downstream tasks [ji2023survey], including abstractive summarization [maynez2020faithfulness; cao2022hallucinated], dialogue generation [shuster2021retrieval], and table-to-text generation [parikh2020totto]. Notably, these works mainly study task-specific fine-tuned models with a focus on _faithfulness_, i.e., factual consistent with the provided source content [filippova2020controlled; zhou2021detecting]. The truthfulness evaluation for open-ended text generation is less studied, especially for LLMs which may only be accessed via APIs. We fill this gap by letting the black-box LLMs interact with external tools to verify their own output. Our method is also inspired by fact-checking in journalism [wang2017liar] that assesses whether a claim made by a human is true [thorne2018fever].

### Natural Language Feedback

The technique of using natural language (NL) feedback is adopted to improve various tasks [rupprecht2018guide; scheurer2022training]. There are two main forms of feedback: scalar signals [dasgupta2019teaching] are commonly used for reinforcement learning [ziegler2019finetuning; quark22] and generate-then-rank framework [chen2023codet; li2022advance], while natural language feedback [saunders2022self] is commonly used for text editing using prompted LLMs [gao2022attributed; shinn2023reflexion] or trained correctors [bai2022constitutional]. Sources of feedback include human demonstration [saunders2022self] and evaluation [stiennon2020], existing corpora such as wiki edits [schick2022peer], automatically constructed data [bai2022constitutional], external metrics [welleck2023generating] or knowledge [peng2023check], and even the LLM itself [saunders2022self; weng2022large]. Nevertheless, LLM's intrinsic self-feedback has limited and task-specific performance compared to human feedback [saunders2022self] and LLMs struggle with verification on truthfulness [kadavath2022language; kuhn2023semantic] and reasoning correctness [ye2022the; huang2022large; huang2023large]. To address such issues, we focus on fully exploiting the emergent ability of LLMs for evaluation [fu2023gptscore] by empowering them with external tools.

[IMAGE: method_prompts.pdf - CRITIC prompts on example tasks, simplified for presentation. CRITIC initially verifies the desired aspects (e.g., "plausibility" and "truthfulness") of the initial answer by interacting with appropriate tools (e.g., search engine, code interpreter), and subsequently generate a corrected answer based on the critiques from verification. The critiques are generated by LLMs in response to the prompt "What's the problem with the above answer?" with demonstration, including all content preceding the corrected answer. If the generation process involves API calls, the API call results are concatenated following the model-generated query.]

### Tools Augmented Language Models

Beyond relying entirely on memorization [tirumala2022memorization], interacting with tools enhances the fidelity and potency of LLMs [parisi2022talm], enabling them to fully leverage their inherent reasoning and compositionality capabilities [yao2023react]. Studies show that we can augment generation with retrievers [Khandelwal2020Generalization; guu2020retrieval] or search engines [nakano2021webgpt; komeili2022internet; press2022measuring], enhance math reasoning with a calculator [andor2019giving; cobbe2021gsm8k], leverage an interpreter to execute the generated code [gao2022pal; chen2022program], use mathematical prover to prove mathematical theory [jiang2023draft], or use multiple tools automatically [schick2023toolformer]. We can teach the LLMs to use tools by pre-training [taylor2022galactica], fine-tuning [nakano2021webgpt], or in-context learning [paranjape2023art]. CRITIC avoids task-specific training and employs in-context learning, which is more simple and general.

# CRITIC: Correcting with Tool-Interactive Critiquing

We can get an overview of the CRITIC method through Figure 1. Given any input, LLMs first generate an initial output based on parametric knowledge, then interact with appropriate external tools (possibly multi-round) through text-to-text APIs to verify the output. The critiques generated by the verification step are concatenated with the initial output, and serve as feedback to allow LLMs to correct the output. We can iterate the cycle of "_Verify_ -> _Correct_ -> _Verify_" to continuously improve the output until a specific stopping condition is met. See Algorithm 1 for a summary of CRITIC method, and the following sections for details.

## In-context Learning for LLMs

CRITIC utilizes the emergent abilities of chain-of-thought reasoning [wei2022chain] and few-shot in-context learning [brown2020language; min-etal-2022-rethinking] of LLMs. Few-shot in-context learning is a powerful approach that exploits the capabilities of LLMs to solve a task given a small set of input-output examples at test time [liu2023pre]. The few-shot setting typically involves only a handful of examples (`latex $k$ `). To accomplish this task, the examples `latex $\{(x_i, y_i)\}_{i=1}^{k}$ ` are combined into a prompt `latex $p$ `, which concatenates the input and output pairs as follows: `latex $\langle{}x_1 \cdot y_1\rangle  \langle{}x_2 \cdot y_2\rangle  \ldots  \langle{} x_k\cdot y_k\rangle$ `. During inference, a test instance `latex $x_\text{test}$ ` is added to the prompt, and the model is then tasked with completing the sequence to generate an output `latex $y_\text{test}$ `.

## Interaction with External Tools

To enable LLMs to use tools, we first construct various external tools such as search engines, code interpreters, and various APIs into text-to-text functions, then interleave the LLMs generations with tool use in in-context demonstrations. As shown in Figure 2, the input for a search engine can be a query generated by LLMs, which returns a parsed search result, whereas the input for a code interpreter is a program, which returns execution information and the final execution result. This free format allows for human-like verify-then-correct trajectories, facilitating the construction of prompts intuitively and concisely while having strong interpretability and trustworthiness [yao2023react].

## Verification with Tool-Interaction

Give model `latex $\mathcal{M}$ ` and input `latex $x$ `, the initial answer is generated with prompt `latex $\wp$ ` by `latex $\hat{y_0} \sim \mathbb{P}_{\mathcal{M}}(\cdot|\wp\oplus x)$ `, where `latex $\oplus$ ` indicates concatenation. Given previous output `latex $\hat{y_{i}}$ `, LLMs interact with external tools to criticize the `latex $\hat{y_{i}}$ ` and produce critiques `latex $c_i \sim \mathbb{P}_{\mathcal{M}}(\cdot|\wp\oplus x\oplus \hat{y_{i}}, \mathcal{T})$ `. If the process involves API calls, we directly concatenate the API call results with the model-generated query to construct the `latex $c_i$ `. The task-specific critiques can be used to detail the attributes of the output we expect to evaluate, such as truthfulness, feasibility, or safety. For different inputs, we can use task-dependent, heuristically selected, or automatically selected appropriate tools for verification. We can implement automatic tool selection with in-context learning, allowing different tools for different input-output pairs. In our implementation, we pre-specify tools for different tasks to facilitate evaluation and experimentation. For example, as shown in Figure 2, the tool used for the QA task is Google, enabling LLMs to verify the truthfulness of output by analyzing and interacting with Google in an interleaved manner.

## Correction with Critiques

LLMs can generate an improved answer conditioned on input `latex $x$ `, previous output `latex $\hat{y_i}$ `, and critiques `latex $c_i$ ` from verification: `latex $\hat{y_{i+1}} \sim \mathbb{P}_{\mathcal{M}}(\cdot|\wp\oplus x\oplus y_{i}\oplus c_i)$ `. Critiques play a crucial role in the correction process as they identify errors, offer actionable suggestions, or provide credible groundings through interaction with external tools, thus guiding a new generation to avoid similar mistakes. Motivated by the human process of iterative drafts refinement, we can iterate the process of _verify-then-correct_ until specific stopping criteria are met, such as satisfying critiques from verification, reaching the maximum iterations `latex $n$ `, or receiving environmental feedback. This method facilitates continuous output improvement by systematically and sample-efficiently verifying and correcting errors resulting from interactions with the world.

# Experiments

We examine CRITIC across diverse tasks: **free-form question answering** concentrates on truthfulness related to open-ended general factual knowledge [kwiatkowski2019natural; min2020ambigqa; joshi2017triviaqa] and multi-hop reasoning [yang2018hotpotqa]; **mathematical program synthesis** emphasizes the correctness and executability of LLM-generated programs for mathematical reasoning; **toxicity reduction** concerns the safety of model generation in open-ended output spaces. We implement our approach using two settings: CRITIC applies corrections to all samples, while CRITIC* employs an *oracle\* setting, correcting only the inaccurate samples. Subsequent sections provide comprehensive implementation details, baselines, and corresponding results for each task.

### LLMs

We present experimental outcomes utilizing the `text-davinci-003` version of InstructGPT trained with RLHF [ouyang2022training], and the `gpt-3.5-turbo` variant of ChatGPT, the most advanced GPT3.5 model tailored for chat applications. To promote reproducibility, we also disclose results employing open-source LLaMA-2 models, encompassing 7B, 13B, and 70B configurations. We deploy the same prompts for the various LLMs.

# Conclusion

We propose CRITIC, a novel plug-and-play framework that empowers frozen LLMs to self-verify and self-correct by interacting with the external environment. Leveraging the intuition of critical thinking with external feedback, CRITIC enables LLMs to validate their knowledge and improve their answers through introspection without requiring further training. Experiments on diverse tasks and LLMs have consistently shown the effectiveness, generality, and interoperability of CRITIC. Moreover, we shed light on the unreliability of LLMs in self-verification, highlighting the potential of external tool interaction to solve this problem. We hope our findings will inspire further exploration into the truthfulness of language models, ultimately leading to more trustworthy AI systems.
