# Abstract

Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLM; then, the same LLM provides _feedback_ for its output and uses it to _refine_ itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner and the feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5 and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test-time using our simple, standalone approach.

Code and data at https://selfrefine.info/

# Introduction

Although large language models (LLMs) can generate coherent outputs, they often fall short in addressing intricate requirements. This mostly includes tasks with multifaceted objectives, such as dialogue response generation, or tasks with hard-to-define goals, such as enhancing program readability. In these scenarios, modern LLMs may produce an intelligible initial output, yet may benefit from further iterative refinement---i.e., iteratively mapping a candidate output to an improved one---to ensure that the desired quality is achieved. Iterative refinement typically involves training a refinement model that relies on domain-specific data. Other approaches that rely on external supervision or reward models require large training sets or expensive human annotations, which may not always be feasible to obtain. These limitations underscore the need for an effective refinement approach that can be applied to various tasks without requiring extensive supervision.

Iterative _self_-refinement is a fundamental characteristic of human problem-solving. Iterative self-refinement is a process that involves creating an initial draft and subsequently refining it based on self-provided feedback. For example, when drafting an email to request a document from a colleague, an individual may initially write a direct request such as "_Send me the data ASAP_". Upon reflection, however, the writer recognizes the potential impoliteness of the phrasing and revises it to "_Hi Ashley, could you please send me the data at your earliest convenience?_". When writing code, a programmer may implement an initial "quick and dirty" implementation, and then, upon reflection, refactor their code to a solution that is more efficient and readable. In this paper, we demonstrate that LLMs can provide iterative self-refinement without additional training, leading to higher-quality outputs on a wide range of tasks.

We present Self-Refine: an iterative self-refinement algorithm that alternates between two generative steps--FEEDBACK and REFINE. These steps work in tandem to generate high-quality outputs. Given an initial output generated by a model M, we pass it back to the same model M to get _feedback_. Then, the feedback is passed back to the same model to _refine_ the previously-generated draft. This process is repeated either for a specified number of iterations or until M determines that no further refinement is necessary. We use few-shot prompting to guide M to both generate feedback and incorporate the feedback into an improved draft.

[IMAGE: figures/autofb_figv3.pdf - Given an input, Self-Refine starts by generating an output and passing it back to the same model M to get feedback. The feedback is passed back to M, which refines the previously generated output. Steps iterate until a stopping condition is met. Self-Refine is instantiated with a language model such as GPT-3.5 and does not involve human assistance.]

We evaluate Self-Refine on 7 generation tasks that span diverse domains, including natural language and source-code generation. We show that Self-Refine outperforms direct generation from strong LLMs like GPT-3.5 (`text-davinci-003` and `gpt-3.5-turbo`) and GPT-4 by 5-40% absolute improvement. In code-generation tasks, Self-Refine improves the initial generation by up to absolute 13% when applied to strong code models such as Codex (`code-davinci-002`). We release all of our code, which is easily extensible to other LLMs. In essence, our results show that even when an LLM cannot generate an optimal output on its first try, the LLM can often provide useful feedback and improve its own output accordingly. In turn, Self-Refine provides an effective way to obtain better outputs from a single model without any additional training, via iterative (self-)feedback and refinement.

# Iterative Refinement with Self-Refine

Given an input sequence, Self-Refine generates an initial output, provides feedback on the output, and refines the output according to the feedback. Self-Refine iterates between feedback and refinement until a desired condition is met. Self-Refine relies on a suitable language model and three prompts (for initial generation, feedback, and refinement), and does not require training.

## Initial generation

Given an input x, prompt p_gen, and model M, Self-Refine generates an initial output y_0:

```latex
$$y_0 = \mathcal{M}\left(p_{\text{gen}}\| x\right)$$
```

For example, the model generates functionally correct code for the given input. Here, p_gen is a task-specific few-shot prompt (or instruction) for an initial generation, and || denotes concatenation. The few-shot prompt contains input-output pairs <x^(k), y^(k)> for the task.

Few-shot prompting (also referred to as "in-context learning") provides a model with a prompt consisting of k in-context examples of the target task, each in the form of input-output pairs <x_i, y_i>.

## FEEDBACK

Next, Self-Refine uses the same model M to provide feedback fb_t on its own output, given a task-specific prompt p_fb for generating feedback:

```latex
$$fb_t = \mathcal{M}\left(p_{\text{fb}}\| x \| y_t\right)$$
```

**The Self-Refine Algorithm:**

```
Input: x, model M, prompts {p_gen, p_fb, p_refine}, stop condition stop(.)
y_0 = M(p_gen || x)
Loop:
  fb_t = M(p_fb || x || y_t)
  if stop(fb_t): break
  y_{t+1} = M(p_refine || x || y_0 || fb_0 || ... || y_t || fb_t)
Return: y_t
```

Intuitively, the feedback may address multiple aspects of the output. For example, in code optimization, the feedback might address the efficiency, readability, and overall quality of the code.

Here, the prompt p_fb provides examples of feedback in the form of input-output-feedback triples <x^(k), y^(k), fb^(k)>. We prompt the model to write feedback that is actionable and specific via fb^(k). By 'actionable', we mean the feedback should contain a concrete action that would likely improve the output. By 'specific', we mean the feedback should identify concrete phrases in the output to change. For example, the feedback is "_This code is slow as it uses a for loop which is brute force. A better approach is to use the formula ... `(n(n+1))/2`_". This feedback is actionable, since it suggests the action 'use the formula...'. The feedback is specific since it mentions the 'for loop'.

## REFINE

Next, Self-Refine uses M to refine its most recent output, given its own feedback:

```latex
$$y_{t+1} = \mathcal{M}\left(p_{\text{refine}}\| x \| y_t \| fb_t\right)$$
```

For example, given the initial output and the generated feedback, the model generates a re-implementation that is shorter and runs much faster than the initial implementation. The prompt p*refine provides examples of improving the output based on the feedback, in the form of input-output-feedback-refined quadruples <x^(k), y_t^(k), fb_t^(k), y*{t+1}^(k)>.

## Iterating Self-Refine

Self-Refine alternates between FEEDBACK and REFINE steps until a stopping condition is met. The stopping condition stop(fb_t, t) either stops at a specified timestep t, or extracts a stopping indicator (e.g. a scalar stop score) from the feedback. In practice, the model can be prompted to generate a stopping indicator in p_fb, and the condition is determined per-task.

To inform the model about the previous iterations, we retain the history of previous feedback and outputs by appending them to the prompt. Intuitively, this allows the model to learn from past mistakes and avoid repeating them. More precisely, the refinement is instantiated as:

```latex
$$y_{t+1} = \mathcal{M}\left(p_{\text{refine}}\| x \| y_0 \| fb_0 \| ... \| y_t \| fb_t\right)$$
```

Finally, we use the last refinement y_t as the output of Self-Refine.

The key idea is that Self-Refine uses the same underlying LLM to generate, get feedback, and refine its outputs given its own feedback. It relies only on supervision present in the few-shot examples.

## Few-shot prompts make Self-Refine supervision-free

In the iterative feedback-and-refine loop of Self-Refine, we use the few shot prompts p_gen, p_fb, and p_refine. These few-shot prompts in Self-Refine equip the base LLM with generation, feedback and refinement capabilities, while being supervision-free.

- Initially, the generation prompt p_gen guides the model to produce the initial output y_0 using task-specific input-output pairs <x_i, y_i>.

- The feedback prompt p_fb provides few-shot examples of feedback in the form of input-output-feedback triples <x_i, y_i, fb_i>. These examples prompt the model to generate specific and _actionable feedback_ on its own output. That is, the prompt contains examples that point concretely to the phrases that should be rephrased, and with suggestions on how to improve them.

- The refinement prompt p*refine provides few-shot examples of refined outputs in the form of input-output-feedback-refined quadruples <x_i, y_i, fb_i, y*{i+1}>, which instruct the model to improve the output based on the feedback.

# Evaluation

We evaluate Self-Refine on 7 diverse tasks: Dialogue Response Generation, Code Optimization, Code Readability Improvement, Math Reasoning, Sentiment Reversal, and we introduce two new tasks: Acronym Generation and Constrained Generation (a harder version of CommonGen with 20-30 keyword constraints instead of 3-5).

## Instantiating Self-Refine

We instantiate Self-Refine following the high-level description above. The FEEDBACK-REFINE iterations continue until the desired output quality or task-specific criterion is reached, up to a maximum of 4 iterations. To make our evaluation consistent across different models, we implemented both FEEDBACK and REFINE as few-shot prompts even with models that respond well to instructions, such as ChatGPT and GPT-4.

### Base LLMs

Our main goal is to evaluate whether we can improve the performance of any strong base LLMs using Self-Refine. Therefore, we compare Self-Refine to the same base LLMs but without feedback-refine iterations. We used three main strong base LLM across all tasks: GPT-3.5 (`text-davinci-003`), ChatGPT (`gpt-3.5-turbo`), and GPT-4. For code-based tasks, we also experimented with Codex (`code-davinci-002`). In all tasks, either GPT-3.5 or GPT-4 is the previous state-of-the-art. We used the same prompts from previous work when available (such as for Code Optimization and Math Reasoning); otherwise, we created prompts as detailed in the Prompts section. We use greedy decoding with a temperature of 0.7 for all setups.

## Metrics

We report three types of metrics:

- **Task specific metric:** When available, we use automated metrics from prior work (Math Reasoning: % solve rate; Code Optimization: % programs optimized; Constrained Gen: coverage %)

- **Human-pref:** In Dialogue Response Generation, Code Readability Improvement, Sentiment Reversal, and Acronym Generation, since no automated metrics are available, we perform a blind human A/B evaluation on a subset of the outputs to select the preferred output.

- **GPT-4-pref:** In addition to human-pref, we use GPT-4 as a proxy for human preference following prior work, and found high correlation (82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Generation) with human-pref. For Code Readability Improvement, we prompt GPT-4 to calculate fraction of the variables that are appropriately named given the context (e.g., `x = []` -> `input_buffer = []`).

## Results

### Self-Refine consistently improves over base models

across all model sizes, and additionally outperforms the previous state-of-the-art across all tasks. For example, GPT-4+Self-Refine improves over the base GPT-4 by 8.7% (absolute) in Code Optimization, increasing optimization percentage from 27.3% to 36.0%. For code-based tasks, we found similar trends when using Codex.

One of the tasks in which we observe the highest gains compared to the base models is Constrained Generation, where the model is asked to generate a sentence containing up to 30 given concepts. We believe that this task benefits significantly from Self-Refine because there are more opportunities to miss some of the concepts on the first attempt, and thus Self-Refine allows the model to fix these mistakes subsequently. Further, this task has an extremely large number of reasonable outputs, and thus Self-Refine allows to better explore the space of possible outputs.

In preference-based tasks such as Dialogue Response Generation, Sentiment Reversal, and Acronym Generation, Self-Refine leads to especially high gains. For example in Dialogue Response Generation, GPT-4 preference score improve by 49.2% -- from 25.4% to 74.6%. Similarly, we see remarkable improvements in the other preference-based tasks across all models.

The modest performance gains in Math Reasoning can be traced back to the inability to accurately identify whether there is any error. In math, errors can be nuanced and sometimes limited to a single line or incorrect operation. Besides, a consistent-looking reasoning chain can deceive LLMs to think that "everything looks good" (e.g., ChatGPT feedback for 94% instances is 'everything looks good'). The gains with Self-Refine on Math Reasoning are much bigger (5%+) if an external source can identify if the current math answer is incorrect.

### Improvement is consistent across base LLMs sizes

Generally, GPT-4+Self-Refine performs better than GPT-3.5+Self-Refine and ChatGPT+Self-Refine across all tasks, even in tasks where the initial base results of GPT-4 were lower than GPT-3.5 or ChatGPT. We thus believe that Self-Refine allows stronger models (such as GPT-4) to unlock their full potential, even in cases where this potential is not expressed in the standard, single-pass, output generation.

# Analysis

The three main steps of Self-Refine are FEEDBACK, REFINE, and repeating them iteratively. In this section, we perform additional experiments to analyze the importance of each of these steps.

## The impact of the feedback quality

| Task               | Self-Refine feedback | Generic feedback | No feedback |
| ------------------ | -------------------- | ---------------- | ----------- |
| Code Optimization  | **27.5**             | 26.0             | 24.8        |
| Sentiment Reversal | **43.2**             | 31.2             | 0           |
| Acronym Generation | **56.4**             | 54.0             | 48.0        |

Prompting to generate generic feedback (or having the model generate no feedback at all) leads to reduced scores, indicating the importance of the FEEDBACK step of Self-Refine. These experiments were performed with ChatGPT (Code Optimization and Sentiment Reversal) and GPT-3.5 (Acronym Generation).

Feedback quality plays a crucial role in Self-Refine. To quantify its impact, we compare Self-Refine, which utilizes specific, actionable feedback, with two ablations: one using generic feedback and another without feedback (the model may still iteratively refine its generations, but is not explicitly provided feedback to do so). For example, in the Code Optimization task: actionable feedback, such as _Avoid repeated calculations in the for loop_, pinpoints an issue and suggests a clear improvement. Generic feedback, like _Improve the efficiency of the code_, lacks this precision and direction.

In Code Optimization, performance slightly dips from 27.5 (Self-Refine feedback) to 26.0 (generic feedback), and further to 24.8 (no feedback). This suggests that while generic feedback offers some guidance -- specific, actionable feedback yields superior results.

This effect is more pronounced in tasks like Sentiment Transfer, where changing from our feedback to generic feedback leads to a significant performance drop (43.2 to 31.2), and the task fails without feedback. Similarly, in Acronym Generation, without actionable feedback, performance drops from 56.4 to 48.0, even with iterative refinements. These results highlight the importance of specific, actionable feedback in our approach. Even generic feedback provides some benefit, but the best results are achieved with targeted, constructive feedback.

## How important are the multiple iterations of FEEDBACK-REFINE?

| Task             | y_0  | y_1  | y_2  | y_3      |
| ---------------- | ---- | ---- | ---- | -------- |
| Code Opt.        | 22.0 | 27.0 | 27.9 | **28.8** |
| Sentiment Rev.   | 33.9 | 34.9 | 36.1 | **36.8** |
| Constrained Gen. | 29.0 | 40.3 | 46.7 | **49.7** |

On average, the quality of the output improves as the number of iterations increases. For instance, in the Code Optimization task, the initial output (y_0) has a score of 22.0, which improves to 28.8 after three iterations (y_3). Similarly, in the Sentiment Reversal task, the initial output has a score of 33.9, which increases to 36.8 after three iterations. This trend of improvement is also evident in Constrained Generation, where the score increases from 29.0 to 49.7 after three iterations. The results also highlight the diminishing returns in the improvement as the number of iterations increases. Overall, having multiple FEEDBACK-REFINE iterations significantly enhances the quality of the output, although the marginal improvement naturally decreases with more iterations.

The performance may not always monotonically increase with iterations: in multi-aspect feedback tasks like Acronym Generation, where the output quality can vary during iteration with improvement in one aspect but decline in another aspect. To counter this, Self-Refine generates numerical scores for different quality aspects, leading to a balanced evaluation and appropriate output selection.

## Can we just generate multiple outputs instead of refining?

Does Self-Refine improve because of the iterative refinement, or just because it generates _more_ outputs? We compare Self-Refine with ChatGPT, when ChatGPT generates k=4 samples (but without feedback and refinement). Then, we compare the performance of Self-Refine against these k initial outputs in a 1 vs. k evaluation. In other words, we assess whether Self-Refine can outperform _all_ k initial outputs. Despite the increased difficulty of the 1 vs. k setting, the outputs of Self-Refine are still preferred by humans over _all_ k initial outputs. This shows the importance of refinement according to feedback over the alternative of just generating multiple initial outputs.

## Does Self-Refine work with weaker models?

The experiments were performed with some of the strongest available models; does Self-Refine work with smaller or weaker models as well? To investigate this, we instantiated Self-Refine with Vicuna-13B, a less powerful base model. While Vicuna-13B is capable of generating initial outputs, it struggles significantly with the refinement process. Specifically, Vicuna-13B was not able to consistently generate the feedback in the required format. Furthermore, even when provided with Oracle or hard-coded feedback, it often failed to adhere to the prompts for refinement. Instead of refining its output, Vicuna-13B either repeated the same output or generated a hallucinated conversation, rendering the outputs less effective. We thus hypothesize that since Vicuna-13B was trained on conversations, it does not generalize as well as instruction-based models to test-time few-shot tasks.

## Qualitative Analysis

We conduct a qualitative analysis of the feedback generated by Self-Refine and its subsequent refinements. We manually analyze 70 samples in total (35 success cases and 35 failure cases) for Code Optimization and Math Reasoning. For both Math Reasoning and Code Optimization, we found that the feedback was predominantly actionable, with the majority identifying problematic aspects of the original generation and suggesting ways to rectify them.

When Self-Refine failed to improve the original generation, the majority of issues were due to erroneous feedback rather than faulty refinements. Specifically, 33% of unsuccessful cases were due to feedback inaccurately pinpointing the error's location, while 61% were a result of feedback suggesting an inappropriate fix. Only 6% of failures were due to the refiner incorrectly implementing good feedback. These observations highlight the vital role of accurate feedback plays in Self-Refine.

In successful cases, the refiner was guided by accurate and useful feedback to make precise fixes to the original generation in 61% of the cases. Interestingly, the refiner was capable of rectifying issues even when the feedback was partially incorrect, which was the situation in 33% of successful cases. This suggests resilience to sub-optimal feedback. Future research could focus on examining the refiner's robustness to various types of feedback errors and exploring ways to enhance this resilience.

### Code Optimization Example

**Slower code (baseline):**

```python
# Slower code
def solve(amount):
  best_price = (amount + 199) // 200 * 380
  # First loop
  for a in range(amount // 200 + 1):
    # ... 4 nested loops ...
    for c1 in range(amount // 1500 + 1):
      if a*200 + b*300 == amount:
        price = a*380 + b*550
        if price < best_price:
          best_price = price
  return best_price
```

**Faster code (after Self-Refine):**

```python
# Faster code
def solve(amount):
  coins = [200, 300]
  prices = [380, 550]
  dp = [float('inf')] * (amount + 1)
  dp[0] = 0
  for i in range(len(coins)):
    for j in range(coins[i], amount+1):
      dp[j] = min(dp[j], dp[j - coins[i]] + prices[i])
  return dp[amount]
```

The initial code by the baseline fails to improve the efficiency and merely alters the logic for reading input. Self-Refine first generates feedback that diagnoses that _"This code is slow because it is using six nested loops to iterate through all possible combinations of coins to pay the amount"_, and suggests that _"a more efficient approach would be..."_. Self-Refine then uses this feedback to generate the revised code, reducing the time complexity to O(amount \* coins).

## Going Beyond Benchmarks

While our evaluation focuses on benchmark tasks, Self-Refine is designed with broader applicability in mind. We explore this in a real-world use case of website generation, where the user provides a high-level goal and Self-Refine assists in iteratively developing the website. Starting from a rudimentary initial design, Self-Refine refines HTML, CSS, and JS to evolve the website in terms of both usability and aesthetics. This demonstrates the potential of Self-Refine in real-world, complex, and creative tasks.

# Related work

Leveraging human- and machine-generated natural language (NL) feedback for refining outputs has been effective for a variety of tasks, including summarization, script generation, program synthesis, and other tasks. Refinement methods differ in the source and format of feedback, and the way that a refiner is obtained.

## Source of feedback

Humans have been an effective source of feedback. Since human feedback is costly, several approaches use a scalar reward function as a surrogate of (or alternative to) human feedback. Alternative sources such as compilers or Wikipedia edits can provide domain-specific feedback. Recently, LLMs have been used to generate feedback for general domains. However, ours is the only method that generates feedback using an LLM on its _own_ output, for the purpose of refining with the same LLM.

| Method                                                         | Supervision-free refiner | Supervision-free feedback | Multi-aspect feedback | Iterative |
| -------------------------------------------------------------- | ------------------------ | ------------------------- | --------------------- | --------- |
| Learned refiners: PEER, Self-critique, CodeRL, Self-correction | No                       | Partial                   | No                    | Partial   |
| Prompted refiners: Augmenter, Re3, Reflexion                   | Yes                      | Partial                   | No                    | Yes       |
| **Self-Refine** (this work)                                    | Yes                      | Yes                       | Yes                   | Yes       |

## Representation of feedback

The form of feedback can be generally divided into natural language (NL) and non-NL feedback. Non-NL feedback can come in human-provided example pairs or scalar rewards. In this work, we use NL feedback, since this allows the model to easily provide _self_-feedback using the same LM that generated the output, while leveraging existing pretrained LLMs such as GPT-4.

## Types of refiners

Pairs of feedback and refinement have been used to learn supervised refiners. Since gathering supervised data is costly, some methods learn refiners using model generations. However, the refiners are trained for each new domain. Finally, some approaches use prompted feedback and refinement specifically tailored for story generation. In this work, we avoid training a separate refiner, and show that the same model can be used as both the refiner and the source of feedback across multiple domains.

## Non-refinement reinforcement learning (RL) approaches

Rather than having explicit refinement, an alternative way to incorporate feedback is by optimizing a scalar reward function, e.g. with reinforcement learning. These methods differ from Self-Refine in that the model does not access feedback on an intermediate generation. Second, these RL methods require updating the model's parameters, unlike Self-Refine.

# Limitations and Discussion

The main limitation of our approach is that the base models need to have sufficient few-shot modeling or instruction-following abilities, in order to learn to provide feedback and to refine in an in-context fashion, without having to train supervised models and rely on supervised data.

Further, the experiments in this work were performed with language models that are not open-sourced, namely GPT-3.5, ChatGPT, GPT-4, and Codex. Existing literature does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases. Further, these models are not free to use, and using them for research requires some funding. Nonetheless, we release our code and model outputs to ensure the reproducibility of our work.

Another limitation of our work is that we exclusively experiment with datasets in English. In other languages, the current models may not provide the same benefits.

Finally, there is a possibility for bad actors to use prompting techniques to steer a model to generate more toxic or harmful text. Our approach does not explicitly guard against this.

# Conclusion

We present Self-Refine: a novel approach that allows large language models to iteratively provide self-feedback and refine their own outputs. Self-Refine operates within a single LLM, requiring neither additional training data nor reinforcement learning. We demonstrate the simplicity and ease of use of Self-Refine across a wide variety of tasks. By showcasing the potential of Self-Refine in diverse tasks, our research contributes to the ongoing exploration and development of large language models, with the aim of reducing the cost of human creative processes in real-world settings. We hope that our iterative approach will help drive further research in this area. To this end, we make all our code, data and prompts anonymously available at https://selfrefine.info/.

# Code Readability

Orthogonal to the correctness, readability is another important quality of a piece of code: though not related to the execution results of the code, code readability may significantly affect the usability, upgradability, and ease of maintenance of an entire codebase. In this section, we consider the problem of improving the readability of code with Self-Refine. We let an LLM write natural language readability critiques for a piece of code; the generated critiques then guide another LLM to improve the code's readability.

## Method

Following the Self-Refine setup, we instantiate INIT, FEEDBACK, and REFINE. The INIT is a no-op --- we directly start by critiquing the code with FEEDBACK and applying the changes with REFINE.

- **FEEDBACK** We prompt an LLM with the given code and an instruction to provide feedback on readability. We give the LLM the freedom to freely choose the type of enhancements and express them in the form of free text.

- **REFINE** The code generator LLM is prompted with the piece of code and the readability improvement feedback provided by FEEDBACK. In addition, we also supply an instruction to fix the code using the feedback. We take the generation from the code generator as the product of one iteration in the feedback loop.

Starting from an initial piece of code y*0, we first critique, c_1 = critique(y_0), and then edit the code, y_1 = editor(y_0, c_1). This is recursively performed N times, where c*{k+1} = critique(y*k) and y*{k+1} = editor(y*k, c*{k+1}).

## Experiments

### Dataset

We use the CodeNet dataset of competitive programming. For our purpose, these are hard-to-read multi-line code snippets. We consider a random subset of 300 examples and apply Self-Refine to them.

We also ask human annotators to edit a 60-example subset to assess human performance on this task. The human annotators are asked to read the code piece and improve its readability.

### Implementation

Both the critique and the editor models are based on the InstructGPT model (text-davinci-003). We consider the temperature of both T=0.0 (greedy) and T=0.7 (sampling) for decoding _Natural Language_ suggestion from the critique model. We always use a temperature T=0.0 (greedy) when decoding _Programming Language_ from the code editor. Due to budget constraints, we run Self-Refine for N=5 iterations.

### Evaluation Methods

We consider a few automatic heuristic-based evaluation metrics:

- **Meaningful Variable Names:** In order to understand the flow of a program, having semantically meaningful variable names can offer much useful information. We compute the ratio of meaningful variables, the number of distinct variables with meaningful names to the total number of distinct variables. We automate the process of extracting distinct variables and the meaningful subset of variables using a few-shot prompted language model.

- **Comments:** Natural language comments give explicit hints on the intent of the code. We compute the average number of comment pieces per code line.

- **Function Units:** Long functions are hard to parse. Seasoned programmers will often refactor and modularize code into smaller functional units.

### Result

For each automatic evaluation metric, the ratio of meaningful variable, of comment, and the number of function units, we compute for each iteration averaged across all test examples. We observe the average of all three metrics grows across iteration of feedback loops. A diverse generation of a higher temperature in the critique leads to more edits to improve the meaningfulness of variable names and to add comments. The greedy critique, on the other hand, provides more suggestions on refactoring the code for modularization.

| Method                   | Meaningful Variable Ratio | Comment Per Line | Function Units |
| ------------------------ | ------------------------- | ---------------- | -------------- |
| Human Annotator Rewrites | 0.653                     | 0.24             | 0.70           |
| Self-Refine (T = 0.0)    | 0.628                     | 0.12             | **1.41**       |
| Self-Refine (T = 0.7)    | **0.700**                 | **0.25**         | 1.33           |

Human vs. Self-Refine performance on 60-example subset. We see Self-Refine can reach similar or achieve even better performance on the metrics compared to rewrites given by human annotator.

### Example

**Starting Code: v_0**

```python
print((int((int(eval(input()))+1)/2)))
```

**Code v_1**

```python
print(
    (int(
        (int(eval(input())) + 1)
        / 2
    )
)
```

**Code v_2**

```python
num_input = eval(input())
num_input = int(num_input)
num_input += 1
num_result = int(num_input / 2)
print(num_result)
```

Self-Refine iterations over a piece of densely composed code. Throughout the iterations, the model first tries to indent out the code and ends up rewriting it into several lines of atomic operations.

# Dialogue Response Generation

Open-domain dialogue response generation is a complex task that requires a system to generate human-like responses to a wide range of topics. Due to the open-ended nature of the task, it is challenging to develop a system that can consistently generate coherent and engaging responses. In this section, we use Self-Refine for automatically generated feedback and applying iterative refinement to improve the quality of the responses.

## Example

**Context:** Speaker A - What are your hobbies? Speaker B - I really just spend the rest of my free time with those that are close to me. Speaker A - I'm very much interested in playing table tennis.

**Response:** That sounds like a lot of fun! I'm not very good at table tennis, but I'm sure it's a great way to socialize and stay active.

**Feedback:**

- Relevant: The response is relevant to the user's input and shows interest in the user's hobby. 3/3
- Informative: The response does not provide any information about table tennis or how to play it. 1/3
- Interesting: The response is not very interesting or intriguing. 1/3
- Consistent: The response is consistent with the tone and context of the conversation. 3/3
- Helpful: The response is not particularly helpful as it does not provide any information or suggest any actions. 1/3
- Engaging: The response is somewhat engaging. It acknowledges the user's interest in table tennis and encourages them to continue playing. 2/3
- Specific: The response is not specific, as it does not address the topic of table tennis in any particular way. 1/3
- Safe: The response is safe and does not contain any offensive, toxic or harmful content and does not touch on any sensitive topics or share any personal information. 3/3
- User understanding: The response shows some understanding of the user's input and acknowledges their interest in table tennis. 2/3
- Fluent: The response is fluent in terms of grammar and flow of words. 3/3
- Total score: 20/30

**New Response:** Table tennis is a great hobby! It's a great way to stay active and socialize with others. Have you ever tried it before? If not, I'd suggest starting with some beginner tutorials on Youtube. They are really helpful in getting a good grasp of the basics.

## Modules

We follow the high-level description of the framework, and instantiate our framework as follows.

### Init

This is the first step in performing the task. The init module takes the dialogue context as input and generates a response that follows the conversation.

### Feedback

We design a FEEDBACK that can provide multifaceted feedback for the quality of the response generated. Specifically, a response is judged along 10 qualitative aspects discussed below. We use 6 in-context examples for feedback generation. In many cases, the feedback explicitly points out the reasons why a response scores low on some qualitative aspect.

- **Relevant** Does the response addresses all important aspects of the context?
- **Informative** - Does the response provide some information relevant to the context?
- **Interesting** - Does the response beyond providing a simple and predictable answer to a question or statement?
- **Consistent** - Is the response consistent with the rest of the conversation in terms of tone and topic?
- **Helpful** - Is the response helpful in providing any information or suggesting any actions?
- **Engaging** - Is the response engaging and encourage further conversation?
- **Specific** - The response contains specific content related to a topic or question.
- **Safe** - Is the response safe and does not contain any offensive, toxic or harmful content and does not touch on any sensitive topics or share any personal information?
- **User understanding** - Does the response demonstrate an understanding of the user's input and state of mind?
- **Fluent** Is the response fluent and easy to understand?

### Iterate

The iterate module takes a sequence of dialogue context, prior generated responses, and the feedback and refines the output to match the feedback better.

## Setup and Experiments

### Model and Baseline

We establish a natural baseline for our approach by using the model directly, without any feedback, which we refer to as INIT. Our implementation of Self-Refine employs a few-shot setup, where each module (INIT, FEEDBACK, ITERATE) is implemented as few-shot prompts, and we execute the self-improvement loop for a maximum k=3 iterations. We provide 3 few-shot in-context examples for the INIT model, and instruct the model to produce a response that is good at the 10 aspects listed above. As in-context examples for FEEDBACK, we use the same 3 contexts and responses shown to the INIT model (including low-scoring variations of those responses), along with scores and explanations for each feedback aspect. The ITERATE model is also shown the same in-context examples, and it consists of contexts-response-feedback followed by a better version of the response. For Self-Refine, we chose the response that gets the highest total score from the FEEDBACK model across all iterations excluding the initial response. We use `text-davinci-003` for all the experiments.

| Method           | GPT-3.5 | ChatGPT | GPT4 |
| ---------------- | ------- | ------- | ---- |
| Self-Refine wins | 36.0    | 48.0    | 54.0 |
| INIT wins        | 23.0    | 18.0    | 16.0 |
| Both are equal   | 41.0    | 50.0    | 30.0 |

Human evaluation results for dialogue response generation.

### Evaluation

We perform experiments on the FED dataset. The FED dataset is a collection of human-system and human-human conversations annotated with eighteen fine-grained dialog qualities at both the turn and the dialogue-level. The dataset was created to evaluate interactive dialog systems without relying on reference responses or training data. We evaluate the quality of the generated outputs using both automated and human evaluation methods. For automatic evaluation, we used zero-shot prompting with `text-davinci-003` and evaluate on a test set of 342 instances. We show the model the responses generated by Self-Refine and the baseline INIT and ask the model to select the better response in terms of the 10 qualities. We report the win rate.

Given a dialogue context with a varying number of turns, we generate outputs from the above mentioned methods. For human evaluation, for 100 randomly selected test instances, we show annotators the 10 response quality aspects, responses from Self-Refine and INIT models and ask them to select the better response. They are also given the option to select "both" when it is hard to show preference toward one response.

### Results

`text-davinci-003` is capable of generating human-like responses of great quality for a wide range of dialogue contexts and hence GPT-3.5 is a strong baseline. Still, Self-Refine beats INIT by a wide margin on both automatic as well as human evaluation. Our manual analysis shows that outputs generated by Self-Refine are more engaging and interesting and generally more elaborate than the outputs generated by INIT.

# Code Optimization

Performance-Improving Code Edits or PIE focuses on enhancing the efficiency of functionally correct programs. The primary objective of PIE is to optimize a given program by implementing algorithmic modifications that lead to improved runtime performance.

Given an optimization generated by PIE, Self-Refine first generates a natural language feedback on possible improvements. Then, the feedback is fed to REFINE for refinement.

| Setup                  | Iteration | % Optimized | Relative Speedup | Speedup  |
| ---------------------- | --------- | ----------- | ---------------- | -------- |
| Direct                 | -         | 9.7         | 62.29            | 3.09     |
| Self-Refine - feedback | 1         | 10.1        | 62.15            | 3.03     |
| Self-Refine - feedback | 2         | 10.4        | 61.79            | 3.01     |
| Self-Refine            | 1         | 15.3        | 59.64            | 2.90     |
| Self-Refine            | 2         | **15.6**    | **65.60**        | **3.74** |

Performance comparison of Self-Refine and ablated variants for code optimization. The table highlights the effectiveness of Self-Refine in optimizing code through iterative feedback and improvement, outperforming both the direct method and the simplified feedback approach, which lacks the introspective feedback mechanism of Self-Refine. This demonstrates the value of our framework's multi-faceted feedback in refining the generated code.

# Math Reasoning

We use the Grade School Math 8k (GSM-8k) dataset for evaluating Self-Refine on math reasoning. In the context of grade school mathematics, Self-Refine aims to enable LLMs to iteratively refine their mathematical problem-solving outputs based on introspective feedback.

Following prior work, we write solutions to the reasoning problems in Python. Consider the following example from the paper, where an error in the code demonstrates a lack of understanding of the problem:

```python
def solution():
    """Twenty dozen cups cost $1200 less than the total cost of
    half a dozen plates sold at $6000 each.
    Calculate the total cost of buying each cup."""
    plates = 6
    plate_cost = 6000
    cups = 12 * 20
    cup_cost = plate_cost
    result = cup_cost
    return result
```

By using Self-Refine, we can identify the error in the code and refine the solution through an iterative process of introspection and feedback:

```python
# Let's go through the error and check step-by-step
plates = 6
plate_cost = 6000

# Let's check the other parts
cups = 12 * 20
cup_cost = plate_cost  # wrong! The cost of a cup is not the same as the cost of a plate.
# The cost of a cup is $1200 less than the total cost of half a dozen plates sold at $6000 each.
half_dozen_plate_cost = 6 * plate_cost
cup_cost = half_dozen_plate_cost - 1200
```

Self-Refine is thus instantiated naturally: the generator generates an initial solution, and FEEDBACK scans the solution to spot errors on which to provide feedback. The feedback is supplied to REFINE to create a new solution. Following prior work, we use the correct label to decide when to go from one point in the loop to the next.

# Sentiment Reversal

We consider the task of long-form text style transfer, where given a passage (a few sentences) and an associated sentiment (positive or negative), the task is to re-write the passage to flip its sentiment (positive to negative or vice-versa). While a large body of work on style transfer is directed at sentence-level sentiment transfer, we focus on transferring the sentiment of entire reviews, making the task challenging and providing opportunities for iterative improvements.

## Instantiating Self-Refine for sentiment reversal

We instantiate Self-Refine for this task following the high-level description of the framework. Recall that our approach requires three components: INIT to generate an initial output, FEEDBACK to generate feedback on the initial output, and REFINE for improving the output based on the feedback.

Self-Refine is implemented in a complete few-shot setup, where each module (INIT, FEEDBACK, ITERATE) is implemented as few-shot prompts. We execute the self-improvement loop for a maximum of k=4 iterations. The iterations continue until the target sentiment is reached.

## Details

### Evaluation

Given an input and a desired sentiment level, we generate outputs Self-Refine and the baselines. Then, we measure the % of times output from each setup was preferred to better align with the desired sentiment level.

We also experiment with standard text-classification metric. That is, given a transferred review, we use an off-the-shelf text-classifier (Vader) to judge its sentiment level. We find that all methods were successful in generating an output that aligns with the target sentiment. For instance, when the target sentiment was positive, both GPT-3.5 with `text-davinci-003` and Self-Refine generates sentences that have a positive sentiment (100% classification accuracy). With the negative target sentiment, the classification scores were 92% for GPT-3.5 and 93.6% for Self-Refine.

We conduct automated and human evaluation for measuring the preference rates for adhering to the desired sentiment, and how dramatic the generations are. For automated evaluation, we create few-shot examples for evaluating which of the two reviews is more positive and less boring. We use a separate prompt for each task. The prompts show examples of reviews of varying degrees of sentiment and colorfulness (more colorful reviews use extreme phrases --- the food was really bad vs. I wouldn't eat it if they pay me.). The model is then required to select one of the outputs as being more aligned with the sentiment and having a more exciting language. We report the preference rates: the % of times a variant was preferred by the model over the outputs generated by Self-Refine.

### Pin-pointed feedback

A key contribution of our method is supplying chain-of-thought prompting style feedback. That is, the feedback not only indicates that the target sentiment has not reached, but further points out phrases and words in the review that should be altered to reach the desired sentiment level. We experiment with an ablation of our setup where the feedback module simply says "something is wrong." In such cases, for sentiment evaluation, the output from Self-Refine were preferred 73% of the time (down from 85% with informative feedback). For dramatic response evaluation, we found that the preference rate went down drastically to 58.92%, from 80.09%. These results clearly indicate the importance of pin-pointed feedback.

# Acronym Generation

Good acronyms provide a concise and memorable way to communicate complex ideas, making them easier to understand and remember, ultimately leading to more efficient and effective communication. Like in email writing, acronym generation also requires an iterative refinement process to achieve a concise and memorable representation of a complex term or phrase. Acronyms often involve tradeoffs between length, ease of pronunciation, and relevance to the original term or phrase. Thus, acronym generation is a natural method testbed for our approach.

We source the dataset for this task from a GitHub repository, and prune the file manually to remove potentially offensive or completely uninformative acronyms. This exercise generated a list of 250 acronyms.

## FEEDBACK

For feedback, we design a FEEDBACK that can provide multifaceted feedback. Specifically, each acronym is judged along five dimensions:

- **Ease of pronunciation:** How easy or difficult is it to pronounce the acronym? Are there any difficult or awkward sounds or combinations of letters that could make it challenging to say out loud?

- **Ease of spelling:** How easy or difficult is it to spell the acronym? Are there any unusual or uncommon letter combinations that could make it tricky to write or remember?

- **Relation to title:** How closely does the acronym reflect the content or topic of the associated title, phrase, or concept? Is the acronym clearly related to the original term or does it seem unrelated or random?

- **Positive connotation:** Does the acronym have any positive or negative associations or connotations? Does it sound upbeat, neutral, or negative in tone or meaning?

- **Well-known:** How familiar or recognizable is the acronym to the target audience? Is it a common or widely-used term, or is it obscure or unfamiliar?

Some of these criteria are difficult to quantify, and are a matter of human preference. As with other modules, we leverage the superior instruction following capabilities of modern LLMs to instead provide a few demonstrations of each task. Crucially, the feedback includes a chain of thought style reasoning --- before generating the score for an acronym for a specific criteria, we generate a reasoning chain explicitly stating the reason for the scores. We use human evaluation to judge the final quality of the acronyms.

### Example: "Sequence to Sequence Learning with Neural Networks"

| Criteria              | GPT3: STSLWN                                                          | Self-Refine: Seq2Seq                                                                       |
| --------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| Ease of pronunciation | Pronounced as ess-tee-ess-ell-double-you-enn which is very difficult. | Pronounced as seq-two-seq which is easy.                                                   |
| Ease of spelling      | Very difficult to spell.                                              | Easy to spell.                                                                             |
| Relation to title     | No relation to the title.                                             | Mentions sequence which is somewhat related to the title.                                  |
| Positive connotation  | Meaningless acronym.                                                  | Positive connotation giving a sense of ease with which the learning algorithm can be used. |
| Well-known            | Not a well-known acronym.                                             | Close to the word sequence which is a well-known word.                                     |
| Total score           | 5/25                                                                  | 20/25                                                                                      |

# Constrained Generation

In this work, we introduce a more challenging variant of the CommonGen task, dubbed "CommonGen-Hard," designed to push the boundaries of state-of-the-art language models. CommonGen-Hard requires models to generate coherent and grammatically correct sentences incorporating 20-30 concepts, as opposed to the original task which presents a set of 3-5 related concepts. This significant increase in the number of concepts tests the model's ability to perform advanced commonsense reasoning, contextual understanding, and creative problem-solving, as it must generate meaningful sentences that encompass a broader range of ideas. This new dataset serves as a valuable benchmark for the continuous improvement of large language models and their potential applications in complex, real-world scenarios.

The increased complexity of the CommonGen-Hard task makes it an ideal testbed for evaluating the effectiveness of our proposed framework, Self-Refine, which focuses on iterative creation with introspective feedback. Given that initial outputs from language models may not always meet the desired level of quality, coherence, or sensibility, applying Self-Refine enables the models to provide multi-dimensional feedback on their own generated output and subsequently refine it based on the introspective feedback provided. Through iterative creation and self-reflection, the Self-Refine framework empowers language models to progressively enhance the quality of their output, closely mimicking the human creative process and demonstrating its ability to improve generated text on complex and demanding natural language generation tasks like CommonGen-Hard.

# Prompts

We include all the prompts used in the experiments:

- **Acronym Generation:** Init, Feedback, and Refine prompts
- **Code Optimization:** Init, Feedback, and Refine prompts
- **Code Readability Improvement:** Feedback and Refine prompts
- **Constrained Generation:** Init, Feedback, and Refine prompts
- **Dialogue Response Generation:** Init, Feedback, and Refine prompts
- **Math Reasoning:** Init, Feedback, and Refine prompts
- **Sentiment Reversal:** Init, Feedback, and Refine prompts

Recall that the Base LLM requires a generation prompt p*gen with input-output pairs <x_i, y_i>, the feedback module requires a feedback prompt p_fb with input-output-feedback triples <x_i, y_i, fb_i>, and the refinement module (REFINE) requires a refinement prompt p_refine with input-output-feedback-refined quadruples <x_i, y_i, fb_i, y*{i+1}>.

- **Sentiment Reversal** We create positive and negative variants of a single review from the training set and manually write a description for converting the negative variant to positive and vice versa. For each variant, the authors generate a response and create a feedback fb_i based on the conversion description.

- **Dialogue Response Generation** We sample six examples as <x*i, y_i> for the few-shot prompt for the Base LLM. For each output y_i, the authors create a response, evaluate it based on a rubric to generate fb_i, and produce an improved version y*{i+1}.

- **Acronym Generation** We provide the Base LLM with a total of 15 (title, acronym) examples. Then, for one title (x*i) we generate an acronym (y_i) using ChatGPT. The authors then score the acronyms based on a 5-point rubric to create the corresponding fb_i, and write improved versions of the acronym to create y*{i+1}. 3 such examples are used for REFINE and FEEDBACK.

- **Code Optimization** We use the slow (x_i) and fast (y_i) versions of programs released by PIE for Base LLM. We use their provided explanations for FEEDBACK and REFINE.

- **Math Reasoning** The prompts for the Base LLM are sourced from PaL as <x*i, y_i>. We select two examples from the training set on which Codex fails when prompted with PaL-styled prompts, and manually write the correct solution (y*{i+1}) and reasoning (fb_i) for REFINE and FEEDBACK.

- **Constrained Generation** We provide ten examples to the Base LLM as <x_i, y_i>. We sample six examples from the training set of Constrained Generation and create variants with missing concepts or incoherent outputs. The missing concepts and the reason for incoherence form fb.
