# Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency

**arXiv:** 2309.17272

**Authors:** Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, Nan Duan

**Affiliations:** Wangxuan Institute of Computer Technology, Peking University; State Key Laboratory of Media Convergence Production Technology and Systems; Microsoft Research Asia

**Code:** https://github.com/skpig/MPSC

## Abstract

Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. Prior works utilize _verification properties_ in software engineering to verify and re-rank solutions in a majority voting manner. But the assumption behind them that generated verification properties have better qualities than solutions may not always hold. In this paper, we treat them equally as different _perspectives_ of LLMs' reasoning processes. We propose the **Multi-Perspective Self-Consistency (MPSC)** framework incorporating both inter- and intra-consistency across outputs from multiple perspectives. Specifically, we prompt LLMs to generate diverse outputs from three perspectives, _Solution_, _Specification_ and _Test case_, constructing a 3-partite graph. With two measure functions of consistency, we embed both inter- and intra-consistency information into the graph. The optimal choice of solutions is then determined based on analysis in the graph. MPSC significantly boosts performance of foundation models (ChatGPT in this paper) on various benchmarks, including HumanEval (+**15.91**%), MBPP (+**6.43**%) and CodeContests (+**9.37**%), even surpassing GPT-4.

## Introduction

In recent years, pre-trained large language models (LLMs) have demonstrated unprecedented proficiency in understanding, generating, and reasoning with human language [brown2020language; chowdhery2022palm; openai2023gpt; touvron2023llama]. Among the diverse applications of LLMs, code generation stands out as pivotal task and has been acknowledged as a fundamental task for benchmarking [liang2023holistic]. This task entails models to generate source codes from provided natural language intents. Many foundation models have exhibited remarkable zero-shot performance in code generation, such as ChatGPT and GPT4 [openai2023gpt], with successful deployments in real-world applications like Github Copilot.

[IMAGE: intro_performance.pdf - Pass@1 of MPSC. With ChatGPT as the foundation model, MPSC even surpasses GPT-4 and achieves SOTA performance on all four benchmarks.]

Despite the remarkable abilities, LLMs often struggle to generate the correct code in a single attempt. Therefore, previous works sample diverse codes from LLMs and re-rank them by introducing _verification properties_ from software engineering. For example, CodeT [chen2022CodeT] generates test cases as a verification property, while ALGO [zhang2023ALGO] generates oracles, the brute-force version of desired algorithms, as a verification property. These methods are essentially variants of majority voting, making the assumption that the correctness of experts (i.e. the verification properties) is better than that of choices (i.e. the desired code outputs). However, both verification properties and desired code outputs are usually generated by the identical model with respect to the same question, and hence the preference over verification properties is not always correct.

Instead, we believe that both desired code outputs and verification properties should be treated equally, since they are different _perspectives_ of LLM's deliberate thinking process in face of identical questions. Aggregating various outputs from different perspectives can lead to a more credible result. To achieve this, we propose the **Multi-Perspective Self-Consistency (MPSC)** framework that incorporates both inter-consistency across multiple perspectives and intra-consistency within a single perspective. In this way, MPSC can fully leverage the consistency information within LLMs and select the model output with the most consistent functionality as the final answer.

In our framework, various verification techniques from software engineering can be included as extended perspectives for better reasoning. Specifically, we prompt the LLM to simultaneously generate diverse outputs from three well-established perspectives in software engineering, namely _Solution_, _Specification_ and _Test case_ [abrahamsson2017agile]. Solutions implement the desired functionality, specifications demonstrate the intended properties in formal language, while test cases outline the expected behavior for some specific inputs. Then, we treat these model outputs as vertices in a graph, and establish connections (i.e. edges) based on the pairwise agreement of vertices from different perspectives. Our goal is to identify the most reliable output using a score function, which evaluates all vertices by considering both intra- and inter-consistency information encoded in the graph. Specifically, the intra-consistency information guides the function to favor the most internally consistent output within a single perspective, while inter-consistency ensures that the scores for two outputs from different perspectives are similar if they reach a consensus. We formalize the learning process of the score function as an optimization problem adhering to these two consistency criteria and leverage an iterative algorithm proposed by zhou2003Ranking to achieve this goal.

We evaluate MPSC on four widely used code generation benchmarks, including HumanEval [chen2021Evaluating], HumanEval+ [liu2023evalplus], MBPP [austin2021Program] and CodeContests [li2022CompetitionLevel]. Experimental results show that our method boosts the performance of ChatGPT by a large margin, 15.91% in HumanEval, 15.64% in HumanEval+, 6.43% in MBPP and 9.37% in CodeContests. Our framework even surpasses GPT-4 [openai2023gpt].

[IMAGE: detailed_example.pdf - Overview of our MPSC code generation method. (a) Stage 1: we require a LLM to generate diverse solutions, specifications and test cases. A detailed example of the three perspectives of function median(l) from HumanEval is presented. (b) Stage 2: we construct a 3-partite graph based on the generated outputs and then calculate both inter- and intra-consistency measures over edges and vertices respectively. The magnitudes of measurements are demonstrated by the shade of colors. (c) Stage 3: Incorporating the multi-perspective consistency information, we then learn a score function to re-rank outputs within each perspective.]

### Related Work

#### Prompting techniques on consistency

Based on Chain-of-thought mechanism [wei2022chain], previous works have adopted various prompting techniques and decoding strategies to reveal the consistency of LLM outputs and further enhance the capabilities of LLMs. One line of approaches decodes multiple times from the same perspective and aggregate the results [wang2022SelfConsistency; zhou2022least; jung2022maieutic; sun2022RecitationAugmented; chen2023universal]. For example, wang2022SelfConsistency targets tasks with fixed answer sets and scores each answer based on the output frequency. Building on this, sun2022RecitationAugmented introduces recitation as context for augmentation. While jung2022maieutic focus on the two-value entailment relations (True or False) between statements and explanations. They treat the inference process as a weighted MAX-SAT problem and utilize a logistic solver to solve it. Another line draws inspiration from the "Dual Process" in cognitive science, which posits that human reasoning is dominated by System 1 and System 2 [daniel2017thinking; sloman1996empirical]. As a result, these approaches require LLMs to play different roles like generator (i.e. System 1) and verifier (i.e. System 2), and optimize the result iteratively by a conversational way [madaan2023SelfRefine; shinn2023Reflexion; zhu2023Solving]. xiong2023Examining also proposes the concept of "inter-consistency". Instead of referring to the consistency within the same LLM, they focus to tackle the inter-inconsistency problem between different models with a formal debate framework.

#### LLM for code

LLMs pretrained on large-scale code data have demonstrated strong capabilities in the field of code generation, such as Codex [chen2021Evaluating], AlphaCode [li2022CompetitionLevel], CodeGen [nijkamp2023codegen], InCoder [fried2022incoder], StarCoder [li2023starcoder], Code Llama [roziere2023codellama], WizardCoder [luo2023wizardcoder], DeepSeekCoder [guo2024deepseekcoder]. However, they remain unreliable, particularly in scenarios involving complex input-output mappings. Because of the low tolerance of compilers and operating systems for bugs, the instability makes LLMs hard to deploy into real-world applications. Several methods have been proposed to mitigate the phenomenon [shi2022Natural; chen2022CodeT; zhang2022Coder; key2022Speak; ni2023lever; dong2023Selfcollaboration; olausson2023Demystifying; chen2023Teaching; zhang2023ALGO]. The line of work with the most direct relevance to ours is to re-rank generated solutions in a post-hoc manner. For example, shi2022Natural matches the execution results of generated solutions for minimum Bayes risk selection. zhang2022Coder prompts another model as a reviewer to check whether generated programs satisfy the given language instruction by measuring `latex $p(\text{instruction}|\text{program})$ `. CodeT [chen2022CodeT] additionally generates test cases to verify the generated solutions. Similarly, ALGO [zhang2023ALGO] additionally generates exhaustive search algorithms as oracle programs to generate high quality test cases for verification.

#### Ranking on graph

In our framework, the final stage is in fact a ranking problem in graph. There exists some renowned graph ranking algorithms like PageRank [Page1998PageRank] and HITS [HITS]. While our approach is inspired by manifold ranking [zhou2003Ranking], which is built upon a regularization framework on discrete spaces (i.e. graphs in this scenario) [zhou2003Learning; zhou2004Regularization; zhou2005Regularization].

## Multi-Perspective Self-Consistency

A single perspective can often lead to an incomplete understanding of a problem, akin to the parable of "blind men describing an elephant". The reasoning process of LLMs follows a similar pattern. LLMs generally cannot guarantee the correctness of generated output in a single attempt, especially in code generation, which necessitates proficient natural language understanding, deliberate reasoning and rigorous controllability.

However, a key aspect of human intelligence is the ability to think from multiple perspectives, resulting in a more comprehensive understanding of situations and more accurate solutions to problems. Inspired by the human cognition process, we propose a novel code generation method by reasoning from three well-established perspectives, solutions, specifications and test cases. Although noisy outputs may inevitably be included in the generated outputs of every perspective, we can leverage both intra-consistency and inter-consistency among the diverse outputs to distinguish the best ones from the noise.

### Solution, Specification and Test Case

Given a user intent in natural language, we introduce solution, specification and test case as three perspectives to describe the desired functionality. A _solution_ is the source code implementing the functionality denoted as `latex $g:\mathbb{X}\rightarrow\mathbb{Y}$ `, which is also the target of code generation. A _test case_ is a pair of valid input and output satisfying the required functionality denoted as `latex $(x,y)\in \mathbb{X}\times\mathbb{Y}$ `. _Specification_ draws inspiration from _Formal Verification_ in software engineering, which mathematically proves the correctness of one program by ensuring its satisfaction of some certain formal specifications. In the context of software engineering, formal verification is usually written in formal programming languages, e.g. Coq [coq] and Dafny [dafny], and conducted by accompanying verification tools. For the generalization of the proposed method in different program language scenarios, we adopt the idea of formal verification and limit the specifications within pre-conditions and post-conditions, which can be written as functions in the same programming language like solutions, without struggling with formal languages. Specifically, a pre-condition constrains the requirements that a valid input should satisfy, while a post-condition constrains the relationships that a pair of valid inputs and outputs should satisfy. We denote them as `latex $h^{pre}: \mathbb{X}\rightarrow\{False,True\}$ ` and `latex $h^{post}:\mathbb{X}\times\mathbb{Y}\rightarrow\{False,True\}$ `.

### Graph Construction

We require LLMs to generate diverse outputs from all three perspectives. We employ a 3-partite graph representation to capture the relationships among these generated outputs. Specifically, we represent the generated solutions `latex $\{g_1,g_2,...,g_I\}$ ` with a vertex set `latex $V^{func}$ `, the specification set `latex $\{(h^{pre}_1,h^{post}_1),...,(h^{pre}_J,h^{post}_J)\}$ ` with `latex $V^{spec}$ `, the test case set `latex $\{(x_1, y_1),...,(x_K,y_K)\}$ ` with `latex $V^{test}$ `, and hence construct a vertex set `latex $V = V^{func}\cup V^{spec}\cup V^{test}$ `. With edges connecting each pair of vertices from two distinct sets, we construct an undirected 3-partite graph `latex $\mathcal{G}=(V,E)$ `. Our goal is to leverage the graph to encode the multi-perspective consistency information, and then learn a score function `latex $f: V\rightarrow \mathbb{R}$ ` (also a vector `latex $\mathbf{f}$ `, `latex $f_i=f(v_i)$ `) from it to choose the most reliable output among all.

### Inter-Consistency between Different Perspectives

We distinguish between two kinds of consistency based on the perspectives involved. Intra-consistency is defined as the degree to which a given output aligns with others within the same perspective, following the original definition in wang2022SelfConsistency. Conversely, inter-consistency is defined as the degree of consensus between a pair of outputs from two different perspectives.

With the well-established definitions of these three perspectives in software engineering, each output implicitly describes a latent functionality regardless of whether it is a solution, a specification or a test case. Consequently, we define the inter-consistency between two vertices from different perspectives as the alignment of their latent functionalities. And the most appealing aspect is that we can quantify the alignments with a code interpreter in a deterministic manner. We formalize the inter-consistency as a measure function `latex $\omega(\cdot,\cdot): V \times V \rightarrow \mathbb{R}$ ` (also the adjacency matrix `latex $\mathbf{W}$ `, where `latex $W_{i,j} = \omega(v_i, v_j)$ `) to weight different edges in different ways.

**Inter-consistency expressions:**

| Vertex Types                                | Expression of `latex $\omega(v_i, v_j)$ `                                                         |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `latex $v_i\in V^{func}, v_j\in V^{spec}$ ` | `latex $\mathbb{E}_{x\in\mathbb{X}}[\mathbf{1}_{h^{pre}_j(x)\rightarrow h^{post}_j(x,g_i(x))}]$ ` |
| `latex $v_i\in V^{func}, v_j\in V^{test}$ ` | `latex $\mathbf{1}_{g_i(x_j)=y_j}$ `                                                              |
| `latex $v_i\in V^{spec}, v_j\in V^{test}$ ` | `latex $\mathbf{1}_{h^{pre}_i(x_j)\wedge h^{post}_i(x_j,y_j)}$ `                                  |
| otherwise                                   | 0                                                                                                 |

We then derive an optimization objective based on inter-consistency measurements:

```latex
$$\mathcal{L}_{inter} = \sum_{(v_i,v_j)\in E}W_{i,j}(f(v_i)-f(v_j))^2 = \mathbf{f}^T \mathbf{L}\mathbf{f}$$
```

where `latex $\mathbf{L}=\mathbf{D}-\mathbf{W}$ ` is the Laplacian matrix of the graph `latex $\mathcal{G}$ `. In the experiment, the symmetric normalized Laplacian `latex $\mathbf{L}^{sym}=\mathbf{D}^{-\frac{1}{2}}\mathbf{L}\mathbf{D}^{-\frac{1}{2}}$ ` is used for more robust performance. The loss function is the weighted sum of the local variation of each edge on the graph. An underlying assumption is that _a pair of outputs exhibiting consistency are either both correct or both incorrect_. Therefore, the difference between scores of two connected vertices should be constrained by the penalty corresponding to the degree of consistency, i.e. edge weight.

### Intra-Consistency within the Same Perspective

Following wang2022SelfConsistency, we define the intra-consistency of one generated output as its similarity to others within the same perspective, which is denoted as a function `latex $\varphi(\cdot): V \rightarrow \mathbb{R}$ ` (also a vector `latex $\mathbf{y}$ `, `latex $y_i=\varphi(v_i)$ `).

wang2022SelfConsistency limits the consistency to mere equalities in final answers, thereby lacking efficacy when applied to open-form tasks. In the scenario of code generation, we extend the scope of intra-consistency to lexical and semantic similarities.

#### Lexical intra-consistency by Bayes risk

Minimum Bayes risk decoding [kumar2004Minimum] selects the hypothesis `latex $h\in\mathbb{H}$ ` that minimizes the expected loss `latex $R(h)=\mathbb{E}_{y\sim P(y)}[L(y,h)]$ ` over the distribution of label `latex $y$ `. Because of the unavailability of `latex $P(y)$ `, `latex $P(h)$ ` is usually used as a proxy distribution in practice. Then the Bayes risk can be rewritten as `latex $R(h)=\sum_{h'\in\mathbb{H}}L(h',h)\cdot P(h')$ `, which is in fact measure the consistency of `latex $h$ ` over the hypothesis space. Specifically, we utilize negative BLEU metrics [papineni2002bleu] as the loss function `latex $L$ ` aiming at lexical similarity and assume the hypothesis distribution is uniform, i.e.

```latex
$$\varphi(v_i) = C\cdot\sum_{v_j\in K(v_i)}\text{BLEU}(v_i,v_j)$$
```

where `latex $C$ ` is the normalizing constant so that measures of outputs in one perspective sum up to 1, `latex $K(v_i)$ ` represents the other outputs within the same perspective as `latex $v_i$ `.

#### Semantic intra-consistency by structural equivalence

In the realm of graph theory, two vertices are deemed structurally equivalent if they share identical weighted connections with the same third-party vertices. Utilizing this equivalence relation, we delineate `latex $V^{func}$ `, `latex $V^{spec}$ `, and `latex $V^{test}$ ` into their respective structural equivalence classes. Noted that the weights assigned to edges reflect the alignments of latent functionalities associated with the vertices, and hence outputs within each equivalence class can be regarded as exhibiting consistent functional behavior. Therefore, we define the intra-consistency measure based on the structural equivalence classes within each perspective. Suppose `latex $v_i$ ` belongs to the solution set `latex $V^{func}$ `. The structural equivalence class of `latex $v_i$ ` is denoted as `latex $S(v_i)\subset V^{func}$ `, and neighbor sets of `latex $v_i$ ` can be partitioned into two subsets `latex $\{N_t(v_i)|t\in\{spec, test\}\}$ ` depending on the perspective they belong to. Overall, the semantic intra-consistency measure is defined as the multiplication of these three sets, i.e.

```latex
$$\varphi(v_i) = C\cdot |S(v_i)|\cdot\prod_t|N_t(v_i)|$$
```

where `latex $C$ ` is the normalizing constant. The notation is similar when `latex $v_i$ ` belongs to other perspectives.

Intra-consistency is in fact an estimate of the LLM's uncertainty [kuhn2023semantic; xiong2023llms] and reflects the confidence of the model on one specific output. Therefore, we can utilize the intra-consistency information as a supervision signal by ensuring the closeness between the score function `latex $f$ ` and the intra-consistency measure `latex $\varphi$ ` with Mean Squared Error (MSE):

```latex
$$\mathcal{L}_{intra}=\frac{1}{2} \sum_{v_i\in V} |f(v_i)-\varphi(v_i)|^2=\frac{1}{2} ||\mathbf{f}-\mathbf{y}||^2$$
```

### Optimization Formulation

Following the criteria of inter- and intra-consistency, we can then formulate the learning process of `latex $f$ ` as an optimization problem that combines both `latex $\mathcal{L}_{inter}$ ` and `latex $\mathcal{L}_{intra}$ `:

```latex
$$\min_{f:V\rightarrow \mathbb{R}}\{\alpha\cdot\mathcal{L}_{inter}+(1-\alpha)\cdot\mathcal{L}_{intra}\}$$
```

To solve this optimization problem on the graph, we adopt the iterative algorithm proposed by zhou2003Ranking.

## Experiment

### Experiment Settings

#### Dataset and metrics

We conduct experiments on four widely used Python code generation benchmarks, including HumanEval, HumanEval+, MBPP and CodeContests. HumanEval [chen2021Evaluating] and MBPP [austin2021Program] are two hand-written Python programming problems. HumanEval+ [liu2023evalplus] adds more unit tests based on HumanEval. CodeContests [li2022CompetitionLevel] is a much more challenging dataset consisting of competition problems from the Codeforces platform. The evaluation metric is Pass@k [chen2021Evaluating], which is an unbiased estimator of the probability that at least one out of the `latex $k$ ` solutions generated by the model passes unit tests.

#### Implementation and baselines

We compare several baselines from different LLMs for code like ChatGPT (i.e. GPT-3.5-Turbo), GPT-4 [openai2023gpt], Code Llama [roziere2024code], WizardCoder [luo2023wizardcoder] and Deepseek Coder [guo2024deepseekcoder], to other post-hoc approaches enhancing LLMs during inference, including Self-consistency [wang2022SelfConsistency], MBR-exec [shi2022Natural], CodeT [chen2022CodeT] and Self-collaboration [dong2023Selfcollaboration]. For both MPSC and other post-hoc augmentation approaches, we employ GPT-3.5-Turbo as the foundation model to generate 200 solutions. MPSC additionally generates 50 specifications and 100 test cases for each problem. Following the original setting in chen2022CodeT, we additionally generate 500 test cases for other baselines.

#### Variants of MPSC

The intra-consistency measure `latex $\varphi(\cdot)$ ` is essentially used as a supervision signal without leveraging the semantics of "consistency". Therefore, we include two variants in our experiments:

1. **MPSC-Uniform** is the baseline without any prior intra-consistency information and treats every vertex equally, i.e. `latex $\varphi(v_i)=C$ `.
2. **MPSC-Label** includes the public example test cases in docstrings as silver labels, i.e. `latex $\varphi(v_i)=C\cdot \mathbf{1}_{v_i\text{ is label}}$ `. (The number of example test cases is two on average. Note that MBPP doesn't provide test cases in docstrings.)

### Main Results

**Table 1: Results on four code generation benchmarks**

| Benchmark        | Method            | Pass@1             | Pass@2             | Pass@5             |
| ---------------- | ----------------- | ------------------ | ------------------ | ------------------ |
| **HumanEval**    | GPT4              | 81.48              | 86.31              | 90.46              |
|                  | GPT-3.5-Turbo     | 68.38              | 76.24              | 83.15              |
|                  | Self-consistency  | 73.86 (+5.48)      | 73.93 (-2.31)      | 74.10 (-9.05)      |
|                  | MBR-exec          | 72.96 (+4.58)      | 76.47 (+0.23)      | 79.00 (-4.15)      |
|                  | CodeT             | 78.05 (+9.67)      | 78.05 (+1.81)      | 78.30 (-4.85)      |
|                  | MPSC-Uniform      | 74.17 (+5.79)      | 77.02 (+0.78)      | 78.53 (-4.62)      |
|                  | MPSC-Lexical      | 82.32 (+13.94)     | 84.76 (+8.52)      | 86.48 (+3.33)      |
|                  | MPSC-Semantic     | 83.38 (+15.00)     | 84.25 (+8.01)      | 84.45 (+1.30)      |
|                  | **MPSC-Label**    | **84.29 (+15.91)** | **86.79 (+10.55)** | 87.13 (+3.98)      |
| **HumanEval+**   | GPT4              | 70.52              | 75.48              | 79.54              |
|                  | GPT-3.5-Turbo     | 58.75              | 66.58              | 73.96              |
|                  | MPSC-Lexical      | **74.39 (+15.64)** | 75.00 (+8.42)      | 77.24 (+3.28)      |
|                  | MPSC-Label        | 73.47 (+14.72)     | **76.66 (+10.08)** | 77.25 (+3.29)      |
| **MBPP**         | GPT4              | 71.26              | 74.27              | 76.99              |
|                  | GPT-3.5-Turbo     | 66.80              | 72.34              | 76.60              |
|                  | **MPSC-Semantic** | **73.23 (+6.43)**  | 73.29 (+0.95)      | 73.55 (-3.05)      |
| **CodeContests** | GPT4              | 6.1                | 8.28               | 11.72              |
|                  | GPT-3.5-Turbo     | 2.57               | 4.22               | 7.16               |
|                  | MPSC-Semantic     | 10.09 (+7.52)      | 10.29 (+6.07)      | 10.30 (+3.14)      |
|                  | **MPSC-Label**    | **11.94 (+9.37)**  | **15.55 (+11.33)** | **18.20 (+11.04)** |

The experimental results show that MPSC consistently enhances the code generation capabilities of the foundation model (i.e. GPT-3.5-Turbo) across all benchmarks with a remarkable margin of improvement. Particularly, when `latex $k$ ` is set to 1, which is the most prevalent scenario in real-world applications, the performance improvement is notably significant (+15.91% on HumanEval, +15.64% on HumanEval+, +6.43% on MBPP and +9.37% on CodeContests). With the foundation model GPT-3.5-Turbo, our MPSC can even outperform GPT-4 in Pass@1 across all benchmarks. Compared to other post-hoc augmentation approaches, even though they utilize more test cases, our MPSC still shows consistent advantages in all benchmarks, excluding the Pass@5 score in MBPP benchmark.

MPSC-Uniform serves as the bottom line of MPSC framework and still achieves great gains for the foundation model, which demonstrates that relying on inter-consistency proves to be entirely effective. Moreover, incorporating various types of intra-consistency information leads to further improvements. Specifically, MPSC-Label and MPSC-Semantic exhibit particularly strong results. They are two representative approaches leveraging the external supervision signals or the internal consistency information respectively. Surprisingly, MPSC-Semantic can match or even surpass MPSC-Label in some benchmarks, which further highlights the significance of consistency information in LLMs.

### Further Analysis

#### Ablation study

We conduct an ablation study to examine the impact of different perspectives on MPSC.

| Benchmark    | Ours  | w/o Specification | w/o Test case | w/o Both |
| ------------ | ----- | ----------------- | ------------- | -------- |
| HumanEval    | 83.38 | 82.32             | 78.30         | 68.38    |
| HumanEval+   | 73.54 | 73.52             | 68.49         | 58.75    |
| MBPP         | 73.23 | 70.18             | 72.00         | 66.80    |
| CodeContests | 10.09 | 9.17              | 8.71          | 2.57     |

Evidently, both the specification and test case perspectives play crucial roles in our framework. Additionally, test cases contribute more to the improvements than specifications. We attribute the observation to the better quality of test cases, as generating an accurate test case is considerably simpler than abstracting a comprehensive and sound specification.

#### Qualities of three perspectives

| Perspective | Solution | Specification | Test Case |
| ----------- | -------- | ------------- | --------- |
| HumanEval   | 68.38    | 45.93         | 63.82     |
| MBPP        | 66.80    | 53.70         | 34.64     |

The quality of all three perspectives is insufficient individually. Indeed, the generated verification properties (i.e. specifications and test cases) are even poorer than the generated solutions. It implies that prior works [zhang2023ALGO; chen2022CodeT] relying on generated verification properties as experts for majority voting on solutions may fail, as these experts perform worse than the choices (i.e. solutions) themselves. Additionally, it indicates that the significant improvements brought by MPSC do not solely depend on the high quality of verification properties. The improvements come from the consistency information within LLMs, which helps to distinguish noise from high-quality solutions.

#### Generalization over different LLMs

MPSC is designed as a model-agnostic framework that assumes black-box access to the underlying foundation model. Experimental results show that MPSC consistently yields significant improvements across all models including GPT-4, Code Llama, WizardCoder, and DeepSeek Coder, demonstrating robust generalization capabilities.

**Table 2: MPSC-Semantic with different foundation models on HumanEval/HumanEval+**

| Model               | Pass@1             | Pass@2             | Pass@5            |
| ------------------- | ------------------ | ------------------ | ----------------- |
| GPT-4               | 81.48              | 86.31              | 90.46             |
| +MPSC               | **92.15 (+10.67)** | **91.62 (+5.31)**  | **91.80 (+1.34)** |
| Code Llama-34B      | 51.78              | 59.24              | 67.07             |
| +MPSC               | **70.97 (+19.19)** | **70.55 (+11.31)** | **71.36 (+4.29)** |
| WizardCoder-13B     | 60.35              | 66.10              | 72.01             |
| +MPSC               | **73.60 (+13.25)** | **74.96 (+8.86)**  | **74.57 (+2.56)** |
| Code Llama-7B       | 39.38              | 45.18              | 52.79             |
| +MPSC               | **58.54 (+19.16)** | **57.83 (+12.65)** | **59.31 (+6.52)** |
| Deepseek Coder-6.7B | 71.73              | 80.92              | 86.73             |
| +MPSC               | **82.38 (+10.65)** | **83.92 (+3.00)**  | 84.71 (-2.02)     |

#### Impact of edge sparsity

Our framework significantly depends on the inter-consistency information between model outputs, which is represented as edges within the constructed graph. The results clearly demonstrate a positive correlation between the number of edges and the overall performance of our framework.

[IMAGE: analysis2.pdf - The correlation between the performance of MPSC and the edge density.]

#### Impact of sampling number

MPSC constantly brings significant performance gains with varied specifications and test cases. MPSC generally suffers a slight degradation in performance when fewer specifications or test cases are used, which is consistent with our intuition. But the performance decline is relatively small (about 4% with only 10% of specifications and test cases retained). The observation indicates the remarkable performance and efficiency of MPSC, suggesting the potential for real-world application with reduced computational costs.

## Conclusion

In this paper, we present a novel code generation method, Multi-Perspective Self-Consistency (MPSC), aimed at enhancing the performance of LLMs in complex code generation tasks where a single attempt may not suffice to ensure the accuracy of the output. Our proposed MPSC strategy capitalizes on both intra- and inter-consistency across three perspectives, solutions, specifications and test cases, to identify the most reliable answer. We systematically validate the effectiveness of MPSC through comprehensive experiments conducted on four widely used code generation benchmarks. The evaluation results demonstrate that MPSC outperforms existing methods and achieves the state-of-the-art performance on all of them.

## Limitations

#### Evaluation in the wild

Even though MPSC has shown remarkable performance on most widely used code generation benchmarks, its effectiveness in real-world scenarios remains largely unexplored. Existing code generation benchmarks often present simplified code generation tasks compared to the intricacies encountered in actual code developments, where user intents are harder to understand and the desired functionalities are more complex.

#### Generalization to other tasks

Since our proposed MPSC framework is designed to be model-agnostic and task-agnostic. We only conduct experiments in the code generation task in this paper. Actually, MPSC can be applied to other textual generation tasks like math problem solving and question answering. However, unlike code generation, where code interpreter can measure the agreement between outputs in a deterministic way, assessing the agreement between natural language outputs is non-trivial. A general task-agnostic inter-consistency measure is to solely rely on LLMs, whose evaluation ability for arbitrary textual inputs has been demonstrated recently. We leave it for future works to discuss.

---

## Appendix

### Details of the Iterative Algorithm

**Algorithm: Iterative Optimization**

**Input:** degree matrix `latex $\mathbf{D}=\text{diag}(d_1,...,d_N)$ `, initialization score vector `latex $\mathbf{y}$ `, weighted adjacency matrix `latex $\mathbf{W}$ `, threshold `latex $\epsilon$ `

**Output:** optimal confidence score vector `latex $\mathbf{f}^*$ `

```
f^(0) <- y
T <- D^(-1/2) W D^(-1/2)
i <- 0
do:
    f^(i+1) <- alpha * T * f^(i) + (1-alpha) * y
    i <- i + 1
while ||f^(i) - f^(i-1)|| > epsilon
f* <- f^(i)
return f*
```

#### Proof of Convergence

We first expand the expression of `latex $\mathbf{f}^{(n)}$ ` according to the recursive formula:

```latex
$$\mathbf{f}^{(n)} = \alpha\mathbf{T}\mathbf{f}^{(n-1)} + (1-\alpha)\mathbf{y} = (\alpha\mathbf{T})^{n-1}\mathbf{f}^{(0)} + (1-\alpha)\sum_{i=0}^{n-1}(\alpha\mathbf{T})^{i}\mathbf{y}$$
```

Notice that `latex $\mathbf{T}$ ` is similar to a stochastic matrix `latex $\mathbf{W}\mathbf{D}^{-1}=\mathbf{D}^{\frac{1}{2}}(\mathbf{D}^{-\frac{1}{2}}\mathbf{W}\mathbf{D}^{-\frac{1}{2}})\mathbf{D}^{-\frac{1}{2}}=\mathbf{D}^{\frac{1}{2}}\mathbf{T}\mathbf{D}^{-\frac{1}{2}}$ `. Therefore the eigenvalues of `latex $\alpha\mathbf{T}$ ` are in `latex $[-\alpha,\alpha]$ `. With `latex $\alpha\in(0,1)$ `, we have:

```latex
$$\lim_{n\rightarrow\infty}(\alpha\mathbf{T})^n=0$$
$$\lim_{n\rightarrow\infty}\sum_{i=0}^{n-1}(\alpha\mathbf{T})^{i}=(\mathbf{I}-\alpha\mathbf{T})^{-1}$$
```

Therefore:

```latex
$$\mathbf{f}^* = \lim_{n\rightarrow\infty}f^{(n)} = (1-\alpha)(1-\alpha\mathbf{T})^{-1}\mathbf{y}$$
```

### Experiment Settings and Baselines

We incorporate various baselines in code generation:

- **ChatGPT** (gpt-3.5-turbo 0614 version)
- **GPT-4** (gpt4-0614 version)
- **Code Llama-Instruct-34B**
- **WizardCoder-Python-34B**
- **DeepSeekCoder-7B-Instruct**

**Inference hyper-parameters:**

| Parameter         | Value |
| ----------------- | ----- |
| Temperature       | 0.8   |
| Top P             | 0.95  |
| Frequency Penalty | 0     |
| Presence Penalty  | 0     |

MPSC requires an additional hyper-parameter `latex $\alpha$ `, which controls the balance between inter-consistency and intra-consistency in the algorithm. Given that the quality of inter-consistency significantly depends on the edge density of the graph, we utilize the mean edge weight to determine the value of `latex $\alpha$ `. Empirically, we assign a relatively small value of `latex $\alpha$ ` (0.01) when the edges are sparse on the graph, indicated by the mean edge weight less than 0.16. Otherwise, we assign a large value of `latex $\alpha$ ` (0.95) to better leverage inter-consistency.

**Baselines:**

- **CodeT:** This baseline first uses generated test cases to verify each solution by code execution. Then it utilizes RANSAC algorithm to create consensus sets based on execution results. The size of consensus set is then used to rank solutions. We generate 500 test cases for CodeT following the original implementation.

- **Self-Consistency:** If two solutions pass the same set of generated test cases and specifications, we regard them "consistent". Then we take a majority voting to rank solutions.

- **MBR-exec:** This baseline ranks solutions by minimum Bayes risk decoding based on the execution results in the 500 generated test cases.

### Prompts for MPSC

**Prompt for Generating Specifications:**

````
I want you to act as a python programmer. Given a docstring about a python method, you need to write its pre-conditions in one test function "def preconditions(input)" and post-conditions in another test function "def postconditions(input, output):". You should ensure invalid input or output of the method will raise error in the two test functions.

```Python
{Demonstration Docstrings 1}
    pass
#Pre-conditions
{Demonstration Preconditions 1}
#Post-conditions
{Demonstration Postconditions 1}
````

```Python
{Demonstration Docstrings 2}
    pass
#Pre-conditions
{Demonstration Preconditions 2}
#Post-conditions
{Demonstration Postconditions 2}
```

```Python
{Docstrings}
    pass
```

**Prompt for Generating Solutions:**

````
I want you to act like a Python programmer. I will give you the declaration of a function and comments about its property. You need to implement the body of the function in the code block. Do not modify any code I provide. Do not provide any explanations.

Here is the question.
```Python
{Docstrings}
````

```

**Prompt for Generating Test Cases:**

```

```Python
# Given a docstring, continue to write the following code with 10 valid assertion statements to check the correctness of the function. Provide diverse test cases.
{Docstrings}
    pass

# check the correctness of with 10 different valid assertion statements in the form of "assert {entry point}(...) == ..."
assert
```
