# Abstract

While Large language models (LLMs) have the capability to iteratively reflect on their own outputs, recent studies have observed their struggles with knowledge-rich problems without access to external resources. In addition to the inefficiency of LLMs in self-assessment, we also observe that LLMs struggle to revisit their predictions despite receiving explicit negative feedback. Therefore, We propose `Mirror`, a **M**ult**i**ple-pe**r**spective self-**r**eflection method for kn**o**wledge-rich **r**easoning, to avoid getting stuck at a particular reflection iteration. `Mirror` enables LLMs to reflect from multiple-perspective clues, achieved through a heuristic interaction between a Navigator and a Reasoner. It guides agents toward diverse yet plausibly reliable reasoning trajectory without access to ground truth by encouraging (1) diversity of directions generated by Navigator and (2) agreement among strategically induced perturbations in responses generated by the Reasoner. The experiments on five reasoning datasets demonstrate that `Mirror`'s superiority over several contemporary self-reflection approaches. Additionally, the ablation study studies clearly indicate that our strategies alleviate the aforementioned challenges. The code is released at https://github.com/hanqi-qi/Mirror.git.

# Introduction

Large Language Models (LLMs) have become an important and flexible building block in a variety of tasks. They can be further improved by iterative correction in many tasks [DBLP:journals/corr/abs-2303-17651; DBLP:journals/corr/abs-2305-11738; Shinn2023ReflexionLA; Pan2023AutomaticallyCL], such as code generation, arithmetic problem solving and reasoning. During iterative refinement, the critic module, which assesses the current response and generates valuable feedback, is crucial to drive performance improvement.

Some research shows that LLMs have self-assessment abilities [DBLP:journals/corr/abs-2303-08896; DBLP:journals/corr/abs-2303-17651]. For example, LLMs can reject its own prediction and generate a response _'I don't know'_ when they are not confident about their predictions [DBLP:journals/corr/abs-2207-05221]. Empirical observations demonstrate LLMs' competence in various reasoning tasks, leading to the utilization of advanced LLMs to evaluate the predictions made by other models [DBLP:journals/corr/abs-2305-14992; DBLP:journals/corr/abs-2310-04406; liu2023training]. However, recent studies suggest that relying directly on LLMs' judgements is not trustworthy and can lead to failures in knowledge-rich iterative reasoning [DBLP:journals/corr/abs-2310-01798]. To guide LLMs through a reasoning loop, existing solutions either incorporate external resources to verify LLMs' outputs [DBLP:journals/corr/abs-2302-12813; DBLP:conf/iclr/YaoZYDSN023], or train a critic module on labelled assessment datasets [DBLP:journals/corr/abs-2305-11738; zelikman2022star]. Furthermore, self-consistency is considered a robust unsupervised method to identify confident and reliable LLM outputs.

In self-refinement, the quality of generated feedback also plays a pivotal role. The Self-Refine method [DBLP:journals/corr/abs-2303-17651] introduced task-specific metrics for multifaceted feedback generation, requiring LLMs to evaluate their outputs across various aspects, such as _fluency, engagement_, and _relevance_ for the dialogue generation task. This process often heavily relies on human expertise, and generating effective feedback for reasoning tasks can be even more difficult as it is obscure to define the essential attributes for different problems. Providing overly general feedback fails to guide LLMs toward generating better outputs in subsequent iterations.

[IMAGE: Figure 1 - Without ground truth for validating LLM-generated outputs, LLMs struggle to consistently improve their own outputs due to their incapability of self-assessment. Autostop and Neverstop provide different generic feedback without leaking the correctness of the current response.]

The inefficiency of self-assessment and feedback generation capabilities largely hinders the performance of iterative refinements. On one hand, as depicted in Figure 1, it is evident that in the absence of a ground truth reference, LLMs fail to consistently improve their predictions, indicating their limitations in self-assessment. On the other hand, even when ground truth labels are available, LLMs often fail to adhere to instructions for revising their incorrect predictions, as shown in Figure 2. Each bar represents the number (averaged over 5 iterations) of revised (blue) and unchanged samples (grey) among the incorrectly predicted samples. It is undesirable to see that a large number of incorrect predictions stay unchanged, suggesting that LLMs can become trapped in a reasoning loop.

To address the aforementioned limitations and generate high-quality feedback without relying on human experts, we propose a novel framework, refer to as `Mirror` (**M**ult**i**ple-pe**r**spective self-**r**eflection method for kn**o**wledge-rich **r**easoning). `Mirror` enables LLMs to reflect from multiple-perspective clues and this is achieved in a heuristic manner between a Navigator and a Reasoner, resembling a typical human tutoring process. For example, when tackling a complex scientific problem, the Navigator generates clues of key elements and rationales behind posing the question, which are crucial in focusing the response on the essential aspects. This information, tailored to the question, serve as instructions for prompting the Reasoner to adjust their predictions accordingly and avoid getting stuck at a particular stage.

To initiate the unsupervised self-reflection properly and avoid being trapped in one reasoning loop, `Mirror` integrates an intrinsically motivated planning algorithm to search for the optimal reasoning trajectory. Inspired by the findings in Section 3.1 and Section 3.2, we propose to reward both the diversity of generated directions and the agreement among strategically induced perturbations in responses. Notably differing from existing tree-based planning methods for reasoning [DBLP:journals/corr/abs-2305-14992; DBLP:journals/corr/abs-2310-04406], `Mirror` avoids deteriorated searching space by encouraging diverse generative outcomes from LLMs at each reflection step, and enhances the self-assessment ability by considering the agreements among multiple-perspective perturbations strategically induced in responses. We evaluate the performance of `Mirror` on two categories of reasoning tasks: MMLU [hendryckstest2021], a knowledge-rich question-answering dataset, and FEVER [Thorne18Fever], a fact-checking dataset. `Mirror` achieves a significant average improvement of over 15% compared to recent popular unsupervised self-refinement methods. The empirical observations demonstrate that the proposed diversity-based reward and answer assessment strategy serve as reliable sources for performance enhancement.

# Related Work

#### Self-Reflection LLMs

Extensive research [honovich-etal-2022-true-evaluating; Xie2023SelfEvaluationGB] has been conducted to enhance LLMs through the concept of self-reflection, where LLMs learn from automatically generated feedback to understand and reflect on their own outputs. This feedback can stem from various sources: the LLM itself [DBLP:journals/corr/abs-2303-17651; Shinn2023ReflexionLA], a separately trained critic module [Gou2023CRITICLL; DBLP:journals/corr/abs-2302-12813] or external sources [DBLP:conf/iclr/YaoZYDSN023], such as Wikipedia or an Internet Browser. Gou2023CRITICLL [DBLP:journals/corr/abs-2302-12813] argued that evaluators trained on task-oriented feedback offer superior performance. For example, Refiner [Paul2023REFINERRF] took context and hypotheses as input to generate templates-based feedback for various error types. Recent studies [DBLP:journals/corr/abs-2302-12813; Shinn2023ReflexionLA; DBLP:journals/corr/abs-2305-14992] have fully utilized the in-context learning capability of LLMs, prompting them to generate high-quality feedback based on their previous generation or potential templates. DBLP:journals/corr/abs-2303-17651 proposed multiple task-oriented metrics and prompted LLMs to evaluate their own outputs based on these criteria. Similarly, DBLP:journals/corr/abs-2302-12813 [Glaese2022ImprovingAO] adopted external tools to predict multi-facet human preference scores. Our solution aligns with this trend by aiming to provide informative and customized instructions tailored to the specific task and query. Moreover, it seeks to achieve this without relying on human intervention or external tools, thereby rendering self-refinement more feasible in practice.

#### Reasoning models augmented with tree search

Recently, tree-based reasoning has attracted significant attention, such as Tree-of-Thought (ToT) [Yao2023TreeOT], Grace [Khalifa2023GRACEDC], and SelfEval-Decoding [Xie2023SelfEvaluationGB]. At each reasoning step, ToT adopts breadth-first search and depth-first search, while the latter two methods select the top-k scoring candidates during the decoding process. Moreover, Monte-Carlo Tree Search (MCTS) is one of the popular search algorithms [DBLP:journals/air/SwiechowskiGSM23], which strikes a balance between exploitation and exploration. Some existing approaches establish a reinforcement learning framework to maximize reward through learning optimal actions/states [DBLP:journals/corr/abs-2302-00111; DBLP:journals/corr/abs-2305-16209; zhu-etal-2023-solving]. Other studies fully utilize the capability of LLMs for interaction and feedback generation. For instance, RAP [DBLP:journals/corr/abs-2305-14992] leveraged step-wise rewards from interactions with the world model to decompose and solve the problem step-by-step, rather than a iterative manner. LATS [DBLP:journals/corr/abs-2310-04406] was the first work in leveraging MCTS for self-reflection. However, their feedback contains information from comparisons with ground truth, which is not applicable in our case. Instead, our approach, `Mirror` has no access to gold labels, and we incorporate a novel diversity reward to avoid the inefficient search in the reflection iteration.

# Lost in the Reasoning Loop

Given the observed challenges in enhancing LLMs' self-improvement without ground truth labels, particularly in knowledge-rich reasoning tasks, our initial experiment aims to address these challenges by breaking them down into two sub-questions.

- Q1: _To what extent can LLMs assess the correctness of a statement?_ This investigation involves enhancing their capabilities through supervised training. The primary goal is to discern if there are viable solutions to enhance the verification ability of LLMs on knowledge-rich statements.

- Q2: _How well can LLMs generate high-quality feedback to guide their own subsequent response update?_ It is especially challenging when the feedback generation models are not trained on high-quality data, relying solely on the in-context learning capability of LLMs.

## LLMs in Knowledge Grounding

We experiment with the multiple-choice dataset, MMLU [hendryckstest2021], covering 57 subjects across STEM, Humanity, Social and other domains. To evaluate the ability of LLMs in assessing the knowledge-rich statements, we construct the positive and negative statements by substituting the question with the correct choice and a randomly selected choice from the other three incorrect choices, respectively. Table [tab:assess_correctness] presents the assessment accuracy of assessing. There are three categories of methods: in-context learning, fine-tuned on statements, and classification based on intermediate activations from LLMs.

As illustrated in the first group results in Table [tab:acc_consistency], an increase in accuracy is observed as the size of Llama-2-13B-chat increases. Notably, GPT-3.5 with 175B parameters consistently achieves the best results across the three domains, although the improvement is not directly proportional to the parameter size. We then apply advanced prompting techniques, i.e., UniLangCheck [Zhang2023InterpretableUL] on the best-performing method, GPT-3.5. Our analysis reveals that the improvements are predominantly driven by self-consistency, while UniLangCheck does not consistently contribute to improvement in grounding. For UniLangCheck, we firstly prompt LLMs to generate a fact about the key elements in a question before making the final assessment. It can be partially explained by the accumulation error, i.e., the inaccurate facts generated by LLMs before reaching the final conclusion can affect the outcome. We also calculate the correlation between accuracy and self-consistency, represented by the probability of generating a single answer through multiple repeated prompting. The average correlation R^2 for questions in the MMLU datasets across three LLMs is about 0.85, indicating that self-consistency can be relied upon as a proxy for assessment.

We also evaluate the performance of some _supervised methods_ (denoted with \* in Table [tab:assess_correctness]). TRUE [honovich-etal-2022-true-evaluating] involves fine-tuning a T5 [JMLR:v21:20-074] model on a collection of natural language inference (NLI) datasets for fact-checking. We further fine-tune its classifier head on our training set. ActivationRegress [Marks2023TheGO] trains classifiers using activations extracted from Llama2-13B 12-layer encodings as inputs. ContrastSearch [DBLP:conf/iclr/BurnsYKS23] is trained using contrastive and consistency loss while having no access to the factual labels. This is achieved by constructing data pairs that include both a positive-labeled and negative-labeled statements, irrespective of the true factual labels. It is surprising that both TRUE and ActivationRegress are inferior than the unsupervised ContrastSearch.

[IMAGE: Figure 2 - The average number (across all iterations) of changed and unchanged samples among those predicted incorrectly. Large percentage of unchanged samples indicate the limited capability for efficient reflection.]

## LLMs in Feedback Generation

Evaluating the quality of generated feedback poses a significant challenge, particularly when such feedback is utilized across diverse tasks [DBLP:journals/corr/abs-2303-17651]. Drawing inspiration from the pivotal role of feedback in the self-improvement, we propose to leverage the performance of LLMs in subsequent iterations for evaluation. Specifically, LLMs can access to ground truth, enabling them to evaluate the correctness of their current responses. This information is then integrated into feedback generation. Consequently, we assess the quality of feedback by examining the percentage of examples that are incorrectly answered, along with the percentage of instances where responses in the next round are revised for the same incorrectly answered examples. This comparison sheds light on the effectiveness of instructions in guiding LLMs to rectify their erroneous responses. Firstly, we follow the settings in [Shinn2023ReflexionLA] to incorporate the assessment results in the feedback: `"Observation: The answer is incorrect."` is inserted after presenting the question and previous attempt, and the LLMs are required to generate refection and response to this question again. From the results in Figure 2, it is consistently observed across different model scales that LLMs struggle to update their predictions despite receiving explicit negative feedback. The average percentage of successfully updated examples for GPT-3.5, Llama, and Vicuna are 65.6%, 51.79% and 74.09%, respectively, indicating an ample room for improvement.

Motivated by the following two observations: (1) LLMs are particularly susceptible to context influence at the beginning or near the end [DBLP:journals/corr/abs-2307-03172], (2) In-Context Learning is highly sensitive to stylistic and emotional words in demonstrations [DBLP:conf/emnlp/MinLHALHZ22; li2023emotionprompt], we develop three prompting strategies for feedback generation. An incorrectly predicted example with different prompting strategies is shown in Figure 9. The results in Table [tab:ans_changes_after] and Table [tab:follow_inst] suggest that based on correct question assessment, enhancing the exploration capability within a diverse answer space could lead to higher accuracy in answering knowledge-rich questions.

The above empirical findings regarding the two research questions provide valuable insights for our proposed model, named `Mirror`. Distinguishing itself from existing self-improvement methods, `Mirror` makes two significant contributions: (1) it features a Navigator module for generating multiple question-adaptive directions, with diversity constraints implemented to prevent invalid reflections. (2) it relies on the consistency of the inherent multiple perspectives for boosted self-assessment.

# The Framework of Mirror

In this section, we introduce our unsupervised self-reflection framework, `Mirror`, depicted in Figure 3. The reward R consists of Diversity and Consistency terms. Diversity is applied to prevent reflection from becoming stuck and to facilitate intra-consistency involved in the stop criteria for self-assessment. The Consistency reward also influences direction generation.

[IMAGE: Figure 3 - An overview of Mirror. It facilitates diverse question-specific directions (represented by different colored dots in the action space) to encourage extensive reflection by the Reasoner. The stopping criterion is based on the consistency among states from multiple perspectives, which also contributes to the direction generation.]

## Problem Setup

Given a question, the Reasoner is to arrive at the final answer through interacting with a Navigator. We consider a Markov Decision Process (MDP) defined by a tuple `latex $(S, A, P, \pi, \gamma, R)$ `, where the `latex $s_t \in S$ ` and `latex $a_t \in A$ ` denote the state and action, respectively in the t-th reflection iteration. In the context of multiple-choice question, `latex $a_t$ ` is the direction generated by the Navigator, and `latex $s_t$ ` is the response generated by the Reasoner, including the answer to the question and the rationale behind. `latex $R(s,a)$ ` is the reward function. Therefore, we have state transition distribution `latex $P(s_t|s_{t-1},a_{t-1})$ ` and action generation distribution `latex $\pi(a_t|s_{t-1},a_{t-1},q,p_0,R)$ `, where `latex $p_0$ ` is the prompt for the Navigator to generate direction `latex $a_t$ `. It is nontrivial to obtain frequent rewards that incentivize self-refinement progress without access to the ground truth. Therefore, we turn to an intrinsically motivated planning algorithm, i.e., Monte-Carlo Tree Search (MCTS) [kocsis2006bandit; 6145622; DBLP:journals/air/SwiechowskiGSM23] to efficiently explore the environment augmenting rewards with auxiliary objectives [mu2022improving; DBLP:conf/icml/DuWWCDA0A23].

Comparing to existing work search-based reasoning methods based on frozen LLMs [DBLP:journals/corr/abs-2305-14992; DBLP:journals/corr/abs-2310-04406], we highlight two notable contributions addressing the vulnerabilities of LLMs as discussed in Section 3: (1) _Step-wise Multiple-perspective self-assessment:_ unlike approaches that rely on ground truth or majority-voting based on several complete generated trajectories, our framework utilizes multiple-perspective consistency as stop criteria at each step t. (2) _Novel Reward Mechanism:_ a novel diversity mechanism is designed to avoid the null space encountered in traditional random search settings. Our method is detailed in Algorithm [alg:cap] in the Appendix.

## Multiple-perspective Assessment

Motivated by the empirical results in Section 3.1 regarding knowledge-grounding, we propose to employ an advanced consistency-based method as a surrogate for factual correctness when external resources are unavailable. This method considers both intra- and inter-consistency of the generated responses. Specially, we employ the Navigator for K question-oriented direction generation, `latex $a \sim \pi(a_t|s_{t-1},a_{t-1},q,p_0,R)$ `. These K directions are intended to provide diverse perspectives for problem-solving, with the agreement among guided responses representing inter-consistency. Meanwhile, the confidence in self-consistency [DBLP:conf/iclr/0002WSLCNCZ23] serves as the measure of intra-consistency.

To integrate consistency considerations into the assessment per reflection iteration, we use **intra-consistency** to determine whether the Reasoner should accept its initial response. If the intra-consistency for our initial answer surpasses a threshold `latex $T_0$ `, we consider it as the final result; otherwise, we integrate the **inter-consistency** as an indicator for stopping criteria in subsequent reflection iterations. We derive the final answer when the inter-consistency exceeds `latex $T_0$ ` or when reach the predefined maximum iterations, selecting the final answer with the highest consistency score. This inter-consistency also becomes part of reward `latex $R_{consistency}$ ` for the current state and contribute to the direction generation. Besides, the intra-consistency value is transformed into verbal form, becoming part of the prompt `latex $p_0$ ` given for Navigator to generate direction. This is inspired by our observation that higher intra-consistency implies a higher likelihood of correctness, so we offer this additional information to assist in feedback generation. This is similar to [DBLP:journals/corr/abs-2305-08848], where ICL performance benefits from accessing to the prediction of a supervised fine-tuned smaller model. Our assessment method is different from majority vote, which treats every node with the same weight when aggregating the final result. The comparison results are shown in Table [tab:answer_asses].

## Diverse and Valid Search Space

Obtaining a meaningful and diverse action space is challenging due to the absence of a dense and well-defined reward function in the planning algorithm. One of the predominant reasons is that different action sequences can lead to similar outcomes [DBLP:journals/ras/BaranesO13]. In our context, considering the limitation of LLMs in following instructions, the Reasoner may ignore the differences among multiple directions and generate identical responses merely based on the question. Therefore, some intrinsically motivated reinforcement learning algorithms choose to explore outcomes rather than actions [oudeyer2007intrinsic; ladosz2022exploration]. MCTS addresses the limitation of sparse rewards by visiting novel states or transitions through random exploration [DBLP:conf/icml/DuWWCDA0A23]. The most popular algorithm in the MCTS family, Upper Confidence Bound for Trees (UCT) [kocsis2006bandit] is treated as the choice of child node:

```latex
$\text{UCT} = \overline{R}_j + 2C_p\sqrt{\frac{2 \ln N(n)}{N(n_j)}}$
```

where `latex $\overline{R}_j$ ` is the average reward for child node j, while the second term encourages sampling from nodes whose children are less visited. `latex $N(n)$ ` is the number of times current node (parent) has been visited in previous iterations, and `latex $N(n_j)$ ` is times of the child node has been visited. The `latex $C_p > 0$ ` is a constant to control balance between exploitation (first term) and exploration (second term). In our case, we specifically promote diversity between the parent and child node, i.e., the response in previous attempt `latex $s_{t-1}$ ` and the current attempt `latex $s_t$ `. For multiple-choice questions in MMLU, we assess if the predicted choices are the same across two reflection iterations. The discrepancy in responses indicates the alleviation of null direction space and the avoidance of being stuck, especially given the relatively low consistency with the response from the previous iteration. The relationship between task performance and the diversity of responses in the generated tree, as illustrated in Figure 5, confirms our motivation for diversity enhancement.

However, maximizing diversity of outcomes may not always be enough, as less relevant states might be collected [DBLP:conf/icml/DuWWCDA0A23]. Therefore, to ensure valid search space, we filter out states whose associated responses are not in the correct form, such as failing to provide a final choice, or refusing to answer questions for moral considerations. For search efficiency, our proposed stopping criteria is to terminate the search process once its inter-consistency surpasses a threshold, thereby avoiding unnecessary search and expansion costs.

# Can Mirror Steer LLMs in Iterative Improvements?

We evaluate our proposed `Mirror` on MMLU and FEVER [Thorne18Fever]. FEVER is a fact-checking dataset featuring three labels for knowledge-rich statements, i.e., `supports`, `refutes` and `not enough info`.

## Experimental Setup and Results

#### Comparison methods

The evaluation models are GPT-3.5, Llama2-13B-Chat [Touvron2023Llama2O], and Vicuna-v1.5-7B [Zheng2023JudgingLW]. We equip the LLMs with different reasoning mechanisms, including Chain-of-Thought (CoT) [wei2022chain], Self-consistency [DBLP:conf/iclr/0002WSLCNCZ23], Self-Correction [DBLP:journals/corr/abs-2310-01798] and Reflexion(w.GT) [Shinn2023ReflexionLA]. We implement CoT by prompting LLMs to first generate step-by-step thoughts and then generate answers based on those thoughts. We repeat this process for five times, resulting in Self-Consistency^(5). The remaining two methods are self-improvement techniques where LLMs are first prompted to generate reflections, followed by updating their current response accordingly if applicable. Self-Correction relies on LLM's internal knowledge for answer assessment, while Reflexion compares the current answer with the ground truth for evaluation.

[IMAGE: Figure 4 - Reasoning process of self-correction and Mirror. Text in red are generated directions. Our diversity is characterised in (i) generating directions tailored to questions (ii) encouraging exploration in multiple plausible reasoning paths. The final answer is derived through an agreement among multiple trajectories.]

#### Results

The results are shown in Table [tab:main_results]. By comparing CoT with Self-Correction, we observe the performance degradation after two rounds of self-Correction across almost all datasets and models. This observation aligns with our findings in Section 3.1 and in [DBLP:journals/corr/abs-2310-01798]. Equipped with self-consistency^(5), significant performance improvements are evident across all settings. `Mirror` considers additional inter-consistency, achieves the most notable improvements, with a relative increase of more than 15% across the three models. Figure 4 illustrates the reasoning process of Self-correction and `Mirror`. Both methods fail to answer correctly in the first trial. With question-oriented direction, the Reasoner better identify errors in the initial response, such as, _the error in score direction_ and _inconsistency between rationales and selection_. The consistency-based criteria built in the tree further improves the fact assessment. During backpropagation, node s^(1)\_1 receives a higher reward, leading to the leftmost reasoning path (details of direction a^(1)\_1, a^(1)\_2, a^(2)\_2 and corresponding responses are shown in the text frame). By contrast, Self-correction seems to engage in groundless inference by switching answers without explicit clues. Even comparing `Mirror` with Relexion(w.GT), we find comparable results for GPT-3.5 on the STEM dataset, for Llama on all datasets except for STEM and for Vicuna on STEM and Humanity. From the perspective of the model, the average improvements over baselines for GPT-3.5 are particularly prominent, partly explained by its better ability to adhere to provided directions. This can also explain the marginal improvements even ground truth are accessible to the smaller models.

## Analysis

We discuss the effects of key strategies in `Mirror`.

#### Question-Oriented Direction

Motivated by the findings in Section 3.2 that LLMs struggle to effectively reflect on themselves with generic feedback, `Mirror` is equipped with a Navigator for generating question-oriented directions. To study the effects of these directions (results in Table [tab:direction_ablation]), we adopt our Navigator for direction generation for CoT settings, in which the direction (GenerativeDirect) is introduced before the LLM generates its thought on the previous trial. We then replace all adaptive directions with a single generic direction (FixedDirect) which reads: `Read the question and choices carefully and diagnose the previous response by locating the incorrect clues and update the response if applicable.` Comparing with CoT, the inclusion of GenerativeDirect boosts the performance across all settings with significant improvements. Conversely, FixedDirect sometimes results in performance degradation for Llama13B. The impact of FixedDirect is similar to advanced instruction intended to provide general direction for the task, whereas GenerativeDirect offers question-specific advice to accurately summarize clues for solution. Referencing to the example in Figure 10, `Mirror` (bottom) firstly prompts the Navigator for direction generation (highlighted in red), which captures the key elements, such as _"the characteristics of a connected and undirected graph"_. The Reasoner then follows this direction to explain the key concepts of this graph, laying a solid foundation for reaching the correct conclusion. Without such direction, the Reasoner may overlook or misinterpret knowledge about this graph, leading to errors in the conclusion.

#### Diversity of the Search Space

We demonstrate the impact of multiple-perspective directions, aiming at guiding the Reasoner out of reflection traps. To this end, we compute the percentage of generated trajectories containing the correct answers (ans*presence) and the according task performances (acc) across various action space sizes, i.e., the number of generated directions. The results in Figure 5 indicate that lager search space enhanced by the ```latex $R*{diversity}$ ``` can increase the probability of reaching the correct answer.

[IMAGE: Figure 5 - The Accuracy (acc) and the percentage of samples where the ground truth is included in the tree (ans-presence), with different sizes of search space (Num). Results for GPT-3.5 and Llama13B are in Figures 11 and 12.]

We provide a case to show the effects of diversity in directions (highlighted difference are in bold). The quantitative results of semantic similarity between multiple directions are shown in Table [tab:diversity].

**Case study:**

**Question**: The price elasticity of demand for a product is greater if

**Answer without Direction**: C. the number of substitute products is limited.

**Answer with Direction**:

**Direction 1**: Pay close attention to nuances among options and prioritize selections that resonate with **economic principles** and the factors influencing consumer behavior.

- **Answer 1**: Given the advice, the most suitable choice would be: _A. the proportion of the good of the consumer's budget is high._ This aligns with **economic principles of demand elasticity**.

**Direction 2**: Prioritize options that align with factors impacting consumer responsiveness to price changes, such as **budget allocation** and **substitute availability**, while considering the time frame for consumer reactions.

- **Answer 2**: Given the advice, the most suitable choice would be: _A. the proportion of the good of the consumer's budget is high._ This choice suggests that when a significant portion of **a consumer's budget** is allocated to a particular product, they are likely to be more sensitive to price changes.

#### Performance of Answer Assessment Criteria

As discussed in Section 3.1, LLMs struggle to assess the correctness of knowledge-rich statements, a capability that can be consistently enhanced through self-consistency. We further reform the majority-voting assessment process by considering the inter-consistency built in the hierarchical decision-making tree. To study the effects of our answer assessment criteria described in Section 4.2, we compare them with two other voting methods, i.e., self-consistency and majority vote within our generated tree-trajectories. We average the results from Table [tab:main_results] for CoT and Self-consistency^(5) across four domains in MMLU and denote them as CoT^(1) and CoT^(5), respectively. We also compare with CoT^(15) because our generated trees have at most 3 layers and 5 branches in each layer, so there are 15 candidate nodes. For Majority^(tree), we select the final answer through majority-voting among all intermediate nodes in our generated tree-trajectories. The results of different final answer assessments are presented in Table [tab:answer_asses]. The performance improvements are observed on self-consistency^(15) over self-consistency^(5), although the improvements percentage isn't pronounced as that seen in the comparison between self-consistency^(5) over CoT. We observe a performance increase after applying majority-voting in the CoT settings, while this simple strategy doesn't yield improvements in the generated tree. This is because undesirable responses may be generated during the node expanding phase, and majority voting treats all nodes equally. In contrast, our reward-based search tends to focus on reliable nodes with higher confidence in each reflection step, thereby avoiding search cost on less desirable nodes.

#### Search Efficiency Analysis

Mirror is a tree-like searching algorithm that benefits from iterative reasoning process, which enhances the reasoning ability at the computation cost. To mitigate the search cost, we (i) incorporate the Monte-Carlo tree search for its selective search and expansion. (ii) introduce early-stop criteria to encourage a shallow and avoid multiple playouts. We summarise the tree-depth in Table [tab:tree_depth]. The results below show that our resulting tree, with a maximum depth of 3, is heavily unbalanced and shallow.

# Conclusion

In this paper, we present a multiple-perspective reflection method, called `Mirror`, for knowledge-enriched reasoning. To tackle the limitations of LLMs in fact assessment and the generation of high-quality feedback, `Mirror` is equipped with a directional Navigator, enabling the Reasoner to identify multiple key clues in problem-solving. Furthermore, the consistency among responses generated under different directions enhances the validity of answer assessment, particularly when ground truth is not accessible. Experiments conducted demonstrate `Mirror`'s superiority over several contemporary CoT-based and self-consistency-based reasoning approaches without access to ground truth. Moreover, the ablation study results clearly show that our strategies effectively alleviate the aforementioned challenges.

# Limitations

In this study, our primary focus is to identify optimal reasoning trajectories based on generated outputs and frozen states. However, the ability to assess facts and generate reflections may be limited by the unaltered decoding process and pre-training. To fully leverage the potential of LLMs in complex reasoning, it is beneficial to explore two directions: (1) Strategically guiding fine-grained generation, such as token-level generation during the decoding phase within the expansive generation space. (2) Fine-tuning LLMs through access to limited task-oriented data to enhance their responses to more complex problems.

Although we can force the model to follow the instructions and change its initial prediction, large percentage of predictions are not changed across different context perturbation. Moreover, in practical setting, we actually do not have access to ground truth as oracle critic to give accurate feedback. In this sense, we would like to explore if the model could check its True/False output in an unsupervised way. One popular way is to identify the activations when feeding the input, recent work has shown that the activations from True and False statements can be mapped into clearly separate spaces.

## Out-of-Distribution Evaluation

# Appendix

## More Experimental Details for Initial Study

### Experiment for Figure 1

The prompt used in _Autostop_ is `''You were either successful or unsuccessful in your previous trial. Stick to your previous answer if it is correct, otherwise consider a new answer''`. The prompt used for _NeverStop_ is `''You failed in your previous trial and reconsider a new answer''`. The motivation behind _Autostop_ is that we totally rely on the LLM's internal knowledge to check the correctness of its own outputs. However, LLM fails in this setting as the performance is even worse than initial stage. For _NeverStop_, we hope to identify that some correctly answered samples will be kept unchanged even the negative feedback provided. However, we didn't find a pattern between the changed and unchanged predicted samples.

### Implementation for Knowledge Grounding and Results

#### Dataset

We evaluate LLMs' knowledge grounding ability on knowledge-rich multiple-choice dataset, MMLU. It consists of four domains: STEM, Social, Humanity and Other, totaling 56 subjects. All methods are evaluated on 50 randomly selected samples for each subject (excluding those in the Other domain), and the remaining samples are used as the training set where applicable.

#### Models and Baselines

In addition to Llama2-13B, Llama2-70B, and GPT-3.5 for prompting, we also leverage unified language checking, _UniLangCheck_ [Zhang2023InterpretableUL], for statement assessment. _UniLangCheck_ aims to check if language input is factual and fair via prompting LLMs to generate groundings for fact-checking. Therefore, we firstly prompt LLMs to generate a fact about the key element in the question before proceeding to the final assessment. We repeatedly prompt the LLMs for 5 times and use the majority-voted answer as the result for _Self-Consistency_ [DBLP:conf/iclr/0002WSLCNCZ23]. _TRUE_ [honovich-etal-2022-true-evaluating] is the T5-11B [JMLR:v21:20-074] model fine-tuned on a collection of natural language inference (NLI) datasets to check factual correctness, and has been used by previous works within similar contexts [gao-etal-2023-rarr; gao2023enabling]. We further fine-tune its classifier head on our training set, which is annotated as factually correct or not, before evaluation. Both Contrastive Consistent Search (_ContrastSearch_) [DBLP:conf/iclr/BurnsYKS23] and _ActivationRegress_ [Marks2023TheGO] train classifiers whose inputs are activations extracted from Llama2-13B 12-layer encodings. _ActivationRegress_ trains a logistic classifier on the activations with factual labels as supervision. _ContrastSearch_, instead, operates without factual labels. For a statement s_i, we firstly construct a data-pair x+ and x- by annotating _True_ and _False_ to this statement, regardless of its factual correctness. Then, we derive the probabilities by mapping x to a number between 0 and 1, i.e., p+ = p_theta(phi(x_i+)) and p- = p_theta(phi(x_i-)). The mapping function p_theta is updated such that the probabilities are both confident (p_i+ approximately equal to 1-p_i-) and consistent (p_i+ not approximately equal to p_i-).

#### Prompt Settings

The basic prompt for knowledge grounding is shown in Figure 6. This is used for Llama2, GPT-3.5 and Self-Consistency. The advanced prompt inspired by _UniLangCheck_ is illustrated in Figure 8. For each subject, we randomly select 50 samples and extract their question and choice to build a statement for knowledge checking. The correctness of this statement is deemed _True_ if the selected choice is exactly the correct one, otherwise it is labeled _False_.

[IMAGE: Figure 6 - Basic prompt for knowledge grounding. Text in gray is extracted from datasets, in red shadow is generated by LLMs.]

[IMAGE: Figure 8 - Fact-extract prompt applied to UniLangCheck for knowledge grounding. Text in gray shadow is extracted from datasets, in red shadow is generated by LLMs. Comparing to the basic prompt, it includes additional fact generation.]

#### Correlation between Self-consistency Confidence and Accuracy

For the self-consistency(5) baseline, we calculate the R^2 for confidence (the frequency of the current answer among all generated answers, totaling 5) and the accuracy. The results are shown in Table [tab:acc_consistency]. We observe a high correlation between the two variables, which inspires our design of multiple-consistency for answer assessment.

### Implementation for Direction Generation

Based on the observation that existing feedback has limited effects to guide LLMs to update their current incorrect response, we propose several simple strategies to enhance the effectiveness of generated feedback in the self-improvement process. These strategies are mainly inspired by the following two observations: (1) LLMs are more susceptible to context influence at the beginning or near the end [DBLP:journals/corr/abs-2307-03172] (2) ICL is highly sensitive to the stylish and emotional words in demonstrations [DBLP:conf/emnlp/MinLHALHZ22; li2023emotionprompt]. We summarize the different strategies in the diagram shown in Figure 9.

[IMAGE: Figure 9 - Given the question and the LLM's previous trial, it is asked to generated feedback under different prompts to facilitate reflection and potentially update its previous response. The four candidate instructions, Baseline, Observation, NewAnswer and NegPrefix, are enclosed in dashed frames, and they will be positioned differently to exert their respective effects.]

We show relative percentage of changed samples those incorrectly predicted ones before and after applying the _NegReflect_ in Table [tab:ans_changes_after]. The percentages have been greatly improved with the instruction which has been inserted closer to the end of prompt. To verify whether this change could lead to task performance, we display the detailed performances over three LLMs after applying different instructions in Table [tab:follow_inst]. It is clear that _NegPrefix_ demonstrates the most significant improvements across all the datasets and models. In contrast, _NewAnswer_ has the same sentences _NegPrefix_ as but its position is far away from the generating point for LLMs. _This can be explained that position of instruction is important in ICL._ And the performance of _NewAnswer_ is slightly better than baseline, it can be partly explained that the _NewAnswer_ explicitly show the negative attitude towards and guide the model to generate a different answer. Among the three models, the average promotion on GPT3.5 is the most negligible. _This can be explained that larger model are more confident with its internal knowledge and less vulnerable to given noisy text._

## Mirror algorithm

We introduce the pipeline of the proposed Mirror in Algorithm [alg:cap] involves iteratively conducting a UCT-Search until predefined iteration constraint is reached, and the best action a(BestChild(v_0,0)) leading to the best child of the root node v_0 returns. Node in the tree is v and its associated state is s(v), representing the response generated by Reasoner. The action is a(v), reward is R and N(.) is the times of the node having been visited. r(v) is the reward for the terminate state at each iteration.

The overall process consists of three steps: (1) SearchPolicy to obtain the terminal node v_l through which expands the tree until fully expanded. Specially, we randomly add one or more nodes to the root node according to the possible actions. In our case, we generate multiple responses to the given question and previous attempts/response. When the current node is fully expanded, we apply the UTC algorithm to select the best child node. (2) Simulation the reward r for v_l through SimulationPolicy. This phrase is to simulate the future rewards of the current node through multiple interactions. For simplicity, we follow the similar process as expansion and return the reward r for selected action-state pair. (3) BackPropagate the simulation results to the selected nodes to accelerate SearchPolicy in next iteration.

```latex
Algorithm: Mirror MCTS

function UCT-SEARCH(s_0)
  create root node v_0 with state s_0
  while within computational iteration do
    v_l <- SearchPolicy(v_0)
    r <- SimulationPolicy(s_{v_l})
    BackPropagate(v_l, r)
  return a(BestChild(v_0, 0))

function SearchPolicy(v)
  while g(v) == 0 do
    if v not fully expanded then
      return Expand(v)
    else
      v <- BestChild(v, C_p)
  return v

function Expand(v)
  choose a in untried actions from A(s(v))
  add a new child v' to v with s(v)' = f(s(v),a) and a(v') = a
  return v'

function BestChild(v, C_p)
  return argmax over v' in children of v: R_{v'}/N(v') + 2*C_p*sqrt(2*ln(N(v))/N(v'))

function SimulationPolicy(s)
  while s is non-terminal do
    a = argmax over a in A: R(a,s)
    s <- f(s,a)
  return reward for s

function BackPropagate(v, r)
  while v is not null do
    N(v) <- N(v) + 1
    R(v) <- R(v) + r(v)
    v <- parent of v
```

## Experiments for Mirror

We will introduce the implementation details and provide complementary results experimented on `Mirror` in this section.

### Implementation Details

#### Hyper-parameter settings

In order to encourage diverse direction generation, we set the generation temperature as 0.8 for all the models, and we set do_Sample=True for llama and vicuna to avoid greedy search. For the threshold T_0 in self-assessment to deriving the final answer, we set 0.8 for GPT35, and 0.5 for llama and Vicuna according to the results on limited validation data. These results reveal that larger language models are more consistent in their multiple outputs, which is more difficult for the smaller models. Hence, we adopt a relatively lower threshold for smaller models. This observation can be partially explained by the tendency of larger LMs to rely on their parametric memory [Xie2023AdaptiveCO].

#### Prompt Settings

We provide 5 demonstrations along with instruction when prompting LLMs. We show the prompts/instructions provided to LLMs in direction generation and response generation process. (a) p*0 in direction generation in pi(a_t|s_t,p_0,R). The guidance in the upper is for initial response, the bottom one is for reflection in the subsequent iterations. (b) Prompt for response generation given previous response and direction. P(s_t|s*{t-1},a\_{t-1};q).

**Prompt for direction generation (MMLU):**

> As a tutor, your focus is on guiding the student to navigate multiple-choice question-answering problems strategically. Encourage them to dissect the question, identifying key elements and nuances within each choice. Emphasize the importance of understanding subtle differences that could distinguish correct from incorrect options. As a tutor, your are supposed to meticulously evaluate the student's approach to multiple-choice problems. Question, Choices and the student's previous thought and answer are given, check if the facts mentioned in the thought is correct and if there might be a more appropriate option than the one chosen. If the student's reasoning thought is accurate and the proposed answer is the most appropriate, encourage them to adhere to their initial trial. Otherwise, guide the student to revisit specific details, explore alternative choice.

**Prompt for direction generation (FEVER):**

> As a tutor, your focus is on guiding the student to navigate fact-checking problems strategically. Encourage them to dissect the claim, identifying key elements and associate facts. Emphasize the correct relation between important elements that could distinguish SUPPORTS from REFUTES options. Also, lacking of enough information will lead to NOT ENOUGH INFO. As a tutor, your are supposed to meticulously evaluate the student's approach to fact verification task. Claim and the student's previous thought and answer are given, check if the relations mentioned in the Thought is correct and if there might be a more appropriate answer. If the student's reasoning thought is accurate and the proposed answer is the most appropriate, encourage them to adhere to their initial trial. Otherwise, guide the student to revisit specific details, explore alternative answer.

**Prompt for response generation (MMLU):**

> You are an expert in multiple-choice question answering. Each problem will provide you with a question and answer choices. Read the question and all the choices carefully, along with the provided advice, and solve the problem by having a thought. Thought can reason about the current situation. Finish[answer] returns the answer and finishes the task. You're an advanced reasoning agent capable of self-reflection and continuous improvement. Your objective is to tackle multiple-choice question answering problems. Each problem will provide you with a question, answer choices, your previous line of reasoning, and the detailed analyses from an experienced tutor. In a succinct review, assess the accuracy of your earlier answer based on your expertise and the advice, subsequently arrive at the definitive response.

**Prompt for response generation (FEVER):**

> You are a knowledgeable and accurate fact verifier. Please verify the correctness of the following claim based on your expertise and provided advice. Return SUPPORTS or REFUTES a Claim, or if there is NOT ENOUGH INFO. You're an advanced reasoning agent capable of self-reflection in fact verification task. Claim and the your previous response and answer are given, along with the advice. In a succinct review, assess the accuracy of your earlier answer based on your expertise and the advice, subsequently arrive at the definitive response.

#### Computational budget

The total running costs for using GPT-3.5 in our experiments are approximately $500. In addition, the running time for Llama2 and Vicuna in our experiments totalled 320 hours, utilising one 40G A100 graphics cards.

### Additional Results

We provide additional results as complementary to our main results.

#### Results on GSM8K dataset

To further verify the effectiveness of Mirror in various reasoning tasks, we include the math problem, i.e., GSM8k [DBLP:journals/corr/abs-2110-14168]. The performance superiority is evident when comparing with the best-performing unsupervised baseline, i.e., self-consistency.

#### Effects of question-oriented direction

To save computational resources, we randomly select 20 samples from each of the four domain datasets in MMLU and from FEVER. We show an example of generated direction in the CoT settings.

[IMAGE: Figure 10 - With question-oriented direction, the Reasoner answers questions with explicit clues.]

#### The diversity of Search Space

One of our motivations is to broaden the diversity of actions available for more effective exploration. Consequently, we compute the upper bound results for our generated tree, indicating the presence of the correct answer in the tree signifies a correctly answered sample. Results are shown in Figure 13.

[IMAGE: Figure 11 - GPT-3.5 results]

[IMAGE: Figure 12 - Llama results]

[IMAGE: Figure 13 - The task performance, Accuracy (acc) and the percentage of samples where the ground truth is included in the tree (ans-presence), with different size of search space (Num).]

To analyse the effects of different LLMs quantitatively, we calculate the average pairwise semantic similarity between multiple directions for one question, then 1-similarity to obtain the diversity measurement shown below. The pretrained model, all-MiniLM-L6-v2 (https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is used for sentence-pair similarity calculation. The results is consistent with the intuition that sophisticated LLMs incline to less diverse instruction although such diverse directions are already capable to improve task performances.

## Ethics Statement

We utilized two publicly available datasets: Massive Multitask Language Understanding (MMLU) and FEVER (Fact Extraction and Verification). MMLU is a multiple-choice question-answering dataset covering 57 subjects across STEM, social sciences, humanities, and more. Notably, some subjects, such as _moral disputes_ and _moral scenarios_, contain statements that may raise ethical concerns. Here, LLMs could be misused or misinterpret the information. We strongly recommend thorough consideration of safety implications before applying such techniques in real-world scenarios. For the FEVER dataset, positive claims (facts) are extracted from Wikipedia, and negative claims are generated by contrasting these facts and subsequently verified without knowledge of their original source sentences. However, due to Wikipedia's editable nature, the extracted facts may not always be entirely accurate. Consequently, we advise against solely relying on our work as the truth source for any fact-checking task to avoid potential confusion and bias.
