# Abstract

Iterative human engagement is a common and effective means of leveraging the advanced language processing power of large language models (LLMs). Using well-structured prompts in a conversational manner, human users can effectively influence an LLM to develop more thoughtful and accurate responses. Motivated by this insight, we propose the Iteration of Thought (IoT) framework for enhancing LLM responses by generating "thought"-provoking prompts _vis a vis_ an input query and the current iteration of an LLM's response. Unlike static or semi-static approaches, _e.g._ Chain of Thought (CoT) or Tree of Thoughts (ToT), IoT adapts its reasoning path dynamically, based on evolving context, and without generating alternate explorative thoughts which are ultimately discarded. The three components of the IoT framework are (1) an Inner Dialogue Agent (IDA) responsible for generating instructive, context-specific prompts; (2) an LLM Agent (LLMA) that processes these prompts to refine its responses; and (3) an iterative prompting loop that implements a conversation between the former two components. We introduce two variants of our framework: Autonomous Iteration of Thought (AIoT), where an LLM decides when to stop iterating, and Guided Iteration of Thought (GIoT), which always forces a fixed number iterations. We investigate the performance of IoT across various datasets, spanning complex reasoning tasks from the GPQA dataset, explorative problem-solving in _Game of 24_, puzzle solving in _Mini Crosswords_, and multi-hop question answering from the HotpotQA dataset. Our results show that IoT represents a viable paradigm for autonomous response refinement in LLMs, showcasing significant improvements over CoT and thereby enabling more adaptive and efficient reasoning systems that minimize human intervention.

[IMAGE: Illustration of different prompting strategies for enhancing LLM reasoning capabilities. The Input-Output (IO) method uses a direct input-output approach with no intermediate reasoning. Chain-of-Thought (CoT) prompts introduce a single, linear reasoning path, while Tree-of-Thought (ToT) methods expand this by exploring multiple reasoning paths in parallel. The proposed Iteration-of-Thought (IoT) framework introduces an Inner Dialogue Agent (IDA) to dynamically adjust reasoning paths, enabling adaptive cross-path exploration to enhance response accuracy.]

# Introduction

The development of Large Language Models (LLMs) like `GPT-3`, `PaLM` [anil2023palm], and their successors, including `GPT-4` [openai2023gpt4], `Gemini` [team2023gemini], `LLaMA` [dubey2024llama], and `Claude`, has revolutionized natural language processing. LLMs have empowered AI systems to perform a wide range of tasks with remarkable proficiency. In the context of human-LLM interaction, a critical observation from practical experience is that the quality of LLM responses tends to improve with repeated prompting and user feedback. Recent research demonstrated that naive prompting can lead to calibration errors, while more sophisticated, iterative prompting strategies significantly improve both accuracy and reliability [krishna2024understandingeffectsiterativeprompting]. These results suggest that, given context-appropriate sequences of inputs, LLMs can much more effectively leverage their internal knowledge base [jiang2020can; petroni2019language; talmor2020olmpics; roberts2020much] to provide richer, more nuanced answers [sloman1996empirical].

A human user's interaction with in an LLM often proceeds as follows: the user poses a question to the LLM, receives an initial response, and, if the answer is incomplete or suboptimal, provides additional guidance to the LLM by reiterating contextual clues (_e.g._ by reminding the LLM of its role, suggesting additional information to consider, or highlighting specific parts of the response that need refinement). This back-and-forth process helps narrow the focus of the LLM while reducing the research effort required from the user, since the LLM is responsible for the bulk of the reasoning and information retrieval.

We identify two predominant forms of human-LLM interaction. In the first form of interaction, the user simply guides an LLM through its own internal knowledge base. For example, consider a scenario where an LLM generates code that is syntactically incorrect due to a missing bracket. The user might prompt it to "verify the syntax," leading the LLM to correct the error in a subsequent response. In the second for of interaction, the user introduces new information to improve the LLM's response. For example, an LLM may be asked to provide up-to-date weather information for a specific city, but lacks access to real-time data. In this case, the user can supply this information (using a tool or API), then prompt the LLM to _e.g._ recommend weather-appropriate clothing or destination to visit in that locale. All together, the first form an interaction leads the LLM to better utilize its _internal_ knowledge, whereas the second form of interaction involves augmenting the LLM's knowledge with new information.

The potential of iterative prompting to improve LLM responses is supported by research showing that prompt phrasing can significantly influence a model's performance in various settings [brown2020language; opsahl2024optimizing]. Figure 1 illustrates the progression from simple Input-Output (IO) approaches to more advanced methods like Chain-of-Thought (CoT) [wei2022chain] and Tree-of-Thought (ToT) [yao2024tree]. CoT introduces sequential reasoning steps along a single linear path, while ToT explores multiple reasoning pathways in parallel, forming a branching structure to optimize the output.

These methods represent "reasoning frameworks" [wei2022chain] that rely on static or semi-static prompts, which may struggle to adapt to the evolving context of each query and response, potentially limiting the quality of LLM responses. CoT prompting encourages LLMs to articulate its intermediate reasoning steps, which leads to better performance on complex tasks. Similarly, the related ToT approach (among other methods [sahoo2024systematic]) reasons along multiple paths to consider a wider breadth of potential responses, most of which are generated then discarded, leading to better performance on more explorative tasks like solving puzzles or crosswords. Other frameworks like _Self-Refine_ [madaan2024self] and _Self-Verification_ [weng2022large] enable LLMs to iteratively critique and refine their outputs, but still rely on static or semi-static prompts. In a broader context, the value of pursuing improved reasoning with inference techniques, as opposed to extensive training, is underscored by more recent advancements such as OpenAI's new series of `o1` models [openai2024o1]. These proprietary models are specifically designed to spend more time "thinking" through problems before responding, focusing on inference to solve complex tasks in science, coding, and math. Such developments highlight a broader shift in the AI community toward post-training enhancement of reasoning capabilities as a more scalable approach.

In this work, noting the lack of reasoning frameworks that strive to replicate the dynamic nature of human-LLM interactions, we propose IoT as an autonomous, iterative, and adaptive approach to LLM reasoning without human feedback.

## Iteration of thought (IoT)

Unlike the aforementioned static and semi-static frameworks, IoT utilizes an Inner Dialogue Agent (IDA) to adjust and refine its reasoning path during each iteration. This enables adaptive exploration across different reasoning trees, fostering a more flexible and context-aware response generation process. A comparison to existing methods is shown schematically in Figure 1.

The core IoT framework is composed of three main components. Further details are also provided in Section 2.

- **Inner dialogue agent (IDA):** The IDA functions as a "guide" that dynamically generates context-sensitive prompts based on the original user query and the LLM's previous response. The adjusted prompts serve to iteratively lead the LLM toward more refined and accurate answers. Mathematically, the IDA can be represented as a function `latex $C: \mathcal{Q} \times \mathcal{R} \times \mathcal{K}^\prime \rightarrow \mathcal{P}$ `, where `latex $\mathcal{Q}$ ` is the space of possible queries, `latex $\mathcal{R}$ ` is the space of potential LLM responses, and `latex $\mathcal{P}$ ` is the space of generated prompts. At each step, it takes the current query `latex $q \in \mathcal{Q}$ ` and the previous response `latex $r \in \mathcal{R}$ ` to generate a new prompt `latex $p \in \mathcal{P}$ `. This process makes prompt generation _dynamic_, differentiating IoT from more rigid approaches like CoT and allowing it to adapt to an evolving context.

- **LLM agent (LLMA):** The LLMA embodies the core reasoning capabilities of an LLM and processes the IDA's dynamically generated prompts. It uses an LLM's internal knowledge base `latex $K$ ` to refine its responses. Formally, we model the LLMA as a function `latex $L: \mathcal{Q} \times \mathcal{P} \times K \rightarrow \mathcal{R}$ `. The LLMA takes as input a query `latex $q$ `, prompt `latex $p$ ` and a knowledge base `latex $K$ ` then generates a refined response `latex $r$ `. The LLMA also identifies areas of uncertainty or gaps in its own reasoning, providing feedback for the IDA to adjust prompts accordingly. This interaction creates a closed-loop system that continuously improves the quality of answers without external inputs.

- **Iterative prompting loop:** The iterative process in IoT involves a back-and-forth between the IDA and LLMA. At each iteration `latex $i$ `, the IDA generates a new prompt `latex $p_i = C(q, r_{i-1})$ ` based on the original query `latex $q$ ` and the LLM's previous response `latex $r_{i-1}$ `. The LLMA then responds to `latex $p_i$ ` with `latex $r_i = L(q, p_i, K)$ `. This loop continues until a satisfactory answer `latex $r^*$ ` is found or the arbitrary maximum iteration count is reached. This back-and-forth approach allows IoT to navigate complex reasoning paths to efficiently explore various potential solutions. Moreover, introducing distinct LLMs for the IDA and LLMA respectively can allow each agent to function as an open system [von1950theory] where internal knowledge is exchanged. In this scenario, the overall system behaves as a _closed_ system with a combined knowledge base, enhancing internal reasoning without external input.

In the sections that follow, we present a detailed analysis of our IoT framework, describe our experimental methodology, and discuss empirical results. We also demonstrate the framework's effectiveness with experimental results on the various datasets, where significant improvements are observed over existing reasoning methods.

# Framework and implementation

In this work, we use two distinct variants of IoT: _Autonomous Iteration of Thought_ (AIoT) and _Guided Iteration of Thought_ (GIoT). In the AIoT variant, the LLMA itself decides when it has generated a satisfactory response. This decision is reflected in a Boolean output signal, `iteration_stop`. Termination following a positive signal usually leads to fewer iterations than the enforced maximum. This, in turn, leads to faster evaluation with less exploration, but risks premature stops when facing more complex queries. Conversely, GIoT employs a more regimented strategy by mandating a fixed number of iterations. GIoT employs the opposite strategy, aiming for comprehensive exploration of reasoning paths to minimize premature convergence, at additional computational cost and with the risk of redundant or repetitive iterations.

We implemented the IoT framework, including both variants, as a Python library [multi_agent_llm_2024], using Pydantic [pydantic_2024] to provide output schemas for raw responses from LLMs.

[IMAGE: Schematic example of processing a sample query with the IoT framework. A simplistic question is asked for illustrative purposes. The guided IoT variant (GIoT) is utilized here, with the number of iterations set to 2. Each grey box contains an individual iteration of IoT, with the IDA shown in yellow and the LLMA in green.]

## Autonomous iteration of thought (AIoT)

In AIoT, the LLM also makes a determination at each step on whether the answer it has generated is sufficient. This is represented by an output signal, `iteration_stop`, which, if set to `True`, indicates that the LLM believes its answer is final and complete. The full AIoT process is shown in pseudocode Algorithm 1, below. A sample AIoT sequence is also provided in Appendix 6.

```latex
Algorithm 1: AIoT
Input: Query $q \in \mathcal{Q}$, LLM configuration with IDA given by $C: \mathcal{Q} \times \mathcal{R} \times \mathcal{K}^\prime \rightarrow \mathcal{P}$, LLMA given by $L: \mathcal{Q} \times \mathcal{P} \times \mathcal{K} \rightarrow \mathcal{R}$, a maximum number of iterations $T \in \mathbb{N}^+$, and a stopping criterion given by $\mathcal{F}: \mathcal{R} \times \mathcal{C} \rightarrow \{0,1\}$.
$r_0 \gets L(q, \text{"Initial Prompt"}, \mathcal{K})$
$i \gets 1$
$\texttt{iteration\_stop} \gets \mathcal{F}(r_0, \mathcal{C})$
while not iteration_stop and i < T:
    $p_i \gets C(q, r_{i-1})$
    $r_i \gets L(q, p_i, \mathcal{K})$
    $\texttt{iteration\_stop} \gets \mathcal{F}(r_i, \mathcal{C})$
    $i \gets i + 1$
return $r_{i-1}$
```

## Guided iteration of thought (GIoT)

The guided variant of Iteration of Thought (GIoT) represents a more controlled iterative process. In GIoT, the iteration continues for a predefined number of steps `latex $N-1$ `, and only in the `latex $N$ `-th iteration is the LLM allowed to decide if it has reached the final answer. Here, the IDA continues to generate new prompts `latex $p_i = C(q, r_{i-1})$ ` for the first `latex $N-1$ ` iterations without allowing the LLM to conclude early. In the final iteration, the LLMA is asked to provide a conclusive answer `latex $r^*$ ` based on the accumulated information from previous steps.

Like AIoT, GIoT ensures that the LLM thoroughly explores its internal knowledge space and refines its output to a greater extent. However, unlike AIoT, GIoT admits the cost of additional generations as a compromise to prevent premature conclusion. The full GIoT process is shown in pseudocode in Algorithm 2. A sample GIoT sequence is also provided in Figure 2.

```latex
Algorithm 2: GIoT
Input: Query $q \in \mathcal{Q}$, LLM configuration with IDA given by $C: \mathcal{Q} \times \mathcal{R} \times \mathcal{K}^\prime \rightarrow \mathcal{P}$, LLMA given by $L: \mathcal{Q} \times \mathcal{P} \times \mathcal{K} \rightarrow \mathcal{R}$, and a maximum number of iterations $N \in \mathbb{N}^+$.
$r_0 \gets L(q, \text{"Initial Prompt"}, \mathcal{K})$
for i = 1 to N-1:
    $p_i \gets C(q, r_{i-1})$
    $r_i \gets L(q, p_i, \mathcal{K})$
$p_N \gets C(q, r_{N-1})$
$r^* \gets L(q, p_N, \mathcal{K})$
return $r^*$
```

To summarize, the choice of AIoT or GIoT defines the mode of iteration in the core IoT framework. Each variant allows the framework to approach the iterative refinement of LLM responses from different angles.

# Results

To comprehensively evaluate the IoT framework, we conducted a series of experiments across various models, datasets, and reasoning strategies. Given the computational expense these evaluations, we selected specific model-dataset combinations to investigate performance and scalability under different conditions. This approach enables us to provide a targeted understanding of how reasoning capability and iteration strategy affect the overall quality of LLM responses. The following sections describe the experiments designed to explore these aspects, including their setups, objectives, and the insights derived from each.

## Assessing IoT on the GPQA questionnaire

In this experiment, we quantify IoT's ability to accurately answer questions from the GPQA Diamond dataset [rein2023gpqa]. These questions are known to require deep reasoning and comprehensive internal knowledge, with even the highly capable LLMs yielding overall scores under 50% [dubey2024llama].

We compare AIoT/GIoT with CoT on `GPT-4o mini`, a proprietary model [openai2023gpt4]. CoT is a widely used reasoning strategy that involves guiding the model through a step-by-step thinking process. By comparing CoT with AIoT/GIoT, we aim to understand the potential improvements resulting from iterating _dynamically_ rather than following predefined steps.

| **Method** | **Acc.**  | **Improvement vs. IO (%)** |
| ---------- | --------- | -------------------------- |
| IO         | 0.405     | 0.00%                      |
| CoT        | 0.406     | 0.12%                      |
| GIoT       | 0.416     | 2.62%                      |
| AIoT       | **0.463** | **14.11%**                 |

[IMAGE: Comparison of GPQA evaluation accuracies for different methods.]

The results of these experiments are presented in the table above and Figure 3, where it is clear that conventional CoT performs about on par with the baseline IO approach, indicating that rigid step-by-step reasoning may not be effective for GPQA. Meanwhile, GIoT performs significantly better than IO and CoT with a modest 2.62% higher accuracy (on average). However, GIoT also exhibits larger variance than IO and CoT in its distribution of accuracy scores. One interpretation of this result is that forced iterations can lead to divergence due to hallucination [huang2023survey] in cases where a correct and complete thought pattern is established well before the mandated number of iterations have been performed. AIoT, on the other hand, is more effective at avoiding this issue.

AIoT emerges as the most effective strategy overall, with a 14.11% improvement in average accuracy over the IO baseline and the lowest variance among all methods tested. Lower variance in AIoT's accuracy scores implies more consistent performance across different types of questions. Together with a higher average score, this superior result is attributed to AIoT's dynamic autonomous, context-aware termination of iterations, which prevents unproductive or counterproductive exploration of the response space. Notably, our analysis shows that AIoT completes approximately 60% of tasks within a single iteration and approximately 90% within two iterations. This reflects AIoT's efficiency in navigating the reasoning space without over-iteration. We therefore infer that AIoT's advantage, wherever applicable, is avoiding the pitfalls of both under- (as seen in IO and CoT) and over-exploration (a risk associated with GIoT).

## Assessing IoT on explorative problem-solving tasks

To evaluate the effectiveness of our IoT (Iterative of Thought) framework against the state-of-the-art, we conduct a comparative analysis using the _Game of 24_ and _Mini Crosswords_ tasks. These games, featured prominently in the ToT genesis paper by Yao et al. [yao2024tree], are easy to understand, challenging to solve, but easy to verify. ToT is well-suited for problems that benefit from a wide variety of exploratory reasoning paths, owing to its systematic search strategy that traverses many possible solution graphs to find the optimal answer. Our motivation for this experiment is to assess whether our IoT method can effectively iterate towards optimal solutions without generating a multitude of alternate, discarded responses. With this in mind, our goal in this experiment is to compare the relative advantage of our IoT framework compared to CoT, recognizing the inherent advantages of ToT, at least in terms its overall solution ability, in contexts benefiting from a broader exploratory approach.

The _Game of 24_ involves generating an arithmetic expression using four given numbers and basic operations and brackets `latex $\{+, -, \times, \div,\left(,\right)\}$ ` to arrive at the number 24. This task requires not only computational ability but also strategic reasoning to explore different combinations of operations. The dataset for this task, as used in the ToT study, consists of various instances where the challenge lies in finding the most efficient path to the solution amidst multiple possibilities. Similarly, the _Mini Crosswords_ task involves solving 5x5 crossword grids based on a set of clues. Solving these grids requires lexical reasoning and pattern recognition, as well as the ability to generate coherent word sequences that fit both vertical and horizontal constraints. The complexity of the _Mini Crosswords_ task also stems from the need to test the compatibility of multiple potential word fits, refining choices based on feedback and constraints. Both datasets are therefore valuable for assessing a model's ability to try out various solutions within a reasonable search space.

[IMAGE: Performance comparison across different methods (GIoT, AIoT, CoT, IO) on Mini Crossword: Letters, Mini Crossword: Words, and Game of 24 tasks. Box plots represent the distribution of mean accuracy percentages across different trials.]

Results for these tasks are visualized in Figure 4, which reveals distinct performance differences between the various methods. Notably, GIoT on average outperforms the AIoT, CoT, and IO methods across both tasks. This result is consistent with the understanding that GIoT is a more exploratory alternative to AIoT. By compelling the model to explore multiple reasoning paths, GIoT enhances the likelihood of arriving at a correct answer, aligning with the ToT approach in terms of beneficial brute-force exploration.

Regarding the _Mini Crosswords_ task, the original ToT study (which used `GPT-4`) demonstrated substantial improvements over CoT, with success rate increases of 92.1% for letters and 284.6% for words [yao2024tree]. In comparison, our experiments using the less capable `GPT-4o mini` model show that GIoT achieves a success rate of 35.5% for letters and a 90.6% success rate for words, as compared to CoT. Meanwhile, AIoT shows gains of 28.3% and 74.5%, respectively. Although these differences are smaller than those reported for ToT, they should be considered in context with the limitations of `GPT-4o mini` versus `GPT-4`. It is also important to note that the superior performance of ToT in this task is primarily due to its capacity to explore a broader range of answers, potentially admitting a higher computational cost than GIoT.

The higher variance observed in the IoT results, particularly in the _Mini Crosswords_ task, suggests a more diverse albeit not always productive exploration of solutions compared to CoT and IO. While this diversity can be advantageous in some scenarios, it may lead to sub-optimal convergence in more constrained problem spaces.

A similar pattern of performance differences emerges in the _Game of 24_ task. Here, the ToT framework showed a dramatic improvement, with success rates increasing from 4.0% with CoT to 74% with ToT (at a breadth of 5), marking a relative improvement of 1750% [yao2024tree]. In comparison, our GIoT method achieves a notable 266.4% improvement over CoT, while AIoT shows a 168.4% increase. These results reflect the effectiveness of our iterative refinement approach in arithmetic problem-solving scenarios, even though a performance gap remains compared to ToT. The structured, multi-step reasoning of GIoT ensures a more thorough exploration of the solution space, which aligns with the exploratory nature of ToT but operates within the constraints of our closed-system approach. A key distinction between our method and ToT is the feedback mechanism: while ToT benefits from its ability to explore more extensive solution spaces or receive external correctness checks, our methods, especially AIoT, can lead to cases where incorrect answers are confidently selected. Integrating external validation tools or feedback could therefore significantly enhance IoT's performance on this and similar tasks.

## Assessing IoT on multi-context reasoning and retrieval tasks

In our final experiment, we evaluate IoT on the HotpotQA-Hard dataset, a challenging benchmark for multi-hop question answering that demands sophisticated aggregate reasoning. Unlike simpler tasks that require straightforward information retrieval, HotpotQA involves complex information synthesis across multiple documents, requiring models to shift focus between various contexts to build a coherent answer. This necessitates bridging implicit information gaps, resolving ambiguities, and integrating scattered evidence.

Answering a HotpotQA question often involves several interconnected steps where initial findings must be used to guide further evidence retrieval. This process mirrors the key strengths of IoT: its ability to adaptively explore different reasoning paths, dynamically integrate context, and iteratively refine conclusions. The IoT's IDA plays a pivotal role here by guiding the LLMA to revisit and adjust its focus based on intermediate outputs, promoting more comprehensive exploration of the problem space. Such a mechanism is crucial for HotpotQA tasks, where the model must constantly re-evaluate earlier conclusions in light of newly synthesized information, ultimately leading to a more robust and accurate final answer.

For this experiment, again using `GPT-4o mini` as our engine, we benchmark the performance of AIoT against CoT using on three evaluation metrics: Exact Match (EM), F1 score, and ROUGE-L score. These metrics capture different facets of multi-hop QA performance: EM measures the proportion of exact matches with the ground truth, providing a stringent gauge of model accuracy; the F1 score balances precision and recall, capturing partial correctness; and ROUGE-L evaluates the longest common sub-sequence between generated and reference answers, highlighting semantic coherence.

The dynamic nature of (A)IoT allows it to autonomously adapt the depth of reasoning based on the complexity of the query, facilitating a flexible exploration of reasoning paths that CoT's static, step-by-step approach may lack. This flexibility enables the IoT framework to better handle the inherent ambiguities of HotpotQA, such as resolving conflicts or disambiguating entities across contexts. This can also serve as a self-correcting mechanism, helping to recognize gaps or errors in reasoning early on and prompting further exploration in subsequent iterations.

[IMAGE: Performance comparison between AIoT and CoT on HotPotQA-Hard dataset]

Our results on the HotpotQA-Hard dataset reveal a clear advantage for AIoT over CoT. As shown in Figure 5, AIoT achieves an Exact Match (EM) score of 0.53, an F1 score of 0.699, and a ROUGE-L score of 0.72, significantly outperforming CoT on almost every instance of the task. These metrics demonstrate the effectiveness of AIoT in managing the complexities inherent in multi-hop question answering tasks, where models must dynamically integrate and synthesize information from various documents.

The observed performance gains are consistent with the core principles underlying the IoT framework. Using an Inner Dialogue Agent that steers the reasoning path, AIoT can better navigate ambiguities and implicit connections between different pieces of information. Improvements in the F1 score are indicative of AIoT's ability to partially correct initial errors and refine its answers over multiple iterations. Similarly, the higher ROUGE-L score reflects AIoT's capacity to generate answers that are not only factually correct but also maintain semantic alignment with the ground truth. These results validate our hypothesis that iterative, adaptive reasoning is essential for tasks requiring complex information synthesis across disjoint contexts.

To contextualize our findings, we compare our AIoT approach with the _AgentLite_ framework introduced by Liu et al. [liu2024agentlite]. _AgentLite_ is built on a novel hierarchical multi-agent orchestration technique that supports structured multi-agent systems, where a manager agent coordinates a set of team agents, each handling different aspects of reasoning. In their experiments on the HotpotQA dataset, Liu et al. utilized _AgentLite_ to implement agents capable of multi-hop reasoning across multiple documents. The action space for these agents was designed with three primary members: _WikipediaSearch_, _Think_, and _Finish_. Various models were tested within this framework, including `GPT-4`, a generally more knowledgeable model [OpenAI2024eval] than `GPT-4o mini` utilized in our experiments.

| **Method**                | **Model**        | **F1 Score** | **Exact Match (EM) Score** |
| ------------------------- | ---------------- | ------------ | -------------------------- |
| Multi-Agent _(AgentLite)_ | `GPT-4-0613`     | 0.527        | 0.38                       |
| Multi-Agent _(AgentLite)_ | `GPT-4-32k-0613` | 0.520        | 0.37                       |
| AIoT _(Ours)_             | `GPT-4o mini`    | **0.699**    | **0.53**                   |

**Table 1:** Performance Comparison on HotpotQA-Hard Dataset between AIoT Framework and AgentLite Benchmarks.

Comparing results from our AIoT approach to _AgentLite_ (Table 1) shows that AIoT achieves higher F1 and EM scores across the board. The AIoT framework's F1 score of 0.699 and EM score of 0.53 surpass the results of even the most potent models used in the _AgentLite_ experiments, such as `GPT-4-0613` and `GPT-4-32k-0613`. This suggests that while _AgentLite_ offers a robust approach to structured reasoning, it may lack the adaptability and refinement capabilities that AIoT provides. By focusing on an autonomous, self-guided iteration process, AIoT effectively revisits and recalibrates its reasoning, allowing for deeper context integration and a more comprehensive exploration of the problem space. This comparison also validates the advantages of our approach in leveraging dynamic reasoning to outperform more static agentic frameworks in a multi-hop QA scenario.

To further illustrate the effectiveness of the IoT framework, we note that it also outperforms recent methods for multi-hop reasoning on HotpotQA, such as those described by Wang et al. [wang2024adapting], Wang et al. [wang2024knowledge], and Jiapeng et al. [jiapeng2024tree]. Although these studies demonstrate improvements over the CoT approach, the increase in F1 and EM scores achieved by IoT is larger than those reported in the aforementioned works. While not all these studies utilize `GPT-4o mini`, which makes direct comparisons less straightforward, it remains evident that the jump in accuracy from CoT to our IoT framework is much more pronounced.

# Strengths and weaknesses of IoT

One qualitative benefit of IoT is its inherent conceptual transparency and explainability. Like CoT and similar methods, IoT provides a clear trace of its reasoning process through a sequence of evolving outputs. However, unlike other "multi-thought" methods, IoT's sequence also includes explicit guidance generated by the IDA. This means each step is accompanied by a rationale that the underlying LLM treats equivalently to prompts from a human user. As a result, _post hoc_ analysis of IoT's output sequences (example in Appendix 6) can reveal the model's capacity to self-correct when provided with course-adjusting instructions. In addition to enhancing the model's explainability, this insight can inform more efficient interactions with LLMs in general.

It is important to note that the IoT framework is not inherently orthogonal to CoT nor _Self-Consistent_ CoT [wang2022self]. One could in principle combine IoT with CoT to create a hybrid method, IoT-CoT, where both the Inner Dialogue Agent (IDA) and LLM Agent (LLMA) use CoT-based reasoning. Such combinations could amplify the benefits of structured reasoning while retaining the flexibility of iterative refinement. Additionally, while our experiments used the same base LLM for both IDA and LLMA, these agents can be made distinct to leverage different models or architectures, changing the total base knowledge of the system to be `latex $K \otimes K^\prime$ ` [creswell2022faithful].

Owing largely to the versatility of LLMs, agentic LLM-based frameworks are not difficult to expand and compose. Recent work has suggested that larger ensembles of agents can lead to better reasoning performance [li2024more], with the rate of improvement diminishing beyond 10-15 agents. A natural progression of IoT could therefore be an expansion of the IDA into a _meta-agent_ consisting of specialized sub-agents, which may or may not be dynamically defined on a per-query basis. Taking the knowledge base of the IDA to be `latex $K^\prime = \bigotimes_j^M K^j$ `, the size of the IDA ensemble, `latex $M \in \mathbb{N}^+$ `, becomes an arbitrary parameter indicating the number of distinct LLMs behind the IDA's constituent sub-agents. IoT as introduced in this work (with `latex $K = K^\prime$ `) represents the "smallest" member of this generalized family and still suffices to deliver powerful reasoning capabilities. Based on Li et al. [li2024more], increasing the ensemble size `latex $M$ ` could be expected to improve reasoning performance in IoT, though at cost of additional complexity and a larger hardware budget to power a multitude of distinct LLMs. `latex $M>1$ ` also introduces the potentially challenging task of ranking sub-agent outputs or resolving conflicts in their guidance. Regarding complexity, using larger LLMs in a smaller ensemble may be a viable alternative for increasing the size of `latex $K^\prime$ ` without increasing `latex $M$ `.

IoT's autonomous iteration also offers significant advantages in situations where human intervention is impractical or impossible -- such that systems are constrained to function independently. Human oversight is difficult to achieve in contexts that demand rapid and continuous decision-making, for example. Here, IoT's autonomous reasoning capabilities can be a valuable asset. Moreover, the thought sequences generated by IoT (see Figure 2 and Appendix 6) could serve as a valuable resource for fine-tuning existing models, potentially enhancing their reasoning capabilities. This dual benefit of autonomy and improved model training makes IoT a powerful tool in building more robust, self-sufficient systems.

Regarding the two variants of IoT, our results demonstrate that while AIoT provides an efficient approach with autonomous decisions to stop iterating, it also often misjudges the completeness of its responses, leading to premature convergence. This limitation could be addressed by incorporating feedback agents [chen2023teaching], using techniques like _maieutic prompting_ [jung2022maieutic], or even allowing for human intervention or external knowledge checks. This would create a semi-autonomous framework that balances efficiency with robustness [wu2022survey]. On the other hand, GIoT forces a fixed number of iterations, which can improve performance in multi-step reasoning tasks, but may also increase the risk of hallucination if the model confidently drifts into incorrect reasoning. Appropriate techniques to reduce hallucination could further refine GIoT's utility in complex tasks [tonmoy2024comprehensive].

# Conclusion and future work

In this work, we introduced the Iteration of Thought (IoT) framework, in which an Inner Dialogue Agent (IDA) iteratively converses with an LLM Agent (LLMA) to perform various complex reasoning tasks like solving puzzles (_Game of 24_, _Mini Crosswords_) and answering difficult questionnaires (GPQA, HotpotQA). We employed two variants of this framework in our experiments, qualified as "autonomous" (AIoT) and "guided" (GIoT) respectively, to compare iteration-terminating mechanisms across these tasks. GIoT, the variant that always performs a fixed number of iterations, was seen to perform better than AIoT, the variant that self-determines termination, in _Game of 24_. On the other hand, AIoT had superior performance on GPQA. Both variants performed similarly on _Mini Crosswords_ and always performed better than the well-known Chain of Thought (CoT) framework, wherever compared. We also compared our IoT framework against the hierarchical _AgentLite_ framework on the multi-context HotpotQA task, finding improvements of approximately a 35% in the F1 score and 44% in the EM score over _AgentLite_. All together, our results demonstrate that IoT can successfully introduce productive dynamism into low-complexity agentic frameworks.

Determining the scale and diversity of the IDA's knowledge base represents a promising direction for future work aiming to maximize the real-world utility of IoT. In pursuit of strictly framework-to-framework comparisons, we used only off-the-shelf, general-purpose LLMs in all our experiments to establish IoT. Moving forward, specialized language models like fine-tuned LLMs or LLMs equipped with additional tools and/or data sources could yield further performance gains, whether by increasing the effective knowledge base or directly addressing challenges like hallucination and the premature termination of iterations.

# Appendix

## Examples

In this section, we provide an example to demonstrate how AIoT (Autonomous Iteration of Thought) works in practice. This example highlights the unique characteristics of AIoT, illustrating how the method improves reasoning, adaptability, and response accuracy.

### Example of AIoT

This example highlights AIoT's strength in efficiently navigating a complex GPQA example.

**Query:**
A textile dye containing an extensively conjugated pi-electrons emits light with energy of 2.3393 eV. What color of light is absorbed by the organic compound? Pick an answer from the following options:
A. Red
B. Yellow
C. Blue
D. Violet
