# Abstract

Recent advancements in large language models (LLMs) have demonstrated that progressive refinement, rather than providing a single answer, results in more accurate and thoughtful outputs. However, existing methods often rely heavily on supervision signals to evaluate previous responses, making it difficult to effectively assess output quality in more open-ended scenarios. Additionally, these methods are typically designed for specific tasks, which limits their generalization to new domains. To address these limitations, we propose Progressive Thought Refinement (PTR), a framework that enables LLMs to progressively refine their responses. PTR operates in two phases: (1) Thought data construction stage: We propose a _weak and strong model collaborative selection_ strategy to build a high-quality progressive refinement dataset to ensure logical consistency from thought to answers, and the answers are gradually refined in each round. (2) Thought-Mask Fine-Tuning Phase: We design a training structure to mask the "thought" and adjust loss weights to encourage LLMs to refine prior thought, teaching them to implicitly understand "how to improve" rather than "what is correct." Experimental results show that PTR significantly enhances LLM performance across ten diverse tasks (avg. from 49.6% to 53.5%) without task-specific fine-tuning. Notably, in more open-ended tasks, LLMs also demonstrate substantial improvements in the quality of responses beyond mere accuracy, suggesting that PTR truly teaches LLMs to self-improve over time.

# Introduction

> "_Think thrice before you act._"
>
> --- Confucius

Recent advancements in large language models (LLMs) have highlighted that progressive refinement is more important than simply providing a single answer [yang2023leandojo; madaan2023self]. Humans often rely on a combination of two thinking systems to solve problems, known as _System 1_ and _System 2_ [kahneman2011thinking]. _System 1_ facilitates quick, intuitive responses but often lacks the depth required to handle complex reasoning tasks. In contrast, _System 2_ engages in progressive refinement, gradually improving a solution by starting with a rough approximate thought and iteratively adding detail and accuracy. Recent work, such as GPT-o1 [openai2024gpto1], demonstrates that LLMs perform better by adopting progressive thought refinement. This approach leads to more accurate and thoughtfully considered outcomes, similar to how the human brain processes complex tasks.

Progressive refinement ability is imperative for LLMs because it significantly enhances the quality of responses by gradually improving accuracy and depth. Previous methods heavily rely on supervision signals, such as correctness assessments, to assess response quality. For example, labeled datasets with feedback are used to fine-tune models as verifiers [han2024smalllanguagemodelselfcorrect; havrilla2024glore; welleck2023generating], facilitating self-assessment and iterative improvement. Additionally, Reinforcement Learning (RL) reward functions are also employed to guide models toward generating better answers [chen2024self; yuan2024self; rosset2024direct; akyurek-etal-2023-rl4f]. However, evaluating answers based on supervision signals has limitations, as annotators often **struggle to provide accurate labels** without clear, comprehensive criteria. This is particularly challenging in open-ended tasks, such as text generation and summarizing, where the distinction between "correct" and "incorrect" is blurred, making it difficult to define and evaluate response quality.

Due to significant variations in supervision signals and evaluation criteria across tasks, previous self-improvement approaches have primarily aimed to enhance accuracy within specific domains. Examples include enabling LLMs to self-debug for improved code generation [chen2023teaching; tony2024promptingtechniquessecurecode; liang2023code] and solving math problems through progressive step validation [wang2023math; lightman2023let; uesato2022solving]. These methods often rely on task-specific pipelines or reward models, making generalization difficult. The key limitation is that errors addressed in one domain may not apply to other tasks, since different tasks exhibit varying error types. Consequently, transferring these approaches to new tasks often fails [tian2024selfimprovementllmsimaginationsearching], and models trained with these methods have **limited generalization capabilities**, struggling to improve performance beyond their training domains.

To address these challenges, we introduce PTR (**P**rogressive **T**hought **R**efinement), a framework specifically designed to stimulate the model's intrinsic refinement ability. Our PTR method comprises a progressive refinement dataset construction phase and a weighted thought-mask fine-tuning phase. During the progressive refinement dataset construction phase, we obtain queries from open-domain datasets and employ a _weak-strong model collaborative selection_ strategy to construct high-quality _thoughts_ and _refined answers_ dataset. This strategy not only ensures improvement from thoughts to answers but also **eliminates the need for accurate labels**. In the fine-tuning phase, we employ _weighted thought-mask fine-tuning_ to teach LLMs to implicitly understand "how to improve" rather than supervising them with "what is correct". Specifically, we reformulate the masked data structure and redesign the loss of weighting to encourage LLMs to improve responses based on previous thoughts and ensuring logical consistency between the thought process and the final answer.

Our experimental results show that LLMs trained with PTR can improve the quality of their previous answers across ten tasks, including knowledge reasoning, code generation, mathematical reasoning, comprehension, summarizing, and text generation. The average performance across these tasks improved from 49.6% to 53.5%, with a significant improvement on the MMLU task, where accuracy increased from 57.1% to 64.1% for Qwen2-8B. Notably, these improvements occur **without task-specific fine-tuning**, demonstrating that our method activates the model to learn progressive refinement from the PTR dataset. Moreover, in more open-ended tasks, LLMs have also demonstrated further improvements in answer quality and formatting beyond correctness.

Our contributions are threefold:

- We propose the PTR method to stimulate models' progressive refinement abilities and enhance generalization across various tasks without additional task-specific fine-tuning.

- We design an efficient _weak-strong model collaborative selection_ strategy to construct high-quality PTR datasets without extra feedback.

- We introduce a novel _weighted thought-mask fine-tuning_ method to instill general progressive refinement capabilities in LLMs.

# Related Work

[IMAGE: figs/main_fig.png - Illustration Our approaches. (A) Pipeline of our progressive refinement Dataset construction. We first prepare queries from the general open domain datasets, and pre-processing queries in three steps. Then we use a strong weak model collaborative selection strategy to generate thoughts and answers for each query. We also implement In-context Learning (ICL) and Consistency Filtering to ensure the quality of the thought process. (B) The illustration of Weighted Thought Masking Fine-tuning. Aiming at training the model to produce a better response in the next attempt and ensure logical consistency during the thought process. The difference between our method and IFT is that we use thought-mask techniques to ask model to generate better responses. (C) Pipeline of our PTR. Given a query Q, LLMs think progressively and refine their responses based on their own previous thought and refinement instruction. LLMs refined its mistakes on the second attempt, as well as gave a more thoughtful answer at a later iteration.]

**Progressive Refinement with External Feedback** Existing work often relies on external tools or stronger LLMs to provide feedback for refinement. For example, external tools are used to critique and provide feedback on the primary model's responses [yang2023auto; chen2023teaching; charalambous2023new; nijkamp2022codegen; yao2022react; gou2023critic]. Models have improved their code generation capabilities by leveraging error messages from the Python interpreter [wang2023leti] and by teaching large language models to debug and explain their own code, allowing them to identify and fix errors without human feedback [chen2023teaching]. Similarly, compiler feedback has been utilized in code generation [chen2024self; olausson2023self]. Additionally, some approaches utilize criticisms or constraints generated by stronger models [pan2023automatically; du2023improving; bai2022constitutional; huang2023agentcoder], such as using a strong model to verify the correctness of another model's math solutions [wang2023math], thereby relying on external information sources to guide improvements. Although models can self-correct through external feedback [pan2023automatically], this approach does not fully tap into their intrinsic progressive refinement capabilities. Moreover, it requires task-specific feedback models or tools, increasing the cost of adapting to a broader range of tasks. Furthermore, current LLMs struggle to self-correct reasoning errors without external feedback [huang2023large]. Our work aims to unlock the model's inherent Progressive Refinement ability, enabling it to perform progressive refinement across all domains without relying on external tools.

**Prompting for Progressive Refinement** Various Prompting methods have been introduced to enhance Progressive Refinement, such as prompting LLMs to generate explanations and self-correct code [li2023explainingcompetitivelevelprogrammingsolutions], or encouraging them to generate alternative solutions and revision suggestions [zhang2024selfcontrastbetterreflectioninconsistent]. Some methods iteratively improve outputs by generating feedback through task-specific prompts [madaan2023selfrefineiterativerefinementselffeedback], or guide models to generate fine-grained feedback in mathematical problem-solving, further enhancing solution accuracy and quality [xue2023rcot]. The Reflexion method enables language models to operate effectively in specific environments by allowing them to reflect and adjust their actions when encountering errors [shinn2023reflexion]. However, these approaches often require carefully designed, task-specific prompts or even oracle ground-truth answers [shinn2023reflexionlanguageagentsverbal], making LLMs highly sensitive to evaluating response and achieving optimal performance [wu2024how]. Without external tools, LLMs have limited self-correction capabilities when relying solely on prompting [huang2023large; zheng2024natural].

**Fine-Tuning for Progressive Refinement** In current progressive refinement work, fine-tuning typically relies on reward models or verifiers to assess the accuracy of model outputs based on predefined criteria [wang2023math; lightman2023let; uesato2022solving]. For instance, some research focuses on improving the model's ability to identify and correct mistakes [han2024smalllanguagemodelselfcorrect], while others progressively validate solutions, such as in solving math problems [uesato2022solvingmathwordproblems]. Additionally, reinforcement learning (RL) [chen2024self; yuan2024self; rosset2024direct; akyurek-etal-2023-rl4f] has been applied to align model outputs with correct responses. For example, researchers create preference-based datasets to align outputs with human values and reduce harmful content [wang2024enablinglanguagemodelsimplicitly; rosset2024directnashoptimizationteaching]. Similarly, ROUGE has been used as a reward function in text summarizing tasks to optimize generated summaries [akyurek-etal-2023-rl4f]. While these methods effectively train models, they focus on building task-specific datasets and reward functions tailored to particular objectives. In contrast, our approach redefines the fine-tuning objective to bolster the model's capacity for progressive refinement. Rather than relying on domain-specific datasets, our model is trained to iteratively enhance its responses---starting from initial thoughts and evolving toward increasingly refined answers.

# Progressive Thought Refinement Framework

Our proposed framework, Progressive Thought Refinement (PTR), comprises two stages, as illustrated in Figure 1: (1) Progressive Thought Refinement Dataset Construction and (2) Progressive Weighted Thought-Mask Fine-tuning. The primary objective of this framework is to enhance models' progressive refinement abilities, enabling them to handle diverse and unfamiliar tasks without relying on task-specific fine-tuning. Since fine-tuning models for every task is impractical, our approach utilizes general queries, thoughts, and answers to help models comprehend progressive refinement. This strategy gradually improves their capacity to tackle complex tasks through progressive refinement.

## Progressive Thought Refinement Dataset Construction

In the first stage, we construct a progressive refinement dataset that includes Queries, Thoughts, and Answers. The Thoughts capture a sequence of different reasoning attempts, which may be varied, incomplete, or even incorrect, reflecting the model's initial exploration of the problem. In contrast, the Answers provide more confident and well-reasoned responses. This structured approach helps the model implicitly understand the difference between initial thoughts and improved answers, enabling it to generate more thoughtful and in-depth responses over time.

### Query Preparation

To enhance the model's generalization, we avoid creating domain-specific datasets. Instead, we use queries from open-domain general datasets (details in Appendix), ensuring the model develops general refinement abilities rather than specializing in specific areas. Our data preprocessing involves three key steps. First, we perform data cleaning to remove noise and irrelevant content, such as images or URLs. Second, to prevent data leakage, we exclude domain-specific testing queries during training. Finally, we incorporate traditional SFT data (queries and answers) into our dataset to mitigate the risk of catastrophic forgetting.

### Thought-Answer Preparation

We strategically select weak and strong models to generate sequences of thoughts and improved answers from an initial query. The objective is to ensure that the final answer is progressively improved through multiple iterations rather than relying on a single-step response. We also employ **In-Context Learning (ICL)** [dong2024surveyincontextlearning] and consistency filtering to ensure logical coherence between thoughts and answers.

**Weak-Strong Model Collaborative Selection Criteria** To ensure the final answer shows significant improvement over the initial thought sequence, we adopt a weak-strong model collaborative selection strategy. Let `latex $\theta_w$ ` and `latex $\theta_s$ ` represent the abilities of the weak and strong models, respectively, with the goal of ensuring `latex $\theta_s \gg \theta_w$ `. We employ three key strategies: _Model Parameter Strength_, _Model Version (New vs. Old)_, and _Domain-Specific Fine-Tuning_. These selection strategies ensure the quality of the final answer surpasses that of the previous thoughts. Additionally, we validate that the strong model performs significantly better than the weak model through Wilcoxon significance tests, as shown in Appendix.

**Thought Generation by the Weak Model** The weak model generates a sequence of thoughts based on the input query `latex $q_i$ `, with `latex $\hat{y}_{i,w}^t$ ` representing the initial thought at the `latex $t$ `-th attempt. We denote the strong model as `latex $\pi_{\text{strong}, \theta_s}$ ` and the weak model as `latex $\pi_{\text{weak}, \theta_w}$ `. These initial thoughts may contain errors but provide a foundation for further refinement:

```latex
$$\begin{align}
  S_{\text{i, thought}} = \left[\hat{y}_{i,w}^1, \hat{y}_{i,w}^2, \dots, \hat{y}_{i,w}^t\right] = \pi_{\text{weak}, \theta_w}(\cdot \mid q_i).

\end{align}$$
```

Multiple weak models can be used to generate these thoughts, or a single weak model can produce multiple attempts. Since the weak model's thoughts need not be correct, constructing these thoughts remains cost-effective.

**Answer Refinement by the Strong Model** To achieve progressive refinement, we leverage the strong model to produce increasingly improved answers. We use ICL to ensure logical coherence between the outputs of the strong and weak models and to avoid randomness. This guides the strong model to generate better answers based on prior thoughts. Specifically, the strong model takes the sequence of thoughts `latex $S_{\text{i, thought}}$ ` and query `latex $q_i$ ` as input and generates the final answer `latex $\hat{y}_{i,s,\text{icl}}$ `:

```latex
$$\begin{align}
  \hat{y}_{i,s,\text{icl}} = \pi_{\text{strong}, \theta_s}(\cdot \mid S_{\text{i, thought}}, q_i).
\end{align}$$
```

**Thoughts-Answer Consistency Filtering** To further ensure that the thought process exhibits logical coherence, we apply consistency filtering to remove inconsistent outputs. If the consistency score is below a certain threshold, the pair is considered inconsistent and removed, ensuring that only coherent thought sequences are used for the final output (see Appendix).

## Progressive Weighted Thought-Mask Fine-tuning

In the second stage, we perform weighted thought-mask fine-tuning using the datasets constructed previously, consisting of the input query `latex $q_i$ `, the initial thought sequence `latex $S_{\text{i, thought}}$ `, and the final answer `latex $\hat{y}_{i,s,\text{icl}}$ `. Formally, the dataset is represented as:

```latex
$$\begin{align}
  \tilde{\mathcal{D}} = \left\{ \left( q_i, S_{\text{i, thought}}, \hat{y}_{i,s,icl} \right) \right\}_{i=1}^N
\end{align}$$
```

**Thought Mask Mechanism** To help the model understand the improvement between the thought process and the answer---rather than focusing solely on the final answer---we introduce a thought mask mechanism. This mechanism selectively hides parts of the thought process during training, as shown in Figure 1 (B). It calculates the loss based only on the accuracy of the refined final answer, ensuring the model focuses on enhancing the quality of its ultimate response. Additionally, we provide refinement instructions (e.g., "Please continue thinking and refine your answer") after each thought process to prompt better refinement in subsequent iterations.

**Weighted Supervised Learning** We adopt a weighted supervised learning approach that enables the model to focus on refining its answers by progressively improving its thought process. Specifically, we perform weighted supervised learning that emphasizes both the accuracy of the final answers and the logical consistency of the thought process. The loss function optimizes the model in three key areas: generating accurate final answers, maintaining consistency in reasoning and ensuring that the model's confidence increases progressively throughout the thought process.

```latex
$$\begin{align}
    \mathcal{L}_{\text{PTR}}(\theta) &= \sum\limits_{(q_i, S_{i, \text{thought}}, y_n) \in \tilde{\mathcal{D}}} \Bigg[ -\lambda_1 \log \Pr(y_n \mid q_i, S_{i,\text{thought}}; \theta) \nonumber \\
    &\quad + \lambda_2 \sum\limits_{t=2}^{n} \mathcal{F}_{\text{cons}}(y_t, y_{t-1}) + \lambda_3 \sum\limits_{t=1}^{n} \beta_t \left(1 - \Pr(y_t \mid q_i, S_{i,\text{thought}}; \theta)\right) \Bigg].
\end{align}$$
```

Unlike standard supervised fine-tuning, which trains the model to produce a single response `latex $\hat{\boldsymbol{y}}$ ` given `latex $\boldsymbol{x}$ `, the equation above focuses exclusively on the accuracy of the final response generated after the thought refinement process. It also ensures that the final response remains logically consistent with the previous thought sequence. We encourage the model to maintain higher confidence in its predictions during subsequent reasoning steps. Here, `latex $\lambda_1$ `, `latex $\lambda_2$ `, and `latex $\lambda_3$ ` are dynamically adjusted according to the model's needs, with their sum constrained to 1.

# Experiments

The goal of our experiments is to demonstrate the effectiveness of PTR in enabling language models to progressively enhance their responses. Specifically, we aim to answer the following questions: (1) Can the PTR method activate the model's progressive refinement ability? (2) Does our method demonstrate generalization? (3) Does progressive refinement ability emerge during training? (4) Is our method robust across different LLMs and instructions? (5) How many iterations are required for our method to achieve optimal performance?

**Datasets Sources** Our model has trained on our PTR dataset, derived from the WizardLM dataset [xu2023wizardlm]. After thorough cleaning, we reduced the original dataset from approximately 50k QA pairs to 40k high-quality QA pairs.

**Evaluation Tasks** In our experiments, we perform generalization over ten datasets across different tasks. For general tasks, we use MMLU [hendrycks2020measuring], and for coding tasks, we use HumanEval [chen2021evaluating] (abbreviated as H-Eval). DROP [dua2019drop] is used for comprehension tasks (abbreviated as Comp), and XSum [narayan2018dontdetailsjustsummary] is applied for summary tasks. We use GSM8K [cobbe2021trainingverifierssolvemath] and MATH [hendrycks2021measuringmathematicalproblemsolving] for math-related tasks. For complex reasoning tasks, we use ARC and GPQA [rein2023gpqagraduatelevelgoogleproofqa]. For knowledge reasoning, we utilize Winogrande [sakaguchi2019winograndeadversarialwinogradschema] (abbreviated as Wino) and CommonsenseQA [talmor2019commonsenseqaquestionansweringchallenge] (abbreviated as Comm).

**Evaluation Settings** We use greedy decoding (with temperature set to 0) for final generation, as lower temperature yields better performance shown in Appendix. We utilize zero-shot prompting [kojima2023largelanguagemodelszeroshot] for both answer sampling and evaluations, observing that zero-shot prompting outperforms few-shot prompting for LLMs fine-tuned on specific tasks. All of our experiments are conducted on workstations equipped with eight NVIDIA A800 PCIe GPUs with 80GB memory, running Ubuntu 20.04.6 LTS and PyTorch 2.0.1.

**Baselines** We compare our model with base models and prior approaches: (1) **Prompt**: Directly prompting the model to refine its answer [huang2023large]. (2) **IFT**: Instruction Fine-Tuning by directly fine-tuning the input-output pairs from strong models on the PRD dataset to show that improvements are not due to knowledge distillation. (3) **RL**: Perform one reinforcement learning training [wu2024progressregressselfimprovementreversal] iteration on the PRD dataset to compare with our method. Specifically, we use the thoughts and answers of the PRD dataset to construct preference data, and prefer model to produce stronger answers through DPO [rafailov2024directpreferenceoptimizationlanguage]. We compare these methods on the PRD dataset under the same settings as in the previous section. Detailed settings are in Appendix.

## Can the PTR method activate the model's progressive refinement ability?

**PTR Activates Progressive Refinement Ability** As shown in Table 1, to emphasize the progressive refinement ability, we conduct tests on a broad range of tasks. The result demonstrates that our PTR activate models substantially refine their responses across multiple iterations in the majority of tasks. For instance, in the Qwen2-7B model, accuracy on MMLU increased by 7.0%, from 57.1% (Base model Prompt Iteration 1) to 64.1% (PTR Iteration 3). On several additional tasks, PTR also showed improvements, with the average score across all tasks increasing by 3.9%-rising from 49.6% to 53.5%. However, the **Prompting** method results show that both two base models degrade in performance when asked to refine, producing worse answers compared to initial responses. These results indicate that PTR effectively enables base models to improve based on previous thoughts.

**PTR vs. Knowledge Distillation (IFT)** We also compare our PTR with Knowledge Distillation (IFT). We find that PTR is not equivalent to knowledge distillation. At the first iteration, we observe that when models are trained on general datasets rather than domain-specific tasks, its initial performance tends to decline at first. We found that this performance drop largely stems from supervised fine-tuning amplifying the initial biases of the base model. When trained on general datasets, the base models tend to accumulate biases that may not apply to specific domains, leading to poorer performance on domain-specific tasks. For example, in ARC tasks, accuracy drops from 60.6% (Base model Iteration 1) to 54.9% (IFT Iteration 1). This demonstrates that our datasets are clean and that knowledge distillation alone cannot improve accuracy in specific domains. Instead of correcting these biases at first response, we focus on correcting them through iterative refinement. However, the IFT approach fails to activate the model's progressive refinement ability and does not significantly increase the performance after the first attempts. On some reasoning benchmarks, such as CommonsenceQA, The IFT approach does not perform a better response at the second iteration (40.3%) compared to their first attempt (46.1%). In contrast, PTR approach improves through iterative attempts without an approach on domain-specific knowledge. This suggests that our method is not simply **distilling knowledge** but effectively **activating** the model to refine outputs and enhance performance through self-driven iterative improvement.

**Refinement beyond Correction** Deeper analysis reveals that in open-ended tasks without clear ground truth, LLMs refine responses to be more thoughtful and comprehensive, regardless of correctness. For example, in the code task shown in Appendix, the LLM iteratively improves its response over three iterations, considering additional aspects of the problem. This highlights PTR's ability to enhance not just correctness but also the quality and usability of outputs (Shown in Appendix).

## Does our method demonstrate generalization?

**PTR vs. Other Progressive Refinement methods** Unlike previous approaches, our method activates the model's inherent progressive refinement ability rather than merely boosting accuracy in specific domains. To validate PTR's generalization capability, we use datasets with general queries and evaluate whether the model can iteratively refine responses across various tasks. As seen in Table 1, our model refines responses across multiple iterations, significantly improving accuracy across tasks, and demonstrating effective generalization. We also compare PTR with other progressive refinement methods like RL to assess generalization. Our results show that methods like RL, when fine-tuned only on general-domain tasks, fails to activate iterative refinement in specialized tasks, often showing decreased accuracy. This suggests that our method is more robust in diverse environments, as it enables the model to iteratively refine its responses without being limited to domain-specific fine-tuning. By leveraging the model's inherent progressive refinement capabilities, PTR achieves consistent improvements across a wide range of tasks.

## Is our method robust across different LLMs and instructions?

**Prompt Robustness** We also evaluated PTR robustness with different prompts and LLMs. Table 2 shows the model's performance using three different prompts across various tasks, refined over four iterations. Across all prompts, we find that PTR achieves iterative improvement across different prompts. Specifically, In the math (GSM8K) tasks, PTR is well-performed(78.1%) compared with initial responses (75.1%). On reasoning tasks (ARC), PTR see substantial improvements, especially with Prompts 1 (62.8%) and Prompts 3 (63.2%). DROP tasks also improve steadily, with accuracy increasing to 22.5% by Iteration 3 in Prompt 2. Our approach enables the model to learn from previous thoughts, rather than relying on the instruction used during training. This PTR enables the model to consistently improve its performance on different prompts, demonstrating the robustness of the PTR mechanism.

**LLMs Robustness** The table also demonstrates that both Llama3-8B and Qwen2-7B exhibit robustness across different prompts and tasks. While Llama3-8B often outperforms Qwen2-7B, both models show consistent improvements with iterative refinement. This robustness ensures that PTR can be applied effectively to a wide variety of open-source LLMs.

[IMAGE: figs/Training_Steps2-14.pdf - Plot A: Multi-line plot showing the performance trends for multiple tasks, along with average performance and variance. Plot B: Bar plot comparing initial, base, and final performance for each task. Plot C: Box plot displaying the performance distribution across tasks. Plot D: Heat map representing task performance across training steps.]

## Does progressive refinement ability exhibit emergence during training?

**Overall Performance** Figures (A) and (B) show a clear upward trend in performance, as shown in Figure 2. Notably, after 24,000 training steps (equivalent to 93 million tokens), significant improvements indicate the emergence of inference capabilities. As training continues, we observe that the average performance of PTR increases from 40.1% to 55.6%, showing an overall improvement across different tasks.

**Task Complexity and Learning Curve** We also find that tasks of varying difficulty exhibit different emergence timings and improvement rates. Plots (C) and (D) reveal that simpler tasks such as MMLU and DROP show early and steady improvements around 22,000 steps. More complex inference tasks such as ARC and GPQA exhibit delayed emergence, with ARC improving from 36.3% to 65.2% and GPQA from 23.2% to 25.6% after 24,000 steps. This shows that as training continues, the model's ability to handle complex reasoning and other tasks significantly improves, showing clear emergent behavior in different task types.

## How many thinking steps are required to achieve optimal performance?

We investigate how iterative thinking steps influence performance across tasks by conducting experiments over ten iterations using the Qwen2-8B model. Figure 3 illustrates performance trends.

**Improvements in the First Three Iterations** In the first three iterations, we saw significant improvements in model performance. In the mathematical reasoning task **GSM8K**, the accuracy improved from 75.0% in the first iteration to 79.9% in the second iteration. Similarly, the **ARC** dataset improves from 58.6% to 65.2% in the third iteration. This shows that PTR quickly refines its problem-solving through progressive refinement.

After the third iteration, the performance improvements for most tasks stabilize. In **GSM8K**, the accuracy fluctuates slightly between the third and tenth iterations, ranging from 79.9% to 80.1%. In **MATH**, the accuracy remains around 50.2% to 50.6% after reaching a peak in the second iteration. This indicates that the marginal gains decrease over time, indicating that the performance ceiling of the model is converging.

**Sustained Performance Without Overfitting** PTR consistently improves across iterations without signs of overfitting. Performance remains stable or improves slightly, with no notable declines. For instance, in **DROP** and **XSum**, accuracy increases from 19.0% and 45.9% to 21.6% and 49.7%, respectively, over ten iterations.

**More Computation for Hard Tasks** Complex tasks benefit more from iterative thinking and may require additional iterations for optimal performance. Accuracy in **CommonsenseQA** improves from 47.9% to 58.6% by the eighth iteration, suggesting that tasks with higher cognitive demands allow PTR to leverage iterative refinement more effectively. While **GSM8K** reaches near-optimal performance within a few iterations, tasks like **MATH** require more computation to achieve substantial gains, likely due to the challenging nature of logical reasoning involved.

[IMAGE: figs/iter.png - Performance of PTR over ten iterations across different tasks. The left plots show accuracy improvements in mathematical reasoning (GSM8K and MATH), reasoning tasks (ARC, GPQA, Winogrande, CommonsenseQA), comprehension tasks (MMLU, DROP, XSum), and coding tasks (HumanEval). Baseline performance is indicated by dashed lines. The right plots show performance over six iterations with radar charts, illustrating improvement over the first three iterations.]

# Conclusion

We propose PTR, an approach designed to stimulate the progressive thought refinement capabilities inherent in LLMs, allowing them to improve their responses through multiple rounds of iterations. PTR adopts an annotation-free strategy to gradually build refined thoughts and answers through a weak and strong models collaborative selection process, and combines thought-answer consistency filtering to ensure logical coherence. Our weighted thought mask fine-tuning further activates the model's internal refinement ability by learning the improvement from initial thoughts to refined answers. Experimental results show that PTR simply trained with general open-domain datasets, but significantly improves the model's progressive refinement capabilities in ten different tasks, including knowledge reasoning, code generation, and mathematical reasoning, achieving a generalization level not observed by previous methods.
