# Abstract

We introduce *Directional Stimulus Prompting*, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model (e.g., T5) to generate an auxiliary *directional stimulus prompt* for each input instance. These directional stimulus prompts act as nuanced, instance-specific hints and clues to guide LLMs in generating desired outcomes, such as including specific keywords in the generated summary. Our approach sidesteps the challenges of direct LLM tuning by optimizing the policy model to explore directional stimulus prompts that align LLMs with desired behaviors. The policy model can be optimized through 1) supervised fine-tuning using labeled data and 2) reinforcement learning from offline or online rewards based on the LLM's output. We assess our method across summarization, dialogue response generation, and chain-of-thought reasoning tasks. Our experiments demonstrate that the framework consistently improves LLMs' (e.g., ChatGPT, Codex, InstructGPT) performance on these supervised tasks using minimal labeled data. Notably, using just 80 dialogues on the MultiWOZ dataset, our approach enhances ChatGPT's performance by an impressive 41.4%, matching or surpassing some fully supervised start-of-the-art models. Additionally, the instance-specific chain-of-thought prompt generated by our approach improves InstructGPT's reasoning accuracy compared to human-crafted or automatically generated prompts. The code and data are publicly available.

[IMAGE: Comparison of Directional Stimulus Prompting and standard prompting using LLMs such as ChatGPT for the summarization task. DSP utilizes directional stimulus/hints (highlighted in orange), which are keywords in this case, to provide instance-specific guidance to LLMs in generating summaries (highlighted in blue) that better align with the desired reference summary with higher ROUGE scores or other measures like human preferences.]

# Introduction

In recent years, a new paradigm has emerged in natural language processing (NLP) with the rise of large language models (LLMs) such as Codex [codex], InstructGPT, ChatGPT [instructgpt], GPT-4 [openai2023gpt4], PaLM [palm], and others. These models exhibit emergent abilities [emergent] such as strong in-context learning and few-shot prompting capabilities, which were not present in previous "smaller" language models (LMs) like BERT [bert], RoBERTa [roberta], GPT-2 [gpt2], and T5 [t5]. This shift in paradigm has led to remarkable advancements in NLP, with LLMs demonstrating impressive general-purpose power. However, due to commercial considerations and the risk of misuse, most LLMs do not publicly release their parameters and only allow users to access them through black-box APIs. While there also exists open-sourced LLMs, fine-tuning them for specific tasks or use cases can be computationally inefficient. In this scenario, the standard approach for utilizing LLMs to perform diverse tasks is crafting task-specific text prompts to query LLMs through black-box APIs. While LLMs have demonstrated considerable performance on a wide range of language tasks, they still struggle to generate outputs that fully align with desired behaviors and directions on some specific tasks and use cases [gpt3sum; evalchatgpt].

Since directly optimizing LLMs for specific tasks is either inefficient and infeasible for most users and developers, researchers resort to optimizing prompts instead. Prompt engineering approaches, which involve manually or automatically designing optimal task-specific natural language instructions and selecting appropriate training samples for demonstration in the prompt, have been the focus of many researchers [gpt3; promptprogramming; ape; promptpg]. Despite these efforts, effectively steering LLMs to generate desired results and effectively exploiting labeled data remains a significant challenge.

To address the challenge, we propose a novel framework called **Directional Stimulus Prompting (DSP)**. This framework introduces a new component called the "*directional stimulus*" into the prompt to provide nuanced, instance-specific guidance and control over LLMs. Specifically, the directional stimulus prompt acts as "hints" and "clues" for the input query to guide LLMs toward the desired output. Notably, this differs from the methods that augment LLMs with additional knowledge retrieved from external sources [dsp; replug], as the directional stimulus prompt is generated solely based on the input query in our framework. Figure 1 compares our proposed prompting approach, DSP, with standard prompting for the summarization task. Our approach incorporates keywords in the prompt as the directional stimulus prompt to hint at key points the desired summary should cover. By providing this instance-specific guidance through directional stimulus prompt, LLMs can generate outputs that more closely align with the desired reference summary.

We utilize a relatively small and tunable LM (e.g., T5), as the policy model to generate the directional stimulus prompt for each input query. This approach enables us to sidestep the direct optimization of black-box LLMs by optimizing the small tunable policy model instead. We train the policy model through supervised fine-tuning (SFT) using a few collected labeled data. After supervised fine-tuning, we further optimize the policy model to explore better directional stimulus prompts with reinforcement learning (RL). During RL training, we aim to maximize the reward defined as downstream performance measures or any other measures of the LLM's output conditioned on the stimulus generated by the policy model.

Figure 2 provides the overview of our framework, using the summarization task as an illustrative example. We employ a compact, tunable policy model to generate the directional stimulus prompt, which specifies keywords that should be included in the LLM-generated summaries. The policy model can be trained with SFT and RL, where the reward is typically defined as the downstream task performance measure, such as the ROUGE score for the summarization task, or other alignment measures like human preferences.

Our framework can be flexibly adapted to a wide range of LLMs and tasks by choosing the appropriate directional stimulus and associated rewards. We conducted experiments on summarization, dialogue response generation, and chain-of-thought reasoning tasks to evaluate the effectiveness of our framework. Our results demonstrate that our DSP approach can effectively guide ChatGPT toward the desired targets with a small collection of labeled data. Specifically, we conduct experiments with the black-box LLMs: ChatGPT, Codex, and InstructGPT. For the policy model, we employ a 750M Flan-T5-Large [t5; flan] and 220M T5-Base. For the summarization task, we use keywords as the directional stimulus, which hints at key points that the desired summary should include. Despite ChatGPT's already considerable performance, the policy model T5 trained with only 4,000 samples from the CNN/Daily Mail dataset [cnndm] improved the ROUGE and BLEU scores by 4-13%. For the dialogue response generation task, we train the policy model to generate dialogue acts that indicate the underlying intentions behind target responses on dialogues from MultiWOZ dataset [multiwoz]. Guided by the policy model trained with only 80 dialogues, ChatGPT's performance improved by up to 41.4% in combined scores, achieving comparable or even better performance than some state-of-the-art models trained on the full dataset with 8,438 dialogues. For the chain-of-thought reasoning, we train the policy model to generate a trigger prompt for each input query to trigger the LLM chain-of-thought reasoning, achieving better performance than the hand-crafted and automatically generated prompts.

[IMAGE: Overview of proposed framework DSP, where we learn a small tunable policy model to generate the directional stimulus (keywords in this case) that provide input-specific guidance for the LLM toward the desired target. The policy model can be trained with SFT and/or RL, where the reward is defined as the downstream task performance measure, such as the ROUGE score for the summarization task, or other alignment measures like human preferences.]

# Directional stimulus prompting

For a downstream task, there is an input space X, a data distribution D over X, and an output space Y. Due to the strong in-context learning and few-shot prompting abilities, LLMs can perform diverse tasks and generate the output y by including instructions that describe the task, a few demonstration examples, and the input query x in the prompt [gpt3]. However, such prompts cannot always steer LLMs toward desired outputs, especially when it comes to fine-grained instance-specific desired behaviors. For instance, in the case of the summarization task, the input x is an article, and the output y is the corresponding summary. Different summarizers have distinct styles and emphasize different aspects of an article [gpt3sum]. In this case, it may not be enough to effectively steer LLMs toward generating summaries that closely match reference summaries relying solely on task-specific instructions or demonstration examples to describe such nuanced differences for each sample.

To this end, our Directional Stimulus Prompting (DSP) approach introduces a small piece of discrete tokens z named "*directional stimulus*" into the prompt, which acts as hints and clues to provide LLMs with fine-grained guidance toward the desired direction. For example, for the summarization task, the directional stimulus z might consist of keywords that should be included in the desired summary. To generate this stimulus for each input query, we use a small tunable policy language model, p_POL(z|x). We then use this generated stimulus, z, along with the original input, x, to construct the prompt that steers the LLM toward generating its output, p_LLM(y|x,z), through black-box API calls. It's important to note that the parameters of the LLM, p_LLM, are not accessible or tunable. Overall, when using the LLM with DSP to perform a downstream task, the output is obtained via y ~ p_LLM(.|x,z), z ~ p_POL(.|x).

## Supervised fine-tuning

To train the policy model that generates directional stimulus for LLMs, we first perform supervised fine-tuning (SFT) on a pre-trained LM (e.g., T5, GPT-2, etc) on a small collection of labeled data. To collect the data, we could heuristically select or annotate the "pseudo-stimulus" z* for each input query x and target output y pair based on the downstream task. For example, for the summarization task, we use keywords that the reference summary includes as pseudo-stimulus, while for the dialogue response generation task, we use dialogue acts that indicate the underlying meaning of the desired system response (see Section 3 for details). The resulting dataset D'={(x, z*)} consists of input-stimulus pairs. We then fine-tune the policy model by maximizing the log-likelihood:

```latex
$$\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{({\bm{x}},{\bm{z}}^*) \sim \mathcal{D}'}\text{log}p_{\text{POL}}({\bm{z}}^*|{\bm{x}}).$$
```

Supervised fine-tuning can provide a good initial point for the policy model. However, it is important to note that the heuristically selected or annotated pseudo-stimulus may not always be optimal, and the supervised fine-tuned policy model may not generate the most preferred directional stimulus for the LLMs toward the desired outputs. To overcome this limitation, we can also incorporate reinforcement learning (RL) to further fine-tune the policy model. By directly optimizing the LLM's output toward desired targets, RL training enables the policy model to explore and generate more effective directional stimulus.

## Reinforcement learning

**Optimization objective** Our goal is to steer the LLM's generation toward the desired target by maximizing an alignment measure R, which can take various forms such as downstream task performance measures (e.g., ROUGE score for summarization), human preferences, or other customized measures. Mathematically, we aim to maximize the below objective:

```latex
$$\mathbb{E}_{{\bm{x}}\sim\mathcal{D},{\bm{z}}\sim p_{\text{POL}}(\cdot|{\bm{x}}), {\bm{y}}\sim p_{\text{LLM}}(\cdot|{\bm{x}},{\bm{z}})}[\mathcal{R}({\bm{x}},{\bm{y}})].$$
```

Since the parameters of the black-box LLM are not accessible or tunable, we resort to optimizing the policy model to generate the directional stimulus that guides the LLMs' generation toward maximizing the objective. To achieve that, we define another measure R_LLM that captures how well the LLM performs when conditioned on a given stimulus z:

```latex
$$\mathcal{R}_{\text{LLM}}({\bm{x}},{\bm{z}})=\mathcal{R}({\bm{x}},{\bm{y}}), {\bm{y}}\sim p_{\text{LLM}}(\cdot|{\bm{x}},{\bm{z}}).$$
```

This allows us to cast the original objective of maximizing R into optimizing the policy model to generate stimulus that maximizes R_LLM. By doing so, the LLM is effectively used as an evaluation function to guide the policy model toward generating more effective directional stimulus. Thus, the optimization objective for LLMs in the equation above is equal to the optimization objective for the policy model:

```latex
$$\text{max}_{p_{\text{POL}}}\mathbb{E}_{{\bm{x}}\sim\mathcal{D},{\bm{z}}\sim p_{\text{POL}}(\cdot|{\bm{x}})}[\mathcal{R}_{\text{LLM}}({\bm{x}},{\bm{z}})].$$
```

**RL formulation** However, the above optimization is intractable for the policy model. To address the issue, we formulate the policy model optimization as an RL problem and employ proximal policy optimization (PPO) [ppo]. We use the policy model to initialize a policy network pi_0 = p_POL and then update pi using PPO. The process that the policy model generates a sequence of tokens as stimulus z can be seen as a Markov decision process (MDP) <S, A, r, P> with a state space S, action space A, reward function r, and state-transition probability P. In each time step t of an episode, the agent selects an action (token) from the vocabulary V according to the distribution of the current policy network pi(z|x, z_<t). The episode ends when an end-of-sequence token is selected, and the stimulus z is generated. We can fine-tune the policy network pi by optimizing the reward r:

```latex
$$\mathbb{E}_{\pi}[r] = \mathbb{E}_{{\bm{x}}\sim\mathcal{D},{\bm{z}}\sim\pi(\cdot|{\bm{x}})}[r({\bm{x}},{\bm{z}})].$$
```

**Reward function** Recall that our goal is to maximize the objective in the equation above, which can be used as the reward r. To keep the policy network pi from moving too far from the initial policy model p_POL, we also add a KL-divergence penalty reward. Therefore, the final reward becomes:

```latex
$$r({\bm{x}},{\bm{z}})=\mathcal{R}_{\text{LLM}}({\bm{x}},{\bm{z}})-\beta\text{log}\frac{\pi({\bm{z}}|{\bm{x}})}{p_{\text{POL}}({\bm{z}}|{\bm{x}})}.$$
```

Following [rlhf1; rl4lm], we dynamically adapt the coefficient beta during training:

```latex
$${\bm{e}}_t = \text{clip}\left ( \frac{\text{KL}(\pi_t,p_{\text{POL}})-\text{KL}_{\text{target}}} {\text{KL}_{\text{target}}}, -0.2, 0.2 \right ),$$
$$\beta_{t+1} = \beta_{t} \left (1+K_{\beta}{\bm{e}}_t \right ).$$
```

#### Implementation

To optimize the policy network pi, we use the NLPO version of PPO from [rl4lm], which is specifically designed for language generators. To address the issue of large action spaces in PPO, NLPO learns to mask out less relevant tokens in the vocabulary using top-p sampling. This technique restricts the action space to the smallest set of tokens whose cumulative probability is greater than the given probability parameter p, which we set to 0.9 in our experiments. Both the policy network pi and value network are initialized from the supervised fine-tuned policy model p_POL, with the final layer of the value network randomly initialized to output a scalar value using a regression head.

# Experiments

Our proposed framework DSP can be flexibly applied to various types of LMs and generation tasks. In this work, we focus on the summarization, dialogue response generation, and automatic prompt generation tasks. We mainly use pre-trained T5 or Flan-T5 [t5; flan] to initialize the policy model and evaluate the OpenAI's **ChatGPT (`gpt-3.5-turbo`)**, **Codex (`code-davinci-002`)**, and **InstructGPT (`text-davinci-002`)**. Our experiments aim to assess the effectiveness of our approach in guiding the generation of black-box LLMs toward desired outputs.

## Summarization

Recent studies [gpt3sum; benchmarkinggpt; evalchatgpt] have shown that LLMs, such as GPT-3, InstructGPT, and ChatGPT, are capable of generating high-quality summaries with zero- or few-shot prompting. However, their reference-based evaluation benchmark performances, such as ROUGE scores, still lag behind fine-tuned methods, indicating that the generated summaries may not completely match the style and emphasis of the reference summaries. In our experiments, we seek to guide LLMs to generate summaries that more closely align with the reference summaries by providing keywords that should be mentioned in the desired summaries as hints. We evaluate the effectiveness using metrics that compare the generated summaries against reference summaries. Notably, other desired directions, such as better alignment with human preferences, can also be pursued.

**Dataset and evaluation** We conduct our experiments on the CNN/Daily Mail dataset, a widely-used news summarization benchmark. To keep the cost of API usage low, we train on a subset of 1,000, 2,000, and 4,000 article-summary pairs from the total 287,113 samples in the training set. For evaluation, we randomly select 500 samples, following previous work [gpt3sum; mbrd], which has been proven to provide sufficient statistical power [card2020little]. We use the overlap-based metrics, including ROUGE [rouge], BLEU [bleu], and Meteor [meteor], and the similarity-based metric, BERTScore [bertscore], to compare the generated summaries with the references. The reported evaluation scores are averaged over three inferences of ChatGPT for each query, using a temperature of 0.7 and top_p of 1.0. We use the same three demonstration examples in the prompt for standard prompting and add keywords as directional stimulus in the prompt for our approach, DSP. The exact prompts used in our experiments are provided in the Appendix.

**Supervised fine-tuning details** We use keywords as the pseudo-stimulus to train the policy model with supervised fine-tuning as discussed in Section 2.1. To collect the data, we employ textrank [textrank; textrankVariants] to automatically extract the keywords from the article and summary and only keep those that appear in the reference summary. As a result, we obtain a list of extracted keywords for each article-summary pair in the dataset. To convert them into a sentence that serves as the stimulus, we concatenate them using a split token ";", resulting in the stimulus formated as "[Keyword1]; [Keyword2]; ... ; [KeywordN].". We use the constructed article-stimulus pairs to train the policy model via supervised fine-tuning. The input format for training is "Extract the keywords: [Article]", while the output is the target stimulus consisting of keywords. The policy model was trained for 5 epochs with a 2e-5 learning rate.

**RL training details** As we aim to guide ChatGPT in generating summaries that more closely match the reference summaries, we adopt the automatic reference-based metric scores as the alignment measure reward. Specifically, we calculate the ROUGE-Avg score between the generated summaries and the reference summaries as the reward, with a rescaling coefficient of 10. We experimentally found that other automatic evaluation metrics, such as BLEU and Meteor, perform similarly. To reduce variance, we generate four outputs per input query using ChatGPT with a temperature of 0.7 and compute the average reward. Additionally, we assign a step-wise reward, which we found could improve the efficiency and stability of the training process. Specifically, the policy model generates a sequence of keywords in each episode, during which we assign a reward of 1 if a keyword appears in the reference summary and a penalty reward of -0.2 is given otherwise. We train the policy network for 51k episodes, with 5 epochs per batch, a batch size of 8, and a learning rate of 2e-6. The KL_target and beta_0 are set to 0.5 and 0.005, respectively.

[IMAGE: Performance comparison of ChatGPT with standard prompting and DSP trained with SFT and SFT+RL, using varying numbers of training samples from the CNN/Daily Mail dataset.]

[IMAGE: Training curve showing reward and ROUGE-1 score on validation set during training process on 1,000 samples.]

**Results** We evaluate the performance of ChatGPT with standard prompting and our approach DSP trained with SFT or SFT and then RL (SFT+RL) on varying sizes of training data and present the results in Figure 3. As can be seen, all the evaluation scores improve with our proposed DSP compared with standard prompting. Specifically, the supervised fine-tuned policy model generates the stimulus that effectively guides ChatGPT to generate summaries that closely align with the reference summaries, leading to improved benchmark performance. Furthermore, the additional fine-tuning of the policy model with RL results in further performance improvement, indicating the effectiveness of RL in exploring better directional stimulus that maximizes the reward. As the size of the training data increases, the performance improvement becomes more significant. Despite using a small collection of only 1,000 to 4,000 samples to keep API usage costs low, our DSP approach still consistently enhances ChatGPT's ROUGE, BLEU, and Meteor scores by 1-2 points, even though ChatGPT has already achieved considerable performance. However, due to the discrepancy between the semantic-based metric BERTScore and the overlap-based metric ROUGE, which are used as the reward, the improvement in BERTScore after RL training may be relatively less significant. The training is relatively stable using the NLPO algorithm.

## Dialogue response generation

In recent years, there has been a rise in LLM-based chatbots such as ChatGPT and Sparrow. These chatbots are typically targeted at open-domain conversations to engage with users on a wide range of topics without a specific goal in mind. However, these chatbots still face challenges in handling task-oriented dialogues where they need to assist users in completing specific goals or tasks, such as making reservations or ordering food [evalchatgpt; llm4tod]. Unlike open-domain conversations, task-oriented dialogues often require the chatbot to follow task-specific business logic and respond based on reliable information from API calls or database queries. To address this limitation, we train a small policy model to learn the underlying dialogue policy from the training data and thus guide the LLMs in generating reliable system responses that assist users in completing tasks.

**Dataset and evaluation** We conduct experiments on the popular task-oriented dialogue dataset MultiWOZ [multiwoz], including both the MultiWOZ2.0 (the original version) and MultiWOZ2.1 version [multiwoz21]. The dataset provides annotations for user utterances, dialogue acts, and system responses for each dialogue turn. The goal is to generate the system response given the history dialogue context as input. We utilize the dialogue act, which represents the communicative intention of the target system response, as the pseudo-stimulus for our experiment. There are 8,438 dialogues in the training set. We only use 1% (80 dialogues) and 10% (800 dialogues) to train the policy model and evaluate the performance on the full validation and test set, which contains 1,000 dialogues. We use the standard evaluation metrics: **Inform**, which measures the rate that the appropriate entity that satisfies the user's requirements is provided; **Success**, which measures the rate that all requested attributes are answered; **BLEU**: the corpus-level BLEU score with reference responses; and an overall measure **Combined score** = (Inform+Success)*0.5+BLEU. Likewise, we report the average score over three inferences. We use the same three demonstration examples when using DSP or standard prompting.

**Supervised fine-tuning details** To conduct supervised fine-tuning on the policy model, we format the input of each sample as "Translate dialogue to dialogue action: [Dialogue context]", with the target being the verbalized dialogue acts in the same format as [damd; pptod]. For instance, a dialogue act <hotel, inform, choice>, <hotel, inform, type>, <hotel, request, area> will be converted to "[hotel] [inform] choice type [request] area", which indicates that the system should inform available hotel choices and their types and ask for the area that the user would like (see the Appendix for examples). Note that the provided dialogue act annotations may not be the only valid dialogue act for the same dialogue content [damd], and thus we hope to explore diverse valid dialogue acts (directional stimulus) through RL training.

**RL training details** The evaluation metrics Success and Inform rates are defined at the dialogue level, while the BLEU score is computed on the corpus level. However, our training and inference on conducted on the turn level. We thus use the sentence-level SacreBLEU [sacrebleu] score as the reward. Same as in the summarization experiments, we generate four outputs per input using the LLM with a temperature of 0.7. The policy network is trained 52k episodes, 5 epochs per batch with a batch size of 8 and a learning rate of 2e-6. Since the generated dialogue acts should adhere to the business logic and ontology, we ensure that the updated policy network does not deviate significantly from the original policy model. We thus set the KL_target and beta_0 as 0.2 and 0.01, respectively. During training, we use top-k sampling and set k to 50 to explore the action space. During inference, we use beam search decoding with a beam size of 5.

**Results** We evaluate the impact of our approach DSP on Codex and ChatGPT and compare the performance with several representative task-oriented dialogue models trained on the full training set (8438 dialogues), including DAMD [damd], MinTL [mintl], Soloist [soloist], SimpleTOD [simpletod], DoTS [dots], PPTOD [pptod], UBAR [ubar], and GALAXY [galaxy]. The results show the following observations: (1) Our approach DSP significantly improves the success and inform rates of Codex and ChatGPT, indicating that they better understand the scenario and generate appropriate responses that help users in completing their tasks. (2) However, there is no improvement in the corpus-level BLEU score, possibly because the LLMs generate responses with different speaking styles and vocabulary since they do not see oracle system responses. Nevertheless, the high success and inform rates demonstrate the usefulness of our approach in delivering helpful and reliable responses. (3) Increasing the number of supervised fine-tuning samples does not guarantee performance improvement, but further fine-tuning the policy model using RL consistently provides performance gains. This suggests that RL training encourages the policy model to explore more model-preferred stimulus, while supervised fine-tuning may merely generate stimulus closely aligned with the pseudo-labeled data, which is not necessarily optimal. (4) Our approach achieves notable success with only 80 dialogues, surpassing several fully trained TOD models, particularly in terms of Success and Inform rates. With 10% of the training data (800 dialogues), our approach delivers comparable performance to current SOTA methods trained with full training data (8438 dialogues). We have also provided the performance of these compared methods in the low-resource settings (1% and 10%) and a running example in the Appendix.

## Chain-of-Thought reasoning

While current methods primarily use general task-specific prompts, LLMs show sensitivity to them. Studies [cot; zero-shot-cot; ape] demonstrate that LLMs can vary in performance based on the prompt used. As a result, much of the previous work has centered on manually [reynolds2021prompt] or automatically [autoprompt; ape] crafting better prompts. However, these efforts mainly focus on task-specific prompts, which may not be optimal for every instance of a task. In our experiment, we employ our approach to generate instance-specific trigger prompts to elicit Chain-of-Thought (CoT) reasoning. Specifically, we train a policy model (`t5-base`) to generate instance-specific CoT trigger prompts, such as "Let's think step by step", to optimally prompt varying samples.

**Dataset and evaluation** We adopted the experimental setup from previous work [zero-shot-cot; ape], where we tested zero-shot CoT reasoning abilities of InstructGPT (`text-davinci-002`) with different trigger prompts. There are 600 examples in the MultiArith dataset [multiarith], which we divided into 300/50/250 for training/validation/test set. As for the AQuA dataset [aqua], we use the standard test set with 254 samples, 300 samples from the standard training set for our training, and 100 samples for the standard validation set for our validation. We report the reasoning accuracy.

**Supervised fine-tuning details** For supervised fine-tuning (SFT), we first run inference on the training set with the 14 human-crafted prompts tested in [zero-shot-cot], respectively. We then selected those prompt and query pairs which resulted in a correct CoT reasoning outcome to form the training set for SFT. These query-prompt pairs were used to train a t5-base policy model for 2 epochs, with the model input being the query instance and the target output a trigger prompt.

**RL training details** After SFT, the prompts generated by the policy model were used to trigger InstructGPT for zero-shot CoT prompting. Reasoning accuracy was utilized as the reward for reinforcement learning (RL). A reward of 1 was assigned for correct reasoning results and 0 otherwise. We conducted 20 training iterations (106k episodes), with 5 epochs per batch, a batch size of 8, and a learning rate of 2e-6. The parameters for KL_target and beta_0 were set to 0.5 and 0.001, respectively.

**Results** We compare the performance of using our generated instance-specific prompts with using the 14 human-crafted prompts which we used as the pseudo-stimulus to constitute the training set for SFT and also the prompt automatically discovered by the APE approach [ape]. Note that all these 15 prompts are general task-specific and are used for the whole test set while ours are instance-specific. The results show that InstructGPT's performance varies significantly when using different task-specific prompts. Compared to the 14 task-specific human-designed prompts, DSP enhances the performance with instance-specific prompts. It also outperforms the prompt discovered by the APE approach. Solely relying on supervised fine-tuning of the policy model with the dataset comprising the 14 human-designed prompts doesn't lead to its peak performance. After fine-tuning with RL, the policy model is encouraged to explore better instance-specific trigger prompts, further improving performance.

**Table: Zero-shot chain of thoughts performance of InstructGPT (`text-davinci-002`) with different prompts:**

| No. | Category | Chain-of-Thought Trigger Prompt | MultiArith | AQuA |
|-----|----------|--------------------------------|------------|------|
| 1 | Human-Designed | *Let's think step by step.* | 79.6 | 31.9 |
| 2 | | *We should think about this step by step.* | 81.2 | 28.7 |
| 3 | | *First,* | 78.0 | 38.2 |
| 4 | | *Before we dive into the answer,* | 54.8 | 27.2 |
| 5 | | *Proof followed by the answer.* | 58.4 | 37.8 |
| 6 | | *Let's think step by step in a realistic way.* | 59.6 | 33.9 |
| 7 | | *Let's think step by step using common sense and knowledge.* | 80.0 | 34.3 |
| 8 | | *Let's think like a detective step by step.* | 73.6 | 24.0 |
| 9 | | *Let's think about this logically.* | 75.2 | 34.7 |
| 10 | | *Let's think step by step. First,* | 78.8 | 32.3 |
| 11 | | *Let's think* | 56.8 | 38.2 |
| 12 | | *Let's solve this problem by splitting it into steps.* | 72.4 | 33.2 |
| 13 | | *The answer is after the proof.* | 42.8 | 34.3 |
| 14 | | *Let's be realistic and think step by step.* | 69.6 | 29.9 |
| 15 | APE [ape] | *Let's work this out in a step by step way to be sure we have the right answer.* | 81.6 | 34.3 |
| 16 | DSP w/ SFT | (*Generated instance-specific prompt) | 75.2 | 35.8 |
| 17 | DSP w/ SFT+RL | (*Generated instance-specific prompt) | **84.0** | **38.6** |

# Related work

**Black-box large language models** Recent years have witnessed the emergence of LLMs such as GPT-3 [gpt3], Codex [codex], InstructGPT, ChatGPT [instructgpt], PaLM [palm], and LaMDA [lamda], which show significant promise in the field of NLP. These LLMs typically have a large number of parameters and require vast amounts of training data. Due to their scaling, these models have exhibited many emergent abilities, such as in-context learning, few-shot prompting, chain-of-thought prompting, and instruction following [gpt3; instructgpt; cot]. However, most LLMs are not open-sourced and can only be accessed via black-box APIs, through which the users send prompt queries and receive responses. While there exist open-source LLMs such as OPT-175B [opt] and Bloom [bloom], their local execution and fine-tuning require significant computational resources that may be infeasible for most researchers and users. However, despite their considerable performance on various tasks, LLMs often fall short of generating outputs that fully align with desired outputs on specific downstream tasks and use cases [gpt3sum; moradi2021gpt; gutierrez2022thinking]. Our approach seeks to address this limitation by introducing directional stimulus generated by a small tunable LM into the prompt to provide more fine-grained guidance and control over black-box LLMs.

**Prompt optimization and engineering** Efficiently optimizing pre-trained LMs on downstream tasks by finding optimal prompts has been a focus of prior research. One approach involves tuning soft prompts, which are continuous embedding vectors that can be optimized using gradient descent methods [prefixtuning; softprompt; spot; inputtuning; bbt]. However, the requirements of gradients and the challenge of passing gradients and continuous prompts through black-box APIs, making them less practical for the black-box LLMs. Researchers have also tried to seek optimal prompts by designing task-specific natural language instructions and selecting proper training samples as in-context demonstrations in the prompt. These methods include manual engineering [lmkb; gpt3; reynolds2021prompt], editing [autoprompt; tempera], reinforcement learning [rlprompt; promptpg], and automatic generation [ape]. Despite these efforts, such prompts are not always effective at steering LLMs to generate desired outputs, especially for fine-grained instance-specific behaviors that are difficult to describe using task-specific instructions and demonstration examples. To address this limitation, our approach is able to provide more **fine-grained instance-specific** guidance generated by a small tunable policy model optimized with supervised fine-tuning and/or reinforcement learning.

**Controllable text generation** The control of language models (LMs) has been extensively studied. Early approaches fine-tuned LMs on datasets containing desired attributes [gururangan2020don]. [ctrl] proposed class-conditioned LMs, generating text with predefined control codes. However, direct LM training is costly. To address this, PPLM [pplm] trains an attribute model and passes gradients to control generation. GeDi [gedi] and DExperts [dexperts] use class-conditional distributions as generative discriminators to guide generation, reducing computation complexity. These methods require either additional LM training or internal gradients and logistics, making them not applicable to black-box LLMs. Our approach proposes a solution to control black-box LLMs by inserting directional stimulus into the input query prompt and optimizing based on the return output.

**Reinforcement learning for NLP** Reinforcement learning has been successfully applied to various NLP tasks, such as syntactic parsing [rl4parser; rl4sp], machine translation [wu2016google; kumar2019reinforcement], summarization [rl4sum; rlhf4sum], conversational systems [rl4dialog], etc. Language models define probability distributions over tokens in their vocabulary, and the text generation problem can be naturally formulated as selecting an action in an RL setting. Therefore, there have been extensive research efforts on optimizing LMs with RL, usually by aligning them with human preferences [rlhf1; rlhf2; quark; rlhf4sum]. For example, the LLM InstructGPT [instructgpt] is optimized with RL to better follow users' instructions and intent. In contrast with these works that directly update the LLMs to align with human preferences, our work optimizes a small policy model that generates text (stimulus) to guide LLMs to generate more human-preferred output instead of directly optimizing the LLMs, bypassing the inefficient LLM's optimization.

# Conclusions and future work

In this paper, we introduce *Directional Stimulus Prompting* (DSP), a new prompting framework to provide black-box LLMs with fine-grained and instance-specific guidance toward the desired outputs. We use a tunable policy model to generate the directional stimulus to provide such guidance and convert the optimization of black-box LLMs to that of the policy model. Experimental results demonstrate the effectiveness of our approach. DSP not only enables better control and guidance for black-box LLMs, but also effectively utilizes labeled data. Furthermore, the generated stimulus provides valuable insights and interpretations of LLMs' behaviors. In this work, we use heuristically selected or annotated pseudo-stimulus data for supervised fine-tuning of the policy model. For future work, we hope to explore the possibility of using a "machine language" between the policy model and the LLMs that might not be intuitively preferred by humans but can better convey guidance information, as well as other forms of directional stimulus beyond text.

# Appendix A: Implementation Details

## A.1 Summarization

We use the representative benchmark dataset CNN/Daily Mail for news summarization [cnndm]. This dataset contains 287,113 training examples, 13,368 validation examples, and 11,490 test examples. To keep the API usage cost low, we use a subset of 1,000, 2,000, and 4,000 for training, 500 for validation, and 500 for testing. Each example in the dataset consists of a news article along with its corresponding highlight/summary written by human authors. In order to train the policy model through supervised fine-tuning, we employed the textrank [textrank] algorithm to automatically extract keywords from each article and only retained those mentioned in the corresponding reference summary. We initialize the policy model using the 780M FLAN-T5-large model [flan; t5], and use it to guide the black-box LLM ChatGPT. All the experiments are run on a server equipped with 8 NVIDIA RTX A6000 GPUs.

## A.2 Dialogue response generation

The MultiWOZ dataset is a widely-used task-oriented dialogue dataset consisting of 8,438 dialogues for training, 1,000 dialogues for validation, and 1,000 dialogues for testing. For each turn of the dialogues, in addition to the user utterances and system response, the annotations of belief state, database query results, and dialogue act are also provided. To process the data, we followed the approach used in UBAR [ubar]. Specifically, we employed delexicalization by replacing specific slot values with corresponding placeholders. These placeholders can be filled based on the results of a database search. The annotated dialogue acts serve as the stimulus in our approach.

We used 780M Flan-T5-Large for our policy model to guide the ChatGPT and Codex LLMs. During the supervised fine-tuning of the policy model, we trained it to generate stimulus converted from the dialogue acts based on the given dialogue context. The policy model was trained for 25 epochs using 80 dialogues from the MultiWOZ2.0 and MultiWOZ2.1 datasets. When 800 dialogues are given, it was trained for 8 epochs on the MultiWOZ2.0 dataset and 20 epochs on the MultiWOZ2.1 dataset.

## A.3 Chain of Thought reasoning

We use our approach to generate instance-specific chain of thought (CoT) trigger prompts. Following previous work [zero-shot-cot; ape], we evaluate two widely used arithmetic reasoning datasets MultiArith [multiarith] and AQuA [aqua]. We compare with the 14 human-crafted chain-of-thought prompts evaluated in [zero-shot-cot], part of which are collected from [ahn2022can; reynolds2021prompt]. We also compare with the prompt automatically designed by the APE approach [ape].

# Appendix B: Additional results

## B.1 Summarization

**Analysis of generated hints/keywords** We outlined changes in the number of generated keywords, hit keywords (those matched in the reference summary), and corresponding ROUGE-1 scores throughout the training process. As the training progresses, the policy model appears to generate keywords with increasing precision, which aligns positively with the increasing ROUGE-1 score. However, it is observed that even when keywords are generated with high precision if their quantity is too limited, the performance doesn't necessarily improve.

We also employed the spacy package [spacy2] for Part-of-Speech (POS) and Named Entity Recognition (NER) tagging on the generated keywords. For the POS tagging, we observe that nouns (NOUN) and proper nouns (PROPN) are the most frequently generated keywords, which can serve as informative keywords. As for the NER tagging, the most commonly generated keywords include persons (PERSON), geopolitical entities (GPE), dates (DATE), organizations (ORG), and numerals (CARDINAL).

**GPT-4 Evaluation** To gain a better understanding of generated summaries guided by keywords, we employed GPT-4 to evaluate the summaries. It has been shown that the LLM, especially GPT-4 is able to produce consistently high-quality assessments of text generation, showing high human alignment and thus being a good alternative to human evaluations [llm-judge; gpt4-eval]. As we employ ROUGE scores as rewards for tuning the policy model to generate keywords that guide the LLM towards generating summaries more aligned with the reference summary, we leveraged GPT-4 to assess the overlap of key points (hints) between generated and reference summaries. Specifically, we use GPT-4 to compare the summaries generated with our proposed DSP and the original standard prompting. GPT-4 was instructed to first generate an explanation, followed by the corresponding answer.

The GPT-4 evaluation results show that GPT-4 can produce reasonable and detailed explanations of their assessment. From our test set of 500 samples: DSP-generated summaries were favored 255 times (51.0%), summaries generated with original standard prompting were favored 222 times (44.4%), while a tie was observed in 23 cases (4.6%).

**Zero-shot prompting** In our main experiments, we employ few-shot prompting with 3 examples in the prompt during training and evaluation. To test whether the approach performed well under the zero-shot setting, we evaluated the following two experimental settings on the CNNDM dataset with 4,000 training samples: (1) few(3)-shot during training and zero-shot during evaluation; and (2) zero-shot during both training and evaluation.

Results show that when both training and testing are conducted using zero-shot prompting, the performance improvement over standard prompting is still comparable to the scenario where both are conducted using few-shot prompting. In addition, we observed that our approach exhibits robustness when different numbers of examples are used in prompts during training and evaluation, as our approach with few(3)-shot training still outperforms standard prompting under zero-shot testing.

## B.2 Dialogue response generation

**Low-resource results** In addition to the performance of compared baseline models with full training data as shown in the main paper, we also present their performance in the low-resource setting. It is important to note that most of these methods struggle to achieve acceptable performance with only 1% of the training data (80 dialogues), and thus their results in the 1% setting are not reported. As for those with reported performance with 80 dialogues, their results are significantly worse compared to Codex and ChatGPT guided by the policy model. Furthermore, even with around 800 dialogues, their Inform and Success rates were still much lower than those achieved by ChatGPT and Codex.

## B.3 Chain-of-Thought reasoning

**Newly discovered prompts** After fine-tuning with RL, the policy model is encouraged to discover instance-specific trigger prompts. Some of these newly generated trigger prompts deviate from those present in the training data for SFT. Some are modifications or combinations of prompts from the training data, such as "First step:" and "Let's think like a detective step by step. First,". Others include new information, like "Let's think step by step using both the above information and the testing." and "Let's think step by step using proven methods.".

**Table: Some generated trigger prompts by the fine-tuned policy model:**

- *First step:*
- *Let's think like a detective step by step. First,*
- *Let's solve this problem by splitting it into steps. First,*
- *Let's think step by step using common sense.*
- *Let's think step by step using our creative brains.*
- *Let's think step by step using both the above information and the testing.*
- *Let's think step by step using proven methods.*
- *The answer is following the proof.*

# Appendix C: Running examples

**Example from CNN/Daily Mail dataset:**

| | |
|---|---|
| **Input article** | The winter of 2014-15 won't be easily forgotten in Boston after the endless snow broke countless records and the city had to pay volunteers $30 an hour to help dig out the battered city. The shere volume of snow that fell earlier this year, nearly 65 inches fell in February alone, means that huge piles of the white stuff still remain. Except the remaining 'snow' isn't very white any more but rather a disgusting black color riddled with trash including broken pieces of glass, plastic shards and goodness knows what else. Vlad Tarasov couldn't resist filming himself ski down the slopes at Boston's largest snow farm located in the city's Seaport District. The one-minute video gives a first-person perspective of pushing through the filthy, trash-filled ice pile that served as a dumping ground for the snow. 'I've been skiing for 20 years, but never like this,' he told about the 'surreal' experience. Tarasov recalls having to avoid junk including rusted lawn chairs, parking cones, broken bottles, and 'pretty much every kind of trash you'd ever find on a Boston street.' He hit a dead seagull, he said. |
| **Reference summary** | Vlad Tarasov couldn't resist filming himself ski down the slopes at Boston's largest snow farm located in the city's Seaport District. His one-minute video gives a first-person perspective of pushing through the filthy, trash-filled ice pile that served as a dumping ground for the snow. 'I've been skiing for 20 years, but never like this,' he said about the 'surreal' experience. Junk in the filthy snow included rusted lawn chairs, parking cones, broken bottles and even a dead seagull. |
| **Generated summary (ChatGPT)** | A video has emerged of a skier named Vlad Tarasov skiing on a pile of snow in the Seaport District of Boston. The pile of snow was where the city put the snow that fell during the record-breaking winter season. Unfortunately, the snow is now a disgusting black color riddled with trash, including broken pieces of glass, plastic shards and other debris. |
| **ROUGE-1** | **37.58** |
| **Generated keywords (T5 SFT)** | Vlad Tarasov; 65 inches; Boston; Seaport District; the city; $30 an hour; volunteers. |
| **Generated summary (ChatGPT w/ SFT)** | Vlad Tarasov filmed himself skiing down the slopes of Boston's largest snow farm located in the city's Seaport District, which served as a dumping ground for the historic winter snowfall. The shere volume of snow that fell earlier this year, nearly 65 inches fell in February alone, means that huge piles of the white stuff still remain and the city had to pay volunteers $30 an hour to help dig out the battered city. |
| **ROUGE-1** | **46.91** |
| **Generated keywords (T5 SFT+RL)** | Vlad Tarasov; ski; Boston; large snow farm; Seaport District; surreal; video; filthy; snowfall. |
| **Generated summary (ChatGPT w/ SFT+RL)** | Vlad Tarasov couldn't resist filming himself skiing down the slopes of Boston's largest snow farm located in the city's Seaport District. To some avid skiers, snow is still snow and Tarasov said it was a 'surreal' experience. The one-minute video gives viewers a first-person perspective of the experience as Tarasov pushes through the filthy, trash-filled ice pile that served as a dumping ground for the historic winter snowfall. |
| **ROUGE-1** | **71.25** |

**Example from MultiWOZ dataset:**

| | |
|---|---|
| **Dialogue context** | **Assistant:** i have [value_choice] available for you! is there a certain area or cuisine that interests you? **User:** yes i would like the restaurant to be located in the center of the attractions. [db_3] |
| **Oracle dialogue acts** | [restaurant] [inform] choice [request] food |
| **Oracle system response** | there are [value_choice] restaurant-s available in the [value_area]. how about a specific type of cuisine? |
| **Generated response (ChatGPT)** | the [value_name] is located in the center of the attractions. would you like me to book a table? |
| **SacreBLEU** | **14.09** |
| **Generated dialogue acts (T5 SFT)** | [restaurant] [inform] food name area choice [recommend] price |
| **Generated response (ChatGPT w/ SFT)** | i would suggest [value_name] in the [value_area] area. they have [value_food] cuisine and are moderate in price. are you interested? |
| **SacreBLEU** | **16.40** |
| **Generated dialogue acts (T5 SFT+RL)** | [restaurant] [inform] choice [request] food |
| **Generated response (ChatGPT w/ SFT+RL)** | i have [value_choice] restaurants in the area. do you have a specific cuisine in mind? |
| **SacreBLEU** | **22.80** |
