# Abstract

Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated development environments, for complex tasks like software engineering, we posit that LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use. We investigate how interface design affects the performance of language model agents. As a result of this exploration, we introduce SWE-agent: a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively, far exceeding the previous state-of-the-art achieved with non-interactive LMs. Finally, we provide insight on how the design of the ACI can impact agents' behavior and performance.

# Introduction

Recent work has demonstrated the efficacy of LM agents for code generation with execution feedback [shinn2023reflexion]. However, applying agents to more complex code tasks like software engineering remains unexplored. To solve programming tasks, LM agents are typically designed to use existing applications, such as the Linux shell or Python interpreter [wu2024oscopilot; xie2024osworld; yang2023intercode]. However, to perform more complex programming tasks such as software engineering [jimenez2024swebench], human engineers benefit from sophisticated applications like VSCode with powerful tools and extensions. Inspired by human-computer interaction (HCI) studies on the efficacy of user interfaces for humans [cooperaboutface], we investigate whether LM agents could similarly benefit from better-designed interfaces for performing software engineering tasks.

[IMAGE: SWE-agent overview - SWE-agent is an LM interacting with a computer through an agent-computer interface (ACI), which includes the commands the agent uses and the format of the feedback from the computer.]

Consider the simple setting of an agent interacting directly with a Linux shell [yang2023intercode]. In practice, we find that LM agents can struggle to reliably take actions in this environment. For example, it fails to provide simple commands to edit a small file segment, and does not provide any feedback if the user makes an invalid edit. These deficits substantially hamper performance, motivating the need for an agent-computer interface (ACI), i.e., an abstraction layer between the LM agent and computer, to enhance the LM agent's abilities in computer environments.

From this effort, we introduce SWE-agent, an agent composed of an LM and ACI, that can interact with a computer to solve challenging real-world software engineering problems, such as those proposed in SWE-bench [jimenez2024swebench]. In contrast to the Linux Shell's granular, highly configurable action space, SWE-agent's ACI instead offers a small set of simple actions for viewing, searching through and editing files. The ACI uses guardrails to prevent common mistakes, and an agent receives specific, concise feedback about a command's effects at every turn. _We show that ACIs tailored specifically for LMs outperform existing user interfaces_ (UIs) _designed for human users_, such as the Linux shell.

Using GPT-4 Turbo as a base LM, SWE-agent solves 12.47% of the 2294 SWE-bench test tasks, substantially outperforming the previous best resolve rate of 3.8% by a non-interactive, retrieval-augmented system [jimenez2024swebench]. We perform an ablation study on a subset of 300 SWE-bench test instances (SWE-bench Lite) to analyze our ACI design choices. The results show that SWE-agent solves 10.7 percentage points _more_ instances than the baseline agent, which uses only the default Linux shell. Although our ACI was developed for GPT-4 Turbo, we show that it is portable to a different LM; SWE-agent with Claude 3 Opus can solve 10.5% of the benchmark tasks.

Our contributions are twofold. First, we introduce the concept of the agent-computer interface (ACI) and demonstrate how careful ACI design can substantially improve LM agent performance without modifying the underlying LM's weights. Second, we build, evaluate, and open-source SWE-agent, a system that provides LMs an ACI for solving real-world software engineering tasks. Unlike prior works that independently explore the merits of tool use, prompting techniques, and code execution in interactive settings, our approach unifies these factors within the ACI framework. We show that crafting LM-centric interactive components has meaningful effects on downstream task performance.

# The Agent-Computer Interface

An LM acts as an agent when it interacts with an environment by iteratively taking actions and receiving feedback [sumers2023cognitive; yao2023react]. Typically, the environment has hard constraints, as in robotics, where agents control actuators in the physical world. On the other hand, digital environments can be molded by abstractions in the form of application programming interfaces and user interfaces for software and humans respectively. Naturally, existing interfaces have been designed with one of these users in mind. We argue that LM agents represent a new category of end user, with their own needs and abilities. We refer to the interface LM agents use to interact with computers as the _agent-computer interface_ (ACI).

[IMAGE: ACI-HCI comparison - Specialized applications like IDEs (e.g., VSCode, PyCharm) make scientists and software engineers more efficient and effective at computer tasks. Similarly, ACI design aims to create a suitable interface that makes LM agents more effective at digital work such as software engineering.]

Disparities in humans' and LMs' abilities and limitations motivates different interface design guidelines. For instance, the current generation of LMs lack the visual understanding abilities to directly operate GUI-based applications with rich visual components and signals. However, many of the features provided by these applications, such as syntax checking and navigation tools, could be useful to LM agents if they were presented in a suitable manner. Additionally, humans can flexibly ignore unnecessary information, whereas all content has a fixed cost in memory and computation for LMs and distracting context can harm performance [liu2023lost]. Therefore, LM agents may be more effective at interacting with computers when provided an interface that was built informed by these differences.

Ultimately, a well-designed ACI should help the LM agent understand the state of the application given previous changes, manage history to avoid unnecessary context from prior observations, and provide actions that models can use efficiently and reliably. The ACI specifies both the commands available to the LM and how the environment state is communicated back to the LM. It also tracks the history of all previous commands and observations and, at each step, manages how these should be formatted and combined with high-level instructions into a single input for the LM.

In this paper, we assume a fixed LM and focus on designing the ACI to improve its performance. This means that we shape the actions, their documentation, and environment feedback to complement an LM's limitations and abilities. We draw inspiration from the field of HCI, where user studies elicit insights about how compatible different interfaces are with respect to human intuition and performance [cooperaboutface]. We use two approaches to enhance performance on a development set: (1) manually inspect agent behavior to identify difficulties and propose improvements, and (2) run a grid search to select the best ACI configuration.

Taking these two actions resulted in several insights about design principles that seem especially important for building effective ACIs:

1.  **Actions should be simple and easy to understand for agents.** Many bash commands have documentation that includes dozens of options. Simple commands with a few options and concise documentation are easier for agents to use, reducing the need for demonstrations or fine-tuning. This is a defining principle for all SWE-agent commands that we describe in Section 3.

2.  **Actions should be compact and efficient.** Important operations (e.g., file navigation, editing) should be consolidated into as few actions as possible. Efficient actions help agents make meaningful progress towards a goal in a single step. A poor design would therefore have many simple actions that must be composed across multiple turns for a higher order operation to take effect. We show this idea in action in the Editing and Search interface analyses in Section 5.1.

3.  **Environment feedback should be informative but concise.** High quality feedback should provide the agent with substantive information about the current environment state (and the effect of the agent's recent actions) without unnecessary details. For instance, when editing a file, updating the agent about revised content is helpful.

4.  **Guardrails mitigate error propagation and hasten recovery.** Like humans, LMs make mistakes when editing or searching and can struggle to recover from these errors. Building in guardrails, such as a code syntax checker that automatically detects mistakes, can help agents recognize and quickly correct errors.

Analysis and ablation studies in Section 5 demonstrate how alternative ACIs affect LM performance. Our studies shows how these principles appear recurrently across actions, feedback, and workflows.

# SWE-agent: Designing an ACI for Software Engineering

Here we describe how SWE-agent provides an ACI for LMs to act as software engineering agents, enabling them to effectively search, navigate, edit, and execute code commands. The ACI comprises several principal components, including search/navigation, file viewer, file editor, and context management. At each step, SWE-agent generates a thought and a command, then incorporates the feedback from the command's execution in the environment (ReAct; yao2023react). Built atop the Linux shell, SWE-agent also allows access to common Linux commands and utilities when needed.

**Search and navigation.** Navigating codebases requires finding the relevant file and content. A common strategy to do this involves looking up terms that might be useful, e.g., files, functions, or class definitions mentioned in an issue. We introduce the special commands `find_file`, `search_file`, and `search_dir`, which output a summary of search results when searching for filenames and strings within files or directories. The `find_file` command searches for filenames in the repository, while the `search_file` and `search_dir` locates strings in a file(s) of a subdirectory. Our interface encourages efficient searches by suppressing verbose results. The search commands return at most 50 results for each search query; if a search exceeds this number, we do not report the results and instead suggest that the agent write a more specific query.

[IMAGE: File viewer and editor - The file viewer and edit command are integrated. (a) The file viewer shows the agent the open file's content with line numbers. (b) The agent invokes the edit function to replace lines 404-407 in the open file. After the edit, the file viewer shows the agent the now updated version of the file.]

**File viewer.** After finding a file they want to view, agents use the interactive file viewer by calling the command `open` on the relevant file path. The file viewer presents a window of at most 100 lines of the file at a time. The agent can move this window with the commands `scroll_down` and `scroll_up` or access a specific line with the `goto` command. To facilitate in-file navigation and code localization, we display: the full path of the open file, the total number of lines in the file, the number of lines omitted before and after the current window, and the line number (prepended to each visible line).

**File editor.** We provide a few commands that let LMs create and edit files. The `edit` command works in conjunction with the file viewer, allowing agents to replace a specific range of lines in the open file. This command takes 3 required arguments: the start line, end line, and replacement text. In a single step, agents can replace all lines between the start and end lines with the replacement text. After edits are applied, the file viewer automatically displays the updated content, helping the agent observe the effects of its edit immediately without invoking additional commands.

Similar to how humans can use tools like syntax highlighting to help them notice format errors when editing files in an IDE, we integrate a code linter into the `edit` function to alert the agent of mistakes it may have introduced when editing a file. Select errors from the linter are shown to the agent along with a snippet of the file contents before/after the error was introduced. Invalid edits are discarded, and the agent is asked to try editing the file again.

**Context management.** The SWE-agent system uses informative prompts, error messages, and history processors to keep agent context concise and informative. Agents receive instructions, documentation, and demonstrations on the correct use of bash and ACI commands. At each step, the system instructs them to generate both a _thought_ and an _action_ [yao2023react]. Malformed generations trigger an error response, asking the agent to try again, which is repeated until a valid generation is received. Once received, all past error messages except the first are omitted.

The agent's environment responses display computer output using a template; however, if no output is generated, a specific message ("Your command ran successfully and did not produce any output") is included to enhance clarity. To further improve context relevance, observations preceding the last 5 are each collapsed into a single line. By removing most content from prior observations, we maintain essential information about the plan and action history while reducing unnecessary context, which allows for more interaction cycles and avoids showing outdated file information.

# Experimental Setup

**Datasets.** We primarily evaluate on the SWE-bench dataset, which includes 2294 task instances from 12 different repositories of popular Python packages [jimenez2024swebench]. We report our main agent results on the full SWE-bench test set and ablations and analysis on the SWE-bench Lite test set, unless otherwise specified. SWE-bench Lite is a canonical subset of 300 instances from SWE-bench that focus on evaluating self-contained functional bug fixes. We also test SWE-agent's basic code editing abilities with HumanEvalFix, a short-form code debugging benchmark [muennighoff2023octopack].

**Models.** All results, ablations, and analyses are based on two leading LMs, GPT-4 Turbo (`gpt-4-1106-preview`) [openai2023gpt4] and Claude 3 Opus (`claude-3-opus-20240229`) [chiang2024chatbot]. We experimented with a number of additional closed and open source models, including Llama 3 and DeepSeek Coder [deepseek-coder], but found their performance in the agent setting to be subpar. Many LMs' context window is too small, such as Llama 3's context window of 8k. GPT-4 Turbo and Claude 3 Opus have 128k and 200k token context windows, respectively, which provides sufficient room for the LM to interact for several turns after being fed the system prompt, issue description, and optionally, a demonstration.

**Baselines.** We compare SWE-agent to two baselines. The first setting is the non-interactive, retrieval-augmented generation (RAG) baselines established in [jimenez2024swebench]. Here, a BM25 retrieval system retrieves the most relevant codebase files using the issue as the query; given these files, the model is asked to directly generate a patch file that resolves the issue.

The second setting, called Shell-only, is adapted from the interactive coding framework introduced in [yang2023intercode]. Following the InterCode environment, this baseline system asks the LM to resolve the issue by interacting with a shell process on Linux. Like SWE-agent, model prediction is generated automatically based on the final state of the codebase after interaction.

**Metrics.** We report **% Resolved** or **pass@1** as the main metric, which is the proportion of instances for which all tests pass successfully after the model generated patch is applied to the repository [jimenez2024swebench]. We also report the **$ Avg. Cost** metric, the API inference cost incurred by SWE-agent averaged over all successfully resolved instances. Due to budget constraints, we set the per-instance budget to $4; if a run exceeded this budget, existing edits were submitted automatically.

**Configuration search.** During the design process of SWE-agent, we arrived at the final ACI design through qualitative analysis of system behavior on a small set of hand-picked examples from the development split of SWE-bench. For the remaining hyperparameter choices, we performed a sweep over the window size, history processing, and decoding temperature.

# Results

Across all systems, SWE-agent w/ GPT-4 Turbo achieves the best performance all-around, successfully solving 12.47% (286/2294) of the full SWE-bench test set and 18.00% (54/300) of the Lite split. Compared to RAG on Lite, SWE-agent is 8-13x more costly but yields a 6.7-fold improved % Resolved rate. An LM-friendly ACI's value is confirmed by SWE-agent's 64% relative increase compared to Shell-only, both with GPT-4 Turbo.

SWE-agent yields strong performance on HumanEvalFix with 88.3% pass@1 rate. Analysis reveals that average performance variance is relatively low, but per-instance resolution can change considerably.

## Analysis of ACI Design

We perform several ablations of the SWE-agent interface, specifically with respect to the SWE-agent w/ GPT-4 configuration. Our case studies shed light on interesting agent behavior along with the impact of different ACI designs.

**Human user interfaces are not always suitable as agent-computer interfaces.** Current LMs are vulnerable to a number of pitfalls when searching for relevant content in a Linux shell environment. Some exploration patterns (e.g., chains of `cd`, `ls`, `cat`) are extremely inefficient. `grep` or `find` look ups can perform better but occasionally produce many lines of irrelevant results. We hypothesize that better localization is possible with faster navigation and a more informative search interface.

[IMAGE: Edit comparison - Three different Edit interfaces for task instance sympy__sympy-24102. Editing with bash commands requires several actions to successfully modify a file. The Editing component defines an edit command that leverages the File Viewer component to replace the bash style of editing workflow with a single command. Linting is beneficial for stymieing cascading errors that often start with an error-introducing edit by the agent.]

The comparison shows the Shell-only setting versus two different search interfaces. _Iterative_ search, directly inspired by traditional user interfaces for search, e.g., `Vim` or VSCode, shows results one by one via the file viewer. Agents can look through results using `next` and `prev` actions. Each result displays the matching line along with surrounding lines of context. An advantage is that an agent can begin editing directly after seeing the relevant code in its search. However, when given a large number of search results, agents tend to look through every match exhaustively, calling `next` until each result has been inspected. This inefficient behavior can exhaust an agent's cost budget or context window, leading to even worse performance than not having additional search tools at all (15.7% [-2.3] for No search vs. 12.0% [-6.0] with Iterative search).

**Compact, efficient file editing is critical to performance.** SWE-agent's file editor and viewer are designed to consolidate the editing process into a single command that enables easy multi-line edits with consistent feedback and automatically updates the agent's view of the file after editing. In the No `edit` setting, editing options are restrictive and prone to errors; the primary methods available are either replacing entire files through redirection and overwriting or using utilities like `sed` for single-line or search-and-replace edits. Both methods have significant drawbacks. Redirection involves copying and rewriting entire files for even minor changes, which is both inefficient and error-prone. Although `sed` can facilitate specific edits, executing multi-line edits is cumbersome and can lead to unintended consequences that are challenging to detect. Moreover, both strategies lack immediate feedback about file updates, making these silent operations potentially confusing for models to interpret and increasing the risk of errors. Without SWE-agent's file editor interface, performance drops to 10.3% [-7.7]. We also find that agents are sensitive to the number of lines the file viewer displays. Either too little content (30 lines, 14.3% [-3.7]) or too much (entire file, 12.7% [-5.3]) lowers performance.

**Guardrails can improve error recovery.** A prominent failure mode occurs when models repeatedly `edit` the same code snippet. The usual suspect for this behavior is an agent introducing a syntax error (e.g., incorrect indentation, extra parenthesis) via an errant `edit`. We add an intervention to the `edit` logic that lets a modification apply only if it does not produce major errors. This intervention improves performance considerably (without linting, 15.0% [-3.0]).

## Analysis of Agent Behavior

Recurring problem-solving patterns emerge when LMs are equipped with a useful, intuitive ACI. We describe several model behaviors and problem-solving patterns that can be discerned from model performance and each model's corresponding trajectories.

**Reproduction and/or localization is the first step.** SWE-agent usually begins with either writing reproduction code and/or localizing the issue's cause to specific lines of code. All trajectories begin with either `create` (reproduction) or `find_file`/`search_dir` (localization). To reproduce, models will `create` a new file, add reproduction code to it with an `edit`, then run with `python`; this is the most popular triple of actions. Using this feedback along with file names and symbols in the issue description, an agent will start with a broad, directory-level keyword search, before then zooming into specific files and lines. The most likely actions following localization sequences like (`python`, `find_file`) and (`search_dir`, `open`) are `search_file` and `goto`, indicative of how an agent "zooms in" on a bug.

**Remaining turns are mostly "edit, then execute" loops.** From turn 5 onwards, the most frequent two actions for all turns are `edit` and `python`. Additional localization operations are often interspersed across these later turns, where agents might look at more in-file code with `search_file`, `scroll_up/down`, or other files altogether with `search_dir`, `find_file`. This behavior usually arises in response to new information from re-running the reproduction script. Submissions are distributed normally from turn 10 onwards, although resolved task instances correlate more with earlier submissions.

**Editing remains challenging for agents.** A non-trivial minority of edit actions raise a linting error; out of 2294 task instances, 1185 (51.7%) of SWE-agent w/ GPT-4 Turbo trajectories have 1+ failed edits. While agents generally recover more often than not from failed edits, the odds of recovery decrease as the agent accumulates more failed edits. Recovery refers to a sequence of consecutive failed edits followed immediately by a successful edit. Any attempt at editing has a 90.5% chance of eventually being successful. This probability drops off to 57.2% after a single failed edit.

[IMAGE: Failure modes - Failure mode distribution for SWE-agent w/ GPT-4 Turbo trajectories of unresolved instances. Each instance is labeled automatically using an LM with the categories from the failure categories table.]

**Agents succeed quickly and fail slowly.** We find that runs submitted relatively early are much more likely to be successful compared to those submitted after a larger number of steps or cost. Successful runs complete earlier and at a cheaper cost than unsuccessful ones. In general, successful instances solved by SWE-agent w/ GPT 4 finish with a median cost of $1.21 and 12 steps compared to a mean of $2.52 and 21 steps for unsuccessful ones. Furthermore, we find that 93.0% of resolved instances are submitted before exhausting their cost budget, compared to 69.0% of instances overall. For these reasons, we suspect that increasing the maximum budget or token limit are unlikely to substantially increase performance.

**Most failures are incorrect implementations.** We use GPT-4o to automatically categorize unresolved trajectories (SWE-agent w/ GPT-4 Turbo on SWE-bench Lite, n=248) into one of 9 manually defined categories. On a hand-labeled validation set, the LM's judgment agrees with the authors' on 87% of instances. About half (52.0%) of unresolved instances fall into the Incorrect Implementation or Overly Specific Implementation categories, suggesting that agents' proposed solutions often simply fail to functionally address the issue or are insufficiently general solutions. Cascading failed edits make up another 23.4% of failures.

# Related Work

## Software Engineering Benchmarks

Code generation benchmarks, which evaluate models on the task of synthesizing code from natural language descriptions, have served as a long-standing bellwether for measuring LM performance [chen2021evaluating; austin2021program; hendrycks2021measuring; lu2021codexglue]. Subsequent works have built upon the code generation task formulation to contribute new benchmarks that translate problems to different (programming) languages [cassano2022multiple; wang2023mconala], incorporate third-party libraries [lai2022ds1000; liu2024mlbench], introduce derivative code completion tasks [huang2024mlagentbench; muennighoff2023octopack], increase test coverage [evalplus], change the edit scope [ding2023crosscodeeval; du2023classeval; Yu2023CoderEvalAB], and add robustness to dataset contamination [jain2024livecodebench]. Code generation problems are largely self-contained, with short problem descriptions (~100 lines) and corresponding solutions that are similarly brief, requiring nothing more complex than basic language primitives. Tests are either handwritten or generated synthetically via fuzz testing. In recent months, the rapid development of LMs has begun to saturate many of these benchmarks. For instance, the top method solves 94.4% of HumanEval [zhou2023language].

Gauging future trends with the code generation task paradigm can be limited by the simplicity of this setting and cost of human-in-the-loop problem creation. In response, recent efforts have demonstrated that software engineering (SE) can serve as a diverse, challenging testbed for LM evaluation [zhang2023repocoder; jimenez2024swebench; Liu2023RepoBenchBR]. Repository-level code editing introduces many reasoning challenges grounded in real SE subtasks, such as spotting errant code and identifying cross-file relationships and understanding codebase-specific symbols and conventions. As a field, SE has generally studied tasks in a more isolated manner; prior benchmarks tended to frame problems in isolation from the rest of a codebase [justJE2014defects; Karampatsis2019HowOD].

We use SWE-bench because it unites many separate SE tasks, such as automated program repair [fan2023automated; sobania2023analysis; xia2022less], bug localization [chakraborty2018entropy; yang2024localization], and testing [kang2023large; wang2023software; xia2023universal] under a single task formulation that faithfully mirrors practical SE. Furthermore, SWE-bench task instances are diverse, having been automatically collected from real GitHub issues across 12 different repositories. In addition, SWE-bench performance is based on rigorous, execution-based evaluation with human-written unit tests.

## Language Models as Agents

The co-emergence of stronger LMs, increasingly challenging benchmarks, and practical use cases have together motivated a paradigm shift in LMs' inference setting. Instead of traditional zero/few-shot generation, LM agents [hong2023metagpt; sumers2023cognitive; wang2024survey; xi2023rise] that interact with a real/virtual world have proliferated as the default setting for web navigation [koh2024visualwebarena; nakano2022webgpt; press-etal-2023-measuring; sridhar2023hierarchical; thoppilan2022lamda; yao2023webshop; yao2023react; zhou2023webarena], computer control [packer2024memgpt; wu2024oscopilot; xie2024osworld], and code generation tasks [holt2024lmac; wang2023executionbased; yin2022natural].

Interaction and code generation are increasingly used together, with code as the modality of choice for actions [wang2024executable; yang2023intercode], tool construction [gu2024middleware; wang2024trove; zhang2024training], and reasoning [shinn2023reflexion; zelikman2022parsel; zelikman2024selftaught]. Coding agents have also been applied to offensive security [fang2024llm; shao2024empirical; yang2023language], theorem proving [thakur2024incontext], and clinical tasks [shi2024ehragent; tang2024medagents; wornow2024automating]. To the best of our knowledge, SWE-agent is the first work to explore language agents for end-to-end software engineering (SE).

# Discussion

We introduce SWE-agent, an agent composed of an LM and ACI capable of autonomously solving software engineering tasks. Through our design methodology, results, and analysis, we demonstrate the value of ACIs tailored to leverage LMs' strengths and mitigate their weaknesses. Beyond empirical applications, we hope the further study of ACIs can also make principled use of and contribute to our understanding of language models and agents, analogous to the synergy between human-computer interaction (HCI) and psychology [carroll1997human]. Humans and LMs have different characteristics, training objectives, specialities, and limitations [griffiths2020understanding; mccoy2023embers], and the interaction design processes can be seen as systematic behavioral experimentation that could reveal more insights into these differences towards establishing a comparative understanding of human and artificial intelligence.

# Appendix

In the appendix, we provide additional analyses and more extensive discussions about SWE-agent, agent-computer interface (ACI) design, and model performance on various evaluation benchmarks. We also provide several thorough case studies of SWE-agent behavior on select task instances. Data, code, and leaderboard at [swe-agent.com](https://swe-agent.com/).

# SWE-agent Design

In this section, we go into greater discussion about the design methodology, appearance, and implementation of each of the SWE-agent components. As described in Section 3, the SWE-agent interface consists of several components that enable agents to accomplish key sub-tasks that are fundamental to solving software engineering problems. These are generally the following:

1.  _Localization_: Identify file(s)/line(s) causing the issue.

2.  _Editing_: Generate fixes addressing the given issue.

3.  _Testing_: Write new scripts or modify existing test files to reproduce the issue and/or verify if fixes are correct.

To enable LM-based agents to efficiently carry out these individual functions and progress towards the overarching goal of resolving a codebase issue, we provide a file viewer, file editor, search / navigation system, and context management system.

## ACI Design

In this section, we revisit each component discussed in Section 3. Per section, we first briefly review the component. We then discuss the underlying motivation for the component with respect to existing software tools. Finally, we note any additional thoughts that influenced the design process of the component with some occasional discussion of what aspects of the component heavily influence language model behavior.

#### File viewer.

As discussed in Section 3, the File Viewer is fundamental to a language agent's ability to understand file content and understand how different programmatic entities relate to one another. The File Viewer refers to an interface that consists of the four commands, and a customized standard output for displaying n lines of a file at a time. Using the file viewer, an agent can look at n lines of a file at a time and jump around the file. The File Viewer enables agents to perform fine-grained localization steps and also understand relationships between intra-file entities.

First, we discuss why existing software systems and graphical user interfaces are sub-optimal for LM use. In a Shell-only setting, there are several commands that can be used to inspect file content. However, out of the box command line tools are sub-optimal or limiting for language agents for several reasons. First, commands that print files to standard output (e.g. `cat`, `printf`) can easily flood a language agent's context window with too much file content, the majority of which is usually irrelevant to the issue. Enabling a language agent to filter out distractions and focus on relevant code snippets is crucial to generating effective edits. While commands like `head` and `tail` reduce length to the first/last n lines, it is not intuitive to use bash commands to perform in-file navigation. It is either impossible or requires a long list of arguments to show specific file lines. Furthermore, since such Bash commands are stateless, "scrolling" up/down relative to the current file position typically requires regenerating the same lengthy command with minor changes. Interactive tools like `more` and `less` accommodate this, but (1) representing navigation actions (multiple key up/down clicks) is intuitive for humans, but is verbose and costly for language agents, and (2) even if jumping to a specific line number is allowed, it is not possible to quickly identify what classes/methods/symbols are declared in a file and then immediately go to their definitions.

There are a couple features of the File Viewer interface that make it friendlier and more operable than the Shell-only setting. First, the File Viewer standard output contextualizes code snippets with prepended line numbers and indicators of the number of lines above/below the current region. These details give a more focused view of a file without compromising easy viewing of other parts of the codebase. This kind of file presentation also makes precise and consistent editing commands possible, as we discuss more thoroughly in the following section.

Another advantage of the File Viewer is that the commands are designed to be complementary and grounded in the File Viewer standard output. This saves the model from having to do repetitive or additional actions that unnecessarily increase the potential for error. As a concrete example, if an agent used a `sed` command to view the first 100 lines of a file and wants to look at the next 100 lines, it will have to recalculate parameters such as the start line and end line and reflect these updates correctly in the subsequent generation. As a rule of thumb, reducing the need for models to do this arithmetic by constructing actions and standard output that complement one another and build upon the effects of prior actions is highly preferable.

[IMAGE: SWE-agent components - The File Viewer and Search components of the SWE-agent interface. The corresponding commands for each component are shown in blue. These examples are copied from trajectories generated by SWE-agent w/ GPT-4 Turbo on the pvlib__pvlib-python-1603 task instance.]

#### File editor.

The File Editor, working in conjunction with the File Viewer, primarily refers to the `edit` command and the guardrails it enforces to protect models against self-incurred cascading edit errors. Editing and testing are crucial to language agents' success on programming tasks, and a well-designed interface directly influences how well an agent's capabilities can be elicited. In other words, a bad interface undermines model performance.

As discussed in Section 3, editing can be very difficult in a Shell-only setting. Built in commands (e.g., `sed`) often require a lengthy list of arguments, and the mis-specification of an argument can easily throw a model off track as it attempts to correct self-incurred errors. We also observe that when agents use such commands directly, they struggle with the arithmetic skills required to generate an edit. Details such as including the correct indentation level, inserting delimiters at specific points in a line, and adhering to stylistic preferences of the codebase all require some amount of planning or calculation. Similar to the Shell-only file viewing process, file editing may also require repeating many commands. For instance, performing a multi-line edit can only be represented as multiple `sed` calls with requisite, delicate tweaks to the arguments for every turn. Furthermore, as referenced in Section 5.1, editing in Shell-only is usually a "silent" procedure. Confirming whether an edit succeeded and viewing its effects requires additional steps that can bloat the editing process with extra, needless commands.

The `edit` command addresses the Shell-only failure modes by being grounded in the File Viewer standard output. The line numbers argument eliminates the need for any additional arithmetic, and the find-and-replace edit mechanism is a format that existing models are more used to. With this functionality, agents can also perform multi-line edits in a single action.

Finally, an important feature of the `edit` command is that it does not apply changes which incur a linting error. A fair and verified assumption we make when considering this feature is that the original codebase associated with each task instance is well-formed. In other words, we assume that codebase maintainers will only push syntactically sound code that can be compiled successfully. When an agent issues an edit, it is applied to the codebase. Then, we run the following linting command:

```
flake8 --isolated --select=F821,F822,F831,E111,E112,E113,E999,E902 "$CURRENT_FILE" 2>&1
```

The arguments for `select` are error codes that refer to syntax issues such as indentation. F821 and F822 indicate undefined names/symbols. F831 indicates a duplicate argument in a function definition. E111, E112, E113 are indentation errors. E999 denotes a syntax error and an E902 occurs if flake8 cannot read the source file.

If the edit does not introduce any of these errors, this command will produce no output. The edit is kept and the updated file content is shown using the File Viewer centered around the lines the edit occurred. If however the linting command produces output, which indicates the edit introduces a syntax error, the edit is reverted. In place of the file viewer, a message is displayed to the agent which shows the error that was caused, what the edit would have looked like, and the original file content. During the development process, we experimented with variations to this message, including the omission of one or more parts. Our takeaway was that having all three messages is helpful. Without the error type, the agent might misdiagnose what the mistake was. Without a snippet of the changed file content, the agent will re-issue the same command more frequently. Without a snippet of the original file content, the agent has to attend to the same content from several turns ago; agents also sometimes generate edits with respect to wrong, errant file content because it is from a more recent turn.

The editing guardrail has a drawback. To a certain degree, it forces some edits to be done in a particular order. For instance, if the model's intention was in fact to remove an argument, due to the SWE-agent editing guardrails, it would have to remove all references from the function implementation either at the same time in a single action, or before removing it from the method header if split into two separate actions. This example highlights the trade-offs between the flexibility and guardrails of a command. Deciding whether to introduce a guardrail depends on how well it reduces common model errors compared to whether such restrictions hamper models' preferred workflows.

#### Search & navigation.

The File Viewer and File Editor together allow agents to make edits, write tests, and perform localization at a file level. The Search & navigation module complements these capabilities by giving agents the tools to perform keyword-driven localization at both a directory level and file level.

As discussed, the main struggles with using built in Shell-only search commands such as `grep` and `find` are (1) given a general enough term, they are prone to producing too many search results that can consume an inordinate amount of space in the context window, and (2) they are highly configurable, making search result outcomes potentially inconsistent in appearance. The alternative to these search utilities is to navigate the file system directly with `cd` and look at what's in each folder with variations of `ls` and `cat`; this kind of approach can take a large number of turns without yielding any particularly useful information.

The `search_dir` and `find_file` helps agents perform directory level searches. The reason we provide two commands is due to the kinds of keywords that are present in an issue description (e.g., class references, file names). The `search_file` command allows agents to search for terms at a file-level, which is helpful for efficient fine-grained localization. Taking a step back, the goal of these search commands is to make it easy for the agent to utilize any signal (e.g., line number, stack trace, natural language) about the root cause of an issue that may be present in the issue description or codebase. Once again, simpler command usage patterns with consistent output formats are easier for agents to use and reduces the chance for mistakes or irrelevant outputs.

The main guardrail in place for all three search commands is curbing the number of search results to 50 or fewer. The downside is that reporting an error forces the model to generate another search query which can be an expensive operation. This reflects a trade-off between keeping observations concise and making additional calls to the base LM.

## Implementation

The SWE-agent codebase is generally composed of three modules: the environment, the agent, and the logging mechanism for saving task episodes into trajectories and patch generations.

**Environment.** The SWE-agent environment is heavily influenced by the InterCode library [yang2023intercode]. For the general pipeline of agent interactions with the environment, our work directly adopts InterCode's interactive coding task formulation. The environment integrates large parts of the interaction handling logic from the InterCode-Bash environment, which is essentially the Shell-only setting referenced in the paper. As a part of this adoption, SWE-agent also uses Docker containers to ensure reproducible and safe execution. Because of this, SWE-agent's infrastructure makes it easy for a user to swap out the Dockerfile (a domain specific language for defining a container) to support other codebases and programming languages beyond the scope of SWE-bench task instances. One difference is that SWE-agent makes minor adjustments to the underlying communication logic that transfers actions and observations between the Docker container and agent entity.

**Agent.** Beyond serving as an agentic wrapper for facilitating multi-turn queries from an LM, the agent module defines the functions that render the ACI (e.g., context management, commands, interface logic, input/output format) and supports inference for closed/open, API-based/local language models. The main workflow is to define an interface as a class and/or set of commands, which can then be specified via a configuration file.

**Logging.** For each task episode, the main artifacts produced are the trajectory, which contains a history of the interactions between the agent and environment, and the final patch generation, which can represents a summary of the changes proposed by the agent during the interaction. The patch generation can be used directly for SWE-bench [jimenez2024swebench] evaluation.

## Configuration

The SWE-agent system is instantiated by three components: an LM, a SWE-bench style dataset or GitHub issue, and a configuration file. The configuration file serves to specify the design of the ACI. Iteratively refining the configuration file is the main way we achieved better agent performance and carried out different analyses for the main paper.

An agent-computer interface is generally made up of four categories of configurable components:

1.  Prompt templates: These prompt templates are used to inform the language model of the task setting, show the list of available commands, augment environment responses with the values of state variables, and provide the initial task setting.

2.  Command files: These files contain the source code of bash or Python functions and scripts. Commands are easily modified, added, and removed through manipulating these files' code contents directly. Documentation added in these files can also be injected into prompts to inform the model of the available commands.

3.  Control flow: Methods for parsing model responses and processing history can be specified through these configuration arguments.

4.  Environment variables: Initial values of variables that may interact with commands and the shell can also be specified in the configuration.

The prompt templates are explained in detail in the Prompts section. The environment variables and command files work in tandem; environment variables make the interfaces stateful, and when commands are invoked, the corresponding states are updated to reflect the changes to the environment and interface. The `parse_command`, `parse_function`, and `history_processor` all reference implementations declared within the agent module. The `parse_command` file describes how command documentation should be presented to the agent. The `parse_function` is what enforces the input/output formats for the agent. The `history_processor` points to the logic for controlling and modifying the message history enforced at each turn throughout a single task episode.

The configuration-based workflow of SWE-agent makes it easy to test new ACIs by incorporating novel commands, input/output formats, context managers, and more into the existing codebase.

**Commands.** Commands are declared as a list of one or more file paths in the `command_files` argument. Individual commands must be declared as separate functions in `.py` or `.sh` files.

The choice of Python or Bash based implementations of commands means they can be written to do a wide variety of actions, and the use of Docker means that the commands and system can be co-designed. Here is a list of guidelines around how to implement commands correctly.

- Command arguments can be referenced via positional parameters notation (i.e. `$1`).
- If there are no arguments, omit the `arguments` section.
- The implementation for your command is unconstrained. There are no limitations on the form of the underlying command code.
- The minimal documentation requirements are `signature` and `docstring`.
- Global variables can be used to make stateful changes to the environment. For instance, for the commands associated with the File Viewer, you'll see we define the `CURRENT_LINE` variable for the file viewer. This variable is modified across multiple commands, including `open`, `goto`, `scroll_up`, `scroll_down`, and `edit`.
- Third party libraries can be freely imported and used by commands (e.g., `flake8`).
- To show effects of a command, print to standard output (e.g., with `echo`). The command parsing logic is implemented such that it does not look for a return value.

Once the file path containing the command is added to `command_docs` as an argument, the command is immediately available for use in subsequent task episodes. Including a demonstration that uses more complicated commands can be helpful to showcase proper use and may increase the frequency with which the agent uses the command.

**Input/output format.** The input/output format defines what a correctly formatted response for an agent should look like. Selecting a suitable format greatly affects how well agents can interact with the environment. The methods for communicating and enforcing the input/output format are separated across several arguments. The value of `parse_function` should point to a class definition that enforces the format and actually parses the agent's responses. Informing the agent of the expectations around the input/output format should take place in `system_template`, and the agent can be reminded of these standards via the `format_error_template`. New input/output formats can be easily devised and enforced by updating these arguments to point to a new class or display different natural language instructions.

**Context management.** Context management is implemented as a class within the agent module. The `history_processor` argument allows one to specify which context manager to use via the configuration file. Underneath the hood, the context manager is invoked per turn of the interactive loop. From the entire recorded history of the agent's interactions so far, the context manager constructs the literal history to be fed to the agent to invoke the next response. The general design of `history_processor`s allows for easy experimentation towards more sophisticated strategies for managing history.

# Extended Results

In this section, we provide additional results, including performance marginalized against different dimensions, patch generation statistics, and problem solving patterns reflected by SWE-agent trajectories. Per analysis, we provide numerical or qualitative evidence that supports our findings, describe our takeaways from each finding, and discuss both the strengths of SWE-agent relative to prior baselines along with future directions based on improving common failure modes.

# Prompts

In this section, we go through the prompt templates that make up the agent's history, discussing them in the order of presentation to SWE-agent. Per template, we describe its purpose, walk through its content, and note any additional motivations that influenced how we wrote the template. The template content can and should be adapted slightly to fit the agent's intended use case. The purpose of this section is to describe our thought process for how we designed each template for these tasks to serve as reference for future work. Across templates, we find that providing tips which tell agents to not make specific mistakes, avoid common pitfalls, and use helpful execution signals are effective for eliciting more successful problem solving.

**Prompt Workflow.** The order of prompt templates throughout a single SWE-agent task instance episode consists of the system, demonstration, and issue templates shown all together at the beginning of the task episode, followed by turn-specific prompts that are shown depending on whether the agent response is well-formatted and whether the action has standard output.

[IMAGE: Prompt flow - The flow of prompt templates throughout a single SWE-agent task instance episode. The system, demonstration, and issue templates are shown all together at the beginning of the task episode, followed by turn-specific prompts that are shown depending on whether the agent response is well-formatted and whether the action has standard output.]

**System Template.** The system template describes the interactive task setting, the commands at the agent's disposal, and the expected response format. It is the first message for any episode, does not change in content across task instances, and is not removed or collapsed at any point from the message history. The agent is told of the general task setting, which is a command line that comes with a special file viewer interface. After this, the agent is presented the command documentation, which shows a usage example and docstring for every custom command. As discussed before, from manual observation, we find that agents need a lot of support to make effective use of the `edit` command.

An agent will occasionally generate an `edit` with either the wrong level of indentation or incorrectly specified line numbers. Because of this, we include a note telling the agent to pay attention to proper indentation. Finally, the system prompt describes what the agent's response should look like, communicated with an example (e.g. JSON format, XML delimiters) followed by a paragraph reinforcing the importance of issuing a _single_ thought/action pair per turn. Because of the constraints imposed by Docker containers, we include one last point about the command line environment not supporting any interactive session commands, such as `vi` or `python`. The system template does not introduce any task instance specific information.

**Demonstration Template.** If provided, the demonstration template immediately follows the system template as the second message showing the agent a trajectory which resulted in the successful resolution of a task instance from the development set. Including a demonstration slightly helps agents with understanding proper command usage and reduces the rate of errant responses. Rather than being written out entirely like the system template, the demonstration template is put together based on two fields as shown in the configuration file. First, the `demonstrations` argument points at 1+ `.traj` file(s), each containing a multi-turn, SWE-agent style history of interactions.

Second, the `demonstration_template` and `put_demos_in_history` arguments control how the demonstration is represented in the message history. If the `put_demos_in_history` argument is set `True`, every turn of the demonstration trajectory is used as a separate message in the history. The alternative is to write out the entire demonstration as a single message, wrapped in a natural language instruction indicating it is a demonstration. The effectiveness of each method varies by model. The first method of adding a demonstration as separate messages can be preferable because it gives an agent the impression that it has already solved a task instance and is working on the next one. However, for a different model, it can be confusing to have the contexts from two tasks in a single trajectory, which is why the second method of explicitly telling the model that it's being given a demonstration can be better.

We are unsure if demonstrations actually help agents understand the nuances of domain specific problem solving. Because of the diversity of software engineering issues, we think the role the demonstration plays is primarily to help the agent learn to issue properly formatted commands. Prior work has demonstrated that fine tuning may have the potential to imbue agents with a certain degree of expertise around how to adaptively solve task instances that may vary in terms of what strategy is most successful.

**Instance Template.** The instance template introduces the agent to the task instance. The problem statement is shown, followed by a brief set of instructions that reiterate important points from the system template. These points are the one thought/action per-turn requirement, mentioning the lack of support for interactive shell commands, and a reminder of the importance of editing indentation. Finally, a notably effective part of the instance template is the inclusion of tips which serve as an additional guidelines for how to operate successfully in the bash environment. These tips were developed manually and iteratively; after running SWE-agent with a particular configuration on the development set, we manually looked at the trajectories for failure modes. The tips were born out of these failures, and through repeated inspection, we found that such tips did reduce the frequency of errant problem solving strategies that they are meant to address. While our manual approach to writing tips certainly does not scale, representing feedback for common mistakes as tips is surprisingly effective. Developing better methods for this process of identifying failure modes and writing natural language instructions that describe the correct alternative behavior could be an avenue to better performance for future SWE-agent based systems. Finally, at the end of the message, the agent is presented with a command line prompt indicating that the task has begun and that the agent should issue its first command.

**Next Step Template.** Assuming an agent's response is well formed and contains an action, there are two simple templates used to present the corresponding output of the agent's action. If an agent's action produces some standard output, the agent is simply shown this output with a command line prompt to indicate that the agent should respond with the next action.

However, if an agent's action runs silently and produces no output (e.g. `rm abc.py`, `touch abc.py`), we found that simple showing no output along with a prompt can be confusing for agents to interpret, and it will often run additional, unnecessary commands to determine the effect of the prior action. To guard against this situation, the agent is informed verbosely that the command ran successfully and did not produce output. While the System, Demonstration, and Instances template are only used a single time, the next step template is used repeatedly. In the SWE-agent configuration described in this work, the next step templates are fairly simple, as they essentially just add the command line prompt to the end of the execution standard output. We have not explored other variations to this style.

**Collapsed Observation Template.** Old observations are _collapsed_; meaning that the structure and order of the agent's interaction history is preserved, but the content of old observations are replaced with a one-line placeholder. This summary simply states that the observation is omitted with the number of lines that were removed.

The purpose of collapsing old observations is twofold. First, it simply reduces the number of tokens needing to be processed at each interaction step. Second, by removing old observations' content, it can also reduce the presence of outdated or duplicate information, such as directory and file contents or command output. We think that both of these purposes can serve to improve agent performance.

**Error Message Template.** An agent's response is not always well formed. As discussed, an agent is generally expected to respond with a single thought and action per turn. In our work, we experimented with asking the agent to generate one of three different styles of responses for communicating one thought and one action (same type of response across any single episode).

1.  A string where the action is wrapped as a markdown-style code block.
2.  A JSON style dictionary with "thought" and "action" keys.
3.  An XML style generation with "<thought>" and "<action>" delimiters.

On occasion, an agent may generate a response that doesn't conform to the requested format. If this occurs, we show an error message to the agent indicating that the prior message was malformed and to issue another response that does not make the same mistake. If a model generates 3 malformed responses in a row, the episode will terminate early.

Another context management trick is that if models generate a malformed response, but then subsequently respond with a valid one, the message history is modified such that the action and response correspond to the malformed generation is removed. This kind of de-noising reduces unnecessary context and helps prevent future malformed generations. Each well-formatted response becomes an additional in-context demonstration of how to interact with the environment correctly; this "momentum" of correct responses is effective for helping agents continue to issue correct actions at later turns in trajectories when there is a lot of information in the message history.

# Qualitative Analysis

We choose 2 successful and unsuccessful task instances from SWE-bench Lite to perform a full qualitative analysis, based on the trajectories generated by the default configuration using GPT-4 Turbo.

## Analysis for psf/requests-2317

**Issue Summary.** The agent was tasked with resolving a problem in the `requests` library where the `method` variable, a bytes object (e.g., `b'GET'`), was being incorrectly converted to a literal string (`"b'GET'"`) by the `builtin_str` function. This caused the `requests` library to raise a `404 Not Found` response due to the incorrect method string. The issue was observed in Python 3.4 and not in Python 2.6, indicating a version-specific difference in handling bytes.

**Interaction Summary**

1.  **Locate the Target File**: The agent began by finding the `sessions.py` file in the `requests` library directory. It correctly identified that the relevant file was in the `requests` directory rather than the build directory.

2.  **Search for the Issue**: It opened `sessions.py` and used the `search_file` command to locate the `builtin_str` function call. It found the relevant line (line 428) where the `method` variable was being converted.

3.  **Edit the Code**: The agent edited the `sessions.py` file to check if the `method` was a bytes object and, if so, decoded it to ASCII before applying the `builtin_str` function.

4.  **Test Changes**: To verify the fix, the agent created a new file, `reproduce_issue.py`, which attempted to send an HTTP request using a binary method string. The script printed the response status code to check if the request was handled correctly. It ran the `reproduce_issue.py` script, which executed successfully and received a `200 OK` response, indicating that the issue was resolved.

5.  **Clean Up and Submit**: Finally, the agent deleted the reproduction script and submitted the changes to the code base.

**Evaluation.** Upon evaluation, the agent's proposed changes successfully pass the instance's unit tests. When comparing the agent's generation to the 'gold patch', we notice that the gold patch uses the `to_native_string` function, defined in requests/utils.py, which behaves identically to the agent's proposed changes. Ultimately, while the agent successfully resolves the issue, it fails to use existing utility functions, which could lead to less maintainable or portable code in production.

## Analysis for pylint-dev/pylint-5859

**Issue Summary.** The issue involves the `--notes` option in `pylint`, which fails to report fixme warnings (W0511) for note tags that are entirely punctuation. Specifically, when running `pylint` with the `--notes="YES,???"` option, only the "YES" tag is recognized, and the "???" tag is ignored. The expected behavior is to recognize the punctuation-only tag normally.

**Interaction Summary**

1.  **Reproducing the Issue**:
    - The agent started by creating a `test.py` file and added comments to replicate the issue described. This was followed by running `pylint` to confirm the issue, which only reported the "YES" note and not the "???" note.

2.  **Searching for Relevant Code**:
    - The agent attempted to search for the handling of the `--notes` option using the `search_dir` command. After some initial confusion with the command options, it successfully identified the `pylint/checkers/misc.py` file as containing the relevant code.

3.  **Editing the Code**:
    - The agent located the regular expression responsible for matching note tags within the `open` method of the `EncodingChecker` class. It modified the regular expression to handle punctuation-only note tags by replacing the word boundary `\b` with a lookahead assertion `(?:\W|$)`.

4.  **Test Changes with Reproduction Code**:
    - After making the modification, the agent re-ran `pylint` on the `test.py` file. The output confirmed that both note tags ("YES" and "???") were now correctly identified.

5.  **Clean Up and Submit**:
    - Finally, the agent removed the `test.py` file and submitted the changes.

**Evaluation** The agent effectively replicated the issue, identified and navigated to the relevant code, made a targeted modification to the regular expression, and verified the fix through testing. While the agent's solution successfully passes the necessary tests, there are slight differences in behavior between the agent's solution and the gold patch. In particular, the agent replaced the `\b` word boundary in the regular expression to handle note tags with the regex pattern `(\W|$)` to mark the tag boundary. Comparing this to the gold patch solution which uses the pattern `(:|\s|\Z)` in the same way, it may be that the agent's solution causes other unintended consequences that aren't captured by the existing tests.

## Analysis for sympy/sympy-21614

**Issue Summary.** The issue is reporting an `UndefinedKind` type returned when printing a `sympy.Derivative`'s `kind` attribute. They expect something like a `DerivativeKind` instead of `UndefinedKind`.

**Interaction Summary**

1.  **Reproduce the Issue**: The agent began by creating a Python script (`reproduce_issue.py`) to reproduce the issue, using the reproduction code provided in the issue report. It ran the script and successfully reproduced the issue.

2.  **Searching for Derivative Code**: The agent begins to search for code where the `Derivative` class is defined. It makes several attempts with different strategies to find the relevant code. After a few unsuccessful approaches, looking for particular filenames, it searches simply for "class Derivative", which provides a single search result. It continues to read the `Derivative` class definition, which spans many hundreds of lines.

3.  **Exhausting Budget**: It continuously scrolls through the `function.py` file where `Derivative` is defined until is exhausts its budget and terminates the episode without submitting.

**Evaluation** The agent makes a few mistakes using the search function effectively. The agent should have made more efficient searches by using more commonsensical patterns (e.g. searching for "class Derivative") early on, instead of more speculative search attempts. In this particular setting, the gold patch shows that the updated code involves overriding an inherited property `kind` from the **sympy.basic** object. This fix would require a more thorough understanding of the codebase and inheritence structure of `Derivative` to be effective. An agent attempting to resolve this issue would need to have more targeted and efficient search approaches to improve its ability to identify this problem.

## Analysis for django/django-14411

**Issue Summary.** The issue involves the `ReadOnlyPasswordHashWidget` in the Django admin interface. The label element for this widget has a 'for' attribute that points to a non-labelable element. Since the `ReadOnlyPasswordHashWidget` renders text rather than an input element, the label should not have a 'for' attribute.

**Interaction Summary**

1.  **Locate the Relevant File**: The agent began by searching for the `ReadOnlyPasswordHashWidget` definition using the `find_file` command, but found no results. It then used the `search_dir` command to find references to the widget within the repository and located relevant matches in several files. The agent chose to first examine the `helpers.py` file, but found it only contained a reference to the widget and did not include the label rendering code.

2.  **Explore Related Files**: Next, the agent opened the `forms.py` file in the `django/contrib/auth` directory, which defined the `ReadOnlyPasswordHashWidget` and its associated `ReadOnlyPasswordHashField`. It identified that the widget used a template (`read_only_password_hash.html`) for rendering, but the template did not include relevant information relating to the Widget's `for` attribute.

3.  **Edit the Class Definition**: Returning to `forms.py`, the agent found the `UserChangeForm` which used the `ReadOnlyPasswordHashField`. It reasoned that Django's form rendering system likely added the 'for' attribute automatically. To address the issue, the agent attempted to write a `label_tag` method for the `ReadOnlyPasswordHashField` to exclude the 'for' attribute. The agent made several edits to this effect, importing the necessary utilities from `django.utils.html` and adding a custom `label_tag` method.

4.  **Submit Without Testing**: After the desired changes were made, the agent proceeds to submit without testing as it assumes that a Django server cannot be run and tested in this environment.

**Evaluation** The agent correctly identified the need to customize the label rendering for the `ReadOnlyPasswordHashField` class. However, its solution failed to solve the issue since it did not understand how labels were defined for a `Widget` object. In actuality, the model needed to override the `id_for_label` property defined in the `Widget` class to change this behavior properly. Similar functionality can be seen in other parts of the codebase.

# Statements

## Ethics & Broader Impacts

Prior works have established how the growing usage of LMs that can code also carries a certain amount of risk. We identify three main risks that could arise when building and using a system like SWE-agent, then discuss how we incorporates measures that mitigate such problems.

First is the security risks that come with executing LM-generated code on device. When evaluating on software engineering tasks that feature an incredibly diverse number of issue descriptions, running code generations on a personal computing device can have negative side effects, such as the unintentional removal of digital assets (e.g., `rm -rf asset/`). To defend against this, we design SWE-agent to use ephemeral containers for both inference and evaluation. SWE-agent's execution environment and the SWE-bench evaluation framework are both carried out in sand-boxed code environments, which is made possible with Docker. Executing code in a Docker container ensures that its effects are mostly isolated from the rest of the system. While not considered as secure as virtualized hardware isolation, the namespace isolation provided by Docker containers is deemed sufficient for code that is not deliberately engineered to exploit recent container escape vulnerabilities.

Second, if the wider community develops interest for SWE-agent and builds upon it, it is also possible that illegitimate evaluation datasets or infrastructure can be used to inject testing devices with malicious code or instructions to generate malicious code. For instance, an unofficial repository claiming to host an inference/evaluation harness for SWE-agent/bench could include a task instance with an issue description that tells the LM agent to build key logging functionality and store it in a hidden folder. To eliminate confusion and reduce the possibility of such an event, we provide clear guidelines listed on our GitHub repositories, data stores, and websites indicating the official repositories and channels that we actively maintain. We also encourage third parties to incorporate any improvements into our codebase and help with integrating such contributions.

Lastly are the consequences of software engineering agents being deployed in the real world. Prior works have conceptualized and put forth prototypes of agents that can carry out offensive security measures. It is also not difficult to imagine that a system like SWE-agent can be incorporated into pipelines resulting in the production of malicious code. SWE-agent's strong performance on SWE-bench implies that future AI systems will likely be increasingly adept in the aforementioned use cases. Releasing SWE-agent as an open source tool can support research towards designing sound, effective constraints for what software engineering agents are permitted to do. It can also serve as a system that legal experts and policy-making entities can experiment with to shape the future of what AI-driven end to end software engineering could look like.

## Reproducibility

To help the greater community reproduce the results presented in this paper and build on the SWE-agent platform, we open source all of our resources that were created for this project. The source code for the interactive pipeline, context management logic, command implementations, interface design, and everything else is entirely available in a GitHub repository. We provide extensive text and video documentation describing how to run and modify different parts of the codebase. Practitioners should be able to easily recover our findings by running the agent with simple scripts. We also open source all inference and evaluation artifacts (e.g., trajectories, code generations, evaluation execution traces, analysis notebooks). The results presented in the main and supplementary parts of this paper can be fully rendered from the data. Finally, we also maintain an active online help forum to assist with any reproduction problems or questions about how to build on ACI design and SWE-agent.

## Limitations & Future Work

The final SWE-agent configuration has a small toolkit, albeit highly effective. With SWE-agent's highly extensible design, we're excited by the prospect of adding more tools, such as web browsing or static analysis, that can leverage more signals from an issue description and codebase to improve the % Resolved performance. Many tools trialed by prior works from software engineering and language model agents, such as static/dynamic analysis, spectrum based fault localization, or test generation via fuzzing could prove useful.

Second, in this work, the ACI development process and case studies are done manually. Many components of SWE-agent were crafted from observations of recurring behavior within a single trajectory or across multiple trajectories. Automating part or all of this process could not only accelerate work built on top of SWE-agent, but also provide greater insights into developing ACI principles for agentic software engineering. Contemporary works have explored automated prompting to improve performance on traditional sequence to sequence tasks, supplanting the need for manual prompt design. Thinking about automating ACI design raises immediately interesting questions around how such systems can scrutinize and iterate upon their own designs. Ensuring such horizon leads to incremental performance improvements across a longer horizon is also a challenging question.

Finally, the scope of SWE-agent is exclusively focused on programmatic tasks like software engineering and code generation. We're curious to see whether the same principles of ACI and our observations of agent behavior are transferable to different domains. Recent work around applying LM agents to a variety of digital work applications have proliferated, such as use cases in education technology, data analysis, and enterprise workflows. We hope that thinking about improving performance of agentic workflows on these domains through the lens of ACI design can be a symbiotic process. For instance, for a task such a shopping on the web, in place of a typical Google-style search tool, could agents benefit from additional information beyond a list of each page's title and snippet? Would the design vary if the nature of the downstream task were to change slightly? For a completely different task, such as navigating an internal company knowledge base to help a recently on-boarded employee, how might the search interface be best adjusted to the agent?

Similar to the progression of the field of User Experience (UX) and Human Computer Interaction (HCI) research, applying ACI to other domains could not only yield improvements in downstream task performance, but also further expand the list of ACI principles. We believe that the fundamental motivations for ACI, the foundational principles we put forth, and our case study of SWE-agent as an instantiation of implementing and improving an ACI can motivate such work.
