--- title: "WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale" author: Shuyan Zhou, Duke University contact: {shuyanzhdot@gmail.com} date: March 2026 ---

Collage of browser-use agent trajectories across generated environments — browser-use agents navigating generated environments across diverse application domains.

Realistic web environments with verifiable tasks are essential for training and evaluating general-purpose browser-use agents. However, constructing such environments remains prohibitively labor-intensive, requiring manual effort across sandbox creation, data curation, task design, and programmatic verifier authoring[^webarena][^osworld][^visualwebarena][^agentcompany]. The required expertise, often including significant programming effort, further limits scalability, confining large-scale environment construction largely to well-resourced organizations. We aim to automate what has traditionally been a manual, expert-driven process. In this work, we propose an approach for automatically generating high-authenticity, high-complexity, and RL-ready environments, along with verifiable tasks, from static artifacts such as recorded workflows and user manuals using a multi-agent system. The resulting pipeline is both time- and cost-efficient: each environment can be generated within ten hours at a cost under $100, and the process is highly parallelizable. A key insight of our approach is that coding agents with *privileged access* to source code and execution artifacts (e.g., task trajectories) can drive targeted and effective improvements. Rather than treating environments as black boxes, these agents can directly inspect, debug, and modify both environment implementations and task logic. This capability enables an iterative, self-improving pipeline that performs cycles of generation, verification, and refinement to continuously improve (1) the functional correctness of the environment and (2) the correctness and difficulty of the verifiable tasks ({@sec:approach}). We further introduce a lightweight environment protocol that supports fast resets, easy replication, and scalable deployment, making the resulting environments well-suited for RL training without sacrificing realism ({@sec:env-protocol}). Despite being fully automatically generated, our environments present a meaningful challenge for current browser-use agents. Strong models with a sophisticated harness (Gemini-3-Flash with Browser Use[^browseruse]) achieve an average success rate of 69.3%, while vision-based approaches such as Kimi-K2.5[^kimik25] and Qwen-3.5-Plus[^qwen35] achieve 45.9% and 49.1% respectively ({@sec:performance}). These results fall below their performance on established, manually constructed benchmarks such as WebArena and OSWorld, suggesting that our approach captures meaningful task complexity without manual design. Ablations further confirm the importance of key components ({@sec:ablations}). In this initial release, we provide ten environments along with 1,260 verifiable tasks spanning multiple domains, including careers, DevOps, finance, healthcare, productivity, and project management. We additionally release 2,070 successful trajectories collected from various browser-use agents to support training and further research. --- ## Self Auditing and Self Challenging Environment and Verifiable Task Generation {#approach} [toc: Approach]

Pipeline overview — Overview of our approach

Our approach coordinates two types of agents: a *coding agent*, which generates and repairs the web environment, tasks, and verifiers, and a *browser-use agent*, which executes tasks through the front-end UI in the same way as an end user. The pipeline proceeds in five stages. (1) The coding agent generates a web application conditioned on static artifacts such as recorded workflows and user manuals. (2) The coding agent produces functional testing tasks together with programmatic verifiers; the browser-use agent executes these tasks, and the coding agent audits failures, iterating up to N times to repair environment bugs. (3) The coding agent generates realistic, difficulty-graded tasks with verifiers, followed by the same evaluate-audit loop. (4) The coding agent analyzes the browser-use agent’s prior trajectories to synthesize progressively more challenging tasks over R rounds. (5) A final regression evaluation runs the full task suite to verify that edits introduced in later stages have not degraded functional correctness. We describe each component in detail below. ### Web Environment Generation Although coding agents can generate arbitrary websites, unconstrained generation often produces environments that lack the structural fidelity needed to approximate real-world deployment conditions. We therefore condition generation on artifacts derived from real applications, specifically product user manuals. User manuals provide a strong foundation for environment construction: they describe core concepts, document task workflows, and often specify UI layout and interaction patterns (e.g., “click the top-right button to access settings”). We select ten functional sections from eight applications, each corresponding to a coherent subset of product capabilities (e.g., the planning and tracking module of GitLab). The coding agent is given a detailed generation prompt together with the relevant manual sections; the full prompt is provided in the appendix. We defer discussion of specific implementation choices, such as UI element constraints, to the following section. ### Functional Correctness Verification Although coding agents such as Claude Code may run many tests during web application generation, these checks do not guarantee functional correctness, particularly front-end interaction. We therefore introduce a functional correctness verification stage in which the coding agent generates tasks and corresponding verifiers to test the basic functionality of the website. A browser-use agent then executes these tasks through the front-end UI to assess whether they can be completed successfully in practice. These tasks are intentionally simple and fundamental; a capable browser-use agent should be able to complete them reliably. In our experiments, this set of tasks achieves a pass rate above 90% on average. We therefore use performance on this stage as an indicator of the website’s functional correctness. **Auditing:** After the browser-use agent completes the tasks, we launch a separate coding agent as an auditing agent to review the recorded trajectories, with particular attention to failed cases, and produce an audit summary. The auditing agent determines whether each failure is attributable to an implementation bug or to limitations of the browser-use agent. It then fixes confirmed environment bugs accordingly. The auditing process terminates once the auditing agent concludes that the remaining failures are best explained by browser-agent error rather than implementation issues. As a concrete example, one of the generated function tasks inside GitLab Plan & Track environment tests whether a user can delete an issue:

Task definition

{
  "id": "task_7",
  "instruction": "Delete the issue 'Markdown preview
    renders incorrectly with nested code blocks'.",
  "verify": "function-tasks/task_7.py"
}

Verifier

def verify(server_url: str) -> tuple[bool, str]:
    resp = requests.get(f"{server_url}/api/state")
    state = resp.json()
    issues = state.get("issues", [])
    target = next((i for i in issues
        if i["title"] == "Markdown preview renders "
           "incorrectly with nested code blocks"),
        None)
    if target is not None:
        return False, "Issue still exists."
    return True, "Issue has been deleted."

While the backend logic supports this action, the browser-use agent failed to attempt this task due to missing the `delete` UI element. The auditing agent identified this by reading the failed trajectory and applied the fix:

  html += '<button class="btn btn-outline" data-action="edit-issue" ...>Edit</button>';
  html += '<button class="btn ..." data-action="toggle-issue-status" ...>Close issue</button>';
+ html += '<button class="btn btn-danger" data-action="delete-issue" ...>Delete</button>';

### Verifiable Realistic Task Generation and Hardening The procedure for realistic task generation and auditing closely follows that of the previous stage, although it uses different prompts tailored to realistic task construction. In particular, we introduce explicit definitions of difficulty levels and provide examples of appropriate and inappropriate realistic tasks. **Hardening:** Even with explicit difficulty-level guidance, coding agents still tend to generate tasks that are comparatively simple. We therefore introduce a hardening step in which the model reviews prior agent execution results and logs to assess the current capability frontier, and then continuously generates additional tasks that are more challenging. Below, we present several progressions from initial tasks to hardened tasks. In each case, the hardened task is not an artificial chain of unrelated actions; instead, it becomes more difficult through implicit prerequisites, conditional reasoning, or domain-specific constraints that arise naturally from the environment.

	Increasing difficulty ▶
Career portal jobs & events	"Save the Microsoft Software Engineer Intern position."	"Leave a comment on Jessica Park's post about her Meta return offer saying 'Congrats! Would love to hear more about your experience.'"	"RSVP to every upcoming event hosted by an employer (not the career center), and save each of those employers' active jobs you haven't saved yet."
Prescriptions pharmacy queue	"Approve the pending Gabapentin refill request."	"Approve the Gabapentin refill request but change the sig to 'Take 1 capsule by mouth twice daily'."	"The patient's Penicillin allergy was cleared by recent testing. Remove it from the chart. Then discontinue the current Amoxicillin 500mg (the patient was hesitant to take it while the allergy was listed). Prescribe a fresh course of Amoxicillin-Clavulanate 875mg/125mg tablet twice daily for 10 days for acute sinusitis [..] send to CVS #4521."
Presentations transition normalization	"Change the Thank You slide's transition type to push."	"Replace every dissolve transition in the deck with a push-left transition."	"Change the deck's default template style to Warm Sunset and set the default transition to slide_out from the bottom with bounce easing at 500ms. Also apply the 'with total' slide number format to every slide that currently uses the Minimal Dark template."
Digital wallet payments & offers	"Stop getting those weekly digest emails."	"Save all the food and drink cashback offers that are currently available."	"Transfer $1,000 from savings to my main balance, then split it: convert $500 to euros and invest $500 in Bitcoin."

### Configurations We use Claude Code (Opus 4.6, high-effort setting with planning mode) as the coding agent, and Gemini-3-Flash with the Browser Use harness as the browser-use agent. We additionally equip the coding agent with a frontend-design skill to support web development. We note that a stronger browser-use agent can further improve the generation quality. However, such improvements introduce additional computational cost and reduce iteration efficiency. --- ## Design Choices In this section, we highlight several key design choices that improve generation quality and make the environments more suitable for RL training at scale. ### Text-based Interaction as a First-Class Citizen We deliberately design our environments to be compatible with browser-use agents that rely primarily on text-based observations, such as enhanced accessibility trees. This choice is motivated by the observation that language models remain more robust and generalize better when operating over textual representations. To support this, we encourage the use of accessibility best practices and browser-native UI elements that can be fully represented in the DOM (e.g., popups, dropdowns). This ensures that critical interface elements and interactions are consistently exposed to the agent. In addition, we disable vision input when using the browser-use agent to improve efficiency and execution speed, making the agent purely text-based. ### Enhancing Verifier Reliability Verifier correctness is a critical requirement for RL training. To ensure reliability, we introduce two layers of guardrails: (1) Programmatic Sanity Check: We first run a program with privileged backend access to directly manipulate the application state and then invoke the same verifier used for agent evaluation. Since this execution path should be correct by construction, any failure indicates a likely issue with the verifier itself. This procedure is integrated into the environment generation pipeline, allowing the coding agent to automatically detect and fix verifier inconsistencies. (2) Automatic Auditing: As described earlier, we further employ an auditing agent to analyze failed task trajectories and identify whether failures stem from environment bugs or agent limitations. This provides an additional layer of validation for both task design and verifier correctness. ### Standalone Verifiers Given full access to both application state and functionality, we design verifiers to be fully standalone through an exposed API. This cleanly separates verification from task execution. In contrast to prior work such as WebArena, which relies on browser-based inspection of rendered pages, our approach avoids the need for expensive browser-side validation and enables more efficient and reliable evaluation. ### The Environment Protocol {#env-protocol} We design our environments to be lightweight, easily replicable, and efficiently resettable, without sacrificing realism. Each environment is a self-contained directory of static HTML/CSS/JS files, served by a lightweight Python HTTP server with no database or external dependencies. The key design decision is that the browser owns all application state. On every user interaction, the browser serializes its full state and pushes it to the server, which simply stores the latest copy in memory. This turns the server into a passive state mirror that verifiers can query without touching the UI — cleanly separating task verification from task execution. The server also captures the first state push as an immutable *seed snapshot*, which is all that is needed for resets: restore the snapshot in memory and notify the browser via Server-Sent Events to reload. No database rollbacks, container restarts, or filesystem cleanup required. This contract is implemented through a minimal API:

PUT  /api/state   → browser pushes state on every mutation
GET  /api/state   → verifiers read current state (JSON)
POST /api/reset   → restore seed snapshot, notify browser via SSE
GET  /api/events  → SSE stream the browser listens to for resets

Because the entire environment is defined by its source directory and a few hundred lines of server code, replication reduces to copying files, and running multiple instances simply requires launching separate processes on different ports. This makes it straightforward to scale to many parallel RL rollouts on a single machine. --- ## Environment and Task Statistics

Overview of the ten generated environments.

Environment	Domain	Source	Tasks	Any-Agent Success
Career Exploration	Careers	Handshake	200	144
Project Planning	DevOps	GitLab	140	93
Accounting & Invoicing	Finance	Xero	120	100
Personal Wallet	Finance	PayPal	140	133
Clinical Records	Healthcare	Elation EHR	120	106
Prescription Management	Healthcare	Elation EHR	120	102
Email Management	Productivity	Gmail	60	55
Accounts & Contacts	Productivity	Gmail	120	89
Email Client	Productivity	Superhuman	120	68
Account Settings	Project Mgmt	Linear	120	100
Total			1,260	990

We present summary statistics for the ten environments included in our initial release. As our pipeline is fully automated with minimal human intervention, we report in the final column the number of tasks for which at least one agent succeeds. This metric serves as a conservative lower bound on task validity and overall environment quality. **Filtering Criteria** In our manual examination, we find that the functional test success rate serves as a strong indicator of environment quality. We therefore use it as a primary filtering criterion, excluding any environments with a functional test pass rate below 50% from the release. While many of these issues are likely fixable with additional iterations, we apply this threshold to ensure a consistent quality bar without significant human intervention. --- ## Performance of Browser-use Agents {#performance}

Real-task success rate by app and model — Overall and breakdown success rate (%) of three browser-use agents

**Browser-use Agents** We evaluate a text-based browser-use agent using the Browser Use harness with `gemini-3-flash-preview` as the underlying model. The Browser Use harness supports both ID-based interaction and the ability to write and execute JavaScript, providing additional flexibility for complex operations. We also evaluate two vision-based agents, `kimi-k2.5` and `qwen-3.5-plus`. For these agents, we implement error-handling strategies, such as retry mechanisms, to mitigate limitations introduced by comparatively less sophisticated interaction harnesses. Across 1,260 real tasks spanning ten environments, Gemini-3-Flash with Browser Use achieves 69.3% (873/1260), followed by Qwen-3.5-Plus at 49.1% (619/1260) and Kimi-K2.5 at 45.9% (578/1260). Performance varies considerably across environments, with PayPal My Wallet being the easiest environment for all agents. While Kimi-K2.5 underperforms Qwen-3.5-Plus overall, it surpasses Qwen-3.5-Plus by a significant margin on Gmail-related applications. The success rates observed here are notably lower than those reported on established benchmarks. For example, Kimi-K2.5 achieves 58.9% on WebArena and 63.3% on OSWorld-verified, compared to 45.9% on our environments. Similarly, Qwen-3.5-Plus scores 62.2% on OSWorld-verified versus 49.1% in our environments. These results suggest that our fully automated environment generation produces meaningfully challenging tasks while still leaving substantial room for future improvement, highlighting the potential of our approach at larger scale. --- ## Ablation Studies {#ablations} In this section, we perform ablation studies to validate the necessity and impact of different components in our approach. ### Richer Source Artifacts Yield More Complex and Authentic Environments [toc: Impact of the Artifacts]

Functional complexity comparison: docs vs no-docs — Ablation comparing environments generated with full documentation (■ blue) versus a brief description only (■ red). Top-left: Number of unique functional operations. Top-right: Task diversity measured by mean 5-NN cosine distance of task embeddings. Bottom: Feature authenticity measured by precision and recall against the user manuals

Task diversity comparison: docs vs no-docs — Ablation comparing environments generated with full documentation (■ blue) versus a brief description only (■ red). Top-left: Number of unique functional operations. Top-right: Task diversity measured by mean 5-NN cosine distance of task embeddings. Bottom: Feature authenticity measured by precision and recall against the user manuals

A central design decision in our pipeline is conditioning environment generation on artifacts. A natural question is: *how much does the richness of source artifacts matter?* To answer this, we conduct a controlled ablation comparing environments generated from full product documentation against those generated from a brief, high-level application description. We select five applications and generate paired environments with all other pipeline parameters held constant. We evaluate the impact along three metrics, namely functional complexity, task diversity, and feature authenticity.

Expand to see the detailed metric definitions

Functional Complexity. We use Claude to extract distinct user-facing capabilities from each generated application's source code and count the number of unique functional operations implemented.

Task Diversity. We embed all real task instructions (excluding hardened tasks) using `Gemini Embedding 2` and measure the mean 5-nearest-neighbor cosine distance for each task set—higher values indicate that tasks are more spread out in semantic space rather than clustered around a few repetitive themes.

Feature Authenticity. We measure how faithfully each generated environment reproduces the capabilities of the real product. We use LLMs to extract user-facing capabilities from three sources—the product documentation, the generated application's source code, and the generated task instructions—producing short action phrases (e.g., "approve an invoice"). We then embed all extracted features using `Gemini Embedding 2` and compute precision and recall by matching each environment feature to its nearest documentation feature via cosine distance (threshold 0.2).

We observe consistent gains from documentation conditioning: - **More capabilities implemented.** Manual-conditioned environments produce more unique functional operations in four out of five domains, with the largest gap in Xero Invoicing (96 vs. 64). The one exception, GitLab Plan & Track, we attribute to its documentation structure: a single menu-style markdown with hyperlinks rather than a self-contained folder. - **More diverse tasks.** Tasks generated from documentation-conditioned environments are more spread out in semantic space with the clearest gains in Figma Slides (0.28 vs. 0.24) and Xero Invoicing (0.26 vs. 0.22). - **More faithful to the real product.** Manual-conditioned environments achieve higher precision and recall across four out of five domains when matching generated features against documentation-derived capabilities—indicating that product manuals substantially reduce feature hallucination and improve feature coverage. These results suggest that while coding agents can generate plausible environments from minimal specifications, more comprehensive artifacts improve the functional complexity, task diversity, and feature authenticity of the resulting environments. ### Auditing Improves Functionality and Verification Reliability [toc: Necessity of Auditing]

Pre-audit to post-audit pass rate changes — Left: Pre-audit to post-audit success rate of the browser-use agent on functional testing tasks. Right: Distribution of issues identified during auditing.

Distribution of bug categories found during auditing — Left: Pre-audit to post-audit success rate of the browser-use agent on functional testing tasks. Right: Distribution of issues identified during auditing.

We plot the success rate of the browser-use agent on functional testing tasks before and after multiple rounds of auditing across three environments. The evaluate–audit loop consistently improves functional correctness: all environments exhibit monotonically increasing pass rates over successive audit rounds. For example, in the Xero Invoicing environment, performance improves from 72% to 100% after auditing. To better understand the nature of the issues uncovered, we analyze the auditing agent’s reports and categorize each identified failure. The distribution indicates that the auditing procedure does not simply attribute all failures to limitations of the browser-use agent. Environment- and task-side errors, including application bugs, verifier bugs, and impossible tasks, account for 45% of all findings, while the remaining 55% are attributed to browser-agent limitations. These results highlight the effectiveness of the auditing procedure in improving environment quality, as unresolved issues—particularly incorrect reward signals—would otherwise compromise downstream training. At the same time, we emphasize that the generated websites are by no means bug-free; rather, auditing makes them more likely to implement the functionalities required for the generated tasks correctly. ### Self-Challenging Increases Task Difficulty and Complexity [toc: Effectiveness of Self-Challenging]

Aggregate success rate by task origin — Average success rate of Gemini-3-Flash with Browser Use across different difficulty levels

We compute the average success rate of Gemini-3-Flash with Browser Use across real tasks at different difficulty levels and observe a monotonic decrease in performance. In particular, the self-challenging stage produces tasks that are substantially more difficult than those generated in the initial round. The browser-use agent’s average success rate drops from 54.2% on the original hard tasks to 41.9% on hardened tasks. This gap indicates that, while the original hard tasks are generated solely based on the website implementation, incorporating prior execution trajectories enables the coding agent to further identify and target weaknesses beyond this initial difficulty frontier. ### Limited Functional Regression Across Iterations [toc: Regression Test]

App	Initial	Final	δ
Elation Clinical	100.0%	98.2%	-1.8pp
Xero Invoicing	98.3%	94.8%	-3.4pp
GitLab Plan & Track	98.2%	98.2%	+0.0pp
PayPal My Wallet	96.4%	96.4%	+0.0pp
Superhuman	93.3%	73.3%	-20.0pp
Elation Prescriptions	87.5%	89.3%	+1.8pp
Elation Patient Comm.	75.9%	32.8%	-43.1pp

Functional task pass rate before and after the full pipeline. Green indicates minor or no regression; red indicates significant regression.

A natural concern with an iterative procedure is whether later stages introduce regressions in the base environment’s functional correctness, potentially invalidating previously generated tasks. To assess this, we compare the functional task pass rate at two points: immediately after the initial generation and auditing stage, and at the end of the full pipeline. We observe that five out of seven environments maintain pass rates close to their initial levels, while two exhibit substantial regressions. These results suggest that, in most cases, functional correctness is preserved throughout the iterative process. This analysis also enables us to filter out low-quality environments using simple heuristics without requiring manual inspection. For example, we manually interacted with the Elation Patient Communication environment and found it to be largely unusable, consistent with its degraded functional task pass rate. This reinforces the value of functional pass rate as an effective quality gate. [^webarena]: Zhou et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." ICLR, 2024. [^osworld]: Xie et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." NeurIPS, 2024. [^visualwebarena]: Koh et al. "VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks." ACL, 2024. [^agentcompany]: Xu et al. "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks." ICLR, 2025. [^browseruse]: Browser Use. "Enable AI to Access the Web." GitHub, 2024. [^kimik25]: Kimi Team. "Kimi K2.5: Visual Agentic Intelligence." arXiv:2602.02276, 2026. --- ## Citation ```bibtex @article{zhou2026wainf, title = "WebArena-Infinity: Generating Browser Environments with Verifiable Tasks at Scale", author = "Shuyan Zhou", journal = "shuyanzhou.com", year = "2026", month = "March", url = "https://webarena.dev/webarena-infinity/" } ``` [^qwen35]: Qwen Team. "Qwen3.5: Towards Native Multimodal Agents." 2026.