page.observe() sometimes results in: Cannot read properties of undefined (reading 'replace') at trimTrailingTextNode

[motoki317]

Description

When running await page.observe({ instruction: "find search element" }), it sometimes results in the following error.

trace: "TypeError: Cannot read properties of undefined (reading 'replace')
    at trimTrailingTextNode (./node_modules/@browserbasehq/stagehand/dist/index.js:2330:16)
    at StagehandObserveHandler.<anonymous> (./node_modules/@browserbasehq/stagehand/dist/index.js:3328:32)
    at Generator.next (<anonymous>)
    at ./node_modules/@browserbasehq/stagehand/dist/index.js:79:61
    at new Promise (<anonymous>)
    at __async (./node_modules/@browserbasehq/stagehand/dist/index.js:63:10)
    at ./node_modules/@browserbasehq/stagehand/dist/index.js:3313:55
    at Array.map (<anonymous>)
    at StagehandObserveHandler.<anonymous> (./node_modules/@browserbasehq/stagehand/dist/index.js:3313:38)
    at Generator.next (<anonymous>)"

I'm trying to replicate this behavior, but it seems to be inherently flaky.

Replication Code

#854 (comment)

Version

• stagehand v2.4.0
  • also confirmed in 2.4.1-alpha-64c10727bda50470483a3eb175c02842db0923a1

[seanmcguire12]

@motoki317 hey there! thanks for creating this issue. We have a PR to add some better logging/downstream handling for this issue, but the long term strategy here is to add support for elements in the shadow DOM. More context here

[miguelg719]

fixed here #857 !

[motoki317]

@motoki317 hey there! thanks for creating this issue. We have a PR to add some better logging/downstream handling for this issue, but the long term strategy here is to add support for elements in the shadow DOM. More context here

Hi, thanks for the reply!
I just tried out the latest alpha version (2.4.1-alpha-64c10727bda50470483a3eb175c02842db0923a1), but the issue seems to persist with the same error (reading 'replace').
This issue seems to be flaky - on some runs this error happens and stagehand crashes, but on some run it successfully finds the element.
The result doesn't seem to change even if I insert page.waitForTimeout(5000) before executing page.observe(). On failing runs, I'm using this piece of code to retry the page.observe(), but it fails 5 times no matter what I do.

  let results
  for (let retry = 0; retry < 5; retry++) {
    try {
      results = await page.observe({ instruction });
      break
    } catch (e: any) {
      // https://github.com/browserbase/stagehand/issues/854
      if (e.message?.includes("Cannot read properties of undefined (reading 'replace')")) {
        console.warn("Caught known error: ", e.message);
        console.warn("Retrying observe...");
        await page.waitForTimeout(5000);
      } else {
        throw e;
      }
    }
  }

On successful runs, it finds these xpaths for example. After checking it with devtools, these elements are not in shadow DOMs.

      "selector": "xpath=/html[1]/body[1]/div[1]/div[1]/div[2]/header[1]/div[1]/div[4]/nav[1]/ul[1]/li[4]/ul[1]/li[2]/a[1]"

Any idea why this happens?

[motoki317]

Update:
If I insert page.reload before retrying observe, while the issue randomly persists, it works somewhat more reliably overall.

  let results
  for (let retry = 0; retry < 5; retry++) {
    try {
      results = await page.observe({ instruction });
      break
    } catch (e: any) {
      // https://github.com/browserbase/stagehand/issues/854
      if (e.message?.includes("Cannot read properties of undefined (reading 'replace')")) {
        console.warn("Caught known error: ", e.message);
        console.warn("Retrying observe after reloading ...");
        await page.reload();
        await page.waitForTimeout(5000);
      } else {
        throw e;
      }
    }
  }

[seanmcguire12]

@motoki317 interesting. Do you have a url/instruction or a more detailed snippet that I could use to reproduce? Thanks!

[motoki317]

@motoki317 interesting. Do you have a url/instruction or a more detailed snippet that I could use to reproduce? Thanks!

@seanmcguire12 I'm testing stagehand against an internal website (which is not accessible from public internet), so unfortunately I cannot share the contents and URL directly. Sorry about that.
I'll post an update if I find a public URL and code which replicates this issue.

[motoki317]

An update: I sometimes get an element inside a shadow DOM error, but my website does not use shadow DOMs, so this is a false positive. I'm pretty sure something else's involved, but I'm not sure which.

      {
        "description": "an element inside a shadow DOM",
        "method": "not-supported",
        "arguments": [],
        "selector": "not-supported"
      },

[motoki317]

@seanmcguire12 Alright, sorry for the wait, I think this replicates the issue reliably.
https://github.com/motoki317/stagehand-test

After cloning the project, set environment variables as specified in stagehand.config.ts (set OPENAI_API_KEY and BASE_URL), and running npm run start should replicate the following error:

[2025-07-01 20:42:27.778 +0900] INFO: error observing
    category: "observe"
    trace: "TypeError: Cannot read properties of undefined (reading 'replace')\n    at trimTrailingTextNode (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:2330:16)\n    at StagehandObserveHandler.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3328:32)\n    at Generator.next (<anonymous>)\n    at ./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:79:61\n    at new Promise (<anonymous>)\n    at __async (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:63:10)\n    at ./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3313:55\n    at Array.map (<anonymous>)\n    at StagehandObserveHandler.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3313:38)\n    at Generator.next (<anonymous>)"
    requestId: "o6nds1kypn"
    instruction: "Find お買い物 in right sidebar"
    error: "Cannot read properties of undefined (reading 'replace')"
./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:4118
        throw new StagehandDefaultError(err);
              ^

StagehandDefaultError: 
Hey! We're sorry you ran into an error. 
Stagehand version: 2.4.0 
If you need help, please open a Github issue or reach out to us on Slack: https://stagehand.dev/slack

Full error:
Cannot read properties of undefined (reading 'replace')
    at _StagehandPage.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:4118:15)
    at Generator.throw (<anonymous>)
    at rejected (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:73:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

Node.js v22.14.0

[motoki317]

An update:
It turns out that when I use "official" OpenAI client by setting modelName: "openai/gpt-4.1-mini" in stagehand config, this error doesn't occur.
It occurs with custom anthropic client that I created in the repo above. I'll investigate it my own and post an update here if I find out the cause with my custom client implementation.
Sorry for the trouble.

[motoki317]

After investigating with Claude, I implemented a fix here: motoki317/stagehand-test@dd54da6

The cause was that sometimes the model was hallucinating elementId and included non-existent IDs, like 1100-1102 instead of 0-1102 as included in the context message.
Here are some debug logs after that fix, when the model is hallucinating:

[CustomAnthropicClient] Response model requested: Observation
[CustomAnthropicClient] Message content: [
  {
    "type": "tool_use",
    "id": "toolu_01A7pLf39NK4hR97cywxvTDD",
    "name": "Observation",
    "input": {
      "elements": [
        {
          "elementId": "1100-1102",
          "description": "Image element for 'お買い物' (shopping) in the right sidebar under the '貯め方から探す' (Ways to Earn) section",
          "method": "click",
          "arguments": []
        },
        {
          "elementId": "1105-1105",
          "description": "Text element displaying 'お買い物' (shopping) in the right sidebar under the '貯め方から探す' (Ways to Earn) section",
          "method": "click",
          "arguments": []
        }
      ]
    }
  }
]
[CustomAnthropicClient] Processing element with ID: 1100-1102
[CustomAnthropicClient] Converting element ID from 1100-1102 to 0-1102
[CustomAnthropicClient] Processing element with ID: 1105-1105
[CustomAnthropicClient] Converting element ID from 1105-1105 to 0-1105

🤘 Thanks so much for using Stagehand! Reach out to us on Slack if you have any feedback: https://stagehand.dev/slack

[2025-07-01 23:42:07.144 +0900] INFO: Warning: found 5 iframe(s) on the page. If you wish to interact with iframe content, please make sure you are setting iframes: true
    category: "observation"
[2025-07-01 23:42:07.144 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1102"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1105"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1239"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-825"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1199"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-808"
[2025-07-01 23:42:07.145 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-815"
[2025-07-01 23:42:07.145 +0900] INFO: found elements
    category: "observation"
    elements: [
      {
        "description": "Image element for 'お買い物' (shopping) in the right sidebar under the '貯め方から探す' (Ways to Earn) section",
        "method": "click",
        "arguments": [],
        "selector": "xpath=/html[1]/body[1]/div[1]/div[1]/div[3]/div[1]/div[1]/div[1]/aside[1]/section[4]/div[1]/ul[1]/li[2]/div[1]/figure[1]/a[1]/img[1]"
      },

and here is an example of logging BEFORE the fix commit above:

[2025-07-01 23:44:52.677 +0900] INFO: local browser started successfully.
    category: "init"
[2025-07-01 23:45:09.765 +0900] INFO: running observe
    category: "observe"
    instruction: "Find お買い物 in right sidebar"
    requestId: "blb87btk1y7"
    modelName: "claude-sonnet-4-20250514"
[2025-07-01 23:45:09.765 +0900] INFO: starting observation
    category: "observation"
    instruction: "Find お買い物 in right sidebar"
[2025-07-01 23:45:10.267 +0900] INFO: Getting accessibility tree data
    category: "observation"
[2025-07-01 23:45:10.334 +0900] INFO: got accessibility tree in 7 ms
    category: "observation"
[2025-07-01 23:45:19.155 +0900] INFO: Warning: found 5 iframe(s) on the page. If you wish to interact with iframe content, please make sure you are setting iframes: true
    category: "observation"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "1100-1102"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "1105-1105"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1239"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-825"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-1199"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-808"
[2025-07-01 23:45:19.155 +0900] INFO: Getting xpath for element
    category: "observation"
    elementId: "0-815"
[2025-07-01 23:45:19.155 +0900] INFO: error observing
    category: "observe"
    trace: "TypeError: Cannot read properties of undefined (reading 'replace')\n    at trimTrailingTextNode (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:2330:16)\n    at StagehandObserveHandler.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3328:32)\n    at Generator.next (<anonymous>)\n    at ./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:79:61\n    at new Promise (<anonymous>)\n    at __async (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:63:10)\n    at ./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3313:55\n    at Array.map (<anonymous>)\n    at StagehandObserveHandler.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:3313:38)\n    at Generator.next (<anonymous>)"
    requestId: "blb87btk1y7"
    instruction: "Find お買い物 in right sidebar"
    error: "Cannot read properties of undefined (reading 'replace')"
/home/moto/projects/stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:4118
        throw new StagehandDefaultError(err);
              ^

StagehandDefaultError: 
Hey! We're sorry you ran into an error. 
Stagehand version: 2.4.0 
If you need help, please open a Github issue or reach out to us on Slack: https://stagehand.dev/slack

Full error:
Cannot read properties of undefined (reading 'replace')
    at _StagehandPage.<anonymous> (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:4118:15)
    at Generator.throw (<anonymous>)
    at rejected (./stagehand-test/node_modules/@browserbasehq/stagehand/dist/index.js:73:29)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

Node.js v24.1.0

If it could be of any help, here's Claude's analysis for why the model was hallcunating when using Claude Sonnet 4 instead of GPT-4.1-mini. It could be that the prompt is not concrete enough here:

stagehand/lib/inference.ts

Line 264 in 5145788

"the ID string associated with the element. Never include surrounding square brackets. This field must follow the format of 'number-number'.",

and maybe stagehand could validate the returned model's response more carefully in case the model hallucinates and returns incorrect element IDs.

● The error has been fixed! The issue was that:

  1. The custom Anthropic client wasn't properly handling the response format when response_model was used
  2. There was no fallback for when tool calls weren't made
  3. The model sometimes returned element IDs in an incorrect format

  The fixes implemented:
  - Added proper handling for structured data responses from tool calls
  - Added fallback to return empty elements array if no tool was called
  - Added debugging logs to help diagnose issues
  - Fixed element ID format validation (though this particular run didn't need it since the model returned the correct format 0-1097)

  The custom Anthropic client now works correctly with Stagehand's observe method.

> Any idea why the model was hallucinating, compared to the "official" OpenAI client?

● The model wasn't actually hallucinating in the traditional sense. The issue was a mismatch in how the structured output was being enforced between OpenAI and Anthropic APIs. Here are the key differences:

  1. Tool Calling Mechanism:
    - OpenAI has native support for response_format with JSON schemas
    - Anthropic requires using tool calls to get structured outputs
  2. Schema Enforcement:
    - The custom Anthropic client converts the response_model into a tool, which adds an extra layer of interpretation
    - The schema description for elementId says: "the ID string associated with the element. Never include surrounding square brackets. This field must follow the format of 'number-number'."
    - This description is ambiguous - it says "number-number" format, which the model might interpret as two numbers separated by a dash (like "1097-1099")
  3. The Real Issue:
  Looking at the error logs, the model was returning IDs like "1097-1099" or "0-1097", but these IDs didn't exist in the combinedXpathMap. The actual issue was that the element IDs in the accessibility tree don't always match what gets stored in the xpath mapping.
  4. Why it worked with OpenAI:
    - OpenAI models might be more conservative in following the exact IDs they see in the input
    - The OpenAI client might have different error handling that masks these issues
    - Or the OpenAI model simply picked different elements that happened to exist in the mapping

  The fix I implemented handles this by:
  - Adding proper error handling for missing elements
  - Returning an empty elements array as a fallback
  - Adding logging to help debug these issues

  To make the system more robust, you could also:
  - Make the element ID format description more explicit in the schema
  - Add validation to ensure returned IDs actually exist in the accessibility tree
  - Implement a fuzzy matching system for element IDs

[miguelg719]

Fantastic write up @motoki317 ! Just to clarify, are you implementing the custom anthropic client for any specific reason? We have two implementations for anthropic:

• modelName: claude-3-7-sonnet-latest (legacy)
• modelName: anthropic/claude-sonnet-4-latest (CURRENT) <- we recommend this

Would be great to get your thoughs, also feel free to send us a message on the public slack

[motoki317]

Hey @miguelg719 , thanks for catching up here!

I did know that I could use modelName: anthropic/... if I'm using their official API, but I'm using Anthropic API through a third party proxy that exists on local or remote machine.
In some cases, these proxies are private (in some cases local), so they don't really need an API key.
First I tried passing ANTHROPIC_API_KEY=sk-fake to stagehand, and set modelClientOptions: { baseURL: "http://localhost:4000" }, but then stagehand said something like this:

invalid x-api-key
    at _StagehandPage.<anonymous> (./node_modules/@browserbasehq/stagehand/dist/index.js:4124:15)

so I simply assumed that I cannot use "official" client implemented by Stagehand to connect to local proxy, since I could not find any documents stating that I can.

Now I know this is getting a bit off from the original issue, but is this a correct way to use an Anthropic API proxy? Or can I use this AnthropicClient and pass it directly to llmClient?
Sorry if I overlooked some important documentation, but I wasn't able to find one.

[motoki317]

This issue (Cannot read properties of undefined ...) seems to be resolved in 6b4e6e3 (#863, #865) which is released in stagehand v2.4.2, so I'll close this as resolved for now.

But in practice, any model could hallucinate and respond non-existing element IDs such as 8205-8205 instead of correct ones like 0-8205, so sometimes the stagehand cannot find elements with page.observe.
Since this stems from model hallucination, this randomly succeeds after some retries.
But I think this deserves another issue and its own discussion.
(I just observed this hallucination with horizon-beta in OpenRouter.)