[Feature]: specify timeout ms for networkidle

[DetachHead]

Feature request

sometimes, 500ms isn't long enough

[akshayp7]

await this.page.waitForNavigation({ waitUntil: networkidle, timeout: 20000});

timeout value in ms

You can use above snippet and try

[DetachHead]

that timeout is waiting for the networkidle event itself to be fired. the timeout i'm talking about is the 500ms hardcoded wait once the network is idle before firing the event, mentioned here:

• 'networkidle' - wait until there are no network connections for at least 500 ms.

sorry i should've made that more clear.

[dgozman]

This is a valid request, but seems low-priority. If you have more information about the usecase for this feature, it would help the prioritization.

[DetachHead]

my current use case is a sapui5 application. it seems quite common for ui5 apps to wait for a few seconds after a page has loaded before triggering additional network requests, and this often takes longer than 500ms

[radekBednarik]

I would love this to be implemented as well ❤️

[mindplay-dk]

The real problem here might be the description isn't actually correct?

wait until there are no network connections for at least 500 ms

This does not seem to wait for 500 msec for me - or really any time at all?

I have a while (await ...) loop, and if, in the awaited function, I do await page.waitForLoadState("networkidle"), the script races through 50 iterations in about 2 seconds - if I replace that with await page.waitForTimeout(500), the same loop takes 30-40 seconds to complete, so it's pretty clear that it isn't waiting for 500 msec. It doesn't appear to wait at all.

I suspect maybe it's forgetting to clear some state between calls? Waiting for networkidle seems to work the first time - but when called repeatedly, it appears to wait only the first time. Perhaps the second time, it just looks and goes, "the network is stll idle since the last time you checked"? That doesn't really work - it should wait for "at least 500 ms" with every call, as prescribed in the documentation.

I don't think this really works?

[mindplay-dk]

Since it doesn't do what it says in the manual, for the time being, you can work around this by just using two awaits:

await page.waitForTimeout(500); // "at least 500 msec"
await page.waitForLoadState("networkidle"); // does not wait for at least 500 msec, so 🤷‍♂️

This doesn't quite do what you want, since it will only be checking for network idle for the latter 500 msec - but it does give you a minimum wait time, like the documentation says.

Either way, yeah, we definitely need a way to control this. It also needs to work. 😉

[DetachHead]

@mindplay-dk waitForLoadState will only wait once per navigation. for example:

await page.click("button") //also waits for the initiated navigation
await page.waitForLoadState("networkidle") //this might not wait at all if the navigation has already finished

instead you should use a Promise.all to make sure you're waiting on the right navigation:

await Promise.all([
  page.waitForLoadState("networkidle"),
  page.click("button"), // Click triggers a popup.
])

see https://playwright.dev/docs/events#waiting-for-event

[mindplay-dk]

@mindplay-dk waitForLoadState will only wait once per navigation.

Ah, I see... it wasn't really clear to me from the documentation that this was navigation dependent...

So there is actually no way to just wait for network idle at any given time?

My use-case is, I'm actually trying to wait for the browser to stop "doing things", before ordering it to do another thing - and these aren't just specific resources I could wait for them to load, and they aren't function calls either... I'm waiting for various scripts on various pages to "do things", and I can't know what these things are, so.

Ideally, I'd like to wait until there's nothing in the browser's internal loop - e.g. no pending setTimeout calls, unresolved promises, and so on. Waiting for the event loop to flush would be ideal, but I don't guess there's any easy/obvious way to do that in the browser as such. I might have to do some money-patching of window.setTimeout and the Promise constructor to make that happen... 🤔

[Filipoliko]

We have been dealing with similar issue, that we needed to wait for network to have no pending requests at the moment. We created a helper function to deal with this, which can be used as fixture. It requires wait-for-expect npm package to work properly.

import waitForExpect from 'wait-for-expect';

/**
 * Implements network related helpers
 */
export default class NetworkHelper {
    constructor(page) {
        this._page = page;
        this._pendingRequests = new Set();

        this._initNetworkSettledListeners();
    }

    /**
     * Waits until there are no pending requests.
     * This is very useful, when you are waiting for resources to be loaded,
     * but you don't know, which resources need to be loaded. This method
     * needs to be called when there are already some pending
     * requests, otherwise it will resolve immediately.
     * @param {Number} [timeout=30000]
     */
    waitForNetworkSettled(timeout = 30000) {
        return waitForExpect(() => {
            if (this._pendingRequests.size !== 0) {
                const pendingUrls = [];

                this._pendingRequests.forEach(req => pendingUrls.push(req.url()));

                throw Error(
                    `Timeout: Waiting for settled network, but there are following pending requests after ${timeout}ms.\n${pendingUrls.join(
                        '\n'
                    )}`
                );
            }
        }, timeout);
    }

    /**
     * Initializase page listeners for network request events,
     * which are later used in other methods.
     */
    _initNetworkSettledListeners() {
        this._page.on('request', request => this._pendingRequests.add(request));
        this._page.on('requestfailed', request => this._pendingRequests.delete(request));
        this._page.on('requestfinished', request => this._pendingRequests.delete(request));
    }
}

[mindplay-dk]

@Filipoliko ah, that is a very simple and elegant approach!

My crazy idea was to basically hook and watch everything I could watch in the browser: timeouts, Promise, fetch, XMLHttpRequest, added script/iframe/etc. nodes (via MutationObserver) and on and on - I would monkey-patch them all to watch for activity. That's a lot of complexity, and a lot work. Simply waiting for a period of network inactivity obviously means you'll wait a little longer, so it's not completely ideal - but definitely wins out on simplicity in my book. Good thinking! 🙂👍

[dwiyatci]

@Filipoliko Wish I could have a Python-ported version of your helper function 😋

[rwoll]

Folded #14132 into here.

NB: Sometimes extending the overall time works, other times, you might want to implement a more custom network-based waiter function.

[rolandz83]

Thanks @rwoll, as for you comments #14132 yes you are correct, if the immediate action is click or fill auto-wait takes care of the problem, however there are many times we like to read textcontent after a page load which comes back with an empty string since it hasn't been loaded yet as waitforloadstate stops monitoring after 500ms of idle as shown in the screenshot.
The use of waitForResponse is also tricky as we reuse the same code on different pages which may have different network calls, making waitForResponse with hardcoded url ineffective.
In my case, since we dont know which network resource are being loaded, networkidle is the best tool, if only we can extend the default idle time it would be perfect.

in the meantime I will try @Filipoliko approach, thank you!

[josephwynn-sc]

My use case for this is testing our analytics script which has a "minimum measurement time", i.e. the analytics script will wait for at least the specified time before sending the beacon request. I have implemented a custom network waiter, but it would be nice if we could specify the networkidle timeout in Playwright like we can in Puppeteer.

[coffeegist]

I would also love to see this issue implemented. I have some tests running where I need to validate proper lazy loading of resources, struggling to find a great workaround other than static sleeps in my driver (which is less than ideal)

[pavelfeldman]

It is unlikely that it will be implemented. networkidle is strongly discouraged, so we aren't going to add features into it. Please file a new issue describing your exact use case if relying upon web assertions after the navigation does not resolve your issues!

[josephwynn-sc]

@pavelfeldman if you're closing this issue, could you please re-open #22661? That is an issue with a specific use case that was previously folded into this one.

[zach-betz-hln]

Thanks for your snippet @Filipoliko - we built on it. Here's our variant:

Snippet

import { Page, Request as PlaywrightRequest } from 'playwright';

export async function waitForNetworkIdle(params: {
  page: Page;
  idleInMillis?: number;
  timeoutInMillis?: number;
  intervalInMillis?: number;
  delayByIntervalOnFirstRun?: boolean;
}): Promise<void> {
  const {
    page,
    idleInMillis = 500,
    timeoutInMillis = 5_000,
    intervalInMillis = 50,
    delayByIntervalOnFirstRun = true
  } = params;

  const pendingRequests = new Set<PlaywrightRequest>();
  page.on('request', (request) => pendingRequests.add(request));
  page.on('requestfailed', (request) => pendingRequests.delete(request));
  page.on('requestfinished', (request) => pendingRequests.delete(request));

  function expectation(): void {
    if (pendingRequests.size === 0) {
      return;
    }

    const pendingUrls: string[] = [];
    pendingRequests.forEach((request) => pendingUrls.push(request.url()));
    const message = `Waiting for pending requests to settle:\n${pendingUrls.join('\n')}`;
    logger.debug(message);
    throw Error(message);
  }

  try {
    await waitForExpect({
      expectation,
      timeoutInMillis,
      intervalInMillis,
      delayByIntervalOnFirstRun
    });
    await sleep(idleInMillis);
    logger.info(`Network has been idle for ${idleInMillis}ms`);
  } catch (error) {
    logger.error(error);
    throw new Error(`Timed out after ${timeoutInMillis}ms when waiting for network to be idle for ${idleInMillis}ms`);
  }
}

export async function sleep(millis: number): Promise<void> {
  await new Promise((resolve) => setTimeout(resolve, millis));
}

// Adapted from https://github.com/TheBrainFamily/wait-for-expect
export async function waitForExpect(params: {
  /**
   * This function should throw an error on the failure condition, which will signal a retry.
   */
  expectation: () => void | Promise<void>;
  timeoutInMillis?: number;
  intervalInMillis?: number;
  delayByIntervalOnFirstRun?: boolean;
}): Promise<void> {
  const { expectation, timeoutInMillis = 5_000, intervalInMillis = 50, delayByIntervalOnFirstRun = true } = params;

  const maxTries = Math.ceil(timeoutInMillis / intervalInMillis);
  let tries = 0;

  await new Promise<void>((mainResolve, mainReject) => {
    async function runExpectation(): Promise<void> {
      tries += 1;

      try {
        await Promise.resolve(expectation());
        mainResolve();
      } catch (error) {
        rejectOrRerun(error as Error);
      }
    }

    function rejectOrRerun(error: Error): void {
      if (tries > maxTries) {
        mainReject(error);
        return;
      }

      setTimeout(runExpectation, intervalInMillis);
    }

    setTimeout(runExpectation, delayByIntervalOnFirstRun ? intervalInMillis : 0);
  });
}

Usage:

await waitForNetworkIdle({
  page,
  idleInMillis: 250,
  timeoutInMillis: 5_000,
  intervalInMillis: 50,
  delayByIntervalOnFirstRun: true
});

[karlhorky]

Thanks @zach-betz-hln and @Filipoliko 🙌 Very useful.

My version of the helper function waits for 2s of consecutive network idle (measured every 100ms), up to a timeout of 30s:

1. No dependencies
2. Single function, no options
3. setTimeout from node:timers/promises
4. Deregisters event listeners

import { setTimeout } from 'node:timers/promises';
import type { Page, Request as PlaywrightRequest } from '@playwright/test';

/**
 * Waits for 2s where the network is idle (no pending requests),
 * up to a timeout of 30s.
 */
async function waitForTwoSecondsNetworkIdle(page: Page) {
  const pendingRequests = new Set<PlaywrightRequest>();

  function onRequest(request: any) {
    pendingRequests.add(request);
  }

  function onRequestDone(request: any) {
    pendingRequests.delete(request);
  }

  page.on('request', onRequest);
  page.on('requestfinished', onRequestDone);
  page.on('requestfailed', onRequestDone);

  try {
    const intervalInMillis = 100;
    const start = Date.now();

    let idleStart: number | null = null;

    while (true) {
      if (pendingRequests.size === 0) {
        if (idleStart === null) {
          idleStart = Date.now();
        }

        if (Date.now() - idleStart >= 2000) {
          break;
        }
      } else {
        idleStart = null;
      }

      if (Date.now() - start > 30_000) {
        const urls: string[] = [];

        for (const request of pendingRequests) {
          urls.push(request.url());
        }

        throw new Error(
          `waitForDataLoad: timed out after 30 seconds. Pending requests: ${urls.join(', ')}`
        );
      }

      await setTimeout(intervalInMillis);
    }
  } finally {
    page.off('request', onRequest);
    page.off('requestfinished', onRequestDone);
    page.off('requestfailed', onRequestDone);
  }
}

[mindplay-dk]

There are some nice solutions here.

I wonder when the devs will add something official and supported?

Clearly a lot of people need something like this.