fix: support non-prerendered dynamic routes with fallback false #1541

orinokai · 2022-08-16T16:48:04Z

Summary

Next.js uses an internal cache to persist page creation and updates during incremental static regeneration (ISR).

In order to preserve atomic deploys, Netlify has a different model that uses our ODB functions for creating and updating pages after deployment.

To support this, we currently patch the Next.js server to disable the internal cache, which ensures our ODB functions always receive fresh content and can handle the caching and revalidation directly.

However, our current solution has an incidental bug that bypasses the fallback: false and fallback: true functionality and always serves the page like fallback: blocking. In addition, it creates an unnecessary function invocation and runs getStaticProps unexpectedly.

Fixing this bug enables support for all the fallback options, but doesn’t work in one specific case: pre-rendered ISR pages with fallback: false and a revalidation TTL. These pages get served as a 404 because there is no fallback and the server thinks there is no statically pre-rendered page. This does not happen for ISR pages without revalidation because they are served statically, bypassing the origin entirely.

This PR implements a different strategy: we continue to disable cache writes, but enable cache reads and force manual revalidation mode so fresh content is always served.

Further work is currently ongoing in order to support bypassing the ODB handler for 404 pages.

enable cache reads to support fallback: false and fallback: true, but force manual revalidation for fresh content
add tests

Test plan

See test coverage

Relevant links (GitHub issues, Notion docs, etc.) or a picture of cute animal

Fixes #1179

🧪 Once merged, make sure to update the version if needed and that it was published correctly.

netlify · 2022-08-16T16:48:08Z

✅ Deploy Preview for netlify-plugin-nextjs-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-demo/deploys/631b539851bd0b00098bd53c
😎 Deploy Preview	https://deploy-preview-1541--netlify-plugin-nextjs-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:48:08Z

✅ Deploy Preview for next-hp-edge-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/next-hp-edge-demo/deploys/631b53987bd35600080dc592
😎 Deploy Preview	https://deploy-preview-1541--next-hp-edge-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:48:08Z

❌ Deploy Preview for netlify-plugin-nextjs-nx-monorepo-demo failed.

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-nx-monorepo-demo/deploys/631b53978cacba00089bbc2b

netlify · 2022-08-16T16:49:12Z

✅ Deploy Preview for netlify-plugin-nextjs-export-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-export-demo/deploys/631b53987946e300088503f6
😎 Deploy Preview	https://deploy-preview-1541--netlify-plugin-nextjs-export-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:50:07Z

✅ Deploy Preview for next-plugin-canary ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/next-plugin-canary/deploys/631b5398b6a31d00089d002c
😎 Deploy Preview	https://deploy-preview-1541--next-plugin-canary.netlify.app/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:50:08Z

✅ Deploy Preview for next-plugin-edge-middleware ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/next-plugin-edge-middleware/deploys/631b539875cbc50008e7d332
😎 Deploy Preview	https://deploy-preview-1541--next-plugin-edge-middleware.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:51:15Z

✅ Deploy Preview for netlify-plugin-nextjs-next-auth-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-next-auth-demo/deploys/631b539843978f000a924fd4
😎 Deploy Preview	https://deploy-preview-1541--netlify-plugin-nextjs-next-auth-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:51:32Z

✅ Deploy Preview for next-i18next-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/next-i18next-demo/deploys/631b5398d0f4fd000836eab2
😎 Deploy Preview	https://deploy-preview-1541--next-i18next-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:51:49Z

✅ Deploy Preview for nextjs-plugin-custom-routes-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/nextjs-plugin-custom-routes-demo/deploys/631b539814b301000a927f25
😎 Deploy Preview	https://deploy-preview-1541--nextjs-plugin-custom-routes-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:52:11Z

✅ Deploy Preview for netlify-plugin-nextjs-static-root-demo ready!

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/netlify-plugin-nextjs-static-root-demo/deploys/631b5397e3c11300081349fe
😎 Deploy Preview	https://deploy-preview-1541--netlify-plugin-nextjs-static-root-demo.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

netlify · 2022-08-16T16:52:11Z

❌ Deploy Preview for next-plugin-rsc-demo failed.

Name	Link
🔨 Latest commit	`55de50a`
🔍 Latest deploy log	https://app.netlify.com/sites/next-plugin-rsc-demo/deploys/631b53989d4e7e00087bbcee

hu0p · 2022-08-23T15:29:01Z

adding documentation to warn i18n users who don't prerender all their locales

Is the motivation for this due to Matt's comment or is this a nice to have? I'm not sure this is necessary (unless I'm missing something) as it appears it's the intended behavior according to the docs (excerpt below).

For pages using getStaticProps with Dynamic Routes, all locale variants of the page desired to be prerendered need to be returned from getStaticPaths. Along with the params object returned for paths, you can also return a locale field specifying which locale you want to render. For example:

…fallback: false

…custom 404s

orinokai · 2022-09-02T13:34:40Z

Thanks @hu0p, that last item on the checklist is just because the current behaviour will render all locales regardless of whether a user specifies them in getStaticPaths. The new behaviour will 404 for unspecified locales, which is the correct behaviour, as you pointed out, but probably needs to be called out in the release notes.

packages/runtime/src/helpers/redirects.ts

packages/runtime/src/helpers/files.ts

packages/runtime/src/helpers/redirects.ts

ascorbic

I don't totally understand what's happening with the SSG cache. What happens when it tries to write to disk? Does it handle the failure? Is this being done so that it can return the prebuilt placeholder HTML?

packages/runtime/src/helpers/files.ts

hu0p · 2022-09-02T19:39:30Z

Thanks @hu0p, that last item on the checklist is just because the current behaviour will render all locales regardless of whether a user specifies them in getStaticPaths. The new behaviour will 404 for unspecified locales, which is the correct behaviour, as you pointed out, but probably needs to be called out in the release notes.

Got it. Thanks for the clarification! Watching this as much as I can. I appreciate both of your work on all of this. Thank you! I've been eying that highlighted portion of the redirects file off and on for months. When this lands it'll be a huge resolution for us.

orinokai · 2022-09-05T14:17:57Z

@ascorbic without an ssgCacheKey, ResponseCache.get() does not read the cache and instead runs the responseGenerator callback, which either handles fallback: true, fallback: false or generates fresh page content with doRender; however, the callback will skip fallback handling if ssgCacheKey is not set, which is what is currently happening. Reinstating ssgCacheKey and instead forcing isManualRevalidate = true still has the same effect within ResponseCache.get(), but lets the responseGenerator callback fall through to the fallback handling.

I will switch to the header method, which will remove the need for the 1st patch, and I may be able to avoid the 2nd patch too with the revalidateOnlyGenerated option, but will need testing. The 3rd patch is necessary, afaik, because otherwise Next.js doesn't send SWR cache control headers on 404 pages which means we can't handle ISR 404 pages with our ODBs (whereas Vercel handles them with the internal cache).

ascorbic · 2022-09-06T11:36:59Z

cypress/integration/default/dynamic-routes.spec.ts

@@ -1,7 +1,124 @@
+describe('Static Routing', () => {
+  it('loads show #42 via SSR on a static route', () => {


I don't think you need to include the show numbers here. Just "the correct page" or something

…ugin-nextjs into rs/fallback-false-fix

packages/runtime/src/helpers/files.ts

test/index.js

orinokai self-assigned this Aug 16, 2022

github-actions bot added the type: bug code to address defects in shipped code label Aug 16, 2022

orinokai added 7 commits September 2, 2022 13:57

fix: bypass handler function for non-prerendered dynamic routes with …

61fe286

…fallback: false

fix: remove catch all handler redirect to support fallback false and …

b7a6872

…custom 404s

feat: support custom 404 pages in all locales

b8e1950

feat: add custom 404 handling for static routes

0996adc

chore: refactor hidden path redirects

52ee02c

feat: add support for isr 404 pages

18c2333

feat: swap cache patch for forced manual revalidate

8a361ae

orinokai force-pushed the rs/fallback-false-fix branch from a5b8971 to 8a361ae Compare September 2, 2022 12:59

chore: add docs for patches

2073b27

orinokai commented Sep 2, 2022

View reviewed changes

orinokai changed the title ~~fix: bypass handler function for non-prerendered dynamic routes without fallback~~ fix: force manual revalidate to support non-prerendered dynamic routes without fallback Sep 2, 2022

ascorbic reviewed Sep 2, 2022

View reviewed changes

packages/runtime/src/helpers/files.ts Show resolved Hide resolved

orinokai changed the title ~~fix: force manual revalidate to support non-prerendered dynamic routes without fallback~~ fix: support non-prerendered dynamic routes without fallback Sep 5, 2022

test: update tests while fallback:true works like fallback:blocking

3f5e675

ascorbic reviewed Sep 6, 2022

View reviewed changes

orinokai added 3 commits September 6, 2022 14:56

test: fix jest tests

5806400

test: simplify cypress test names

81d2d80

Merge branch 'main' into rs/fallback-false-fix

bc0aace

orinokai changed the title ~~fix: support non-prerendered dynamic routes without fallback~~ fix: support non-prerendered dynamic routes with fallback false Sep 6, 2022

orinokai added 11 commits September 6, 2022 19:59

fix: revert header revalidation owing to side-effects

94e732c

feat: patch next server to force revalidation only during cache fetches

7a662ee

test: update tests with new env var name

dc424b3

Merge branch 'main' into rs/fallback-false-fix

68461db

test: update test with reverted handler signature

eec91b0

Merge branch 'rs/fallback-false-fix' of github.com:netlify/netlify-pl…

f81b2c3

…ugin-nextjs into rs/fallback-false-fix

fix: reinstate catch-all redirect for Next redirect handling

8732a49

Merge branch 'main' into rs/fallback-false-fix

636d887

test: update snapshot with new redirects

e056209

test: update cypress tests for correct render mode

987defd

Merge branch 'rs/fallback-false-fix' of github.com:netlify/netlify-pl…

4f41b49

…ugin-nextjs into rs/fallback-false-fix

orinokai marked this pull request as ready for review September 7, 2022 16:14

orinokai requested a review from a team September 7, 2022 16:14

ericapisani reviewed Sep 8, 2022

View reviewed changes

packages/runtime/src/helpers/files.ts Show resolved Hide resolved

test/index.js Show resolved Hide resolved

ericapisani approved these changes Sep 8, 2022

View reviewed changes

orinokai added 2 commits September 9, 2022 15:50

Merge branch 'main' into rs/fallback-false-fix

f77d276

Merge branch 'main' into rs/fallback-false-fix

55de50a

orinokai added the automerge label Sep 9, 2022

kodiakhq bot merged commit 92a015c into main Sep 9, 2022

kodiakhq bot deleted the rs/fallback-false-fix branch September 9, 2022 15:37

token-generator-app bot mentioned this pull request Sep 9, 2022

chore: release main #1614

Merged

orinokai mentioned this pull request Nov 11, 2022

fix: ensure isr conditional 404 fix works for next 11 #1765

Closed

orinokai mentioned this pull request Nov 22, 2022

fix: use static 404 for non-prerendered dynamic routes without fallback #1795

Merged

orinokai mentioned this pull request Apr 18, 2023

[Bug]: 404 page for static routes using getStatic... #2043

Closed

3 tasks

		@@ -1,7 +1,124 @@
		describe('Static Routing', () => {
		it('loads show #42 via SSR on a static route', () => {

fix: support non-prerendered dynamic routes with fallback false #1541

fix: support non-prerendered dynamic routes with fallback false #1541

Uh oh!

Conversation

orinokai commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Relevant links (GitHub issues, Notion docs, etc.) or a picture of cute animal

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for netlify-plugin-nextjs-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for next-hp-edge-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for netlify-plugin-nextjs-nx-monorepo-demo failed.

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for netlify-plugin-nextjs-export-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for next-plugin-canary ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for next-plugin-edge-middleware ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for netlify-plugin-nextjs-next-auth-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for next-i18next-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextjs-plugin-custom-routes-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for netlify-plugin-nextjs-static-root-demo ready!

Uh oh!

netlify bot commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for next-plugin-rsc-demo failed.

Uh oh!

hu0p commented Aug 23, 2022

Uh oh!

orinokai commented Sep 2, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ascorbic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hu0p commented Sep 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

orinokai commented Sep 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ascorbic Sep 6, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

orinokai commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

netlify bot commented Aug 16, 2022 •

edited

Loading

hu0p commented Sep 2, 2022 •

edited

Loading

orinokai commented Sep 5, 2022 •

edited

Loading