RangeError: Invalid string length for JSON.stringify

[lustremedia]

Version

@nuxt/content: 1.11.1
nuxt: 2.15.7

Steps to reproduce

Generating or building a nuxt project with more than 12,000 json files in /content fails with error:

JSON.stringify

FATAL: RangeError: Invalid string lenght

Image to error here:
https://discord.com/channels/473401852243869706/476088961010499605/875018331764174888

What is Expected?

I guess that it generates successfully

What is actually happening?

I am assuming that there are to many data files that get bundled into one loki db.json file and it hits a natural string boundry for a json file? Currently with half my content/*.json files included this db-.json file is 500MB large, and it generates fine, however with all of the included I run into this error.

If my assumption is correct, can lokidb be split up into several db files, avoiding this error? How can we handle "large" amounts of data? Can lokidb spit out binary instead of json?

[nozomuikuta]

@lustremedia

The error is thrown at the part below, where Nuxt Content generates a hash based on data of database:

content/packages/content/lib/index.js

Line 177 in 0b6a7be

const dbHash = hash(JSON.stringify(database.items._data)).substr(0, 8)

According to the page about String length in MDN:

ECMAScript 2016 (ed. 7) established a maximum length of 2^53 - 1 elements. Previously, no maximum length was specified. In Firefox, strings have a maximum length of 230 - 2 (~1GB). In versions prior to Firefox 65, the maximum length was 228 - 1 (~256MB).

Nuxt Content is actually run in Node.js environment, but there would be same kind of limitation.
Thus, Nuxt Content has limitation in terms of total amount of contents (1 big file, so many small file, or both).

---

@atinux

If Nuxt Content v2 has same implementation, you might want to reconsider the way to generate hash.
If you share your idea in this issue, I can submit a PR to back-port for the current version.

[lustremedia]

@nozomuikuta is it correct that this hash is only used for decorating/naming/fetching the db? So, there should be an easy fix, no?
The issue lies then within the hasha package, right? It is just a matter of what data to use to generate the hash with, correct?

Maybe not the entire database.items._data as a String, especially if we only use the first 8 chars of that hash? What is the reasoning behind that? @atinux @pi0

[lustremedia]

What kind of a hash is needed here? Could we base this on maybe a timestamp, or number of elements in the db plus timestamp combination, or only on the first or last element of the database? Would also be much faster :-) than parsing the entire db for a substring hash ...

@atinux @pi0

[lustremedia]

So I changed the hash string to this:
const dbHash = hash("Some funky String").substr(0, 8)
and I could build successfully.

However, I now hit another String limit of similar nature, and this time I guess it is lokiDB problem whilst serializing the database, I assume ...

content/packages/content/lib/index.js

Line 193 in 0b6a7be

database.db.serialize(),

content/packages/content/lib/index.js

Line 194 in 0b6a7be

'utf-8'