Skip to content

Performance improvment: Add format "raw" for "string" type (skip escaping) #685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 14 commits into from
Closed
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -630,6 +630,25 @@ integer-like values, such as:
- `'2e4'` - _note this will be converted to `2`, not `20000`_
- `1.5` - _note this will be converted to `1`_

<a name="raw"></a>
#### Raw string
By default the library escape all string. With format "raw" the string isn't escaped. This has a very dangerous potential security issue. Raw format would not escape a double quote char which makes it very easy to inject something into the data. You can use it only if you 200% sure in your data.
The advantage is a massive performance improvement

Example:
```javascript
const stringify = fastJson({
title: 'Example Schema',
type: 'object',
properties: {
'code': {
type: 'string',
format 'raw'
}
}
})
```

##### Benchmarks

For reference, here goes some benchmarks for comparison over the three
Expand Down
24 changes: 24 additions & 0 deletions benchmark/bench.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,14 @@ const benchmarks = [
},
input: 'hello world'
},
{
name: 'short string raw',
schema: {
type: 'string',
format: 'raw'
},
input: 'hello world'
},
{
name: 'short string with double quote',
schema: {
Expand All @@ -61,13 +69,29 @@ const benchmarks = [
},
input: longSimpleString
},
{
name: 'long string without double quotes raw',
schema: {
type: 'string',
format: 'raw'
},
input: longSimpleString
},
{
name: 'long string',
schema: {
type: 'string'
},
input: longString
},
{
name: 'long string raw',
schema: {
type: 'string',
format: 'raw'
},
input: longString
},
{
name: 'number',
schema: {
Expand Down
9 changes: 5 additions & 4 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ function buildExtraObjectPropertiesSerializer (context, location, addComma) {
code += `
if (/${propertyKey.replace(/\\*\//g, '\\/')}/.test(key)) {
${addComma}
json += serializer.asString(key) + ':'
json += serializer.asString(key,null) + ':'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why pass null? It’s type is object while the parameter type is string — to me that gives me the impression that an object argument is expected there

What’s wrong with leaving it as undefined?

Suggested change
json += serializer.asString(key,null) + ':'
json += serializer.asString(key) + ':'

${buildValue(context, propertyLocation, 'value')}
continue
}
Expand All @@ -299,13 +299,13 @@ function buildExtraObjectPropertiesSerializer (context, location, addComma) {
if (additionalPropertiesSchema === true) {
code += `
${addComma}
json += serializer.asString(key) + ':' + JSON.stringify(value)
json += serializer.asString(key,null) + ':' + JSON.stringify(value)
`
} else {
const propertyLocation = location.getPropertyLocation('additionalProperties')
code += `
${addComma}
json += serializer.asString(key) + ':'
json += serializer.asString(key,null) + ':'
${buildValue(context, propertyLocation, 'value')}
`
}
Expand Down Expand Up @@ -726,7 +726,8 @@ function buildSingleTypeSerializer (context, location, input) {
} else if (schema.format === 'time') {
return `json += serializer.asTime(${input})`
} else {
return `json += serializer.asString(${input})`
const format = schema?.format ? `"${schema.format}"` : 'null'
return `json += serializer.asString(${input},${format})`
}
}
case 'integer':
Expand Down
6 changes: 4 additions & 2 deletions lib/serializer.js
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ module.exports = class Serializer {
throw new Error(`The value "${date}" cannot be converted to a time.`)
}

asString (str) {
asString (str, format) {
if (typeof str !== 'string') {
if (str === null) {
return '""'
Expand All @@ -113,7 +113,9 @@ module.exports = class Serializer {
}
}

if (str.length < 42) {
if (format === 'raw') {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would create a different serializer method instead of adding the check here, because you know the if condition at the compile time and there is no need to to it in a serialization time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mhh eg. asStringRaw(str) ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeap

Copy link
Member

@ivan-tymoshenko ivan-tymoshenko Mar 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would use a different name than raw to emphasize that is format has a very dangerous potential security issue. Raw format would not escape a double quote char which makes it very easy to inject something into the data. You can use it only if you 200% sure in your data. I would call it something like unsafe.

@mcollina WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for unsafe ...

return '"' + str + '"'
} else if (str.length < 42) {
return this.asStringSmall(str)
} else if (STR_ESCAPE.test(str) === false) {
return '"' + str + '"'
Expand Down
6 changes: 6 additions & 0 deletions test/basic.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ function buildTest (schema, toStringify) {
})
}

buildTest({
title: 'string',
type: 'string',
format: 'raw'
}, 'hello world')

buildTest({
title: 'basic',
type: 'object',
Expand Down
16 changes: 16 additions & 0 deletions test/string.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ const t = require('tap')
const test = t.test
const build = require('..')

test('serialize short string raw', (t) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add some test cases where it doesn't escape chars

t.plan(2)

const schema = {
type: 'string',
format: 'raw'
}

const input = 'abcd'
const stringify = build(schema)
const output = stringify(input)

t.equal(output, '"abcd"')
t.equal(JSON.parse(output), input)
})

test('serialize short string', (t) => {
t.plan(2)

Expand Down