`body` removal causes `400 - illegal_argument_exception` #2584

afharo · 2025-01-23T16:30:59Z

Regression report

In Kibana, we're migrating to stop using the deprecated body key. An example is elastic/kibana#204882.

However, some of our tests started failing with the following error:

illegal_argument_exception: request [/<data_stream>/_mapping] contains unrecognized parameter: [_data_stream_timestamp]

We managed to ship this because the code tries to update some existing mappings by GET the existing ones and PUT the full updated object. _data_stream_timestamp was automatically populated by ES and returned in GET request.

The old logic using the body param works because it forces the client to ship that parameter in the body of the request. However, after removing the body param, the client sent that property as a query parameter, leading to ES validation issues as this property is only accepted in the body.

Last working version

8.16.0

To reproduce

Try to update the mappings of an existing data stream by GET-ing the mappings first, adding the new field in the mappings, and PUT-ing the resulting object.

Expected behavior

The request is successful.

Node.js version

v20.18.2

TypeScript version

5.1.6

Elasticsearch client version

@elastic/[email protected] - @elastic/[email protected]

Elasticsearch server version

9.0.0-SNAPSHOT

Operating system

No response

Any other relevant environment information.

No response

The text was updated successfully, but these errors were encountered:

rudolf · 2025-01-29T10:36:06Z

Just want to highlight the use case where this surfaced:

https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/alerting/server/alerts_service/lib/create_concrete_write_index.ts#L73-L95

Here Kibana makes a simulate index template request to retrieve the mappings that an index template would apply if a data stream write index was rolled over. It then uses the mappings from the response to also apply these mappings to existing backing indices (presumably to synchronise the mappings?).

Since the simulate index template request responded with _data_stream_timestamp which is undocumented and not in the spec the update mappings requests started failing. Since Elasticsearch can add new response properties at any time to their API without it being considered a breaking change this is a significant risk, a minor upgrade of Elasticsearch can suddenly cause Eleasticsearch-js client calls to start failing.

While it would be a risky and breaking change to Elasticsearch-js, it seems like defaulting to passing unknown properties in the body would preferably over passing them as query params.

This would however break any consumers of Elasticsearch-js that are using query params not in the spec or belonging to a custom plugin (related elastic/elasticsearch-py#2181)

JoshMock · 2025-01-30T17:13:09Z

I do agree that passing unknowns to the body makes more sense than passing as query params. The main concern is, as you said, breaking any existing code that works on the assumption that unknown params will end up in the querystring.

My only other concern with using the body as the catch-all place for unknown params is that there are many requests that do not accept a JSON body. In those cases, query params probably are the better choice. It might be possible to update the code generator to be able to choose the ideal catch-all for both of these cases, but I'm not confident I'd have something stable in time for 9.0. I can look into it next week but it will be a tight squeeze.

@rudolf @afharo would making this change introduce even more last-minute pain for Kibana, or do you think it will be a smooth transition? If it's painful, maybe holding off is the better choice.

afharo · 2025-02-01T15:04:02Z

would making this change introduce even more last-minute pain for Kibana, or do you think it will be a smooth transition? If it's painful, maybe holding off is the better choice.

elastic/kibana#208776 proves that not doing this change is blocking Kibana from removing all body usages. Especially in the putMappings API. While the indices.putSettings wraps all the settings under a property settings, putMappings expects al the params in the root level, and, apparently, there are many root-level mappings fields that are not in the spec. Maybe a safer approach for this API is to wrap the properties inside a top-level mappings field.

WDYT?

JoshMock · 2025-02-03T20:15:33Z

Maybe a safer approach for this API is to wrap the properties inside a top-level mappings field.

Making a spec change like this is a discussion that would need to happen in the spec repo. Because all request types are derived from the spec, for all language clients, that change has to be something that will be compatible with all dynamic and static clients work.

elastic/kibana#208776 proves that not doing this change is blocking Kibana from removing all body usages

Which change? Adding body and querystring parameters to each request? Or changing the catch-all to body for most requests? The former was released in alpha.3 last week; the latter I have not yet implemented because I've been working on #2605, which was deemed a high priority.

JoshMock · 2025-02-04T19:28:02Z

I'm going to work on changing the catch-all from querystring to body (except for APIs that don't accept a body) soon and get it published in the next 9.0.0 alpha.

@ts-expect-error

## Summary Updating the ES client to 9.0. Resolves #116102 ## What changes? **Breaking change**: `body` has been removed. Most of the changes are about bringing all the content inside the body as a root attribute to the API params: ```diff const response = await client.search({ index: 'test', - body: { query: { match_all: {} } - } }) ``` For this reason, enabling the "Hide whitespace changes" option when reviewing is recommended. Some exceptions to this rule: * Bulk APIs replace the `body` array with `operations` array (direct replacement) * Index Put Settings API replace `body` array with `settings` (direct replacement) * Msearch replaces the `body` array with `searches` array (direct replacement) * Document Index API replaces `body` with `document` (direct replacement) * Create Repository replaces `body` with `repository` (direct replacement) Because of a known issue in the client (elastic/elasticsearch-js#2584), there's still an escape hatch to send data in the body in case the specific use case requires it via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`, but it shouldn't be abused because we lose types. In this PR we've used it in those scenarios where we reuse the response of a GET as the body of a PUT/POST. ### Other changes * `estypes` can be imported from the root of the library as `import type { estypes } from '@elastic/elasticsearch';` * `estypesWithBody` have been removed * `requestTimeout`'s 30s default has been removed in the client. This PR explicitly adds the setting in all client usages. ### Identify risks - [x] The client places unknown properties as querystring, risking body params leaking there, and causing 400 errors from ES => Solved by forcing `body` usage there via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. The next version of the client will address this. - [x] We need to run the MKI tests to make sure that we're not breaking anything there => https://elastic.slack.com/archives/C04HT4P1YS3/p1739528112482629?thread_ts=1739480136.231439&cid=C04HT4P1YS3 --------- Co-authored-by: Gloria Hornero <[email protected]>

JoshMock · 2025-02-26T18:29:54Z

As of #2636 the param logic changes to:

if recognized as a body param from the spec, put it in the JSON body
if recognized as a path param, put it in the URL path
if recognized as a query param or a "common" query param (e.g. pretty, error_trace), put it in the querystring
if not recognized and this API accepts a JSON body, put it in the JSON body
if not recognized and this API does not accept a JSON body, put it in the querystring

The first two steps are identical to 8.x. The last three steps replace the logic that put all unrecognized params in the querystring.

The change also updates each API function to only instantiate its arrays of accepted body/path/query param names once per client instance rather than during every function call, by moving the array values up to the constructor or module, depending on which is available. Hopefully that introduces a small performance/memory improvement.

I'm going to merge that PR today and publish 9.0.0-alpha.4 to npm shortly after.

JoshMock · 2025-02-26T19:07:58Z

9.0.0-alpha.4 is now available on npm @afharo @rudolf

@ts-expect-error

## Summary Updating the ES client to 9.0. Resolves elastic#116102 ## What changes? **Breaking change**: `body` has been removed. Most of the changes are about bringing all the content inside the body as a root attribute to the API params: ```diff const response = await client.search({ index: 'test', - body: { query: { match_all: {} } - } }) ``` For this reason, enabling the "Hide whitespace changes" option when reviewing is recommended. Some exceptions to this rule: * Bulk APIs replace the `body` array with `operations` array (direct replacement) * Index Put Settings API replace `body` array with `settings` (direct replacement) * Msearch replaces the `body` array with `searches` array (direct replacement) * Document Index API replaces `body` with `document` (direct replacement) * Create Repository replaces `body` with `repository` (direct replacement) Because of a known issue in the client (elastic/elasticsearch-js#2584), there's still an escape hatch to send data in the body in case the specific use case requires it via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`, but it shouldn't be abused because we lose types. In this PR we've used it in those scenarios where we reuse the response of a GET as the body of a PUT/POST. ### Other changes * `estypes` can be imported from the root of the library as `import type { estypes } from '@elastic/elasticsearch';` * `estypesWithBody` have been removed * `requestTimeout`'s 30s default has been removed in the client. This PR explicitly adds the setting in all client usages. ### Identify risks - [x] The client places unknown properties as querystring, risking body params leaking there, and causing 400 errors from ES => Solved by forcing `body` usage there via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. The next version of the client will address this. - [x] We need to run the MKI tests to make sure that we're not breaking anything there => https://elastic.slack.com/archives/C04HT4P1YS3/p1739528112482629?thread_ts=1739480136.231439&cid=C04HT4P1YS3 --------- Co-authored-by: Gloria Hornero <[email protected]>

afharo · 2025-03-13T10:27:56Z

FYI: I'm working on the upgrade in elastic/kibana#213375.

Some tests are now running OOM, and I think that's caused by our client's mock: https://github.com/elastic/kibana/blob/4a00449bdcd699f89294536a86dffcbeab4daf53/src/core/packages/elasticsearch/client-server-mocks/src/mocks.ts#L186-L189

If my suspicion is correct, It mocks every API in a recursive manner, and it looks like it's reaching a circular reference that ends up OOM'ing the test:
https://github.com/elastic/kibana/blob/4a00449bdcd699f89294536a86dffcbeab4daf53/src/core/packages/elasticsearch/client-server-mocks/src/mocks.ts#L125-L184

I'll see if changing how our mocks are created can address this issue. Any help is appreciated.

JoshMock · 2025-03-18T16:04:21Z

@afharo Any luck? I was out on PTO so just seeing this. I don't know of any changes to the client that would cause a circular reference, unless it was a spec change. If you can narrow in on where the circular reference might be, that'll be helpful.

@ts-expect-error

## Summary Updating the ES client to 9.0. Resolves elastic#116102 ## What changes? **Breaking change**: `body` has been removed. Most of the changes are about bringing all the content inside the body as a root attribute to the API params: ```diff const response = await client.search({ index: 'test', - body: { query: { match_all: {} } - } }) ``` For this reason, enabling the "Hide whitespace changes" option when reviewing is recommended. Some exceptions to this rule: * Bulk APIs replace the `body` array with `operations` array (direct replacement) * Index Put Settings API replace `body` array with `settings` (direct replacement) * Msearch replaces the `body` array with `searches` array (direct replacement) * Document Index API replaces `body` with `document` (direct replacement) * Create Repository replaces `body` with `repository` (direct replacement) Because of a known issue in the client (elastic/elasticsearch-js#2584), there's still an escape hatch to send data in the body in case the specific use case requires it via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`, but it shouldn't be abused because we lose types. In this PR we've used it in those scenarios where we reuse the response of a GET as the body of a PUT/POST. ### Other changes * `estypes` can be imported from the root of the library as `import type { estypes } from '@elastic/elasticsearch';` * `estypesWithBody` have been removed * `requestTimeout`'s 30s default has been removed in the client. This PR explicitly adds the setting in all client usages. ### Identify risks - [x] The client places unknown properties as querystring, risking body params leaking there, and causing 400 errors from ES => Solved by forcing `body` usage there via `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. The next version of the client will address this. - [x] We need to run the MKI tests to make sure that we're not breaking anything there => https://elastic.slack.com/archives/C04HT4P1YS3/p1739528112482629?thread_ts=1739480136.231439&cid=C04HT4P1YS3 --------- Co-authored-by: Gloria Hornero <[email protected]>

afharo · 2025-03-26T11:29:06Z

@JoshMock, I had to move to other tasks. I'm going to try again this week. I'll let you know how it goes.

afharo · 2025-03-26T19:53:31Z

It looks like the issue happened when trying to "mock" acceptedParams: elastic/kibana@b7071cf

@JoshMock

## Summary Upgrading the ES client to v9.0.0-alpha.4 to test the changes mentioned in elastic/elasticsearch-js#2584 This new version introduces some type changes, most notably, the `FieldValue` is `string | number | boolean | null | undefined` instead of `any`, leading to some new type checks to be implemented (like on aggregation results `bucket.key`, `search_after`, and `sort` options). On top of that, it adds the new behavior where unknown properties are placed in the `body` (when the request has a body). If they must be in as a query parameter, they should be placed under the `querystring` option. cc @JoshMock TODO: - [x] Stabilize the type errors - [x] Address all the query parameters that are now placed in the body (by wrapping them inside the option `querystring: {}`) I will address `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584` in a separate PR to reduce noise. Related #208776

@ts-expect-error

## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`.

@ts-expect-error

…217223) ## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`.

@ts-expect-error

## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

@ts-expect-error

…c#217219) ## Summary Follow up to elastic#213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ...

@ts-expect-error

## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

@ts-expect-error

#217225) ## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`.

@ts-expect-error

## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

@ts-expect-error

## Summary Follow up to #213375: The latest version of the ES client fixed the issue elastic/elasticsearch-js#2584. We should be able to remove all usages of `// @ts-expect-error [email protected] https://github.com/elastic/elasticsearch-js/issues/2584`.

JoshMock · 2025-04-10T15:11:39Z

Are we good to close this @afharo?

afharo · 2025-04-10T17:28:19Z

Yes! Thank you! Apologies for not coming back to you.

afharo added the Category: Bug label Jan 23, 2025

JoshMock added the Area: Specification label Jan 23, 2025

JoshMock self-assigned this Jan 23, 2025

JoshMock mentioned this issue Jan 23, 2025

Prepare JS client 9.0 #2433

Closed

8 tasks

afharo mentioned this issue Jan 30, 2025

Upgrade ES client to 9.0.0-alpha.3 elastic/kibana#208776

Merged

2 tasks

ymao1 mentioned this issue Feb 18, 2025

[Response Ops][Alerting] Strip unnecessary fields when updating alert mappings elastic/kibana#211630

Open

JoshMock mentioned this issue Feb 26, 2025

Auto-generated code for main #2636

Merged

afharo mentioned this issue Mar 6, 2025

Upgrade ES client to 9.0.0-alpha.4 elastic/kibana#213375

Merged

2 tasks

This was referenced Apr 4, 2025

[ES 9.0] Remove body workaround (@elastic/security-threat-hunting) elastic/kibana#217224

Merged

[ES 9.0] Remove body workaround (@elastic/security-entity-analytics) elastic/kibana#217225

Merged

JoshMock closed this as completed Apr 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`body` removal causes `400 - illegal_argument_exception` #2584

`body` removal causes `400 - illegal_argument_exception` #2584

afharo commented Jan 23, 2025

rudolf commented Jan 29, 2025

Uh oh!

JoshMock commented Jan 30, 2025

Uh oh!

afharo commented Feb 1, 2025

Uh oh!

JoshMock commented Feb 3, 2025

Uh oh!

JoshMock commented Feb 4, 2025

Uh oh!

JoshMock commented Feb 26, 2025

Uh oh!

JoshMock commented Feb 26, 2025

Uh oh!

afharo commented Mar 13, 2025

Uh oh!

JoshMock commented Mar 18, 2025

Uh oh!

afharo commented Mar 26, 2025

Uh oh!

afharo commented Mar 26, 2025

Uh oh!

JoshMock commented Apr 10, 2025

Uh oh!

afharo commented Apr 10, 2025

Uh oh!

body removal causes 400 - illegal_argument_exception #2584

body removal causes 400 - illegal_argument_exception #2584

Comments

afharo commented Jan 23, 2025

Regression report

Last working version

To reproduce

Expected behavior

Node.js version

TypeScript version

Elasticsearch client version

Elasticsearch server version

Operating system

Any other relevant environment information.

rudolf commented Jan 29, 2025

Uh oh!

JoshMock commented Jan 30, 2025

Uh oh!

afharo commented Feb 1, 2025

Uh oh!

JoshMock commented Feb 3, 2025

Uh oh!

JoshMock commented Feb 4, 2025

Uh oh!

JoshMock commented Feb 26, 2025

Uh oh!

JoshMock commented Feb 26, 2025

Uh oh!

afharo commented Mar 13, 2025

Uh oh!

JoshMock commented Mar 18, 2025

Uh oh!

afharo commented Mar 26, 2025

Uh oh!

afharo commented Mar 26, 2025

Uh oh!

JoshMock commented Apr 10, 2025

Uh oh!

afharo commented Apr 10, 2025

Uh oh!

`body` removal causes `400 - illegal_argument_exception` #2584

`body` removal causes `400 - illegal_argument_exception` #2584