_bulk?refresh=wait_for request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk request at all.
+ You might want to disable the refresh interval temporarily to improve indexing throughput for large bulk requests. + Refer to the linked documentation for step-by-step instructions using the index settings API.
`conflicts property.
Additionally, if you opt to count version conflicts, the operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.
- NOTE: The reindex API makes no effort to handle ID collisions. - The last document written will "win" but the order isn't usually predictable so it is not a good idea to rely on this behavior. - Instead, make sure that IDs are unique by using a script.
-Running reindex asynchronously
-If the request contains wait_for_completion=false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task.
- Elasticsearch creates a record of this task as a document at _tasks/<task_id>.
Reindex from multiple sources
-If you have many sources to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up multiple sources. - That way you can resume the process if there are any errors by removing the partially completed source and starting over. - It also makes parallelizing the process fairly simple: split the list of sources to reindex and run each list in parallel.
-For example, you can use a bash script like this:
-for index in i1 i2 i3 i4 i5; do
- curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{
- "source": {
- "index": "'$index'"
- },
- "dest": {
- "index": "'$index'-reindexed"
- }
- }'
- done
- Throttling
-Set requests_per_second to any positive decimal number (1.4, 6, 1000, for example) to throttle the rate at which reindex issues batches of index operations.
- Requests are throttled by padding each batch with a wait time.
- To turn off throttling, set requests_per_second to -1.
The throttling is done by waiting between batches so that the scroll that reindex uses internally can be given a timeout that takes into account the padding.
- The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
- By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
- wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
- Since the batch is issued as a single bulk request, large batch sizes cause Elasticsearch to create many requests and then wait for a while before starting the next set. - This is "bursty" instead of "smooth".
-Slicing
-Reindex supports sliced scroll to parallelize the reindexing process. - This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.
-NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
-You can slice a reindex request manually by providing a slice ID and total number of slices to each request.
- You can also let reindex automatically parallelize by using sliced scroll to slice on _id.
- The slices parameter specifies the number of slices to use.
Adding slices to the reindex request just automates the manual process, creating sub-requests which means it has some quirks:
slices only contains the status of completed slices.slices will rethrottle the unfinished sub-request proportionally.slices will cancel each sub-request.slices, each sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the previous point about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being reindexed.If slicing automatically, setting slices to auto will choose a reasonable number for most indices.
- If slicing manually or otherwise tuning automatic slicing, use the following guidelines.
Query performance is most efficient when the number of slices is equal to the number of shards in the index.
- If that number is large (for example, 500), choose a lower number as too many slices will hurt performance.
- Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.
Indexing performance scales linearly across available resources with the number of slices.
-Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.
-Modify documents during reindexing
-Like _update_by_query, reindex operations support a script that modifies the document.
- Unlike _update_by_query, the script is allowed to modify the document's metadata.
Just as in _update_by_query, you can set ctx.op to change the operation that is run on the destination.
- For example, set ctx.op to noop if your script decides that the document doesn’t have to be indexed in the destination. This "no operation" will be reported in the noop counter in the response body.
- Set ctx.op to delete if your script decides that the document must be deleted from the destination.
- The deletion will be reported in the deleted counter in the response body.
- Setting ctx.op to anything else will return an error, as will setting any other field in ctx.
Think of the possibilities! Just be careful; you are able to change:
-_id_index_version_routingSetting _version to null or clearing it from the ctx map is just like not sending the version in an indexing request.
- It will cause the document to be overwritten in the destination regardless of the version on the target or the version type you use in the reindex API.
Reindex from remote
-Reindex supports reindexing from a remote Elasticsearch cluster.
- The host parameter must contain a scheme, host, port, and optional path.
- The username and password parameters are optional and when they are present the reindex operation will connect to the remote Elasticsearch node using basic authentication.
- Be sure to use HTTPS when using basic authentication or the password will be sent in plain text.
- There are a range of settings available to configure the behavior of the HTTPS connection.
When using Elastic Cloud, it is also possible to authenticate against the remote cluster through the use of a valid API key.
- Remote hosts must be explicitly allowed with the reindex.remote.whitelist setting.
- It can be set to a comma delimited list of allowed remote host and port combinations.
- Scheme is ignored; only the host and port are used.
- For example:
reindex.remote.whitelist: [otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"]
- The list of allowed hosts must be configured on any nodes that will coordinate the reindex. - This feature should work with remote clusters of any version of Elasticsearch. - This should enable you to upgrade from any version of Elasticsearch to the current version by reindexing from a cluster of the old version.
-WARNING: Elasticsearch does not support forward compatibility across major versions. - For example, you cannot reindex from a 7.x cluster into a 6.x cluster.
-To enable queries sent to older versions of Elasticsearch, the query parameter is sent directly to the remote host without validation or modification.
NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
-Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb.
- If the remote index includes very large documents you'll need to use a smaller batch size.
- It is also possible to set the socket read timeout on the remote connection with the socket_timeout field and the connection timeout with the connect_timeout field.
- Both default to 30 seconds.
Configuring SSL parameters
-Reindex from remote supports configurable SSL settings.
- These must be specified in the elasticsearch.yml file, with the exception of the secure settings, which you add in the Elasticsearch keystore.
- It is not possible to configure SSL in the body of the reindex request.
Refer to the linked documentation for examples of how to reindex documents.
`geo_bounds aggregation on the <field>. The search only includes this aggregation if the exact_bounds parameter is true.with_labels is true, the internal search will include a dynamic runtime field that calls the getLabelPosition function of the geometry doc value. This enables the generation of new point features containing suggested geometry labels, so that, for example, multi-polygons will have only one label.For example, Elasticsearch may translate a vector tile search API request with a grid_agg argument of geotile and an exact_bounds argument of true into the following search
GET my-index/_search
- {
- "size": 10000,
- "query": {
- "geo_bounding_box": {
- "my-geo-field": {
- "top_left": {
- "lat": -40.979898069620134,
- "lon": -45
- },
- "bottom_right": {
- "lat": -66.51326044311186,
- "lon": 0
- }
- }
- }
- },
- "aggregations": {
- "grid": {
- "geotile_grid": {
- "field": "my-geo-field",
- "precision": 11,
- "size": 65536,
- "bounds": {
- "top_left": {
- "lat": -40.979898069620134,
- "lon": -45
- },
- "bottom_right": {
- "lat": -66.51326044311186,
- "lon": 0
- }
- }
- }
- },
- "bounds": {
- "geo_bounds": {
- "field": "my-geo-field",
- "wrap_longitude": false
- }
- }
- }
- }
- The API returns results as a binary Mapbox vector tile. Mapbox vector tiles are encoded as Google Protobufs (PBF). By default, the tile contains three layers:
Learn how to use the vector tile search API with practical examples in the Vector tile search examples guide.
`routing only to hit a particular shard.
+ Use routing only to hit a particular shard.
+ Refer to the linked documentation for detailed examples of how to use this API.
`The document must still be reindexed, but using this API removes some network roundtrips and reduces chances of version conflicts between the GET and the index operation.
The _source field must be enabled to use this API.
- In addition to _source, you can access the following variables through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).
_source, you can access the following variables through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).
+ For usage examples such as partial updates, upserts, and scripted updates, see the External documentation.
`Refreshing shards
+Specifying the refresh parameter refreshes all shards once the request completes.
+ This is different to the update API's refresh parameter, which causes only the shard
+ that received the request to be refreshed. Unlike the update API, it does not support
+ wait_for.
Running update by query asynchronously
+If the request contains wait_for_completion=false, Elasticsearch
+ performs some preflight checks, launches the request, and returns a
+ task you can use to cancel or get the status of the task.
+ Elasticsearch creates a record of this task as a document at .tasks/task/${taskId}.
Waiting for active shards
+wait_for_active_shards controls how many copies of a shard must be active
+ before proceeding with the request. See wait_for_active_shards
+ for details. timeout controls how long each write request waits for unavailable
+ shards to become available. Both work exactly the way they work in the
+ Bulk API. Update by query uses scrolled searches, so you can also
+ specify the scroll parameter to control how long it keeps the search context
+ alive, for example ?scroll=10m. The default is 5 minutes.
Throttling update requests
To control the rate at which update by query issues batches of update operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
@@ -6265,18 +6138,8 @@ async def update_by_query(
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
-Update the document source
-Update by query supports scripts to update the document source.
- As with the update API, you can set ctx.op to change the operation that is performed.
Set ctx.op = "noop" if your script decides that it doesn't have to make any changes.
- The update by query operation skips updating the document and increments the noop counter.
Set ctx.op = "delete" if your script decides that the document should be deleted.
- The update by query operation deletes the document and increments the deleted counter.
Update by query supports only index, noop, and delete.
- Setting ctx.op to anything else is an error.
- Setting any other field in ctx is an error.
- This API enables you to only modify the source of matching documents; you cannot move them.
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
+ Refer to the linked documentation for examples of how to update documents using the _update_by_query API:
Get cluster-wide settings. - By default, it returns only settings that have been explicitly defined.
+Get cluster-wide settings.
+By default, it returns only settings that have been explicitly defined.
`Get a specific running ES|QL query information. + Returns an object extended information about a running ES|QL query.
+ + + :param id: The query ID + """ + if id in SKIP_IN_PATH: + raise ValueError("Empty value passed for parameter 'id'") + __path_parts: t.Dict[str, str] = {"id": _quote(id)} + __path = f'/_query/queries/{__path_parts["id"]}' + __query: t.Dict[str, t.Any] = {} + if error_trace is not None: + __query["error_trace"] = error_trace + if filter_path is not None: + __query["filter_path"] = filter_path + if human is not None: + __query["human"] = human + if pretty is not None: + __query["pretty"] = pretty + __headers = {"accept": "application/json"} + return await self.perform_request( # type: ignore[return-value] + "GET", + __path, + params=__query, + headers=__headers, + endpoint_id="esql.get_query", + path_parts=__path_parts, + ) + + @_rewrite_parameters() + @_stability_warning(Stability.EXPERIMENTAL) + async def list_queries( + self, + *, + error_trace: t.Optional[bool] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + pretty: t.Optional[bool] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Get running ES|QL queries information. + Returns an object containing IDs and other information about the running ES|QL queries.
+ + """ + __path_parts: t.Dict[str, str] = {} + __path = "/_query/queries" + __query: t.Dict[str, t.Any] = {} + if error_trace is not None: + __query["error_trace"] = error_trace + if filter_path is not None: + __query["filter_path"] = filter_path + if human is not None: + __query["human"] = human + if pretty is not None: + __query["pretty"] = pretty + __headers = {"accept": "application/json"} + return await self.perform_request( # type: ignore[return-value] + "GET", + __path, + params=__query, + headers=__headers, + endpoint_id="esql.list_queries", + path_parts=__path_parts, + ) + @_rewrite_parameters( body_fields=( "query", @@ -422,7 +518,9 @@ async def query( parameter, runs it, and returns the results. :param allow_partial_results: If `true`, partial results will be returned if there are shard failures, but the query can continue to execute on other - clusters and shards. + clusters and shards. If `false`, the query will fail if there are any failures. + To override the default behavior, you can set the `esql.query.allow_partial_results` + cluster setting to `false`. :param columnar: By default, ES|QL returns results as rows. For example, FROM returns each individual document as one row. For the JSON, YAML, CBOR and smile formats, ES|QL can return the results in a columnar fashion where one diff --git a/elasticsearch/_async/client/indices.py b/elasticsearch/_async/client/indices.py index f2535776f..adcde9942 100644 --- a/elasticsearch/_async/client/indices.py +++ b/elasticsearch/_async/client/indices.py @@ -338,7 +338,7 @@ async def clear_cache( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param fielddata: If `true`, clears the fields cache. Use the `fields` parameter to clear the cache of specific fields only. :param fields: Comma-separated list of field names used to limit the `fielddata` @@ -563,7 +563,7 @@ async def close( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. :param master_timeout: Period to wait for a connection to the master node. If @@ -656,7 +656,15 @@ async def create( `Delete data stream options. + Removes the data stream options from a data stream.
+ + + `Delete a legacy index template.
+Delete a legacy index template. + IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
`Get data stream options.
+Get the data stream options configuration of one or more data streams.
+ + + `Get data stream settings.
+Get setting information for one or more data streams.
+ + + `Get index templates. +
Get legacy index templates. Get information about one or more index templates.
IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
@@ -3157,7 +3345,7 @@ async def open( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. :param master_timeout: Period to wait for a connection to the master node. If @@ -3416,8 +3604,7 @@ async def put_data_lifecycle( for this data stream. A data stream lifecycle that's disabled (enabled: `false`) will have no effect on the data stream. :param expand_wildcards: Type of data stream that wildcard patterns can match. - Supports comma-separated values, such as `open,hidden`. Valid values are: - `all`, `hidden`, `open`, `closed`, `none`. + Supports comma-separated values, such as `open,hidden`. :param master_timeout: Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error. @@ -3466,6 +3653,167 @@ async def put_data_lifecycle( path_parts=__path_parts, ) + @_rewrite_parameters( + body_fields=("failure_store",), + ) + async def put_data_stream_options( + self, + *, + name: t.Union[str, t.Sequence[str]], + error_trace: t.Optional[bool] = None, + expand_wildcards: t.Optional[ + t.Union[ + t.Sequence[ + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]] + ], + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]], + ] + ] = None, + failure_store: t.Optional[t.Mapping[str, t.Any]] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + master_timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + pretty: t.Optional[bool] = None, + timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + body: t.Optional[t.Dict[str, t.Any]] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Update data stream options. + Update the data stream options of the specified data streams.
+ + + `Update data stream settings.
+This API can be used to override settings on specific data streams. These overrides will take precedence over what + is specified in the template that the data stream matches. To prevent your data stream from getting into an invalid state, + only certain settings are allowed. If possible, the setting change is applied to all + backing indices. Otherwise, it will be applied when the data stream is next rolled over.
+ + + `Update field mappings. Add new fields to an existing data stream or index. - You can also use this API to change the search settings of existing fields and add new properties to existing object fields. - For data streams, these changes are applied to all backing indices by default.
-Add multi-fields to an existing field
-Multi-fields let you index the same field in different ways. - You can use this API to update the fields mapping parameter and enable multi-fields for an existing field. - WARNING: If an index (or data stream) contains documents when you add a multi-field, those documents will not have values for the new multi-field. - You can populate the new multi-field with the update by query API.
-Change supported mapping parameters for an existing field
-The documentation for each mapping parameter indicates whether you can update it for an existing field using this API.
- For example, you can use the update mapping API to update the ignore_above parameter.
Change the mapping of an existing field
-Except for supported mapping parameters, you can't change the mapping or field type of an existing field. - Changing an existing field could invalidate data that's already indexed.
-If you need to change the mapping of a field in a data stream's backing indices, refer to documentation about modifying data streams. - If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.
-Rename a field
-Renaming a field would invalidate data already indexed under the old field name. - Instead, add an alias field to create an alternate field name.
+ You can use the update mapping API to: +Learn how to use the update mapping API with practical examples in the Update mapping API examples guide.
`To revert a setting to the default value, use a null value.
- The list of per-index settings that can be updated dynamically on live indices can be found in index module documentation.
+ The list of per-index settings that can be updated dynamically on live indices can be found in index settings documentation.
To preserve existing settings from being updated, set the preserve_existing parameter to true.
For performance optimization during bulk indexing, you can disable the refresh interval. + Refer to disable refresh interval for an example. + There are multiple valid ways to represent index settings in the request body. You can specify only the setting, for example:
+{
+ "number_of_replicas": 1
+ }
+ Or you can use an index setting object:
{
+ "index": {
+ "number_of_replicas": 1
+ }
+ }
+ Or you can use dot annotation:
+{
+ "index.number_of_replicas": 1
+ }
+ Or you can embed any of the aforementioned options in a settings object. For example:
{
+ "settings": {
+ "index": {
+ "number_of_replicas": 1
+ }
+ }
+ }
+ NOTE: You can only define new analyzers on closed indices. To add an analyzer, you must close the index, define the analyzer, and reopen the index. You cannot close the write index of a data stream. @@ -3856,7 +4225,8 @@ async def put_settings( Then roll over the data stream to apply the new analyzer to the stream's write index and future backing indices. This affects searches and any new data added to the stream after the rollover. However, it does not affect the data stream's backing indices or their existing data. - To change the analyzer for existing backing indices, you must create a new data stream and reindex your data into it.
+ To change the analyzer for existing backing indices, you must create a new data stream and reindex your data into it. + Refer to updating analyzers on existing indices for step-by-step examples. `Create or update an index template. +
Create or update a legacy index template. Index templates define settings, mappings, and aliases that can be applied automatically to new indices. Elasticsearch applies templates to new indices based on an index pattern that matches the index name.
IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
@@ -4172,7 +4542,7 @@ async def refresh( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. """ @@ -4288,6 +4658,105 @@ async def reload_search_analyzers( path_parts=__path_parts, ) + @_rewrite_parameters() + async def remove_block( + self, + *, + index: str, + block: t.Union[str, t.Literal["metadata", "read", "read_only", "write"]], + allow_no_indices: t.Optional[bool] = None, + error_trace: t.Optional[bool] = None, + expand_wildcards: t.Optional[ + t.Union[ + t.Sequence[ + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]] + ], + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]], + ] + ] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + ignore_unavailable: t.Optional[bool] = None, + master_timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + pretty: t.Optional[bool] = None, + timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Remove an index block.
+Remove an index block from an index. + Index blocks limit the operations allowed on an index by blocking specific operation types.
+ + + `Create an inference endpoint.
- When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint.
IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral, Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
+The following integrations are available through the inference API. You can find the available task types next to the integration name:
+completion, rerank, sparse_embedding, text_embedding)completion, text_embedding)completion)completion, text_embedding)completion, text_embedding)completion, rerank, text_embedding)completion, chat_completion)rerank, sparse_embedding, text_embedding - this service is for built-in models and models uploaded through Eland)sparse_embedding)completion, text_embedding)rerank, text_embedding)chat_completion, completion, rerank, text_embedding)chat_completion, completion, text_embedding)chat_completion, completion, text_embedding)text_embedding, rerank)text_embedding)text_embedding, rerank)Create an AlibabaCloud AI Search inference endpoint.
Create an inference endpoint to perform an inference task with the alibabacloud-ai-search service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Amazon Bedrock inference endpoint.
-Creates an inference endpoint to perform an inference task with the amazonbedrock service.
Create an inference endpoint to perform an inference task with the amazonbedrock service.
-info You need to provide the access and secret keys only once, during the inference model creation. The get inference API does not retrieve your access or secret keys. After creating the inference model, you cannot change the associated key pairs. If you want to use a different access and secret key pair, delete the inference model and recreate it with the same name and the updated keys.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Anthropic inference endpoint.
Create an inference endpoint to perform an inference task with the anthropic service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Azure AI studio inference endpoint.
Create an inference endpoint to perform an inference task with the azureaistudio service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
The list of embeddings models that you can choose from in your deployment can be found in the Azure models documentation.
-When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Cohere inference endpoint.
Create an inference endpoint to perform an inference task with the cohere service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a DeepSeek inference endpoint.
+Create an inference endpoint to perform an inference task with the deepseek service.
Create an Google AI Studio inference endpoint.
Create an inference endpoint to perform an inference task with the googleaistudio service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Google Vertex AI inference endpoint.
Create an inference endpoint to perform an inference task with the googlevertexai service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Hugging Face inference endpoint.
-Create an inference endpoint to perform an inference task with the hugging_face service.
You must first create an inference endpoint on the Hugging Face endpoint page to get an endpoint URL.
- Select the model you want to use on the new endpoint creation page (for example intfloat/e5-small-v2), then select the sentence embeddings task under the advanced configuration section.
- Create the endpoint and copy the URL after the endpoint initialization has been finished.
The following models are recommended for the Hugging Face service:
+Create an inference endpoint to perform an inference task with the hugging_face service.
+ Supported tasks include: text_embedding, completion, and chat_completion.
To configure the endpoint, first visit the Hugging Face Inference Endpoints page and create a new endpoint. + Select a model that supports the task you intend to use.
+For Elastic's text_embedding task:
+ The selected model must support the Sentence Embeddings task. On the new endpoint creation page, select the Sentence Embeddings task under the Advanced Configuration section.
+ After the endpoint has initialized, copy the generated endpoint URL.
+ Recommended models for text_embedding task:
all-MiniLM-L6-v2all-MiniLM-L12-v2multilingual-e5-basemultilingual-e5-smallWhen you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
For Elastic's chat_completion and completion tasks:
+ The selected model must support the Text Generation task and expose OpenAI API. HuggingFace supports both serverless and dedicated endpoints for Text Generation. When creating dedicated endpoint select the Text Generation task.
+ After the endpoint is initialized (for dedicated) or ready (for serverless), ensure it supports the OpenAI API and includes /v1/chat/completions part in URL. Then, copy the full endpoint URL for use.
+ Recommended models for chat_completion and completion tasks:
Mistral-7B-Instruct-v0.2QwQ-32BPhi-3-mini-128k-instructFor Elastic's rerank task:
+ The selected model must support the sentence-ranking task and expose OpenAI API.
+ HuggingFace supports only dedicated (not serverless) endpoints for Rerank so far.
+ After the endpoint is initialized, copy the full endpoint URL for use.
+ Tested models for rerank task:
bge-reranker-basejina-reranker-v1-turbo-en-GGUFCreate an inference endpoint to perform an inference task with the jinaai service.
To review the available rerank models, refer to https://jina.ai/reranker.
To review the available text_embedding models, refer to the https://jina.ai/embeddings/.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Mistral inference endpoint.
-Creates an inference endpoint to perform an inference task with the mistral service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint to perform an inference task with the mistral service.
Create an OpenAI inference endpoint.
Create an inference endpoint to perform an inference task with the openai service or openai compatible APIs.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint to perform an inference task with the watsonxai service.
You need an IBM Cloud Databases for Elasticsearch deployment to use the watsonxai inference service.
You can provision one through the IBM catalog, the Cloud Databases CLI plug-in, the Cloud Databases API, or Terraform.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Get data frame analytics jobs usage info.
+Get data frame analytics job stats.
`Get datafeeds usage info. +
Get datafeed stats.
You can get statistics for multiple datafeeds in a single API request by
using a comma-separated list of datafeeds or a wildcard expression. You can
get statistics for all datafeeds by using _all, by specifying * as the
@@ -2033,7 +2033,7 @@ async def get_job_stats(
"""
.. raw:: html
-
Get anomaly detection jobs usage info.
+Get anomaly detection job stats.
`size: Because rollups work on pre-aggregated data, no search hits can be returned and so size must be set to zero or omitted entirely.
highlighter, suggestors, post_filter, profile, explain: These are similarly disallowed.
Searching both historical rollup and non-rollup data
-The rollup search API has the capability to search across both "live" non-rollup data and the aggregated rollup data. - This is done by simply adding the live indices to the URI. For example:
-GET sensor-1,sensor_rollup/_rollup_search
- {
- "size": 0,
- "aggregations": {
- "max_temperature": {
- "max": {
- "field": "temperature"
- }
- }
- }
- }
- The rollup search endpoint does two things when the search runs:
-When the two responses are received, the endpoint rewrites the rollup response and merges the two together. - During the merging process, if there is any overlap in buckets between the two responses, the buckets from the non-rollup index are used.
+For more detailed examples of using the rollup search API, including querying rolled-up data only or combining rolled-up and live data, refer to the External documentation.
`To use this API, you must have at least the manage_own_api_key or the read_security cluster privileges.
If you have only the manage_own_api_key privilege, this API returns only the API keys that you own.
- If you have the read_security, manage_api_key, or greater privileges (including manage_security), this API returns all API keys regardless of ownership.
read_security, manage_api_key, or greater privileges (including manage_security), this API returns all API keys regardless of ownership.
+ Refer to the linked documentation for examples of how to find API keys:
`This API supports updates to an API key's access scope, metadata, and expiration.
The owner user's information, such as the username and realm, is also updated automatically on every call.
NOTE: This API cannot update REST API keys, which should be updated by either the update API key or bulk update API keys API.
+To learn more about how to use this API, refer to the Update cross cluter API key API examples page.
`Analyze a snapshot repository. - Analyze the performance characteristics and any incorrect behaviour found in a repository.
-The response exposes implementation details of the analysis which may change from version to version. - The response body format is therefore not considered stable and may be different in newer versions.
+Analyze a snapshot repository.
+Performs operations on a snapshot repository in order to check for incorrect behaviour.
There are a large number of third-party storage systems available, not all of which are suitable for use as a snapshot repository by Elasticsearch. - Some storage systems behave incorrectly, or perform poorly, especially when accessed concurrently by multiple clients as the nodes of an Elasticsearch cluster do. This API performs a collection of read and write operations on your repository which are designed to detect incorrect behaviour and to measure the performance characteristics of your storage system.
+ Some storage systems behave incorrectly, or perform poorly, especially when accessed concurrently by multiple clients as the nodes of an Elasticsearch cluster do. + This API performs a collection of read and write operations on your repository which are designed to detect incorrect behaviour and to measure the performance characteristics of your storage system.The default values for the parameters are deliberately low to reduce the impact of running an analysis inadvertently and to provide a sensible starting point for your investigations.
Run your first analysis with the default parameter values to check for simple problems.
- If successful, run a sequence of increasingly large analyses until you encounter a failure or you reach a blob_count of at least 2000, a max_blob_size of at least 2gb, a max_total_data_size of at least 1tb, and a register_operation_count of at least 100.
+ Some repositories may behave correctly when lightly loaded but incorrectly under production-like workloads.
+ If the first analysis is successful, run a sequence of increasingly large analyses until you encounter a failure or you reach a blob_count of at least 2000, a max_blob_size of at least 2gb, a max_total_data_size of at least 1tb, and a register_operation_count of at least 100.
Always specify a generous timeout, possibly 1h or longer, to allow time for each analysis to run to completion.
+ Some repositories may behave correctly when accessed by a small number of Elasticsearch nodes but incorrectly when accessed concurrently by a production-scale cluster.
Perform the analyses using a multi-node cluster of a similar size to your production cluster so that it can detect any problems that only arise when the repository is accessed by many nodes at once.
If the analysis fails, Elasticsearch detected that your repository behaved unexpectedly. This usually means you are using a third-party storage system with an incorrect or incompatible implementation of the API it claims to support. If so, this storage system is not suitable for use as a snapshot repository. + Repository analysis triggers conditions that occur only rarely when taking snapshots in a production system. + Snapshotting to unsuitable storage may appear to work correctly most of the time despite repository analysis failures. + However your snapshot data is at risk if you store it in a snapshot repository that does not reliably pass repository analysis. + You can demonstrate that the analysis failure is due to an incompatible storage implementation by verifying that Elasticsearch does not detect the same problem when analysing the reference implementation of the storage protocol you are using. + For instance, if you are using storage that offers an API which the supplier claims to be compatible with AWS S3, verify that repositories in AWS S3 do not fail repository analysis. + This allows you to demonstrate to your storage supplier that a repository analysis failure must only be caused by an incompatibility with AWS S3 and cannot be attributed to a problem in Elasticsearch. + Please do not report Elasticsearch issues involving third-party storage systems unless you can demonstrate that the same issue exists when analysing a repository that uses the reference implementation of the same storage protocol. You will need to work with the supplier of your storage system to address the incompatibilities that Elasticsearch detects.
If the analysis is successful, the API returns details of the testing process, optionally including how long each operation took.
You can use this information to determine the performance of your storage system.
@@ -790,14 +824,17 @@ async def repository_analyze(
This consumes bandwidth on the network between the cluster and the repository, and storage space and I/O bandwidth on the repository itself.
You must ensure this load does not affect other users of these systems.
Analyses respect the repository settings max_snapshot_bytes_per_sec and max_restore_bytes_per_sec if available and the cluster setting indices.recovery.max_bytes_per_sec which you can use to limit the bandwidth they consume.
NOTE: This API is intended for exploratory use by humans. You should expect the request parameters and the response format to vary in future versions.
+NOTE: This API is intended for exploratory use by humans. + You should expect the request parameters and the response format to vary in future versions. + The response exposes immplementation details of the analysis which may change from version to version.
NOTE: Different versions of Elasticsearch may perform different checks for repository compatibility, with newer versions typically being stricter than older ones. A storage system that passes repository analysis with one version of Elasticsearch may fail with a different version. This indicates it behaves incorrectly in ways that the former version did not detect. You must work with the supplier of your storage system to address the incompatibilities detected by the repository analysis API in any version of Elasticsearch.
NOTE: This API may not work correctly in a mixed-version cluster.
Implementation details
-NOTE: This section of documentation describes how the repository analysis API works in this version of Elasticsearch, but you should expect the implementation to vary between versions. The request parameters and response format depend on details of the implementation so may also be different in newer versions.
+NOTE: This section of documentation describes how the repository analysis API works in this version of Elasticsearch, but you should expect the implementation to vary between versions. + The request parameters and response format depend on details of the implementation so may also be different in newer versions.
The analysis comprises a number of blob-level tasks, as set by the blob_count parameter and a number of compare-and-exchange operations on linearizable registers, as set by the register_operation_count parameter.
These tasks are distributed over the data and master-eligible nodes in the cluster for execution.
For most blob-level tasks, the executing node first writes a blob to the repository and then instructs some of the other nodes in the cluster to attempt to read the data it just wrote. @@ -1223,6 +1260,11 @@ async def status(
If you omit the <snapshot> request path parameter, the request retrieves information only for currently running snapshots.
This usage is preferred.
If needed, you can specify <repository> and <snapshot> to retrieve information for specific snapshots, even if they're not currently running.
Note that the stats will not be available for any shard snapshots in an ongoing snapshot completed by a node that (even momentarily) left the cluster. + Loading the stats from the repository is an expensive operation (see the WARNING below). + Therefore the stats values for such shards will be -1 even though the "stage" value will be "DONE", in order to minimize latency. + A "description" field will be present for a shard snapshot completed by a departed node explaining why the shard snapshot's stats results are invalid. + Consequently, the total stats for the index will be less than expected due to the missing values from these shards.
WARNING: Using the API to return the status of any snapshots other than currently running snapshots can be expensive. The API requires a read from the repository for each shard in each snapshot. For example, if you have 100 snapshots with 1,000 shards each, an API request that includes all snapshots will require 100,000 reads (100 snapshots x 1,000 shards).
diff --git a/elasticsearch/_async/client/synonyms.py b/elasticsearch/_async/client/synonyms.py index 26b248a35..2466dfb6c 100644 --- a/elasticsearch/_async/client/synonyms.py +++ b/elasticsearch/_async/client/synonyms.py @@ -90,6 +90,7 @@ async def delete_synonym_rule( filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, human: t.Optional[bool] = None, pretty: t.Optional[bool] = None, + refresh: t.Optional[bool] = None, ) -> ObjectApiResponse[t.Any]: """ .. raw:: html @@ -102,6 +103,9 @@ async def delete_synonym_rule( :param set_id: The ID of the synonym set to update. :param rule_id: The ID of the synonym rule to delete. + :param refresh: If `true`, the request will refresh the analyzers with the deleted + synonym rule and wait for the new synonyms to be available before returning. + If `false`, analyzers will not be reloaded with the deleted synonym rule """ if set_id in SKIP_IN_PATH: raise ValueError("Empty value passed for parameter 'set_id'") @@ -121,6 +125,8 @@ async def delete_synonym_rule( __query["human"] = human if pretty is not None: __query["pretty"] = pretty + if refresh is not None: + __query["refresh"] = refresh __headers = {"accept": "application/json"} return await self.perform_request( # type: ignore[return-value] "DELETE", @@ -299,6 +305,7 @@ async def put_synonym( filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, human: t.Optional[bool] = None, pretty: t.Optional[bool] = None, + refresh: t.Optional[bool] = None, body: t.Optional[t.Dict[str, t.Any]] = None, ) -> ObjectApiResponse[t.Any]: """ @@ -309,12 +316,16 @@ async def put_synonym( If you need to manage more synonym rules, you can create multiple synonym sets.When an existing synonyms set is updated, the search analyzers that use the synonyms set are reloaded automatically for all indices. This is equivalent to invoking the reload search analyzers API for all indices that use the synonyms set.
+For practical examples of how to create or update a synonyms set, refer to the External documentation.
`IMPORTANT: If the specified watch is currently being executed, this API will return an error The reason for this behavior is to prevent overwriting the watch status from a watch execution.
Acknowledging an action throttles further executions of that action until its ack.state is reset to awaits_successful_execution.
- This happens when the condition of the watch is not met (the condition evaluates to false).
When Elasticsearch security features are enabled on your cluster, watches are run with the privileges of the user that stored the watches.
If your user is allowed to read index a, but not index b, then the exact same set of rules will apply during execution of a watch.
When using the run watch API, the authorization data of the user that called the API will be used as a base, instead of the information who stored the watch.
+When using the run watch API, the authorization data of the user that called the API will be used as a base, instead of the information who stored the watch. + Refer to the external documentation for examples of watch execution requests, including existing, customized, and inline watches.
`_bulk?refresh=wait_for request with three documents in it that happen to be routed to different shards in an index with five shards.
The request will only wait for those three shards to refresh.
The other two shards that make up the index do not participate in the _bulk request at all.
+ You might want to disable the refresh interval temporarily to improve indexing throughput for large bulk requests. + Refer to the linked documentation for step-by-step instructions using the index settings API.
`conflicts property.
Additionally, if you opt to count version conflicts, the operation could attempt to reindex more documents from the source than max_docs until it has successfully indexed max_docs documents into the target or it has gone through every document in the source query.
- NOTE: The reindex API makes no effort to handle ID collisions. - The last document written will "win" but the order isn't usually predictable so it is not a good idea to rely on this behavior. - Instead, make sure that IDs are unique by using a script.
-Running reindex asynchronously
-If the request contains wait_for_completion=false, Elasticsearch performs some preflight checks, launches the request, and returns a task you can use to cancel or get the status of the task.
- Elasticsearch creates a record of this task as a document at _tasks/<task_id>.
Reindex from multiple sources
-If you have many sources to reindex it is generally better to reindex them one at a time rather than using a glob pattern to pick up multiple sources. - That way you can resume the process if there are any errors by removing the partially completed source and starting over. - It also makes parallelizing the process fairly simple: split the list of sources to reindex and run each list in parallel.
-For example, you can use a bash script like this:
-for index in i1 i2 i3 i4 i5; do
- curl -HContent-Type:application/json -XPOST localhost:9200/_reindex?pretty -d'{
- "source": {
- "index": "'$index'"
- },
- "dest": {
- "index": "'$index'-reindexed"
- }
- }'
- done
- Throttling
-Set requests_per_second to any positive decimal number (1.4, 6, 1000, for example) to throttle the rate at which reindex issues batches of index operations.
- Requests are throttled by padding each batch with a wait time.
- To turn off throttling, set requests_per_second to -1.
The throttling is done by waiting between batches so that the scroll that reindex uses internally can be given a timeout that takes into account the padding.
- The padding time is the difference between the batch size divided by the requests_per_second and the time spent writing.
- By default the batch size is 1000, so if requests_per_second is set to 500:
target_time = 1000 / 500 per second = 2 seconds
- wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
- Since the batch is issued as a single bulk request, large batch sizes cause Elasticsearch to create many requests and then wait for a while before starting the next set. - This is "bursty" instead of "smooth".
-Slicing
-Reindex supports sliced scroll to parallelize the reindexing process. - This parallelization can improve efficiency and provide a convenient way to break the request down into smaller parts.
-NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
-You can slice a reindex request manually by providing a slice ID and total number of slices to each request.
- You can also let reindex automatically parallelize by using sliced scroll to slice on _id.
- The slices parameter specifies the number of slices to use.
Adding slices to the reindex request just automates the manual process, creating sub-requests which means it has some quirks:
slices only contains the status of completed slices.slices will rethrottle the unfinished sub-request proportionally.slices will cancel each sub-request.slices, each sub-request won't get a perfectly even portion of the documents. All documents will be addressed, but some slices may be larger than others. Expect larger slices to have a more even distribution.requests_per_second and max_docs on a request with slices are distributed proportionally to each sub-request. Combine that with the previous point about distribution being uneven and you should conclude that using max_docs with slices might not result in exactly max_docs documents being reindexed.If slicing automatically, setting slices to auto will choose a reasonable number for most indices.
- If slicing manually or otherwise tuning automatic slicing, use the following guidelines.
Query performance is most efficient when the number of slices is equal to the number of shards in the index.
- If that number is large (for example, 500), choose a lower number as too many slices will hurt performance.
- Setting slices higher than the number of shards generally does not improve efficiency and adds overhead.
Indexing performance scales linearly across available resources with the number of slices.
-Whether query or indexing performance dominates the runtime depends on the documents being reindexed and cluster resources.
-Modify documents during reindexing
-Like _update_by_query, reindex operations support a script that modifies the document.
- Unlike _update_by_query, the script is allowed to modify the document's metadata.
Just as in _update_by_query, you can set ctx.op to change the operation that is run on the destination.
- For example, set ctx.op to noop if your script decides that the document doesn’t have to be indexed in the destination. This "no operation" will be reported in the noop counter in the response body.
- Set ctx.op to delete if your script decides that the document must be deleted from the destination.
- The deletion will be reported in the deleted counter in the response body.
- Setting ctx.op to anything else will return an error, as will setting any other field in ctx.
Think of the possibilities! Just be careful; you are able to change:
-_id_index_version_routingSetting _version to null or clearing it from the ctx map is just like not sending the version in an indexing request.
- It will cause the document to be overwritten in the destination regardless of the version on the target or the version type you use in the reindex API.
Reindex from remote
-Reindex supports reindexing from a remote Elasticsearch cluster.
- The host parameter must contain a scheme, host, port, and optional path.
- The username and password parameters are optional and when they are present the reindex operation will connect to the remote Elasticsearch node using basic authentication.
- Be sure to use HTTPS when using basic authentication or the password will be sent in plain text.
- There are a range of settings available to configure the behavior of the HTTPS connection.
When using Elastic Cloud, it is also possible to authenticate against the remote cluster through the use of a valid API key.
- Remote hosts must be explicitly allowed with the reindex.remote.whitelist setting.
- It can be set to a comma delimited list of allowed remote host and port combinations.
- Scheme is ignored; only the host and port are used.
- For example:
reindex.remote.whitelist: [otherhost:9200, another:9200, 127.0.10.*:9200, localhost:*"]
- The list of allowed hosts must be configured on any nodes that will coordinate the reindex. - This feature should work with remote clusters of any version of Elasticsearch. - This should enable you to upgrade from any version of Elasticsearch to the current version by reindexing from a cluster of the old version.
-WARNING: Elasticsearch does not support forward compatibility across major versions. - For example, you cannot reindex from a 7.x cluster into a 6.x cluster.
-To enable queries sent to older versions of Elasticsearch, the query parameter is sent directly to the remote host without validation or modification.
NOTE: Reindexing from remote clusters does not support manual or automatic slicing.
-Reindexing from a remote server uses an on-heap buffer that defaults to a maximum size of 100mb.
- If the remote index includes very large documents you'll need to use a smaller batch size.
- It is also possible to set the socket read timeout on the remote connection with the socket_timeout field and the connection timeout with the connect_timeout field.
- Both default to 30 seconds.
Configuring SSL parameters
-Reindex from remote supports configurable SSL settings.
- These must be specified in the elasticsearch.yml file, with the exception of the secure settings, which you add in the Elasticsearch keystore.
- It is not possible to configure SSL in the body of the reindex request.
Refer to the linked documentation for examples of how to reindex documents.
`geo_bounds aggregation on the <field>. The search only includes this aggregation if the exact_bounds parameter is true.with_labels is true, the internal search will include a dynamic runtime field that calls the getLabelPosition function of the geometry doc value. This enables the generation of new point features containing suggested geometry labels, so that, for example, multi-polygons will have only one label.For example, Elasticsearch may translate a vector tile search API request with a grid_agg argument of geotile and an exact_bounds argument of true into the following search
GET my-index/_search
- {
- "size": 10000,
- "query": {
- "geo_bounding_box": {
- "my-geo-field": {
- "top_left": {
- "lat": -40.979898069620134,
- "lon": -45
- },
- "bottom_right": {
- "lat": -66.51326044311186,
- "lon": 0
- }
- }
- }
- },
- "aggregations": {
- "grid": {
- "geotile_grid": {
- "field": "my-geo-field",
- "precision": 11,
- "size": 65536,
- "bounds": {
- "top_left": {
- "lat": -40.979898069620134,
- "lon": -45
- },
- "bottom_right": {
- "lat": -66.51326044311186,
- "lon": 0
- }
- }
- }
- },
- "bounds": {
- "geo_bounds": {
- "field": "my-geo-field",
- "wrap_longitude": false
- }
- }
- }
- }
- The API returns results as a binary Mapbox vector tile. Mapbox vector tiles are encoded as Google Protobufs (PBF). By default, the tile contains three layers:
Learn how to use the vector tile search API with practical examples in the Vector tile search examples guide.
`routing only to hit a particular shard.
+ Use routing only to hit a particular shard.
+ Refer to the linked documentation for detailed examples of how to use this API.
`The document must still be reindexed, but using this API removes some network roundtrips and reduces chances of version conflicts between the GET and the index operation.
The _source field must be enabled to use this API.
- In addition to _source, you can access the following variables through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).
_source, you can access the following variables through the ctx map: _index, _type, _id, _version, _routing, and _now (the current timestamp).
+ For usage examples such as partial updates, upserts, and scripted updates, see the External documentation.
`Refreshing shards
+Specifying the refresh parameter refreshes all shards once the request completes.
+ This is different to the update API's refresh parameter, which causes only the shard
+ that received the request to be refreshed. Unlike the update API, it does not support
+ wait_for.
Running update by query asynchronously
+If the request contains wait_for_completion=false, Elasticsearch
+ performs some preflight checks, launches the request, and returns a
+ task you can use to cancel or get the status of the task.
+ Elasticsearch creates a record of this task as a document at .tasks/task/${taskId}.
Waiting for active shards
+wait_for_active_shards controls how many copies of a shard must be active
+ before proceeding with the request. See wait_for_active_shards
+ for details. timeout controls how long each write request waits for unavailable
+ shards to become available. Both work exactly the way they work in the
+ Bulk API. Update by query uses scrolled searches, so you can also
+ specify the scroll parameter to control how long it keeps the search context
+ alive, for example ?scroll=10m. The default is 5 minutes.
Throttling update requests
To control the rate at which update by query issues batches of update operations, you can set requests_per_second to any positive decimal number.
This pads each batch with a wait time to throttle the rate.
@@ -6263,18 +6136,8 @@ def update_by_query(
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
-Update the document source
-Update by query supports scripts to update the document source.
- As with the update API, you can set ctx.op to change the operation that is performed.
Set ctx.op = "noop" if your script decides that it doesn't have to make any changes.
- The update by query operation skips updating the document and increments the noop counter.
Set ctx.op = "delete" if your script decides that the document should be deleted.
- The update by query operation deletes the document and increments the deleted counter.
Update by query supports only index, noop, and delete.
- Setting ctx.op to anything else is an error.
- Setting any other field in ctx is an error.
- This API enables you to only modify the source of matching documents; you cannot move them.
Whether query or update performance dominates the runtime depends on the documents being reindexed and cluster resources.
+ Refer to the linked documentation for examples of how to update documents using the _update_by_query API:
Get cluster-wide settings. - By default, it returns only settings that have been explicitly defined.
+Get cluster-wide settings.
+By default, it returns only settings that have been explicitly defined.
`Get a specific running ES|QL query information. + Returns an object extended information about a running ES|QL query.
+ + + :param id: The query ID + """ + if id in SKIP_IN_PATH: + raise ValueError("Empty value passed for parameter 'id'") + __path_parts: t.Dict[str, str] = {"id": _quote(id)} + __path = f'/_query/queries/{__path_parts["id"]}' + __query: t.Dict[str, t.Any] = {} + if error_trace is not None: + __query["error_trace"] = error_trace + if filter_path is not None: + __query["filter_path"] = filter_path + if human is not None: + __query["human"] = human + if pretty is not None: + __query["pretty"] = pretty + __headers = {"accept": "application/json"} + return self.perform_request( # type: ignore[return-value] + "GET", + __path, + params=__query, + headers=__headers, + endpoint_id="esql.get_query", + path_parts=__path_parts, + ) + + @_rewrite_parameters() + @_stability_warning(Stability.EXPERIMENTAL) + def list_queries( + self, + *, + error_trace: t.Optional[bool] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + pretty: t.Optional[bool] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Get running ES|QL queries information. + Returns an object containing IDs and other information about the running ES|QL queries.
+ + """ + __path_parts: t.Dict[str, str] = {} + __path = "/_query/queries" + __query: t.Dict[str, t.Any] = {} + if error_trace is not None: + __query["error_trace"] = error_trace + if filter_path is not None: + __query["filter_path"] = filter_path + if human is not None: + __query["human"] = human + if pretty is not None: + __query["pretty"] = pretty + __headers = {"accept": "application/json"} + return self.perform_request( # type: ignore[return-value] + "GET", + __path, + params=__query, + headers=__headers, + endpoint_id="esql.list_queries", + path_parts=__path_parts, + ) + @_rewrite_parameters( body_fields=( "query", @@ -422,7 +518,9 @@ def query( parameter, runs it, and returns the results. :param allow_partial_results: If `true`, partial results will be returned if there are shard failures, but the query can continue to execute on other - clusters and shards. + clusters and shards. If `false`, the query will fail if there are any failures. + To override the default behavior, you can set the `esql.query.allow_partial_results` + cluster setting to `false`. :param columnar: By default, ES|QL returns results as rows. For example, FROM returns each individual document as one row. For the JSON, YAML, CBOR and smile formats, ES|QL can return the results in a columnar fashion where one diff --git a/elasticsearch/_sync/client/indices.py b/elasticsearch/_sync/client/indices.py index a63f0319f..512b7d73e 100644 --- a/elasticsearch/_sync/client/indices.py +++ b/elasticsearch/_sync/client/indices.py @@ -338,7 +338,7 @@ def clear_cache( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param fielddata: If `true`, clears the fields cache. Use the `fields` parameter to clear the cache of specific fields only. :param fields: Comma-separated list of field names used to limit the `fielddata` @@ -563,7 +563,7 @@ def close( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. :param master_timeout: Period to wait for a connection to the master node. If @@ -656,7 +656,15 @@ def create( `Delete data stream options. + Removes the data stream options from a data stream.
+ + + `Delete a legacy index template.
+Delete a legacy index template. + IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
`Get data stream options.
+Get the data stream options configuration of one or more data streams.
+ + + `Get data stream settings.
+Get setting information for one or more data streams.
+ + + `Get index templates. +
Get legacy index templates. Get information about one or more index templates.
IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
@@ -3157,7 +3345,7 @@ def open( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. :param master_timeout: Period to wait for a connection to the master node. If @@ -3416,8 +3604,7 @@ def put_data_lifecycle( for this data stream. A data stream lifecycle that's disabled (enabled: `false`) will have no effect on the data stream. :param expand_wildcards: Type of data stream that wildcard patterns can match. - Supports comma-separated values, such as `open,hidden`. Valid values are: - `all`, `hidden`, `open`, `closed`, `none`. + Supports comma-separated values, such as `open,hidden`. :param master_timeout: Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error. @@ -3466,6 +3653,167 @@ def put_data_lifecycle( path_parts=__path_parts, ) + @_rewrite_parameters( + body_fields=("failure_store",), + ) + def put_data_stream_options( + self, + *, + name: t.Union[str, t.Sequence[str]], + error_trace: t.Optional[bool] = None, + expand_wildcards: t.Optional[ + t.Union[ + t.Sequence[ + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]] + ], + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]], + ] + ] = None, + failure_store: t.Optional[t.Mapping[str, t.Any]] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + master_timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + pretty: t.Optional[bool] = None, + timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + body: t.Optional[t.Dict[str, t.Any]] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Update data stream options. + Update the data stream options of the specified data streams.
+ + + `Update data stream settings.
+This API can be used to override settings on specific data streams. These overrides will take precedence over what + is specified in the template that the data stream matches. To prevent your data stream from getting into an invalid state, + only certain settings are allowed. If possible, the setting change is applied to all + backing indices. Otherwise, it will be applied when the data stream is next rolled over.
+ + + `Update field mappings. Add new fields to an existing data stream or index. - You can also use this API to change the search settings of existing fields and add new properties to existing object fields. - For data streams, these changes are applied to all backing indices by default.
-Add multi-fields to an existing field
-Multi-fields let you index the same field in different ways. - You can use this API to update the fields mapping parameter and enable multi-fields for an existing field. - WARNING: If an index (or data stream) contains documents when you add a multi-field, those documents will not have values for the new multi-field. - You can populate the new multi-field with the update by query API.
-Change supported mapping parameters for an existing field
-The documentation for each mapping parameter indicates whether you can update it for an existing field using this API.
- For example, you can use the update mapping API to update the ignore_above parameter.
Change the mapping of an existing field
-Except for supported mapping parameters, you can't change the mapping or field type of an existing field. - Changing an existing field could invalidate data that's already indexed.
-If you need to change the mapping of a field in a data stream's backing indices, refer to documentation about modifying data streams. - If you need to change the mapping of a field in other indices, create a new index with the correct mapping and reindex your data into that index.
-Rename a field
-Renaming a field would invalidate data already indexed under the old field name. - Instead, add an alias field to create an alternate field name.
+ You can use the update mapping API to: +Learn how to use the update mapping API with practical examples in the Update mapping API examples guide.
`To revert a setting to the default value, use a null value.
- The list of per-index settings that can be updated dynamically on live indices can be found in index module documentation.
+ The list of per-index settings that can be updated dynamically on live indices can be found in index settings documentation.
To preserve existing settings from being updated, set the preserve_existing parameter to true.
For performance optimization during bulk indexing, you can disable the refresh interval. + Refer to disable refresh interval for an example. + There are multiple valid ways to represent index settings in the request body. You can specify only the setting, for example:
+{
+ "number_of_replicas": 1
+ }
+ Or you can use an index setting object:
{
+ "index": {
+ "number_of_replicas": 1
+ }
+ }
+ Or you can use dot annotation:
+{
+ "index.number_of_replicas": 1
+ }
+ Or you can embed any of the aforementioned options in a settings object. For example:
{
+ "settings": {
+ "index": {
+ "number_of_replicas": 1
+ }
+ }
+ }
+ NOTE: You can only define new analyzers on closed indices. To add an analyzer, you must close the index, define the analyzer, and reopen the index. You cannot close the write index of a data stream. @@ -3856,7 +4225,8 @@ def put_settings( Then roll over the data stream to apply the new analyzer to the stream's write index and future backing indices. This affects searches and any new data added to the stream after the rollover. However, it does not affect the data stream's backing indices or their existing data. - To change the analyzer for existing backing indices, you must create a new data stream and reindex your data into it.
+ To change the analyzer for existing backing indices, you must create a new data stream and reindex your data into it. + Refer to updating analyzers on existing indices for step-by-step examples. `Create or update an index template. +
Create or update a legacy index template. Index templates define settings, mappings, and aliases that can be applied automatically to new indices. Elasticsearch applies templates to new indices based on an index pattern that matches the index name.
IMPORTANT: This documentation is about legacy index templates, which are deprecated and will be replaced by the composable templates introduced in Elasticsearch 7.8.
@@ -4172,7 +4542,7 @@ def refresh( :param expand_wildcards: Type of index that wildcard patterns can match. If the request can target data streams, this argument determines whether wildcard expressions match hidden data streams. Supports comma-separated values, such - as `open,hidden`. Valid values are: `all`, `open`, `closed`, `hidden`, `none`. + as `open,hidden`. :param ignore_unavailable: If `false`, the request returns an error if it targets a missing or closed index. """ @@ -4288,6 +4658,105 @@ def reload_search_analyzers( path_parts=__path_parts, ) + @_rewrite_parameters() + def remove_block( + self, + *, + index: str, + block: t.Union[str, t.Literal["metadata", "read", "read_only", "write"]], + allow_no_indices: t.Optional[bool] = None, + error_trace: t.Optional[bool] = None, + expand_wildcards: t.Optional[ + t.Union[ + t.Sequence[ + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]] + ], + t.Union[str, t.Literal["all", "closed", "hidden", "none", "open"]], + ] + ] = None, + filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, + human: t.Optional[bool] = None, + ignore_unavailable: t.Optional[bool] = None, + master_timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + pretty: t.Optional[bool] = None, + timeout: t.Optional[t.Union[str, t.Literal[-1], t.Literal[0]]] = None, + ) -> ObjectApiResponse[t.Any]: + """ + .. raw:: html + +Remove an index block.
+Remove an index block from an index. + Index blocks limit the operations allowed on an index by blocking specific operation types.
+ + + `Create an inference endpoint.
- When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint.
IMPORTANT: The inference APIs enable you to use certain services, such as built-in machine learning models (ELSER, E5), models uploaded through Eland, Cohere, OpenAI, Mistral, Azure OpenAI, Google AI Studio, Google Vertex AI, Anthropic, Watsonx.ai, or Hugging Face. For built-in models and models uploaded through Eland, the inference APIs offer an alternative way to use and manage trained models. However, if you do not plan to use the inference APIs to use these models or if you want to use non-NLP models, use the machine learning trained model APIs.
+The following integrations are available through the inference API. You can find the available task types next to the integration name:
+completion, rerank, sparse_embedding, text_embedding)completion, text_embedding)completion)completion, text_embedding)completion, text_embedding)completion, rerank, text_embedding)completion, chat_completion)rerank, sparse_embedding, text_embedding - this service is for built-in models and models uploaded through Eland)sparse_embedding)completion, text_embedding)rerank, text_embedding)chat_completion, completion, rerank, text_embedding)chat_completion, completion, text_embedding)chat_completion, completion, text_embedding)text_embedding, rerank)text_embedding)text_embedding, rerank)Create an AlibabaCloud AI Search inference endpoint.
Create an inference endpoint to perform an inference task with the alibabacloud-ai-search service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Amazon Bedrock inference endpoint.
-Creates an inference endpoint to perform an inference task with the amazonbedrock service.
Create an inference endpoint to perform an inference task with the amazonbedrock service.
-info You need to provide the access and secret keys only once, during the inference model creation. The get inference API does not retrieve your access or secret keys. After creating the inference model, you cannot change the associated key pairs. If you want to use a different access and secret key pair, delete the inference model and recreate it with the same name and the updated keys.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Anthropic inference endpoint.
Create an inference endpoint to perform an inference task with the anthropic service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an Azure AI studio inference endpoint.
Create an inference endpoint to perform an inference task with the azureaistudio service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
The list of embeddings models that you can choose from in your deployment can be found in the Azure models documentation.
-When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Cohere inference endpoint.
Create an inference endpoint to perform an inference task with the cohere service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a DeepSeek inference endpoint.
+Create an inference endpoint to perform an inference task with the deepseek service.
Create an Google AI Studio inference endpoint.
Create an inference endpoint to perform an inference task with the googleaistudio service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Google Vertex AI inference endpoint.
Create an inference endpoint to perform an inference task with the googlevertexai service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Hugging Face inference endpoint.
-Create an inference endpoint to perform an inference task with the hugging_face service.
You must first create an inference endpoint on the Hugging Face endpoint page to get an endpoint URL.
- Select the model you want to use on the new endpoint creation page (for example intfloat/e5-small-v2), then select the sentence embeddings task under the advanced configuration section.
- Create the endpoint and copy the URL after the endpoint initialization has been finished.
The following models are recommended for the Hugging Face service:
+Create an inference endpoint to perform an inference task with the hugging_face service.
+ Supported tasks include: text_embedding, completion, and chat_completion.
To configure the endpoint, first visit the Hugging Face Inference Endpoints page and create a new endpoint. + Select a model that supports the task you intend to use.
+For Elastic's text_embedding task:
+ The selected model must support the Sentence Embeddings task. On the new endpoint creation page, select the Sentence Embeddings task under the Advanced Configuration section.
+ After the endpoint has initialized, copy the generated endpoint URL.
+ Recommended models for text_embedding task:
all-MiniLM-L6-v2all-MiniLM-L12-v2multilingual-e5-basemultilingual-e5-smallWhen you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
For Elastic's chat_completion and completion tasks:
+ The selected model must support the Text Generation task and expose OpenAI API. HuggingFace supports both serverless and dedicated endpoints for Text Generation. When creating dedicated endpoint select the Text Generation task.
+ After the endpoint is initialized (for dedicated) or ready (for serverless), ensure it supports the OpenAI API and includes /v1/chat/completions part in URL. Then, copy the full endpoint URL for use.
+ Recommended models for chat_completion and completion tasks:
Mistral-7B-Instruct-v0.2QwQ-32BPhi-3-mini-128k-instructFor Elastic's rerank task:
+ The selected model must support the sentence-ranking task and expose OpenAI API.
+ HuggingFace supports only dedicated (not serverless) endpoints for Rerank so far.
+ After the endpoint is initialized, copy the full endpoint URL for use.
+ Tested models for rerank task:
bge-reranker-basejina-reranker-v1-turbo-en-GGUFCreate an inference endpoint to perform an inference task with the jinaai service.
To review the available rerank models, refer to https://jina.ai/reranker.
To review the available text_embedding models, refer to the https://jina.ai/embeddings/.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create a Mistral inference endpoint.
-Creates an inference endpoint to perform an inference task with the mistral service.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint to perform an inference task with the mistral service.
Create an OpenAI inference endpoint.
Create an inference endpoint to perform an inference task with the openai service or openai compatible APIs.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Create an inference endpoint to perform an inference task with the watsonxai service.
You need an IBM Cloud Databases for Elasticsearch deployment to use the watsonxai inference service.
You can provision one through the IBM catalog, the Cloud Databases CLI plug-in, the Cloud Databases API, or Terraform.
When you create an inference endpoint, the associated machine learning model is automatically deployed if it is not already running.
- After creating the endpoint, wait for the model deployment to complete before using it.
- To verify the deployment status, use the get trained model statistics API.
- Look for "state": "fully_allocated" in the response and ensure that the "allocation_count" matches the "target_allocation_count".
- Avoid creating multiple endpoints for the same model unless required, as each endpoint consumes significant resources.
Get data frame analytics jobs usage info.
+Get data frame analytics job stats.
`Get datafeeds usage info. +
Get datafeed stats.
You can get statistics for multiple datafeeds in a single API request by
using a comma-separated list of datafeeds or a wildcard expression. You can
get statistics for all datafeeds by using _all, by specifying * as the
@@ -2033,7 +2033,7 @@ def get_job_stats(
"""
.. raw:: html
-
Get anomaly detection jobs usage info.
+Get anomaly detection job stats.
`size: Because rollups work on pre-aggregated data, no search hits can be returned and so size must be set to zero or omitted entirely.
highlighter, suggestors, post_filter, profile, explain: These are similarly disallowed.
Searching both historical rollup and non-rollup data
-The rollup search API has the capability to search across both "live" non-rollup data and the aggregated rollup data. - This is done by simply adding the live indices to the URI. For example:
-GET sensor-1,sensor_rollup/_rollup_search
- {
- "size": 0,
- "aggregations": {
- "max_temperature": {
- "max": {
- "field": "temperature"
- }
- }
- }
- }
- The rollup search endpoint does two things when the search runs:
-When the two responses are received, the endpoint rewrites the rollup response and merges the two together. - During the merging process, if there is any overlap in buckets between the two responses, the buckets from the non-rollup index are used.
+For more detailed examples of using the rollup search API, including querying rolled-up data only or combining rolled-up and live data, refer to the External documentation.
`To use this API, you must have at least the manage_own_api_key or the read_security cluster privileges.
If you have only the manage_own_api_key privilege, this API returns only the API keys that you own.
- If you have the read_security, manage_api_key, or greater privileges (including manage_security), this API returns all API keys regardless of ownership.
read_security, manage_api_key, or greater privileges (including manage_security), this API returns all API keys regardless of ownership.
+ Refer to the linked documentation for examples of how to find API keys:
`This API supports updates to an API key's access scope, metadata, and expiration.
The owner user's information, such as the username and realm, is also updated automatically on every call.
NOTE: This API cannot update REST API keys, which should be updated by either the update API key or bulk update API keys API.
+To learn more about how to use this API, refer to the Update cross cluter API key API examples page.
`Analyze a snapshot repository. - Analyze the performance characteristics and any incorrect behaviour found in a repository.
-The response exposes implementation details of the analysis which may change from version to version. - The response body format is therefore not considered stable and may be different in newer versions.
+Analyze a snapshot repository.
+Performs operations on a snapshot repository in order to check for incorrect behaviour.
There are a large number of third-party storage systems available, not all of which are suitable for use as a snapshot repository by Elasticsearch. - Some storage systems behave incorrectly, or perform poorly, especially when accessed concurrently by multiple clients as the nodes of an Elasticsearch cluster do. This API performs a collection of read and write operations on your repository which are designed to detect incorrect behaviour and to measure the performance characteristics of your storage system.
+ Some storage systems behave incorrectly, or perform poorly, especially when accessed concurrently by multiple clients as the nodes of an Elasticsearch cluster do. + This API performs a collection of read and write operations on your repository which are designed to detect incorrect behaviour and to measure the performance characteristics of your storage system.The default values for the parameters are deliberately low to reduce the impact of running an analysis inadvertently and to provide a sensible starting point for your investigations.
Run your first analysis with the default parameter values to check for simple problems.
- If successful, run a sequence of increasingly large analyses until you encounter a failure or you reach a blob_count of at least 2000, a max_blob_size of at least 2gb, a max_total_data_size of at least 1tb, and a register_operation_count of at least 100.
+ Some repositories may behave correctly when lightly loaded but incorrectly under production-like workloads.
+ If the first analysis is successful, run a sequence of increasingly large analyses until you encounter a failure or you reach a blob_count of at least 2000, a max_blob_size of at least 2gb, a max_total_data_size of at least 1tb, and a register_operation_count of at least 100.
Always specify a generous timeout, possibly 1h or longer, to allow time for each analysis to run to completion.
+ Some repositories may behave correctly when accessed by a small number of Elasticsearch nodes but incorrectly when accessed concurrently by a production-scale cluster.
Perform the analyses using a multi-node cluster of a similar size to your production cluster so that it can detect any problems that only arise when the repository is accessed by many nodes at once.
If the analysis fails, Elasticsearch detected that your repository behaved unexpectedly. This usually means you are using a third-party storage system with an incorrect or incompatible implementation of the API it claims to support. If so, this storage system is not suitable for use as a snapshot repository. + Repository analysis triggers conditions that occur only rarely when taking snapshots in a production system. + Snapshotting to unsuitable storage may appear to work correctly most of the time despite repository analysis failures. + However your snapshot data is at risk if you store it in a snapshot repository that does not reliably pass repository analysis. + You can demonstrate that the analysis failure is due to an incompatible storage implementation by verifying that Elasticsearch does not detect the same problem when analysing the reference implementation of the storage protocol you are using. + For instance, if you are using storage that offers an API which the supplier claims to be compatible with AWS S3, verify that repositories in AWS S3 do not fail repository analysis. + This allows you to demonstrate to your storage supplier that a repository analysis failure must only be caused by an incompatibility with AWS S3 and cannot be attributed to a problem in Elasticsearch. + Please do not report Elasticsearch issues involving third-party storage systems unless you can demonstrate that the same issue exists when analysing a repository that uses the reference implementation of the same storage protocol. You will need to work with the supplier of your storage system to address the incompatibilities that Elasticsearch detects.
If the analysis is successful, the API returns details of the testing process, optionally including how long each operation took.
You can use this information to determine the performance of your storage system.
@@ -790,14 +824,17 @@ def repository_analyze(
This consumes bandwidth on the network between the cluster and the repository, and storage space and I/O bandwidth on the repository itself.
You must ensure this load does not affect other users of these systems.
Analyses respect the repository settings max_snapshot_bytes_per_sec and max_restore_bytes_per_sec if available and the cluster setting indices.recovery.max_bytes_per_sec which you can use to limit the bandwidth they consume.
NOTE: This API is intended for exploratory use by humans. You should expect the request parameters and the response format to vary in future versions.
+NOTE: This API is intended for exploratory use by humans. + You should expect the request parameters and the response format to vary in future versions. + The response exposes immplementation details of the analysis which may change from version to version.
NOTE: Different versions of Elasticsearch may perform different checks for repository compatibility, with newer versions typically being stricter than older ones. A storage system that passes repository analysis with one version of Elasticsearch may fail with a different version. This indicates it behaves incorrectly in ways that the former version did not detect. You must work with the supplier of your storage system to address the incompatibilities detected by the repository analysis API in any version of Elasticsearch.
NOTE: This API may not work correctly in a mixed-version cluster.
Implementation details
-NOTE: This section of documentation describes how the repository analysis API works in this version of Elasticsearch, but you should expect the implementation to vary between versions. The request parameters and response format depend on details of the implementation so may also be different in newer versions.
+NOTE: This section of documentation describes how the repository analysis API works in this version of Elasticsearch, but you should expect the implementation to vary between versions. + The request parameters and response format depend on details of the implementation so may also be different in newer versions.
The analysis comprises a number of blob-level tasks, as set by the blob_count parameter and a number of compare-and-exchange operations on linearizable registers, as set by the register_operation_count parameter.
These tasks are distributed over the data and master-eligible nodes in the cluster for execution.
For most blob-level tasks, the executing node first writes a blob to the repository and then instructs some of the other nodes in the cluster to attempt to read the data it just wrote. @@ -1223,6 +1260,11 @@ def status(
If you omit the <snapshot> request path parameter, the request retrieves information only for currently running snapshots.
This usage is preferred.
If needed, you can specify <repository> and <snapshot> to retrieve information for specific snapshots, even if they're not currently running.
Note that the stats will not be available for any shard snapshots in an ongoing snapshot completed by a node that (even momentarily) left the cluster. + Loading the stats from the repository is an expensive operation (see the WARNING below). + Therefore the stats values for such shards will be -1 even though the "stage" value will be "DONE", in order to minimize latency. + A "description" field will be present for a shard snapshot completed by a departed node explaining why the shard snapshot's stats results are invalid. + Consequently, the total stats for the index will be less than expected due to the missing values from these shards.
WARNING: Using the API to return the status of any snapshots other than currently running snapshots can be expensive. The API requires a read from the repository for each shard in each snapshot. For example, if you have 100 snapshots with 1,000 shards each, an API request that includes all snapshots will require 100,000 reads (100 snapshots x 1,000 shards).
diff --git a/elasticsearch/_sync/client/synonyms.py b/elasticsearch/_sync/client/synonyms.py index 1c9613196..8731f40fd 100644 --- a/elasticsearch/_sync/client/synonyms.py +++ b/elasticsearch/_sync/client/synonyms.py @@ -90,6 +90,7 @@ def delete_synonym_rule( filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, human: t.Optional[bool] = None, pretty: t.Optional[bool] = None, + refresh: t.Optional[bool] = None, ) -> ObjectApiResponse[t.Any]: """ .. raw:: html @@ -102,6 +103,9 @@ def delete_synonym_rule( :param set_id: The ID of the synonym set to update. :param rule_id: The ID of the synonym rule to delete. + :param refresh: If `true`, the request will refresh the analyzers with the deleted + synonym rule and wait for the new synonyms to be available before returning. + If `false`, analyzers will not be reloaded with the deleted synonym rule """ if set_id in SKIP_IN_PATH: raise ValueError("Empty value passed for parameter 'set_id'") @@ -121,6 +125,8 @@ def delete_synonym_rule( __query["human"] = human if pretty is not None: __query["pretty"] = pretty + if refresh is not None: + __query["refresh"] = refresh __headers = {"accept": "application/json"} return self.perform_request( # type: ignore[return-value] "DELETE", @@ -299,6 +305,7 @@ def put_synonym( filter_path: t.Optional[t.Union[str, t.Sequence[str]]] = None, human: t.Optional[bool] = None, pretty: t.Optional[bool] = None, + refresh: t.Optional[bool] = None, body: t.Optional[t.Dict[str, t.Any]] = None, ) -> ObjectApiResponse[t.Any]: """ @@ -309,12 +316,16 @@ def put_synonym( If you need to manage more synonym rules, you can create multiple synonym sets.When an existing synonyms set is updated, the search analyzers that use the synonyms set are reloaded automatically for all indices. This is equivalent to invoking the reload search analyzers API for all indices that use the synonyms set.
+For practical examples of how to create or update a synonyms set, refer to the External documentation.
`IMPORTANT: If the specified watch is currently being executed, this API will return an error The reason for this behavior is to prevent overwriting the watch status from a watch execution.
Acknowledging an action throttles further executions of that action until its ack.state is reset to awaits_successful_execution.
- This happens when the condition of the watch is not met (the condition evaluates to false).
When Elasticsearch security features are enabled on your cluster, watches are run with the privileges of the user that stored the watches.
If your user is allowed to read index a, but not index b, then the exact same set of rules will apply during execution of a watch.
When using the run watch API, the authorization data of the user that called the API will be used as a base, instead of the information who stored the watch.
+When using the run watch API, the authorization data of the user that called the API will be used as a base, instead of the information who stored the watch. + Refer to the external documentation for examples of watch execution requests, including existing, customized, and inline watches.
`