[[api-conventions]] = API Conventions [partintro] -- The *elasticsearch* REST APIs are exposed using <>. The conventions listed in this chapter can be applied throughout the REST API, unless otherwise specified. * <> * <> * <> * <> -- [[multi-index]] == Multiple Indices Most APIs that refer to an `index` parameter support execution across multiple indices, using simple `test1,test2,test3` notation (or `_all` for all indices). It also support wildcards, for example: `test*` or `*test` or `te*t` or `*test*`, and the ability to "exclude" (`-`), for example: `test*,-test3`. All multi indices API support the following url query string parameters: `ignore_unavailable`:: Controls whether to ignore if any specified indices are unavailable, this includes indices that don't exist or closed indices. Either `true` or `false` can be specified. `allow_no_indices`:: Controls whether to fail if a wildcard indices expressions results into no concrete indices. Either `true` or `false` can be specified. For example if the wildcard expression `foo*` is specified and no indices are available that start with `foo` then depending on this setting the request will fail. This setting is also applicable when `_all`, `*` or no index has been specified. This settings also applies for aliases, in case an alias points to a closed index. `expand_wildcards`:: Controls to what kind of concrete indices wildcard indices expression expand to. If `open` is specified then the wildcard expression is expanded to only open indices and if `closed` is specified then the wildcard expression is expanded only to closed indices. Also both values (`open,closed`) can be specified to expand to all indices. If `none` is specified then wildcard expansion will be disabled and if `all` is specified, wildcard expressions will expand to all indices (this is equivalent to specifying `open,closed`). The defaults settings for the above parameters depend on the api being used. NOTE: Single index APIs such as the <> and the <> do not support multiple indices. [[date-math-index-names]] == Date math support in index names Date math index name resolution enables you to search a range of time-series indices, rather than searching all of your time-series indices and filtering the results or maintaining aliases. Limiting the number of indices that are searched reduces the load on the cluster and improves execution performance. For example, if you are searching for errors in your daily logs, you can use a date math name template to restrict the search to the past two days. Almost all APIs that have an `index` parameter, support date math in the `index` parameter value. A date math index name takes the following form: [source,txt] ---------------------------------------------------------------------- ---------------------------------------------------------------------- Where: [horizontal] `static_name`:: is the static text part of the name `date_math_expr`:: is a dynamic date math expression that computes the date dynamically `date_format`:: is the optional format in which the computed date should be rendered. Defaults to `YYYY.MM.dd`. `time_zone`:: is the optional time zone . Defaults to `utc`. You must enclose date math index name expressions within angle brackets, and all special characters should be URI encoded. For example: [source,js] ---------------------------------------------------------------------- # GET //_search GET /%3Clogstash-%7Bnow%2Fd%7D%3E/_search { "query" : { "match": { "test": "data" } } } ---------------------------------------------------------------------- // CONSOLE // TEST[s/^/PUT logstash-2016.09.20\n/] // TEST[s/now/2016.09.20||/] [NOTE] .Percent encoding of date math characters ====================================================== The special characters used for date rounding must be URI encoded as follows: [horizontal] `<`:: `%3C` `>`:: `%3E` `/`:: `%2F` `{`:: `%7B` `}`:: `%7D` `|`:: `%7C` `+`:: `%2B` `:`:: `%3A` `,`:: `%2C` ====================================================== The following example shows different forms of date math index names and the final index names they resolve to given the current time is 22rd March 2024 noon utc. [options="header"] |====== | Expression |Resolves to | `` | `logstash-2024.03.22` | `` | `logstash-2024.03.01` | `` | `logstash-2024.03` | `` | `logstash-2024.02` | `` | `logstash-2024.03.23` |====== To use the characters `{` and `}` in the static part of an index name template, escape them with a backslash `\`, for example: * `` resolves to `elastic{ON}-2024.03.01` The following example shows a search request that searches the Logstash indices for the past three days, assuming the indices use the default Logstash index name format, `logstash-YYYY.MM.dd`. [source,js] ---------------------------------------------------------------------- # GET /,,/_search GET /%3Clogstash-%7Bnow%2Fd-2d%7D%3E%2C%3Clogstash-%7Bnow%2Fd-1d%7D%3E%2C%3Clogstash-%7Bnow%2Fd%7D%3E/_search { "query" : { "match": { "test": "data" } } } ---------------------------------------------------------------------- // CONSOLE // TEST[s/^/PUT logstash-2016.09.20\nPUT logstash-2016.09.19\nPUT logstash-2016.09.18\n/] // TEST[s/now/2016.09.20||/] [[common-options]] == Common options The following options can be applied to all of the REST APIs. [float] === Pretty Results When appending `?pretty=true` to any request made, the JSON returned will be pretty formatted (use it for debugging only!). Another option is to set `?format=yaml` which will cause the result to be returned in the (sometimes) more readable yaml format. [float] === Human readable output Statistics are returned in a format suitable for humans (eg `"exists_time": "1h"` or `"size": "1kb"`) and for computers (eg `"exists_time_in_millis": 3600000` or `"size_in_bytes": 1024`). The human readable values can be turned off by adding `?human=false` to the query string. This makes sense when the stats results are being consumed by a monitoring tool, rather than intended for human consumption. The default for the `human` flag is `false`. [[date-math]] [float] === Date Math Most parameters which accept a formatted date value -- such as `gt` and `lt` in <> `range` queries, or `from` and `to` in <> -- understand date maths. The expression starts with an anchor date, which can either be `now`, or a date string ending with `||`. This anchor date can optionally be followed by one or more maths expressions: * `+1h` - add one hour * `-1d` - subtract one day * `/d` - round down to the nearest day The supported time units differ from those supported by <> for durations. The supported units are: [horizontal] `y`:: years `M`:: months `w`:: weeks `d`:: days `h`:: hours `H`:: hours `m`:: minutes `s`:: seconds Some examples are: [horizontal] `now+1h`:: The current time plus one hour, with ms resolution. `now+1h+1m`:: The current time plus one hour plus one minute, with ms resolution. `now+1h/d`:: The current time plus one hour, rounded down to the nearest day. `2015-01-01||+1M/d`:: `2015-01-01` plus one month, rounded down to the nearest day. [float] [[common-options-response-filtering]] === Response Filtering All REST APIs accept a `filter_path` parameter that can be used to reduce the response returned by elasticsearch. This parameter takes a comma separated list of filters expressed with the dot notation: [source,js] -------------------------------------------------- GET /_search?q=elasticsearch&filter_path=took,hits.hits._id,hits.hits._score -------------------------------------------------- // CONSOLE // TEST[setup:twitter] Responds: [source,js] -------------------------------------------------- { "took" : 3, "hits" : { "hits" : [ { "_id" : "0", "_score" : 1.6375021 } ] } } -------------------------------------------------- // TESTRESPONSE[s/"took" : 3/"took" : $body.took/] // TESTRESPONSE[s/1.6375021/$body.hits.hits.0._score/] It also supports the `*` wildcard character to match any field or part of a field's name: [source,sh] -------------------------------------------------- GET /_cluster/state?filter_path=metadata.indices.*.stat* -------------------------------------------------- // CONSOLE // TEST[s/^/PUT twitter\n/] Responds: [source,sh] -------------------------------------------------- { "metadata" : { "indices" : { "twitter": {"state": "open"} } } } -------------------------------------------------- // TESTRESPONSE And the `**` wildcard can be used to include fields without knowing the exact path of the field. For example, we can return the Lucene version of every segment with this request: [source,js] -------------------------------------------------- GET /_cluster/state?filter_path=routing_table.indices.**.state -------------------------------------------------- // CONSOLE // TEST[s/^/PUT twitter\n/] Responds: [source,js] -------------------------------------------------- { "routing_table": { "indices": { "twitter": { "shards": { "0": [{"state": "STARTED"}, {"state": "UNASSIGNED"}], "1": [{"state": "STARTED"}, {"state": "UNASSIGNED"}], "2": [{"state": "STARTED"}, {"state": "UNASSIGNED"}], "3": [{"state": "STARTED"}, {"state": "UNASSIGNED"}], "4": [{"state": "STARTED"}, {"state": "UNASSIGNED"}] } } } } } -------------------------------------------------- // TESTRESPONSE It is also possible to exclude one or more fields by prefixing the filter with the char `-`: [source,js] -------------------------------------------------- GET /_count?filter_path=-_shards -------------------------------------------------- // CONSOLE // TEST[setup:twitter] Responds: [source,js] -------------------------------------------------- { "count" : 5 } -------------------------------------------------- // TESTRESPONSE And for more control, both inclusive and exclusive filters can be combined in the same expression. In this case, the exclusive filters will be applied first and the result will be filtered again using the inclusive filters: [source,js] -------------------------------------------------- GET /_cluster/state?filter_path=metadata.indices.*.state,-metadata.indices.logstash-* -------------------------------------------------- // CONSOLE // TEST[s/^/PUT index-1\nPUT index-2\nPUT index-3\nPUT logstash-2016.01\n/] Responds: [source,js] -------------------------------------------------- { "metadata" : { "indices" : { "index-1" : {"state" : "open"}, "index-2" : {"state" : "open"}, "index-3" : {"state" : "open"} } } } -------------------------------------------------- // TESTRESPONSE Note that elasticsearch sometimes returns directly the raw value of a field, like the `_source` field. If you want to filter `_source` fields, you should consider combining the already existing `_source` parameter (see <> for more details) with the `filter_path` parameter like this: [source,js] -------------------------------------------------- POST /library/book?refresh {"title": "Book #1", "rating": 200.1} POST /library/book?refresh {"title": "Book #2", "rating": 1.7} POST /library/book?refresh {"title": "Book #3", "rating": 0.1} GET /_search?filter_path=hits.hits._source&_source=title&sort=rating:desc -------------------------------------------------- // CONSOLE [source,js] -------------------------------------------------- { "hits" : { "hits" : [ { "_source":{"title":"Book #1"} }, { "_source":{"title":"Book #2"} }, { "_source":{"title":"Book #3"} } ] } } -------------------------------------------------- // TESTRESPONSE [float] === Flat Settings The `flat_settings` flag affects rendering of the lists of settings. When `flat_settings` flag is `true` settings are returned in a flat format: [source,js] -------------------------------------------------- GET twitter/_settings?flat_settings=true -------------------------------------------------- // CONSOLE // TEST[setup:twitter] Returns: [source,js] -------------------------------------------------- { "twitter" : { "settings": { "index.number_of_replicas": "1", "index.number_of_shards": "1", "index.creation_date": "1474389951325", "index.uuid": "n6gzFZTgS664GUfx0Xrpjw", "index.version.created": ..., "index.provided_name" : "twitter" } } } -------------------------------------------------- // TESTRESPONSE[s/1474389951325/$body.twitter.settings.index\\\\.creation_date/] // TESTRESPONSE[s/n6gzFZTgS664GUfx0Xrpjw/$body.twitter.settings.index\\\\.uuid/] // TESTRESPONSE[s/"index.version.created": \.\.\./"index.version.created": $body.twitter.settings.index\\\\.version\\\\.created/] When the `flat_settings` flag is `false` settings are returned in a more human readable structured format: [source,js] -------------------------------------------------- GET twitter/_settings?flat_settings=false -------------------------------------------------- // CONSOLE // TEST[setup:twitter] Returns: [source,js] -------------------------------------------------- { "twitter" : { "settings" : { "index" : { "number_of_replicas": "1", "number_of_shards": "1", "creation_date": "1474389951325", "uuid": "n6gzFZTgS664GUfx0Xrpjw", "version": { "created": ... }, "provided_name" : "twitter" } } } } -------------------------------------------------- // TESTRESPONSE[s/1474389951325/$body.twitter.settings.index.creation_date/] // TESTRESPONSE[s/n6gzFZTgS664GUfx0Xrpjw/$body.twitter.settings.index.uuid/] // TESTRESPONSE[s/"created": \.\.\./"created": $body.twitter.settings.index.version.created/] By default the `flat_settings` is set to `false`. [float] === Parameters Rest parameters (when using HTTP, map to HTTP URL parameters) follow the convention of using underscore casing. [float] === Boolean Values All REST APIs parameters (both request parameters and JSON body) support providing boolean "false" as the value `false` and boolean "true" as the value `true`. All other values will raise an error. [float] === Number Values All REST APIs support providing numbered parameters as `string` on top of supporting the native JSON number types. [[time-units]] [float] === Time units Whenever durations need to be specified, e.g. for a `timeout` parameter, the duration must specify the unit, like `2d` for 2 days. The supported units are: [horizontal] `d`:: days `h`:: hours `m`:: minutes `s`:: seconds `ms`:: milliseconds `micros`:: microseconds `nanos`:: nanoseconds [[byte-units]] [float] === Byte size units Whenever the byte size of data needs to be specified, eg when setting a buffer size parameter, the value must specify the unit, like `10kb` for 10 kilobytes. Note that these units use powers of 1024, so `1kb` means 1024 bytes. The supported units are: [horizontal] `b`:: Bytes `kb`:: Kilobytes `mb`:: Megabytes `gb`:: Gigabytes `tb`:: Terabytes `pb`:: Petabytes [[size-units]] [float] === Unit-less quantities Unit-less quantities means that they don't have a "unit" like "bytes" or "Hertz" or "meter" or "long tonne". If one of these quantities is large we'll print it out like 10m for 10,000,000 or 7k for 7,000. We'll still print 87 when we mean 87 though. These are the supported multipliers: [horizontal] ``:: Single `k`:: Kilo `m`:: Mega `g`:: Giga `t`:: Tera `p`:: Peta [[distance-units]] [float] === Distance Units Wherever distances need to be specified, such as the `distance` parameter in the <>), the default unit if none is specified is the meter. Distances can be specified in other units, such as `"1km"` or `"2mi"` (2 miles). The full list of units is listed below: [horizontal] Mile:: `mi` or `miles` Yard:: `yd` or `yards` Feet:: `ft` or `feet` Inch:: `in` or `inch` Kilometer:: `km` or `kilometers` Meter:: `m` or `meters` Centimeter:: `cm` or `centimeters` Millimeter:: `mm` or `millimeters` Nautical mile:: `NM`, `nmi` or `nauticalmiles` [[fuzziness]] [float] === Fuzziness Some queries and APIs support parameters to allow inexact _fuzzy_ matching, using the `fuzziness` parameter. When querying `text` or `keyword` fields, `fuzziness` is interpreted as a http://en.wikipedia.org/wiki/Levenshtein_distance[Levenshtein Edit Distance] -- the number of one character changes that need to be made to one string to make it the same as another string. The `fuzziness` parameter can be specified as: `0`, `1`, `2`:: the maximum allowed Levenshtein Edit Distance (or number of edits) `AUTO`:: + -- generates an edit distance based on the length of the term. Low and high distance arguments may be optionally provided `AUTO:[low],[high]`, if not specified, the default values are 3 and 6, equivalent to `AUTO:3,6` that make for lengths: `0..2`:: must match exactly `3..5`:: one edit allowed `>5`:: two edits allowed `AUTO` should generally be the preferred value for `fuzziness`. -- [float] [[common-options-error-options]] === Enabling stack traces By default when a request returns an error Elasticsearch doesn't include the stack trace of the error. You can enable that behavior by setting the `error_trace` url parameter to `true`. For example, by default when you send an invalid `size` parameter to the `_search` API: [source,js] ---------------------------------------------------------------------- POST /twitter/_search?size=surprise_me ---------------------------------------------------------------------- // CONSOLE // TEST[s/surprise_me/surprise_me&error_trace=false/ catch:request] // Since the test system sends error_trace=true by default we have to override The response looks like: [source,js] ---------------------------------------------------------------------- { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Failed to parse int parameter [size] with value [surprise_me]" } ], "type" : "illegal_argument_exception", "reason" : "Failed to parse int parameter [size] with value [surprise_me]", "caused_by" : { "type" : "number_format_exception", "reason" : "For input string: \"surprise_me\"" } }, "status" : 400 } ---------------------------------------------------------------------- // TESTRESPONSE But if you set `error_trace=true`: [source,js] ---------------------------------------------------------------------- POST /twitter/_search?size=surprise_me&error_trace=true ---------------------------------------------------------------------- // CONSOLE // TEST[catch:request] The response looks like: [source,js] ---------------------------------------------------------------------- { "error": { "root_cause": [ { "type": "illegal_argument_exception", "reason": "Failed to parse int parameter [size] with value [surprise_me]", "stack_trace": "Failed to parse int parameter [size] with value [surprise_me]]; nested: IllegalArgumentException..." } ], "type": "illegal_argument_exception", "reason": "Failed to parse int parameter [size] with value [surprise_me]", "stack_trace": "java.lang.IllegalArgumentException: Failed to parse int parameter [size] with value [surprise_me]\n at org.elasticsearch.rest.RestRequest.paramAsInt(RestRequest.java:175)...", "caused_by": { "type": "number_format_exception", "reason": "For input string: \"surprise_me\"", "stack_trace": "java.lang.NumberFormatException: For input string: \"surprise_me\"\n at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)..." } }, "status": 400 } ---------------------------------------------------------------------- // TESTRESPONSE[s/"stack_trace": "Failed to parse int parameter.+\.\.\."/"stack_trace": $body.error.root_cause.0.stack_trace/] // TESTRESPONSE[s/"stack_trace": "java.lang.IllegalArgum.+\.\.\."/"stack_trace": $body.error.stack_trace/] // TESTRESPONSE[s/"stack_trace": "java.lang.Number.+\.\.\."/"stack_trace": $body.error.caused_by.stack_trace/] [float] === Request body in query string For libraries that don't accept a request body for non-POST requests, you can pass the request body as the `source` query string parameter instead. When using this method, the `source_content_type` parameter should also be passed with a media type value that indicates the format of the source, such as `application/json`. [float] === Content-Type Requirements The type of the content sent in a request body must be specified using the `Content-Type` header. The value of this header must map to one of the supported formats that the API supports. Most APIs support JSON, YAML, CBOR, and SMILE. The bulk and multi-search APIs support NDJSON, JSON and SMILE; other types will result in an error response. Additionally, when using the `source` query string parameter the content type must be specified using the `source_content_type` query string parameter. [[url-access-control]] == URL-based access control Many users use a proxy with URL-based access control to secure access to Elasticsearch indices. For <>, <> and <> requests, the user has the choice of specifying an index in the URL and on each individual request within the request body. This can make URL-based access control challenging. To prevent the user from overriding the index which has been specified in the URL, add this setting to the `elasticsearch.yml` file: rest.action.multi.allow_explicit_index: false The default value is `true`, but when set to `false`, Elasticsearch will reject requests that have an explicit index specified in the request body.