[role="xpack"] [testenv="basic"] [[sql-rest]] == SQL REST API * <> * <> * <> * <> * <> * <> * <> [[sql-rest-overview]] === Overview The SQL REST API accepts SQL in a JSON document, executes it, and returns the results. For example: [source,console] -------------------------------------------------- POST /_sql?format=txt { "query": "SELECT * FROM library ORDER BY page_count DESC LIMIT 5" } -------------------------------------------------- // TEST[setup:library] Which returns: [source,text] -------------------------------------------------- author | name | page_count | release_date -----------------+--------------------+---------------+------------------------ Peter F. Hamilton|Pandora's Star |768 |2004-03-02T00:00:00.000Z Vernor Vinge |A Fire Upon the Deep|613 |1992-06-01T00:00:00.000Z Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z Alastair Reynolds|Revelation Space |585 |2000-03-15T00:00:00.000Z James S.A. Corey |Leviathan Wakes |561 |2011-06-02T00:00:00.000Z -------------------------------------------------- // TESTRESPONSE[s/\|/\\|/ s/\+/\\+/] // TESTRESPONSE[non_json] [[sql-kibana-console]] [TIP] .Using Kibana Console ==== If you are using {kibana-ref}/console-kibana.html[Kibana Console] (which is highly recommended), take advantage of the triple quotes `"""` when creating the query. This not only automatically escapes double quotes (`"`) inside the query string but also support multi-line as shown below: image:images/sql/rest/console-triple-quotes.png[] ==== [[sql-rest-format]] === Response Data Formats While the textual format is nice for humans, computers prefer something more structured. {es-sql} can return the data in the following formats which can be set either through the `format` property in the URL or by setting the `Accept` HTTP header: NOTE: The URL parameter takes precedence over the `Accept` HTTP header. If neither is specified then the response is returned in the same format as the request. [cols="^m,^4m,^8"] |=== s|format s|`Accept` HTTP header s|Description 3+h| Human Readable |csv |text/csv |https://en.wikipedia.org/wiki/Comma-separated_values[Comma-separated values] |json |application/json |https://www.json.org/[JSON] (JavaScript Object Notation) human-readable format |tsv |text/tab-separated-values |https://en.wikipedia.org/wiki/Tab-separated_values[Tab-separated values] |txt |text/plain |CLI-like representation |yaml |application/yaml |https://en.wikipedia.org/wiki/YAML[YAML] (YAML Ain't Markup Language) human-readable format 3+h| Binary Formats |cbor |application/cbor |http://cbor.io/[Concise Binary Object Representation] |smile |application/smile |https://en.wikipedia.org/wiki/Smile_(data_interchange_format)[Smile] binary data format similar to CBOR |=== Here are some examples for the human readable formats: ==== CSV [source,console] -------------------------------------------------- POST /_sql?format=csv { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,text] -------------------------------------------------- author,name,page_count,release_date Peter F. Hamilton,Pandora's Star,768,2004-03-02T00:00:00.000Z Vernor Vinge,A Fire Upon the Deep,613,1992-06-01T00:00:00.000Z Frank Herbert,Dune,604,1965-06-01T00:00:00.000Z Alastair Reynolds,Revelation Space,585,2000-03-15T00:00:00.000Z James S.A. Corey,Leviathan Wakes,561,2011-06-02T00:00:00.000Z -------------------------------------------------- // TESTRESPONSE[non_json] ==== JSON [source,console] -------------------------------------------------- POST /_sql?format=json { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,console-result] -------------------------------------------------- { "columns": [ {"name": "author", "type": "text"}, {"name": "name", "type": "text"}, {"name": "page_count", "type": "short"}, {"name": "release_date", "type": "datetime"} ], "rows": [ ["Peter F. Hamilton", "Pandora's Star", 768, "2004-03-02T00:00:00.000Z"], ["Vernor Vinge", "A Fire Upon the Deep", 613, "1992-06-01T00:00:00.000Z"], ["Frank Herbert", "Dune", 604, "1965-06-01T00:00:00.000Z"], ["Alastair Reynolds", "Revelation Space", 585, "2000-03-15T00:00:00.000Z"], ["James S.A. Corey", "Leviathan Wakes", 561, "2011-06-02T00:00:00.000Z"] ], "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl+v///w8=" } -------------------------------------------------- // TESTRESPONSE[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl\+v\/\/\/w8=/$body.cursor/] ==== TSV [source,console] -------------------------------------------------- POST /_sql?format=tsv { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,text] -------------------------------------------------- author name page_count release_date Peter F. Hamilton Pandora's Star 768 2004-03-02T00:00:00.000Z Vernor Vinge A Fire Upon the Deep 613 1992-06-01T00:00:00.000Z Frank Herbert Dune 604 1965-06-01T00:00:00.000Z Alastair Reynolds Revelation Space 585 2000-03-15T00:00:00.000Z James S.A. Corey Leviathan Wakes 561 2011-06-02T00:00:00.000Z -------------------------------------------------- // TESTRESPONSE[s/\t/ /] // TESTRESPONSE[non_json] ==== TXT [source,console] -------------------------------------------------- POST /_sql?format=txt { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,text] -------------------------------------------------- author | name | page_count | release_date -----------------+--------------------+---------------+------------------------ Peter F. Hamilton|Pandora's Star |768 |2004-03-02T00:00:00.000Z Vernor Vinge |A Fire Upon the Deep|613 |1992-06-01T00:00:00.000Z Frank Herbert |Dune |604 |1965-06-01T00:00:00.000Z Alastair Reynolds|Revelation Space |585 |2000-03-15T00:00:00.000Z James S.A. Corey |Leviathan Wakes |561 |2011-06-02T00:00:00.000Z -------------------------------------------------- // TESTRESPONSE[s/\|/\\|/ s/\+/\\+/] // TESTRESPONSE[non_json] ==== YAML [source,console] -------------------------------------------------- POST /_sql?format=yaml { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,yaml] -------------------------------------------------- columns: - name: "author" type: "text" - name: "name" type: "text" - name: "page_count" type: "short" - name: "release_date" type: "datetime" rows: - - "Peter F. Hamilton" - "Pandora's Star" - 768 - "2004-03-02T00:00:00.000Z" - - "Vernor Vinge" - "A Fire Upon the Deep" - 613 - "1992-06-01T00:00:00.000Z" - - "Frank Herbert" - "Dune" - 604 - "1965-06-01T00:00:00.000Z" - - "Alastair Reynolds" - "Revelation Space" - 585 - "2000-03-15T00:00:00.000Z" - - "James S.A. Corey" - "Leviathan Wakes" - 561 - "2011-06-02T00:00:00.000Z" cursor: "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl+v///w8=" -------------------------------------------------- // TESTRESPONSE[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl\+v\/\/\/w8=/$body.cursor/] [[sql-pagination]] === Paginating through a large response Using the example from the <>, one can continue to the next page by sending back the cursor field. In case of text format, the cursor is returned as `Cursor` http header. [source,console] -------------------------------------------------- POST /_sql?format=json { "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWYUpOYklQMHhRUEtld3RsNnFtYU1hQQ==:BAFmBGRhdGUBZgVsaWtlcwFzB21lc3NhZ2UBZgR1c2Vy9f///w8=" } -------------------------------------------------- // TEST[continued] // TEST[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWYUpOYklQMHhRUEtld3RsNnFtYU1hQQ==:BAFmBGRhdGUBZgVsaWtlcwFzB21lc3NhZ2UBZgR1c2Vy9f\/\/\/w8=/$body.cursor/] Which looks like: [source,console-result] -------------------------------------------------- { "rows" : [ ["Dan Simmons", "Hyperion", 482, "1989-05-26T00:00:00.000Z"], ["Iain M. Banks", "Consider Phlebas", 471, "1987-04-23T00:00:00.000Z"], ["Neal Stephenson", "Snow Crash", 470, "1992-06-01T00:00:00.000Z"], ["Frank Herbert", "God Emperor of Dune", 454, "1981-05-28T00:00:00.000Z"], ["Frank Herbert", "Children of Dune", 408, "1976-04-21T00:00:00.000Z"] ], "cursor" : "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWODRMaXBUaVlRN21iTlRyWHZWYUdrdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl9f///w8=" } -------------------------------------------------- // TESTRESPONSE[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWODRMaXBUaVlRN21iTlRyWHZWYUdrdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl9f\/\/\/w8=/$body.cursor/] Note that the `columns` object is only part of the first page. You've reached the last page when there is no `cursor` returned in the results. Like Elasticsearch's <>, SQL may keep state in Elasticsearch to support the cursor. Unlike scroll, receiving the last page is enough to guarantee that the Elasticsearch state is cleared. To clear the state earlier, you can use the clear cursor command: [source,console] -------------------------------------------------- POST /_sql/close { "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWYUpOYklQMHhRUEtld3RsNnFtYU1hQQ==:BAFmBGRhdGUBZgVsaWtlcwFzB21lc3NhZ2UBZgR1c2Vy9f///w8=" } -------------------------------------------------- // TEST[continued] // TEST[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWYUpOYklQMHhRUEtld3RsNnFtYU1hQQ==:BAFmBGRhdGUBZgVsaWtlcwFzB21lc3NhZ2UBZgR1c2Vy9f\/\/\/w8=/$body.cursor/] Which will like return the [source,console-result] -------------------------------------------------- { "succeeded" : true } -------------------------------------------------- [[sql-rest-filtering]] === Filtering using {es} query DSL One can filter the results that SQL will run on using a standard {es} query DSL by specifying the query in the filter parameter. [source,console] -------------------------------------------------- POST /_sql?format=txt { "query": "SELECT * FROM library ORDER BY page_count DESC", "filter": { "range": { "page_count": { "gte" : 100, "lte" : 200 } } }, "fetch_size": 5 } -------------------------------------------------- // TEST[setup:library] Which returns: [source,text] -------------------------------------------------- author | name | page_count | release_date ---------------+------------------------------------+---------------+------------------------ Douglas Adams |The Hitchhiker's Guide to the Galaxy|180 |1979-10-12T00:00:00.000Z -------------------------------------------------- // TESTRESPONSE[s/\|/\\|/ s/\+/\\+/] // TESTRESPONSE[non_json] [[sql-rest-columnar]] === Columnar results The most well known way of displaying the results of an SQL query result in general is the one where each individual record/document represents one line/row. For certain formats, {es-sql} can return the results in a columnar fashion: one row represents all the values of a certain column from the current page of results. The following formats can be returned in columnar orientation: `json`, `yaml`, `cbor` and `smile`. [source,console] -------------------------------------------------- POST /_sql?format=json { "query": "SELECT * FROM library ORDER BY page_count DESC", "fetch_size": 5, "columnar": true } -------------------------------------------------- // TEST[setup:library] Which returns: [source,console-result] -------------------------------------------------- { "columns": [ {"name": "author", "type": "text"}, {"name": "name", "type": "text"}, {"name": "page_count", "type": "short"}, {"name": "release_date", "type": "datetime"} ], "values": [ ["Peter F. Hamilton", "Vernor Vinge", "Frank Herbert", "Alastair Reynolds", "James S.A. Corey"], ["Pandora's Star", "A Fire Upon the Deep", "Dune", "Revelation Space", "Leviathan Wakes"], [768, 613, 604, 585, 561], ["2004-03-02T00:00:00.000Z", "1992-06-01T00:00:00.000Z", "1965-06-01T00:00:00.000Z", "2000-03-15T00:00:00.000Z", "2011-06-02T00:00:00.000Z"] ], "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl+v///w8=" } -------------------------------------------------- // TESTRESPONSE[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl\+v\/\/\/w8=/$body.cursor/] Any subsequent calls using a `cursor` still have to contain the `columnar` parameter to preserve the orientation, meaning the initial query will not _remember_ the columnar option. [source,console] -------------------------------------------------- POST /_sql?format=json { "cursor": "sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl+v///w8=", "columnar": true } -------------------------------------------------- // TEST[continued] // TEST[s/sDXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAEWWWdrRlVfSS1TbDYtcW9lc1FJNmlYdw==:BAFmBmF1dGhvcgFmBG5hbWUBZgpwYWdlX2NvdW50AWYMcmVsZWFzZV9kYXRl\+v\/\/\/w8=/$body.cursor/] Which looks like: [source,console-result] -------------------------------------------------- { "values": [ ["Dan Simmons", "Iain M. Banks", "Neal Stephenson", "Frank Herbert", "Frank Herbert"], ["Hyperion", "Consider Phlebas", "Snow Crash", "God Emperor of Dune", "Children of Dune"], [482, 471, 470, 454, 408], ["1989-05-26T00:00:00.000Z", "1987-04-23T00:00:00.000Z", "1992-06-01T00:00:00.000Z", "1981-05-28T00:00:00.000Z", "1976-04-21T00:00:00.000Z"] ], "cursor": "46ToAwFzQERYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQUVXWjBaNlFXbzNOV0pVY21Wa1NUZDJhV2t3V2xwblp3PT3/////DwQBZgZhdXRob3IBBHRleHQAAAFmBG5hbWUBBHRleHQAAAFmCnBhZ2VfY291bnQBBGxvbmcBAAFmDHJlbGVhc2VfZGF0ZQEIZGF0ZXRpbWUBAAEP" } -------------------------------------------------- // TESTRESPONSE[s/46ToAwFzQERYRjFaWEo1UVc1a1JtVjBZMmdCQUFBQUFBQUFBQUVXWjBaNlFXbzNOV0pVY21Wa1NUZDJhV2t3V2xwblp3PT3\/\/\/\/\/DwQBZgZhdXRob3IBBHRleHQAAAFmBG5hbWUBBHRleHQAAAFmCnBhZ2VfY291bnQBBGxvbmcBAAFmDHJlbGVhc2VfZGF0ZQEIZGF0ZXRpbWUBAAEP/$body.cursor/] [[sql-rest-params]] === Passing parameters to a query Using values in a query condition, for example, or in a `HAVING` statement can be done "inline", by integrating the value in the query string itself: [source,console] -------------------------------------------------- POST /_sql?format=txt { "query": "SELECT YEAR(release_date) AS year FROM library WHERE page_count > 300 AND author = 'Frank Herbert' GROUP BY year HAVING COUNT(*) > 0" } -------------------------------------------------- // TEST[setup:library] or it can be done by extracting the values in a separate list of parameters and using question mark placeholders (`?`) in the query string: [source,console] -------------------------------------------------- POST /_sql?format=txt { "query": "SELECT YEAR(release_date) AS year FROM library WHERE page_count > ? AND author = ? GROUP BY year HAVING COUNT(*) > ?", "params": [300, "Frank Herbert", 0] } -------------------------------------------------- // TEST[setup:library] [IMPORTANT] The recommended way of passing values to a query is with question mark placeholders, to avoid any attempts of hacking or SQL injection. [[sql-rest-fields]] === Supported REST parameters In addition to the `query` and `fetch_size`, a request a number of user-defined fields for specifying the request time-outs or localization information (such as timezone). The table below lists the supported parameters: [cols="^m,^m,^5"] |=== s|name s|Default value s|Description |query |Mandatory |SQL query to execute |fetch_size |1000 |The maximum number of rows (or entries) to return in one response |filter |none |Optional {es} query DSL for additional <>. |request_timeout |90s |The timeout before the request fails. |page_timeout |45s |The timeout before a pagination request fails. |time_zone |`Z` (or `UTC`) |Time-zone in ISO 8601 used for executing the query on the server. More information available https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html[here]. |columnar |false |Return the results in a columnar fashion, rather than row-based fashion. Valid for `json`, `yaml`, `cbor` and `smile`. |field_multi_value_leniency |false |Throw an exception when encountering multiple values for a field (default) or be lenient and return the first value from the list (without any guarantees of what that will be - typically the first in natural ascending order). |index_include_frozen |false |Whether to include <> in the query execution or not (default). |params |none |Optional list of parameters to replace question mark (`?`) placeholders inside the query. |=== Do note that most parameters (outside the timeout and `columnar` ones) make sense only during the initial query - any follow-up pagination request only requires the `cursor` parameter as explained in the <> chapter. That's because the query has already been executed and the calls are simply about returning the found results - thus the parameters are simply ignored.