* [ML] Adds progress reporting for transforms
* fixing after master merge
* Addressing PR comments
* removing unused imports
* Adjusting afterKey handling and percentage to be 100*
* Making sure it is a linked hashmap for serialization
* removing unused import
* addressing PR comments
* removing unused import
* simplifying code, only storing total docs and decrementing
* adjusting for rewrite
* removing initial progress gathering from executor
Today the `_field_caps` API returns the list of indices where a field
is present only if this field has different types within the requested indices.
However if the request is an index pattern (or an alias, or both...) there
is no way to infer the indices if the response contains only fields that have
the same type in all indices. This commit changes the response to always return
the list of indices in the response. It also adds a way to retrieve unmapped field
in a specific section per field called `unmapped`. This section is created for each field
that is present in some indices but not all if the parameter `include_unmapped` is set to
true in the request (defaults to false).
We link to these migraiton docs but we don't specify the id. This
isn't great practice in general and is preventing us from migrating to
Asciidoctor because it generates ids in a slightly different way.
Adds some validation to prevent duplicate source names from being
used in the composite agg.
Also refactored to use a ConstructingObjectParser and removed the
private ctor and setter for sources, making it mandatory.
This adds a gradle task called generateContextDoc in the Painless module. The
task will start a cluster, issue commands against the context rest api for
Painless, and generate documentation for each API per context. Each context
has a first page of classes sorted by package first and class name second,
along with a page per package with each classes' constructors, methods, and
fields. A link is generated for each constructor, method, and field to a JavaDoc
page when possible.
Drops some inline callouts that snuck into 7.x. We're doings this in
preparation for switching the elasticsearch reference to asciidoctor
which doesn't support them.
Implement a more trivial case of the CASE expression which is
expressed as a traditional function with 2 or 3 arguments. e.g.:
IIF(a = 1, 'one', 'many')
IIF(a > 0, 'positive')
Closes: #40917
(cherry picked from commit add02f4f553ad472026dcc1eaa84245a0558a4b0)
The example to delete a remote cluster is missing the `skip_unavailable` setting which results in an error:
```
"type": "illegal_argument_exception",
"reason": "missing required setting [cluster.remote.tiny-test.seeds] for setting [cluster.remote.tiny-test.skip_unavailable]"
```
Implement the ANSI SQL CASE expression which provides the if/else
functionality common to most programming languages.
The CASE expression can have multiple WHEN branches and becomes a
powerful tool for SQL queries as it can be used in SELECT, WHERE,
GROUP BY, HAVING and ORDER BY clauses.
Closes: #36200
(cherry picked from commit 8b2577406f47ae60d15803058921d128390af0b6)
Today's `docker-compose` docs are missing the `discovery.seed_nodes` config on
one of the nodes. With today's configuration the cluster can still form the
first time it is started, because `cluster.initial_master_nodes` requires both
nodes to bootstrap the cluster which ensures that each discover the other.
However if `es02` is elected master it will remove `es01` from the voting
configuration and then when restarted it will form a cluster on its own without
needing to do any discovery. Meanwhile `es01` doesn't know how to find `es02`
after a restart so will be unable to join this cluster.
This commit fixes this by adding the missing configuration.
This change clarifies the documentation around the recommended JVM. The
recommended JVM is the bundled JVM. If a user does not use our
recommended JVM we suggest that they use a supported LTS version of the
JVM.
Closes#41132
Fix a deprecation warning that wasn't rendering correctly in
asciidoctor. This one needed to be explicitly marked as an inline macro
because it is on its own line and it needed to have its text escaped
because it contained a `,`. It also was missing explanitory text for
what the setting was.
Currently enabling profiling disables top-hits optimizations, which is
unfortunate: it would be nice to be able to notice the difference in method
counts and timings depending on whether total hit counts are requested.
This PR makes a few clarifications to the docs for the `enabled` setting:
- Replace references to 'mapping type' with 'mapping' or 'mapping definition'.
- In code examples, clarify that the disabled fields have type `object`.
- Add a section on how disabled fields can hold non-object data.
This section should be at the same sub-level as other sections in the
auto date-histogram docs, otherwise it is rendered on to another page
and is confusing for users to understand what it's in reference to.
The following phrase causes confusion:
> Alternatively the IP addresses or hostnames (if node name defaults to the
> host name) can be used.
This change clarifies the conditions under which you can use a hostname, and
adds an anchor to the note introduced in (#41137) so we can link directly to it
in conversations with users.
Fixes rendering the `version_qualified` attribute in the docs. This
attribute includes `-SNAPSHOT` but does not include `-alpha` or `-beta`
and represents the `version` field as returned by `GET /` or
`GET /_cat/plugins`. Without this change we just drop the entire line on
the floor rather than render it so the output looks bad.
This helps avoid memory issues when computing deep sub-aggregations. Because it
should be rare to use sub-aggregations with significant terms, we opted to always
choose breadth first as opposed to exposing a `collect_mode` option.
Closes#28652.
Added documentation for node repurpose tool and included documentation on how to repurpose nodes safely. Adjusted order of tools in `elasticsearch-node` tool since the repurpose tool is most likely to be used.
Co-Authored-By: David Turner <david.turner@elastic.co>
The explanation given in the completion suggester documentation why we use the
"simple" analyzer as the default is no longer valid. Since we still use "simple"
as the default, we should just delete the explanation that doesn't fit anymore.
Closes#36715
* [DOCS] Fix broken link to Elasticsearch Docker source code
* [DOCS] Link to Dockerfile in elastic/elasticsearch repo
* [DOCS] Link to Docker source files in elastic/elasticsearch repo
* [DOCS] Added settings page for ILM.
* [DOCS] Adding ILM settings file
* [DOCS] Moved the ILM settings to a separate section
* [DOCS] Linked to the rollover docs.
* [DOCS] Tweaked the "required" wording.
This commit is a correction of a doc bug in the docs for the ingest
date-index-name processor. The correct pattern is
yyyy-MM-dd'T'HH:mm:ss.SSSXX. This is due to the transition from Joda
time to Java time where Z does not mean the same thing between the two.
Command needs `?v` so user can see the column headers. Otherwise the instructions in the note about checking the init and relo columns don't make sense
This commit fixes a problem with BWC that was brought up in #40511. A
newer version of the code was emitting a new value for an enum to an
older version, and the older version could not handle that. It caused
the response to error. The MainResponse is now relaxed, and will accept
whatever values the server expose, and holds most of them as Strings
instead of complex objects.
Fixes#40511
We generate two pages with "funny" names:
* _changing_the_client_8217_s_initialization_code.html
* _changing_the_application_8217_s_code.html
The leading `_` comes from us not specifying the name of the page. The
`8217` comes about because of the single quote character. This is a
funny name, but it is the name that we have so we shouldn't change it
without putting in a redirect.
We're looking at switching these docs from being built with the
no-longer-maintained AsciiDoc project to being built with the
actively-maintained Asciidoctor project. Asciidoctor Doesn't include the
`8217`s in the generated ids. That is *better*, but we don't really want
to change the pages. Ultimately we'd prefer none of our pages start with
`_`, but that is a problem for a different time.
Anyway, this pins the ids to their "funny" id so it won't change when we
switch to Asciidoctor. We'll remove it later, when we have more fine
control of our redirects.
Drops the inline callouts from the painless reference book. These
callouts are incompatible with Asciidoctor and we'd very much like to
switch to Asciidoctor for building this book, partially because
Asciidoctor is actively developed and AsciiDoc is not, and partially
because it builds the book three times faster.
This change rejects an illegal combination of flush parameters where
force is true, but wait_if_ongoing is false. This combination is trappy
and should be forbidden.
Closes#36342
- Added square brackets for the optional argument of precision
- Fixed character to lower case after comma
(cherry picked from commit d2f6f3b9ce36875e2eb6145c50464b4d72f2b1df)
After `TIME` SQL data type is introduced, implement
`CURRENT_TIME/CURTIME` functions similarly to CURRENT_TIMESTAMP
that return the system's current time (only, without the date part).
Closes: #40468
(cherry picked from commit 9feede781409d0e264ce45951a25b28ff129b187)
* Add release notes for 7.0.0-rc1.
* [DOCS] Fixes broken link to breaking changes
* [DOCS] Removed old ML PRs; edited titles
* Remove superseded PR.
* Clean up Lucene upgrade PRs/issues.
A full format for a DATETIME would be:
`2019-03-30T10:20:30.123+10:00` which is 29 chars long.
For DATE a full format would be: `2019-03-30T00:00:00.000+10:00`
which is also 29 chars long.
(cherry picked from commit 6be83964ed025528778bca8d35692762e166983b)
Moves the id of the preface in the java-api so it is compatible with
both AsciiDoc and Asciidoctor. As it stands now we apply the id that we
want for the preface to the book itself which is strange and only works
with AsciiDoc.
Support ANSI SQL's TIME type by introductin a runtime-only
ES SQL time type.
Closes: #38174
(cherry picked from commit 046ccd4cf0a251b2a3ddff6b072ab539a6711900)
This updates the casting table to reflect the recent changes for casting consistency in Painless. This also adds a small section on explicitly casting a character to a String which has always been allowed but undocumented.
There is a single example in the Java API docs that contains an inline
callout that is incompatible with Asciidoctor:
```
client.prepareUpdate("ttl", "doc", "1")
.setScript(new Script(
"ctx._source.gender = \"male\"" <1> , ScriptService.ScriptType.INLINE, null, null))
.get();
```
This rewrites the example to use an Asciidoctor compatible end of line
callout. It also looks nicer to me because it fits better on the page.
```
client.prepareUpdate("ttl", "doc", "1")
.setScript(new Script(
"ctx._source.gender = \"male\"", <1>
ScriptService.ScriptType.INLINE, null, null))
.get();
```
We no longer need to mention soft deletes in the getting started guide
now that retention leases exist and default to 12h. This commit removes
mention of soft deletes from the getting started guide, to simplify that
content.
Currently, the docs correctly state that using `now` in range queries will not
be affected by the `time_zone` parameter. However, using date math roundings
like e.g. `now\d` will be affected by the `time_zone`. Adding this example
because it seems to be a frequently asked question and source of confusion.
Relates to #40581
Specifying an inline script in an "inline" field
was deprecated in 5.x. The new field name is
"source".
(Since 6.x still accepts "inline" I will only backport
this docs change as far as 7.0.)
I have been hitting the suite timeout on `DocsClientYamlTestSuiteIT`
As far as I can see, the docs tests are taking quite a while, I assume
it's because more and more docs snippets get added over time, which
means more tests. The current suite timeout is the default 20 minutes.
It takes me just a little less than 20 minutes to run these on my
laptop. On my CI, I end up hitting the suite timeout. Hereby I propose
that we increase the suite timeout to 30 minutes.
This commit changes the note in docs about required java version to note
the existence of the bundled jdk and how to bring your own java. It also
reorganizes the zip/targz docs as zip is no longer suitable on
Linux/MacOS.
The basic models `b, de, p` and the after effect `no`
are not available anymore in Lucene 8 but they are still
listed in the >7x documentation. This change removes these
references that should also be listed in the breaking change
of es 7.0.
Closes#40264
Improve the documentation of parameter --pass of elasticsearch-certutil
Backport of: #40137
Co-Authored-By: Diego Cardozo Sandrim <diegocsandrim@users.noreply.github.com>
Co-Authored-By: Vigneash Sundar <vikene@users.noreply.github.com>
To make script_score query to have the same features
as function_score query, we need to add randomScore
function.
This function produces different
random scores on different index shards.
It is also able to produce random scores
based on the internal Lucene Document Ids.
In #33062 we introduced the `cluster.remote.*.proxy` setting for proxied
connections to remote clusters, but left it deliberately undocumented since it
needed followup work so that it could work with SNI. However, since #32517 is
now closed we can add this documentation and remove the comment about its lack
of documentation.
Adds the search_as_you_type field type that acts like a text field optimized
for as-you-type search completion. It creates a couple subfields that analyze
the indexed terms as shingles, against which full terms are queried, and a
prefix subfield that analyze terms as the largest shingle size used and
edge-ngrams, against which partial terms are queried
Adds a match_bool_prefix query type that creates a boolean clause of a term
query for each term except the last, for which a boolean clause with a prefix
query is created.
The match_bool_prefix query is the recommended way of querying a search as you
type field, which will boil down to term queries for each shingle of the input
text on the appropriate shingle field, and the final (possibly partial) term
as a term query on the prefix field. This field type also supports phrase and
phrase prefix queries however
Today the upgrade assistant identifies that discovery configuration is required
in production mode, but links to docs saying to add one of these settings:
- `discovery.seed_hosts`
- `discovery.seed_providers`
- `cluster.initial_master_nodes`
However these settings do not exist in 6.7 so this is unhelpful advice. This
commit adjusts the docs to give more useful advice to users arriving at this
page from the upgrade assistant.
Relates https://discuss.elastic.co/t/174102
* Document MATCH and QUERY function predicates.
* Polish the functions pages and add a list of functions to the main Functions & Operators page.
(cherry picked from commit 4cec0ae1b962ec7ea011a290aec72740386eb808)
* document ODBC's extra connection string parameters
Document the connection string parameters that are currently not
configurable from the GUI.
Add a note about SmartScreen possible warning on driver MSI
installation.
Add a note about the TLS certificate file not being supported as bundled
or password-protected.
* rephrasing and restructuring the list of params
- addressing PR review notes
* rephrasings and reference linking
- addressing PR review notes
(cherry picked from commit ca90190226c0a8c63f51c2603b27df299ce8d503)
The current documentation recommends disabling allocations of all shards. This prevents shards of
new indices from being allocated as well. That means that during a rolling upgrade, operations like
roll over and index shrinking will operate correctly. This, in turn, can cause issues for data ingestion
and ILM.
* [ML] Add data frame task state object and field
* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction
* Addressing PR comments
* adjusting after master merge
* addressing pr comment
* Adjusting auditor usage with failure state
* Refactor, renamed state items to task_state and indexer_state
* Adding todo and removing redundant auditor call
* Address HLRC changes and PR comment
* adjusting hlrc IT test
To avoid having to specify each spec by hand (which can miss specs to be
added), the test infrastructure now performs classpath discovery so that
each spec added, is automatically considered.
Relates #40358
(cherry picked from commit d0f60b4425c731509aa8ca765d55f563f866ef90)
* [ML] make source and dest objects in the transform config
* addressing PR comments
* Fixing compilation post merge
* adding comment for Arrays.hashCode
* addressing changes for moving dest to object
* fixing data_frame yml tests
* fixing API test
Extend CAST to support all data types notations (whether SQL or ES
specific)
Fix#40282
(cherry picked from commit eb2ee8a344da946920598839a5db76c8bb9bc3fe)
This commit expands the ccr overview page to include more information
about the lifecycle of following an index. It adds information linking
to the remote recovery documentation. And describes how an index can
fall-behind and how to fix it when this happens.
This is the equivalent of the `field_masking_span` query, allowing users to
merge intervals from multiple fields - for example, to search for stemmed tokens
near unstemmed tokens.
This change adds an option to convert a `date` field to nanoseconds resolution
and a `date_nanos` field to millisecond resolution when sorting.
The resolution of the sort can be set using the `numeric_type` option of the
field sort builder. The conversion is done at the shard level and is restricted
to dates from 1970 to 2262 for the nanoseconds resolution in order to avoid
numeric overflow.
The new cluster coordination subsystem introduced in 7.0 will only keep an
unresponsive node in the cluster for 30 seconds, whereas in earlier versions it
might have remained in the cluster for 90 seconds. This commit adds a note to
the migration documentation to that effect.
This commit clarifies how the gateway selection works when configuring
remote clusters for CCR or CCS. Specifically, it clarifies compatibility
between different versions which is a very common question.
The Migration Assistance API has been functionally replaced by the
Deprecation Info API, and the Migration Upgrade API is not used for the
transition from ES 6.x to 7.x, and does not need to be kept around to
repair indices that were not properly upgraded before upgrading the
cluster, as was the case in 6.
For cases where fields can have multi values, allow the behavior to be
customized through a dedicated configuration field.
By default this will be enabled on the drivers so that existing datasets
work instead of throwing an exception.
For regular SQL usage, the behavior is false so that the user is aware
of the underlying data.
Fix#39700
(cherry picked from commit 2b351571961f172fd59290ee079126bbd081ceaf)
The `GET /_cluster/state` API returns an internal representation of the cluster
state that does change from version to version. It's useful for debugging, but
it is not intended for regular use by clients.
This change adjusts the documentation of `GET /_cluster/state` to clarify that
this API yields an internal representation that should not be expected to
remain stable between versions.
Relates #40061, #40016
This change adds an option to the `FieldSortBuilder` that allows to transform the type
of a numeric field into another. Possible values for this option are `long` that transforms
the source field into an integer and `double` that transforms the source field into a floating point.
This new option is useful for cross-index search when the sort field is mapped differently on some
indices. For instance if a field is mapped as a floating point in one index and as an integer in another
it is possible to align the type for both indices using the `numeric_type` option:
```
{
"sort": {
"field": "my_field",
"numeric_type": "double" <1>
}
}
```
<1> Ensure that values for this field are transformed to a floating point if needed.
Currently if a field alias is updated, any percolator queries that contain the
alias will still refer to its old target. This PR documents the issue while we
look into addressing it.
Relates to #37212.
`document_type` is the type to use for parsing the document to percolate, which
is already optional and deprecated. However `percotale` queries also have the
ability to percolate existing documents, identified by an index, an id and a
type. This change makes the latter optional and deprecated.
Closes#39963
This commit, mostly authored by @DaveCTurner,
adds documentation for elasticsearch-node tool #37696.
(cherry picked from commit 09425d5a5158c2d3fdad794411b3bbc4bba47b15)
Lucene 8.0 includes a [change](https://issues.apache.org/jira/browse/LUCENE-8635)
that moves the terms index off-heap for all fields but ID fields. I'm
including this in the migration notes so that users who have queries that match
lots of terms won't be surprised in case of slowdown.
- new `rank_feature`/`script_score` queries
- new `index_phrases`/`index_prefixes` options
- disabling `_field_names` doesn't help anymore
- adaptive replica selection is on by default
We were missing release notes for 7.0.0-alpha1. I generated them by running
the release-notes script with a quick hack that filtered out pull requests
that had been closed on or after 2018-11-15.
Some breaking changes had been documented in the release notes rather than
the migration guide so I moved them.
Jash Gala
Benjamin Trent
Jim Ferenczi
Ignacio Vera
Alexander Reelsen
Nik Everett
Zachary Tong
James Rodewig
Jack Conradson
Guilherme Ferreira
David Turner
Marios Trivyzas
Hendrik Muhs
Melori Arellano
Ioannis Kakavas
Joe Zack
Jason Tedor
clement-tourriere
Arlind
Jay Modi
Bob Blank
Adrien Grand
Lisa Cawley
Julie Tibshirani
David Kyle
Antonio Matarrese
Henning Andersen
Christoph Büscher
Shaunak Kashyap
debadair
DeDe Morton
Michael Basnight
Taylor Scott
Nhat Nguyen
Alan Woodward
Jake Landis
michaelbaamonde
Ryan Ernst
Jim Ferenczi
Omkar Deshpande
Andy Bristol
Andrei Stefan
Mayya Sharipova
David Roberts
Luca Cavanna
aladdam
avshiav
Tim Vernum
Costin Leau
Bogdan Pintea
Boaz Leskes
Costin Leau
alex101101
Tim Brooks
RomainGeffraye
Yannick Welsch
Florian Kelbert
Alex Doerr
Gordon Brown
Daniel Mitterdorfer
Martijn van Groningen
Tal Levy
Andrey Ershov
Przemyslaw Gomulka
MK Swanson