Make SnapshotInfo and CreateSnapshotResponse parsers lenient for backwards compatibility. Remove extraneous fields from CreateSnapshotRequest toXContent.
* Adds a new auto-interval date histogram
This change adds a new type of histogram aggregation called `auto_date_histogram` where you can specify the target number of buckets you require and it will find an appropriate interval for the returned buckets. The aggregation works by first collecting documents in buckets at second interval, when it has created more than the target number of buckets it merges these buckets into minute interval bucket and continues collecting until it reaches the target number of buckets again. It will keep merging buckets when it exceeds the target until either collection is finished or the highest interval (currently years) is reached. A similar process happens at reduce time.
This aggregation intentionally does not support min_doc_count, offest and extended_bounds to keep the already complex logic from becoming more complex. The aggregation accepts sub-aggregations but will always operate in `breadth_first` mode deferring the computation of sub-aggregations until the final buckets from the shard are known. min_doc_count is effectively hard-coded to zero meaning that we will insert empty buckets where necessary.
Closes#9572
* Adds documentation
* Added sub aggregator test
* Fixes failing docs test
* Brings branch up to date with master changes
* trying to get tests to pass again
* Fixes multiBucketConsumer accounting
* Collects more buckets than needed on shards
This gives us more options at reduce time in terms of how we do the
final merge of the buckeets to produce the final result
* Revert "Collects more buckets than needed on shards"
This reverts commit 993c782d117892af9a3c86a51921cdee630a3ac5.
* Adds ability to merge within a rounding
* Fixes nonn-timezone doc test failure
* Fix time zone tests
* iterates on tests
* Adds test case and documentation changes
Added some notes in the documentation about the intervals that can bbe
returned.
Also added a test case that utilises the merging of conseecutive buckets
* Fixes performance bug
The bug meant that getAppropriate rounding look a huge amount of time
if the range of the data was large but also sparsely populated. In
these situations the rounding would be very low so iterating through
the rounding values from the min key to the max keey look a long time
(~120 seconds in one test).
The solution is to add a rough estimate first which chooses the
rounding based just on the long values of the min and max keeys alone
but selects the rounding one lower than the one it thinks is
appropriate so the accurate method can choose the final rounding taking
into account the fact that intervals are not always fixed length.
Thee commit also adds more tests
* Changes to only do complex reduction on final reduce
* merge latest with master
* correct tests and add a new test case for 10k buckets
* refactor to perform bucket number check in innerBuild
* correctly derive bucket setting, update tests to increase bucket threshold
* fix checkstyle
* address code review comments
* add documentation for default buckets
* fix typo
This commit adds the _xpack/usage api to the high level rest client.
Currently in the transport api, the usage data is exposed in a limited
fashion, at most giving one level of helper methods for the inner keys
of data, but then exposing thos subobjects as maps of objects. Rather
than making parsers for every set of usage data from each feature, this
PR exposes the entire set of usage data as a map of maps.
Because this is a static method on a public API, and one that we encourage
plugin authors to use, the method with the typo is deprecated in 6.x
rather than just renamed.
With this commit we introduce a new circuit-breaking strategy to the parent
circuit breaker. Contrary to the current implementation which only accounts for
memory reserved via child circuit breakers, the new strategy measures real heap
memory usage at the time of reservation. This allows us to be much more
aggressive with the circuit breaker limit so we bump it to 95% by default. The
new strategy is turned on by default and can be controlled with the new cluster
setting `indices.breaker.total.userealmemory`.
Note that we turn it off for all integration tests with an internal test cluster
because it leads to spurious test failures which are of no value (we cannot
fully control heap memory usage in tests). All REST tests, however, will make
use of the real memory circuit breaker.
Relates #31767
* master:
[TEST] Mute SlackMessageTests.testTemplateRender
Docs: Explain closing the high level client
[ML] Re-enable memory limit integration tests (#31328)
[test] disable packaging tests for suse boxes
Add nio transport to security plugin (#31942)
XContentTests : Insert random fields at random positions (#30867)
Force execution of fetch tasks (#31974)
Fix unreachable error condition in AmazonS3Fixture (#32005)
Tests: Fix SearchFieldsIT.testDocValueFields (#31995)
Add Expected Reciprocal Rank metric (#31891)
[ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973)
SQL: Add support for single parameter text manipulating functions (#31874)
[ML] Ensure immutability of MlMetadata (#31957)
Tests: Mute SearchFieldsIT.testDocValueFields()
muted tests due to #31940
Work around reported problem in eclipse (#31960)
Move build integration tests out of :buildSrc project (#31961)
Tests: Remove use of joda time in some tests (#31922)
[Test] Reactive 3rd party tests on CI (#31919)
SQL: Support for escape sequences (#31884)
SQL: HAVING clause should accept only aggregates (#31872)
Docs: fix typo in datehistogram (#31972)
Switch url repository rest tests to new style requests (#31944)
Switch reindex tests to new style requests (#31941)
Docs: Added note about cloud service to installation and getting started
[DOCS] Removes alternative docker pull example (#31934)
Add Snapshots Status API to High Level Rest Client (#31515)
ingest: date_index_name processor template resolution (#31841)
Test: fix null failure in watcher test (#31968)
Switch test framework to new style requests (#31939)
Switch low level rest tests to new style Requests (#31938)
Switch high level rest tests to new style requests (#31937)
[ML] Mute test failing due to Java 11 date time format parsing bug (#31899)
[TEST] Mute SlackMessageTests.testTemplateRender
Fix assertIngestDocument wrongfully passing (#31913)
Remove unused reference to filePermissionsCache (#31923)
rolling upgrade should use a replica to prevent relocations while running a scroll
HLREST: Bundle the x-pack protocol project (#31904)
Increase logging level for testStressMaybeFlush
Added lenient flag for synonym token filter (#31484)
[X-Pack] Beats centralized management: security role + licensing (#30520)
HLRest: Move xPackInfo() to xPack().info() (#31905)
Docs: add security delete role to api call table (#31907)
[test] port archive distribution packaging tests (#31314)
Watcher: Slack message empty text (#31596)
[ML] Mute failing DetectionRulesIT.testCondition() test
Fix broken NaN check in MovingFunctions#stdDev() (#31888)
Date: Add DateFormatters class that uses java.time (#31856)
[ML] Switch native QA tests to a 3 node cluster (#31757)
Change trappy float comparison (#31889)
Fix building AD URL from domain name (#31849)
Add opaque_id to audit logging (#31878)
re-enable backcompat tests
add support for is_write_index in put-alias body parsing (#31674)
Improve release notes script (#31833)
[DOCS] Fix broken link in painless example
Handle missing values in painless (#30975)
Remove the ability to index or query context suggestions without context (#31007)
Ingest: Enable Templated Fieldnames in Rename (#31690)
[Docs] Fix typo in the Rollup API Quick Reference (#31855)
Ingest: Add ignore_missing option to RemoveProc (#31693)
Add template config for Beat state to X-Pack Monitoring (#31809)
Watcher: Add ssl.trust email account setting (#31684)
Remove link to oss-MSI (#31844)
Painless: Restructure Definition/Whitelist (#31879)
HLREST: Add x-pack-info API (#31870)
It looks like we weren't clear on when and why you should close the high
level client and folks were closing it after every request which is not
efficient. This explains why you should close the client and when so
this shouldn't be as common.
Closes#32001
* Added lenient flag for synonym-tokenfilter.
Relates to #30968
* added docs for synonym-graph-tokenfilter
-- Also made lenient final
-- changed from !lenient to lenient == false
* Changes after review (1)
-- Renamed to ElasticsearchSynonymParser
-- Added explanation for ElasticsearchSynonymParser::add method
-- Changed ElasticsearchSynonymParser::logger instance to static
* Added lenient option for WordnetSynonymParser
-- also added more documentation
* Added additional documentation
* Improved documentation
* Handle missing values in painless
Throw an exception for `doc['field'].value`
if this document is missing a value for the `field`.
For 7.0:
This is the default behaviour from 7.0
For 6.x:
To enable this behavior from 6.x, a user can set a jvm.option:
`-Des.script.exception_for_missing_value=true` on a node.
If a user does not enable this behavior, a deprecation warning is logged on start up.
Closes#29286
This commit removes the link to an oss-MSI; there is only one version of the MSI, which includes X-Pack.
(cherry picked from commit d2e5db8a806ec8a25162f79db5209aceed4f30f7)
This is the first x-pack API we're adding to the high level REST client
so there is a lot to talk about here!
= Open source
The *client* for these APIs is open source. We're taking the previously
Elastic licensed files used for the `Request` and `Response` objects and
relicensing them under the Apache 2 license.
The implementation of these features is staying under the Elastic
license. This lines up with how the rest of the Elasticsearch language
clients work.
= Location of the new files
We're moving all of the `Request` and `Response` objects that we're
relicensing to the `x-pack/protocol` directory. We're adding a copy of
the Apache 2 license to the root fo the `x-pack/protocol` directory to
line up with the language in the root `LICENSE.txt` file. All files in
this directory will have the Apache 2 license header as well. We don't
want there to be any confusion. Even though the files are under the
`x-pack` directory, they are Apache 2 licensed.
We chose this particular directory layout because it keeps the X-Pack
stuff together and easier to think about.
= Location of the API in the REST client
We've been following the layout of the rest-api-spec files for other
APIs and we plan to do this for the X-Pack APIs with one exception:
we're dropping the `xpack` from the name of most of the APIs. So
`xpack.graph.explore` will become `graph().explore()` and
`xpack.license.get` will become `license().get()`.
`xpack.info` and `xpack.usage` are special here though because they
don't belong to any proper category. For now I'm just calling
`xpack.info` `xPackInfo()` and intend to call usage `xPackUsage` though
I'm not convinced that this is the final name for them. But it does get
us started.
= Jars, jars everywhere!
This change makes the `xpack:protocol` project a `compile` scoped
dependency of the `x-pack:plugin:core` and `client:rest-high-level`
projects. I intend to keep it a compile scoped dependency of
`x-pack:plugin:core` but I intend to bundle the contents of the protocol
jar into the `client:rest-high-level` jar in a follow up. This change
has grown large enough at this point.
In that followup I'll address javadoc issues as well.
= Breaking-Java
This breaks that transport client by a few classes around. We've
traditionally been ok with doing this to the transport client.
* master:
[ML] Fix master node deadlock during ML daily maintenance (#31836)
Build: Switch integ-test-zip to OSS-only (#31866)
SQL: Remove restriction for single column grouping (#31818)
Build: Fix detection of Eclipse Compiler Server (#31838)
Docs: Inconsistency between description and example (#31858)
Re-enable bwc tests now that #29538 has been backported and 6.x intake build succeeded.
QA: build improvements related to SQL projects (#31862)
[Docs] Add clarification to analysis example (#31826)
Check timeZone() argument in AbstractSqlQueryRequest (#31822)
SQL: Fix incorrect HAVING equality (#31820)
Smaller aesthetic fixes to InternalTestCluster (#31831)
[Docs] Clarify accepted sort case (#31605)
Temporarily disable bwc test in order to backport #29538
Remove obsolete parameters from analyze rest spec (#31795)
[Docs] Fix wrong link in Korean analyzer docs (#31815)
Fix profiling of ordered terms aggs (#31814)
Properly mute test involving JDK11 closes#31739
Do not return all indices if a specific alias is requested via get aliases api. (#29538)
Get snapshot rest client cleanups (#31740)
Docs: Explain _bulk?refresh shard targeting
Fix handling of points_only with term strategy in geo_shape (#31766)
For historical reasons SQL restricts GROUP BY to only one field.
This commit removes the restriction and improves the test suite with
multi group by tests.
Close#31793
There have been at least two PRs trying to fix the spelling of "lazi" because it
isn't very clear from the example that the english analyzer will stem each token
in the example. This adds a short description of the analysis process to make
this clearer.
Relates to #31797
This is a followup to #31537. It makes a number of changes requested by
a review that came after the PR was merged. These are mostly cleanups
and doc improvements.
Only the shards that receive the bulk request will be affected by
`refresh`. Imagine a `_bulk?refresh=wait_for` request with three
documents in it that happen to be routed to different shards in an index
with five shards. The request will only wait for those three shards to
refresh. The other two shards of that make up the index do not
participate in the `_bulk` request at all.
Relates to #31819
* master:
REST high-level client: add get index API (#31703)
SQL: Allow long literals (#31777)
SQL: Fix incorrect message for aliases (#31792)
Test: Do not remove xpack templates when cleaning (#31642)
Reduce more raw types warnings (#31780)
Add unreleased version 6.3.2
Scripting: Remove support for deprecated StoredScript contexts (#31394)
[ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
[ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800)
[ML] Fix calendar and filter updates from non-master nodes (#31804)
Fix license header generation on Windows (#31790)
mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix
mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix
Correct exclusion of test on JDK 11
Fix doclint jdk 11
Add JDK11 support and enable in CI (#31644)
Watcher: Fix check for currently executed watches (#31137)
Watcher: Ensure correct method is used to read secure settings (#31753)
SQL: Update CLI logo
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
"query": {
"match": {
"title": "{{query_string}}"
}
}
}
```
and must instead do:
```
POST _scripts/<templatename>
{
"script": {
"lang": "mustache",
"source": {
"query": {
"match": {
"title": "{{query_string}}"
}
}
}
}
}
```
This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
* Upgrade bouncycastle
Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11
* Downgrade bouncycastle to avoid invalid manifest
* Add checksum for new jars
* Update tika permissions for jdk 11
* Mute test failing on jdk 11
* Add JDK11 to CI
* Thread#stop(Throwable) was removed
http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html
* Disable failing tests #31456
* Temprorarily disable doc tests
To see if there are other failures on JDK11
* Only blacklist specific doc tests
* Disable only failing tests in ingest attachment plugin
* Mute failing HDFS tests #31498
* Mute failing lang-painless tests #31500
* Fix backwards compatability builds
Fix JAVA version to 10 for ES 6.3
* Add 6.x to bwx -> java10
* Prefix out and err from buildBwcVersion for readability
```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
[bwc] :buildSrc:compileJava
[bwc] WARNING: An illegal reflective access operation has occurred
[bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
[bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
[bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[bwc] WARNING: All illegal access operations will be denied in a future release
[bwc] :buildSrc:compileGroovy
[bwc] :buildSrc:writeVersionProperties
[bwc] :buildSrc:processResources
[bwc] :buildSrc:classes
[bwc] :buildSrc:jar
```
* Also set RUNTIME_JAVA_HOME for bwcBuild
So that we can make sure it's not too new for the build to understand.
* Align bouncycastle dependency
* fix painles array tets
closes#31500
* Update jar checksums
* Keep 8/10 runtime/compile untill consensus builds on 11
* Only skip failing tests if running on Java 11
* Failures are dependent of compile java version not runtime
* Condition doc test exceptions on compiler java version as well
* Disable hdfs tests based on runtime java
* Set runtime java to minimum supported for bwc
* PR review
* Add comment with ticket for forbidden apis
* master:
[ML] Rate limit established model memory updates (#31768)
[Docs] Correct default window_size (#31582)
S3 fixture should report 404 on unknown bucket (#31782)
Detach Transport from TransportService (#31727)
[ML] Limit ML filter items to 10K (#31731)
[ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
Fixture for Minio testing (#31688)
[DOCS] Add missing get mappings docs to HLRC (#31765)
[DOCS] Starting Elasticsearch (#31701)
Painless: Complete Removal of Painless Type (#31699)
Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
Consolidate watcher setting update registration (#31762)
Build: re-enabled bwc (#31769)
ingest: Introduction of a bytes processor (#31733)
Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
Add analyze API to high-level rest client (#31577)
[DOCS] Typos
DOC: Add examples to the SQL docs (#31633)
Add support for AWS session tokens (#30414)
Watcher: Reenable start/stop yaml tests (#31754)
Implemented XContent serialisation for GetIndexResponse (#31675)
JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
resolveHasher defaults to NOOP (#31723)
Account for XContent overhead in in-flight breaker
Split CircuitBreaker-related tests (#31659)
Add write*Blob option to replace existing blob (#31729)
Painless: Add Context Docs (#31190)
Watcher: Fix chain input toXcontent serialization (#31721)
Docs: Match the examples in the description (#31710)
rest-high-level: added get cluster settings (#31706)
[Docs] Correct typos (#31720)
Clean up double semicolon code typos (#31687)
[DOCS] Check for Windows and *nix file paths (#31648)
[ML] Validate ML filter_id (#31535)
Revert long lines
Fix TransportChangePasswordActionTests
ingest: Introduction of a bytes processor
This processor allows for human readable byte values (e.g. 1kb) to be converted to value in bytes (e.g. 1024). Internally this processor re-uses "ByteSizeValue.parseBytesSizeValue" which supports conversions up to Long.MAX_VALUE and the following units: "b", "kb", "mb", "gb", "tb", pb".
This change also introduces a generic return type for the AbstractStringProcessor to allow for code reuse while supporting a String -> T conversion. (String -> Long in this case).
Significantly improve the example snippets in the documentation.
The examples are part of the test suite and checked nightly.
To help readability, the existing dataset was extended (test_emp renamed
to emp plus library).
Improve output of JDBC tests to be consistent with the CLI
Add lenient flag to JDBC asserts to allow type widening (a long is
equivalent to a integer as long as the value is the same).
AWS supports the creation and use of credentials that are only valid for a
fixed period of time. These credentials comprise three parts: the usual access
key and secret key, together with a session token. This commit adds support for
these three-part credentials to the EC2 discovery plugin and the S3 repository
plugin.
Note that session tokens are only valid for a limited period of time and yet
there is no mechanism for refreshing or rotating them when they expire without
restarting Elasticsearch. Nonetheless, this feature is already useful for
nodes that need only run for a few days, such as for training, testing or
evaluation. #29135 tracks the work towards allowing these credentials to be
refreshed at runtime.
Resolves#16428
So far the in-flight request circuit breaker has only accounted for the
on-the-wire representation of a request. However, we convert the raw
request into XContent internally which increases the overhead.
Therefore, we increase the value of the corresponding setting
`network.breaker.inflight_requests.overhead` from one to two. While this
value is still rather conservative (we assume that the representation as
structured objects has no overhead compared to the byte[]), it is closer
to reality than the current value.
Relates #31613
* master:
Do not check for object existence when deleting repository index files (#31680)
Remove extra check for object existence in repository-gcs read object (#31661)
Support multiple system store types (#31650)
[Test] Clean up some repository-s3 tests (#31601)
[Docs] Use capital letters in section headings (#31678)
[DOCS] Add PQL language Plugin (#31237)
Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
TEST: Fix test task invocation (#31657)
Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
Fix RealmInteg test failures
Extend allowed characters for grok field names (#21745) (#31653)
[DOCS] Fix licensing API details (#31667)
[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
Fix CreateSnapshotRequestTests Failure (#31630)
Configurable password hashing algorithm/cost (#31234)
[TEST] Mute failing NamingConventionsTaskIT tests
[DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
Core: Require all actions have a Task (#31627)
* master:
Docs: Remove duplicate test setup
Print output when the name checker IT fails (#31660)
Fix syntax errors in get-snapshots docs (#31656)
Docs: Fix description of percentile ranks example example (#31652)
Add MultiSearchTemplate support to High Level Rest client (#30836)
Add test for low-level client round-robin behaviour (#31616)
SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
Correct integTest enable logic (#31646)
Fix missing get-snapshots docs reference #31645
Do not check for Azure container existence (#31617)
Merge AwsS3Service and InternalAwsS3Service in a S3Service class (#31580)
Upgrade gradle wrapper to 4.8 (#31525)
Only set vm.max_map_count if greater than default (#31512)
Add Get Snapshots High Level REST API (#31537)
QA: Merge query-builder-bwc to restart test (#30979)
Update reindex.asciidoc (#31626)
Docs: Skip xpack snippet tests if no xpack (#31619)
mute CreateSnapshotRequestTests
HLRest: Fix test for explain API
[TEST] Fix RemoteClusterConnectionTests
Add Create Snapshot to High-Level Rest Client (#31215)
Remove legacy MetaDataStateFormat (#31603)
Add explain API to high-level REST client (#31387)
Preserve thread context when connecting to remote cluster (#31574)
Unify headers for full text queries
Remove redundant 'minimum_should_match'
JDBC driver prepared statement set* methods (#31494)
[TEST] call yaml client close method from test suite (#31591)
The range docs had an introductory section that described how to set up
and index *and* a test setup section in `docs/build.gradle` that
duplicated that section. This is bad because these section can (and do)
drift from one another. This change removes the setup in build.gradle
and marks the introductor snippet with `// TESTSETUP` so it is used on
all the snippets.
Skips tests the require xpack if we run the doc build without xpack. So
this should work:
```
./gradlew -p docs check -Dtests.distribution=oss-zip
```
This is implemented by detecting parts of the doc that look like:
```
[testenv="basic"]
```
Relates to #30665
Added support to the high-level rest client for the create snapshot API call. This required
several changes to toXContent which may need to be cleaned up in a later PR. Also
added several parsers for fromXContent to be able to retrieve appropriate responses
along with tests.
* master:
ingest: Add ignore_missing property to foreach filter (#22147) (#31578)
Fix a formatting issue in the docvalue_fields documentation. (#31563)
reduce log level at gradle configuration time
[TEST] Close additional clients created while running yaml tests (#31575)
Docs: Clarify sensitive fields watcher encryption (#31551)
Watcher: Remove never executed code (#31135)
Add support for switching distribution for all integration tests (#30874)
Improve robustness of geo shape parser for malformed shapes (#31449)
QA: Create xpack yaml features (#31403)
Improve test times for tests using `RandomObjects::addFields` (#31556)
[Test] Add full cluster restart test for Rollup (#31533)
Enhance thread context uniqueness assertion
[DOCS] Fix heading format errors (#31483)
fix writeIndex evaluation for aliases (#31562)
Add x-opaque-id to search slow logs (#31539)
Watcher: Fix put watch action (#31524)
Add package pre-install check for java binary (#31343)
Reduce number of raw types warnings (#31523)
Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
turn GetFieldMappingsResponse to ToXContentObject (#31544)
Close xcontent parsers (partial) (#31513)
Ingest Attachment: Upgrade Tika to 1.18 (#31252)
TEST: Correct the assertion arguments order (#31540)
* Migrate scripted metric aggregation scripts to ScriptContext design #29328
* Rename new script context container class and add clarifying comments to remaining references to params._agg(s)
* Misc cleanup: make mock metric agg script inner classes static
* Move _score to an accessor rather than an arg for scripted metric agg scripts
This causes the score to be evaluated only when it's used.
* Documentation changes for params._agg -> agg
* Migration doc addition for scripted metric aggs _agg object change
* Rename "agg" Scripted Metric Aggregation script context variable to "state"
* Rename a private base class from ...Agg to ...State that I missed in my last commit
* Clean up imports after merge
* master:
Add get field mappings to High Level REST API Client (#31423)
[DOCS] Updates Watcher examples for code testing (#31152)
TEST: Add bwc recovery tests with synced-flush index
[DOCS] Move sql to docs (#31474)
[DOCS] Move monitoring to docs folder (#31477)
Core: Combine doExecute methods in TransportAction (#31517)
IndexShard should not return null stats (#31528)
fix repository update with the same settings but different type (#31458)
Fix Mockito trying to mock IOException that isn't thrown by method (#31433) (#31527)
Node selector per client rather than per request (#31471)
Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
Upgrade to Lucene 7.4.0. (#31529)
[ML] Add ML filter update API (#31437)
Allow multiple unicast host providers (#31509)
Avoid deprecation warning when running the ML datafeed extractor. (#31463)
REST high-level client: add simulate pipeline API (#31158)
Get Mapping API to honour allow_no_indices and ignore_unavailable (#31507)
[PkiRealm] Invalidate cache on role mappings change (#31510)
[Security] Check auth scheme case insensitively (#31490)
In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. (#31514)
[DOCS] Fix REST tests in SQL docs
[DOCS] Add code snippet testing in more ML APIs (#31339)
Core: Remove ThreadPool from base TransportAction (#31492)
[DOCS] Remove fixed file from build.gradle
Rename createNewTranslog to fileBasedRecovery (#31508)
Test: Skip assertion on windows
[DOCS] Creates field and document level security overview (#30937)
[DOCS] Significantly improve SQL docs
[DOCS] Move migration APIs to docs (#31473)
Core: Convert TransportAction.execute uses to client calls (#31487)
Return transport addresses from UnicastHostsProvider (#31426)
Ensure local addresses aren't null (#31440)
Remove unused generic type for client execute method (#31444)
Introduce http and tcp server channels (#31446)
We have made node selectors configurable per request, but all
of other language clients don't allow for that.
A good reason not to do so, is that having a different node selector
per request breaks round-robin. This commit makes NodeSelector
configurable only at client initialization. It also improves the docs
on this matter, important given that a single node selector can still
affect round-robin.
Since #31251, we use the default distribution to test docs. With this
distribution, the cat thread-pool response will include the ccr
thread-pool. This commit adjusts /_cat/thread_pool doc to include ccr.
Relates #31251
A link to the ip_range datatype page provides a way for newer users to know
it exists if they land directly on the ip datatype page first via a search.
The `multiplexer` filter emits multiple tokens at the same position, each
version of the token haivng been passed through a different filter chain.
Identical tokens at the same position are removed.
This allows users to, for example, index lowercase and original-case tokens,
or stemmed and unstemmed versions, in the same field, so that they can search
for a stemmed term within x positions of an unstemmed term.
Folks tend to want to be able to make a single `_reindex` call to
migrate many indices. You *can* do that and we even have an example of
how to do that in the docs but it isn't always a good idea. This change
adds some advice to the docs: generally you want to make one reindex
call per index.
Closes#22920
In 6.3 this was moved to `Strings.toString(XContentBuilder)` as part of the
XContent extraction. This commit fixes the docs to reference the new method.
Resolves#31326
This switches the docs tests from the `oss-zip` distribution to the
`zip` distribution so they have xpack installed and configured with the
default basic license. The goal is to be able to merge the
`x-pack/docs` directory into the `docs` directory, marking the x-pack
docs with some kind of marker. This is the first step in that process.
This also enables `-Dtests.distribution` support for the `docs`
directory so you can run the tests against the `oss-zip` distribution
with something like
```
./gradlew -p docs check -Dtests.distribution=oss-zip
```
We can set up Jenkins to run both.
Relates to #30665
The other metric aggregations (min/max/etc) return `null` as their XContent value and string when nothing was computed (due to empty/missing fields). Percentiles and Percentile Ranks, however, return `NaN `which is inconsistent and confusing for the user. This fixes the inconsistency by making the aggs return `null`. This applies to both the numeric value and the "as string" value.
Note: like the metric aggs, this does not change the value if fetched directly from the percentiles object, which will return as `NaN`/`"NaN"`. This only changes the XContent output.
While this is a bugfix, it still breaks bwc in a minor way as the response changes from prior version.
Closes#29066
This commit adds the is-write-index flag for aliases.
It allows requests to set the flag, and responses to display the flag.
It does not validate and/or affect any indexing/getting/updating behavior
of Elasticsearch -- this will be done in a follow-up PR.
Add a `NodeSelector` so that users can filter the nodes that receive
requests based on node attributes.
I believe we'll need this to backport #30523 and we want it anyway.
I also added a bash script to help with rebuilding the sniffer parsing
test documents.
With #29331 we added support for the cluster health API to the
high-level REST client. The transport client does not support the level
parameter, and it always returns all the info needed for shards level
rendering. We have maintained that behaviour when adding support for
cluster health to the high-level REST client, to ease migration, but the
correct thing to do is to default the high-level REST client to
`cluster` level, which is the same default as when going through the
Elasticsearch REST layer.
This commit removes all the API methods that accept a `Header` varargs
argument, in favour of the newly introduced API methods that accept a
`RequestOptions` argument.
Relates to #31069
Given the weirdness of the response returned by the get alias API, we went for a client specific response, which allows us to hold the error message, exception and status returned as part of the response together with aliases. See #30536 .
Relates to #27205
This adds a thread interrupter that allows us to encapsulate calls to org.joni.Matcher#search()
This method can hang forever if the regex expression is too complex.
The thread interrupter in the background checks every 3 seconds whether there are threads
execution the org.joni.Matcher#search() method for longer than 5 seconds and
if so interrupts these threads.
Joni has checks that that for every 30k iterations it checks if the current thread is interrupted and
if so returns org.joni.Matcher#INTERRUPTED
Closes#28731
Allows users of the Low Level REST client to specify which hosts a
request should be run on. They implement the `NodeSelector` interface
or reuse a built in selector like `NOT_MASTER_ONLY` to chose which nodes
are valid. Using it looks like:
```
Request request = new Request("POST", "/foo/_search");
RequestOptions options = request.getOptions().toBuilder();
options.setNodeSelector(NodeSelector.NOT_MASTER_ONLY);
request.setOptions(options);
...
```
This introduces a new `Node` object which contains a `HttpHost` and the
metadata about the host. At this point that metadata is just `version`
and `roles` but I plan to add node attributes in a followup. The
canonical way to **get** this metadata is to use the `Sniffer` to pull
the information from the Elasticsearch cluster.
I've marked this as "breaking-java" because it breaks custom
implementations of `HostsSniffer` by renaming the interface to
`NodesSniffer` and by changing it from returning a `List<HttpHost>` to a
`List<Node>`. It *shouldn't* break anyone else though.
Because we expect to find it useful, this also implements `host_selector`
support to `do` statements in the yaml tests. Using it looks a little
like:
```
---
"example test":
- skip:
features: host_selector
- do:
host_selector:
version: " - 7.0.0" # same syntax as skip
apiname:
something: true
```
The `do` section parses the `version` string into a host selector that
uses the same version comparison logic as the `skip` section. When the
`do` section is executed it passed the off to the `RestClient`, using
the `ElasticsearchHostsSniffer` to sniff the required metadata.
The idea is to use this in mixed version tests to target a specific
version of Elasticsearch so we can be sure about the deprecation
logging though we don't currently have any examples that need it. We do,
however, have at least one open pull request that requires something
like this to properly test it.
Closes#21888
With `max_concurrent_shard_requests` we used to throttle / limit
the number of concurrent shard requests a high level search request
can execute per node. This had several problems since it limited the
number on a global level based on the number of nodes. This change
now throttles the number of concurrent requests per node while still
allowing concurrency across multiple nodes.
Closes#31192
Full restructure of the spec into new sections for operators, statements, scripts, functions, lambdas, and regexes. Split of operators into 6 sections, a table, reference, array, numeric, boolean, and general. Clean up of all operators sections. Sporadic clean up else where.
* Initial commit of rest high level exposure of cancel task
* fix javadocs
* address some code review comments
* update branch to use tasks namespace instead of cluster
* High-level client: list tasks failure to not lose nodeId
This commit reworks testing for `ListTasksResponse` so that random
fields insertion can be tested and xcontent equivalence can be checked
too. Proper exclusions need to be configured, and failures need to be
tested separately. This helped finding a little problem, whenever there
is a node failure returned, the nodeId was lost as it was never printed
out as part of the exception toXContent.
* added comment
* merge from master
* re-work CancelTasksResponseTests to separate XContent failure cases from non-failure cases
* remove duplication of logic in parser creation
* code review changes
* refactor TasksClient to support RequestOptions
* add tests for parent task id
* address final PR review comments, mostly formatting and such
Adds support for `ignore_unmapped` parameter in geo distance sorting,
which is functionally equivalent to specifying an `unmapped_type` in
the field sort.
Closes#28152
This field is similar to the `feature` field but is better suited to index
sparse feature vectors. A use-case for this field could be to record topics
associated with every documents alongside a metric that quantifies how well
the topic is connected to this document, and then boost queries based on the
topics that the logged user is interested in.
Relates #27552
By default span_multi query will limit term expansions = boolean max clause.
This will limit high heap usage in case of high cardinality term
expansions. This applies only if top_terms_N is not used in inner multi
query.
* Fix index prefixes to work with span_multi
Text fields that use `index_prefixes` can rewrite `prefix` queries into
`term` queries internally. This commit fix the handling of this rewriting
in the `span_multi` query.
This change also copies the index options of the text field into the
prefix field in order to be able to run positional queries. This is mandatory
for `span_multi` to work but this could also be useful to optimize `match_phrase_prefix`
queries in a follow up. Note that this change can only be done on indices created
after 6.3 since we set the index options to doc only in this version.
Fixes#31056
When `lenient=false`, attempts to create match phrase queries with custom analyzers against non-text fields will throw an IllegalArgumentException.
Also changes `*Match*QueryBuilderTests` so that it avoids this scenario
Fixes#31061
The documentation for the account circuit breaker listed the settings for it's limit and overhead to be `network.breaker.accounting.limit` and `network.breaker.accounting.overhead` when in `HieratchyCircuitBreakerService` it seems the settings are actually `indices.breaker.accounting.limit` and `indices.breaker.accounting.overhead`.
* [DOCS] Rewords _field_names documentation
Corrects the language around when we write to `_field_names` and when you might want to disable it given that n recent versions it does not carry the indexing overhead it once did.
Relates to #30862
* Update wording following review
Specifying `index_phrases: true` on a text field mapping will add a subsidiary
[field]._index_phrase field, indexing two-term shingles from the parent field.
The parent analysis chain is re-used, wrapped with a FixedShingleFilter.
At query time, if a phrase match query is executed, the mapping will redirect it
to run against the subsidiary field.
This should trade faster phrase querying for a larger index and longer indexing
times.
Relates to #27049
This change adds an option named `split_queries_on_whitespace` to the `keyword`
field type. When set to true full text queries (`match`, `multi_match`, `query_string`, ...) that target the field will split the input on whitespace to build the query terms. Defaults to `false`.
Closes#30393
This change deprecates completion queries and documents without context that target a
context enabled completion field. Querying without context degrades the search
performance considerably (even when the number of indexed contexts is low).
This commit targets master but the deprecation will take place in 6.x and the functionality
will be removed in 7 in a follow up.
Closes#29222
This commit adds Verify Repository, the associated docs and tests for
the high level REST API client. A few small changes to the Verify
Repository Response went into the commit as well.
Relates #27205
Currently failures to compile a script usually lead to a ScriptException, which
inherits the 500 INTERNAL_SERVER_ERROR from ElasticsearchException if it does
not contain another root cause. Instead, this should be a 400 Bad Request error.
This PR changes this more generally for script compilation errors by changing
ScriptException to return 400 (bad request) as status code.
Closes#12315
This change adds a new option to the composite aggregation named `missing_bucket`.
This option can be set by source and dictates whether documents without a value for the
source should be ignored. When set to true, documents without a value for a field emits
an explicit `null` value which is then added in the composite bucket.
The `missing` option that allows to set an explicit value (instead of `null`) is deprecated in this change and will be removed in a follow up (only in 7.x).
This commit also changes how the big arrays are allocated, instead of reserving
the provided `size` for all sources they are created with a small intial size and they grow
depending on the number of buckets created by the aggregation:
Closes#29380
Our API spec define the tasks API as e.g. tasks.list, meaning that they belong to their own namespace. This commit moves them from the cluster namespace to their own namespace.
Relates to #29546
Include size of snapshot in snapshot metadata
Adds difference of number of files (and file sizes) between prev and current snapshot. Total number/size reflects total number/size of files in snapshot.
Closes#18543
Treats geohashes as grid cells instead of just points when the
geohashes are used to specify the edges in the geo_bounding_box
query. For example, if a geohash is used to specify the top_left
corner, the top left corner of the geohash cell will be used as the
corner of the bounding box.
Closes#25154
The current documentation isn't very clear about how incomplete dates are
treated when specifying custom formats in a `range` query. This change adds a
note explaining how missing month or year coordinates translate to dates that
have the missings slots filled with unix time start date (1970-01-01)
Closes#30634
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
This reverts commit 31251c9 introduced in #30695.
We suspect this commit is causing the OOME's reported in #30811 and we will use this PR to test this assertion.
This commit adds the ability to configure how a docvalue field should be
formatted, so that it would be possible eg. to return a date field
formatted as the number of milliseconds since Epoch.
Closes#27740
Lucene has a new `FeatureField` which gives the ability to record numeric
features as term frequencies. Its main benefit is that it allows to boost
queries with the values of these features and efficiently skip non-competitive
documents at the same time using block-max WAND and indexed impacts.
Adding headers rather than setting them all at once seems more
user-friendly and we already do it in a similar way for parameters
(see Request#addParameter).
Currently the first snippet in the documentation test in script-fields.asciidoc
isn't executed, although it has the CONSOLE annotation. Adding a test setup
annotation to it seems to fix the problem.
This is related to #29500 and #28898. This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
=== Char Group Tokenizer
The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.
=== Configuration
The `char_group` tokenizer accepts one parameter:
`tokenize_on_chars`::
A string containing a list of characters to tokenize the string on.
Whenever a character
from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
Defaults to an empty list.
=== Example output
```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```
When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:
```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
This commit fixes docs failure on language analyzers when compared to the built in analyzers.
The `elision` filters used by the rebuilt language analyzers should be case insensitive to match
the definition of the prebuilt analyzers.
Closes#30557
This commit adds Delete Repository, the associated docs and tests for
the high level REST API client. It also cleans up a seemingly innocuous
line in the RestDeleteRepositoryAction and some naming in SnapshotIT.
Relates #27205
The getDate() and getDates() existed prior to 5.x on long fields in
scripting. In 5.x, a new Date type for ScriptDocValues was added. The
getDate() and getDates() methods were left on long fields and added to date
fields to ease the transition. This commit removes those methods for
7.0.
The docs/README shows how to run the :docs:check goal on a subset of pages,
however using the current example fails. Removing the masking character before
the first wildcard fixes the problem.
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
This pipeline aggregation gives the user the ability to script functions that "move" across a window
of data, instead of single data points. It is the scripted version of MovingAvg pipeline agg.
Through custom script contexts, we expose a number of convenience methods:
- MovingFunctions.max()
- MovingFunctions.min()
- MovingFunctions.sum()
- MovingFunctions.unweightedAvg()
- MovingFunctions.linearWeightedAvg()
- MovingFunctions.ewma()
- MovingFunctions.holt()
- MovingFunctions.holtWinters()
- MovingFunctions.stdDev()
The user can also define any arbitrary logic via their own scripting, or combine with the above methods.
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the
task management API.
Related to #27205
This commit adds Create Repository, the associated docs and tests
for the high level REST API client. A few small changes to the
PutRepository Request and Response went into the commit as well.
This commit exposes the master version to the REST test context. This
will be needed in a follow-up where the master version will be used to
determine whether or not a certain warning header is expected.
This does away with the deprecated `com.google.api-client:google-api-client:1.23`
and replaces it with `com.google.cloud:google-cloud-storage:1.28.0`.
It also changes security permissions for the repository-gcs plugin.
We are starting over on the changelog with a different approach. This
commit removes the existing incarnation of the changelog to remove
confusion that we need to continue adding entries to it.
The geo_bounding_box query might produce false positives alongside
the right and upper edges and false negatives alongside left and
bottom edges. This commit documents the behavior and defines the
maximum error.
Closes#29196
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.
Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
The High Level REST Client's documentation suggested that users should
use the Low Level REST Client for index management activities. This
change removes that suggestion because the high level REST client
supports those APIs now.
This also changes the examples in the migration docs to that still use
the Low Level REST Client to use the non-deprecated varieats of
`performRequest`.
We want copying settings to be the default behavior. This commit
deprecates not copying settings, and disallows explicitly not copying
settings. This gives users a transition path to the future default
behavior.
Dates internally contain milliseconds (which appear when converting them
to Strings) however parsing does not accept them (and is being strict).
The parser has been changed so that Date is mandatory but the time
(including its fractions such as millis) are optional.
Fix#30002
We have a pile of documentation describing how to rebuild the built in
language analyzers and, previously, our documentation testing framework
made sure that the examples successfully built *an* analyzer but they
didn't assert that the analyzer built by the documentation matches the
built in anlayzer. Unsuprisingly, some of the examples aren't quite
right.
This adds a mechanism that tests that the analyzers built by the docs.
The mechanism is fairly simple and brutal but it seems to be working:
build a hundred random unicode sequences and send them through the
`_analyze` API with the rebuilt analyzer and then again through the
built in analyzer. Then make sure both APIs return the same results.
Each of these calls to `_anlayze` takes about 20ms on my laptop which
seems fine.
This commit adds the Snapshot Client with a first API call within it,
the get repositories call in snapshot/restore module. This also creates
a snapshot namespace for the docs, as well as get repositories docs.
Relates #27205
Today we can execute cluster API actions on only master, data or ingest nodes
using the `master:true`, `data:true` and `ingest:true` filters, but it is not
so easy to select coordinating-only nodes (i.e. those nodes that are neither
master nor data nor ingest nodes). This change fixes this by adding support for
a `coordinating_only` filter such that `coordinating_only:true` adds all
coordinating-only nodes to the set of selected nodes, and
`coordinating_only:false` deletes them.
Resolves#28831.
The HTTPClient used in watcher is based on the apache http client. The
current client is using a lot of defaults - which are not always
optimal. Two of those defaults are the maximum number of total
connections and the maximum number of connections to a single route.
If one of those limits is reached, the HTTPClient waits for a connection
to be finished thus acting in a blocking fashion. In order to prevent
this when many requests are being executed, we increase the limit of
total connections as well as the connections per route (a route is
basically an endpoint, which also contains proxy information, not
containing an URL, just hosts).
On top of that an additional option has been set to evict
long running connections, which can potentially be reused after some
time. As this requires an additional background thread, this required
some changes to ensure that the httpclient is closed properly. Also the
timeout for this can be configured.
Deprecate the many arguments versions of `performRequest` and
`performRequestAsync` in favor of the `Request` object flavored variants
introduced in #29623. We'll be dropping the many arguments variants in
7.0 because they make it difficult to add new features in a backwards
compatible way and they create a *ton* of intellisense noise.
We had been using `task_id:1` or `taskId:1` because it is parses as a
valid task identifier but the `:1` part is confusing. This replaces
those examples with `task_id` which matches the response from the list
tasks API.
Closes#28314
Fixes and edge case when using `more_like_this` where TermVectorsWriter
could throw an NPE when a field produced zero tokens after analysis. This
changes the implementation to use an empty list of tokens in this case.
Closes#30148
Auto-expands replicas in the same cluster state update (instead of a follow-up reroute) where nodes are added or removed.
Closes#1873, fixing an issue where nodes drop their copy of auto-expanded data when coming up, only to sync it again later.
Adds verification that geohashes are not empty and contain only
valid characters. It fixes the issue when en empty geohash is
treated as [-180, -90] and geohashes with non-geohash character
are getting resolved into invalid coordinates.
Closes#23579
Jack Conradson
Paul Sanwald
Colin Goodheart-Smithe
Ryan Ernst
Mayya Sharipova
Clinton Gormley
Alan Woodward
Daniel Mitterdorfer
Nhat Nguyen
Nik Everett
Jimi Ford
Lisa Cawley
James Baiera
Sohaib Iftikhar
Jim Ferenczi
Armin Braun
Russ Cam
Costin Leau
Piotr Prądzyński
Christoph Büscher
Tim Brooks
Alpar Torok
Lee Hinman
Jake Landis
Costin Leau
David Turner
Peter Evers
Stéphane Campinas
Jason Tedor
Albert Zaharovits
Fredrik Meyer
ritesh-kapoor
markharwood
DeDe Morton
Yu
Julie Tibshirani
Sue Gallagher
Jonathan Little
Vladimir Dolzhenko
Luca Cavanna
Jonathan Pool
Peter Dyson
Zachary Tong
Tal Levy
Tanguy Leroux
David Pilato
Van0SS
olcbean
Martijn van Groningen
debadair
Simon Willnauer
Igor Motov
Rafał Bigaj
Adrien Grand
Nirmal Chidambaram
Dmitri Moore
Itamar Syn-Hershko
Jim Ferenczi
lipsill
Michael Basnight
Fernando Medina Corey
Lee Jones
Adam Chalkley
William Dearden
lukens
javanna
srini-raman
Yannick Welsch
Mueed Chaudhry
Ke Li
Alexander Reelsen
Karim Frenn
aditya-agrawal
Matija Bruncic
Dave Moore