Commit Graph

49460 Commits

Author SHA1 Message Date
Alexander Reelsen 71054d269b Sync grok patterns with logstash patterns (#50381)
In order to ensure that logstash and Elasticsearch are able to understand
the same patterns, this commit adapts to changes in logstash, adds a few
patterns and changes a few.
2020-01-08 14:59:34 +01:00
James Rodewig d3094f9d23 [DOCS] Fix typo in mapping date format docs 2020-01-08 07:55:51 -06:00
Yannick Welsch f203c2b39d Import replicated closed dangling indices (#50649)
Dangling replicated closed indices are not imported properly (they miss their routing table when imported).
2020-01-08 13:39:20 +01:00
Rory Hunter b1ff74f652 New setting to prevent automatically importing dangling indices (#49174)
Introduce a new static setting, `gateway.auto_import_dangling_indices`, which prevents dangling indices from being automatically imported. Part of #48366.
2020-01-08 13:39:20 +01:00
Andrei Dan 3915d4c055
Make the UpdateRolloverLifecycleDateStep retryable (#50702) (#50730)
This makes the "update-rollover-lifecycle-date" step, which is part of the
rollover action, retryable. It also adds an integration test to check the
step is retried and it eventually succeeds.

(cherry picked from commit 5bf068522deb2b6cd2563bcf80f34fdbf459c9f2)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-01-08 11:45:26 +01:00
Christoph Büscher d8c907d648 Remove _reload_search_analyzer experimental status (#50696)
Removing the experimental status in the docs and the rest specs.
2020-01-08 10:35:19 +01:00
Tim Vernum 293661d62c
Security should not reload files that haven't changed (#50724)
In security we currently monitor a set of files for changes:

- config/role_mapping.yml (or alternative configured path)
- config/roles.yml
- config/users
- config/users_roles

This commit prevents unnecessary reloading when the file change actually doesn't change the internal structure.

Backport of: #50207

Co-authored-by: Anton Shuvaev <anton.shuvaev91@gmail.com>
2020-01-08 15:13:47 +11:00
Mayya Sharipova c1c0b47d5e
Specify the indexname in searches (#50717)
vector REST tests occasionally fail on 7.x because
we don't receive the expected response headers with deprecation warnings.
This happens as searchers were executed against all indices including
internal indices, whose shards did not produce expected warnings.

This PR ensures that searchers are executed only against expected
indices.

Closes #50716
2020-01-07 17:06:52 -05:00
Mayya Sharipova 0b7309ec9c Fix NPE bug inner_hits (#50709)
When there several subqueries on different relations of the join field,
and only one of subqueries is using inner_hits, NPE occurs.
This PR prevents NPE error.

Closes #50539
2020-01-07 14:21:54 -05:00
Armin Braun d0d48311f4
Faster and Simpler GCS REST Mock (#50706) (#50707)
* Faster and Simpler GCS REST Mock

I reworked the GCS mock a little to use less copying+allocation,
log the full request body on failure to read a multi-part request
and generally be a little simpler and easy to follow to track down
the remaining issues that are causing almost daily failures from this
class's multi-part request parsing that can't be reproduced locally.
2020-01-07 20:17:46 +01:00
James Rodewig de6b62f789 [DOCS] Fuzzy wildcard not supported in `query_string` (#50466)
The `query_string` does not support mixing wildcards with fuzziness.
This adds a related warning to the `query_string` docs.
2020-01-07 12:54:50 -06:00
Ryan Ernst 1f6c1df58e Add cliSetup command to test clusters configuration (#50414)
This commit adds a cliSetup command that can be used to run arbitrary
bin scripts to setup a test cluster. This is the same as what was
previously called setupCommands in cluster formation tasks.

closes #50382
2020-01-07 10:29:31 -08:00
James Rodewig 20eba1e410 [DOCS] Reformat reverse token filter docs (#50672)
* Updates the description and adds a Lucene link
* Adds analyze and custom analyzer snippets
2020-01-07 11:01:55 -06:00
Nik Everett deb0991667
Teach ObjectParser a happy pattern (#50691) (#50710)
We *very* commonly have object with ctors like:
```
public Foo(String name)
```

And then declare a bunch of setters on the object. Every aggregation
works like this, for example. This change teaches `ObjectParser` how to
build these aggregations all on its own, without any help. This'll make
it much cleaner to parse aggs, and, probably, a bunch of other things.
It'll let us remove lots of wrapping. I've used this new power for the
`avg` aggregation just to prove that it works outside of a unit test.
2020-01-07 11:57:41 -05:00
James Rodewig 8009b07ccb [DOCS] Reformat truncate token filter docs (#50687)
* Updates the description and adds a Lucene link
* Adds analyze, custom analyzer, and custom filter snippets
* Adds parameter documentation
2020-01-07 10:33:57 -06:00
arkel-s d5f4790f90 [DOCS] Add example format for `date_optional_time` (#50458)
Adds an example format for `date_optional_time` to the `format` mapping
parameter docs.

Closes #50457
2020-01-07 10:13:34 -06:00
Nhat Nguyen c3d207f437 Disable auto refresh in testSegmentsStats (#50689)
If an auto-refresh happens, then version_map_memory is reset to 0. By 
default, the auto-refresh occurs for every second in the first 30
seconds until search becomes idle.

Closes #50362
2020-01-07 10:44:30 -05:00
James Rodewig 0753915eed [DOCS] Update SQL REST API pages for new structure (#50690)
#43007 restructured the SQL REST API docs so they display across several pages.

This updates up a reference that assumes a single page in the "Paginating through a large response" section. It also reformats a tip for the Kibana console.

Closes #50688
2020-01-07 09:27:34 -06:00
Benjamin Trent 060e0a6277
[ML][Inference] Add support for models shipped as resources (#50680) (#50700)
This adds support for models that are shipped as resources in the ML plugin. The first of which is the `lang_ident` model.
2020-01-07 09:21:59 -05:00
Hendrik Muhs 98ca9500e8
implement a workaround for remote cluster validation (#50460)
In 7.x an internal API used for validating remote cluster does not throw, see #50420 for the 
details. This change implements a workaround for remote cluster validation, only for 7.x branches.

fixes #50420
2020-01-07 13:51:51 +01:00
Przemysław Witek 4116452d90
Implement testStopAndRestart for ClassificationIT (#50585) (#50698) 2020-01-07 13:41:37 +01:00
David Roberts 35453e2b0e [ML] Improve uniqueness of result document IDs (#50644)
Switch from a 32 bit Java hash to a 128 bit Murmur hash for
creating document IDs from by/over/partition field values.
The 32 bit Java hash was not sufficiently unique, and could
produce identical numbers for relatively common combinations
of by/partition field values such as L018/128 and L017/228.

Fixes #50613
2020-01-07 10:24:45 +00:00
David Roberts 46d600c446 [ML] Fix off-by-one error in ml_classic tokenizer end offset (#50655)
The end offset of a tokenizer is supposed to point one past the
end of the input, not to the end character of the input.  The
ml_classic tokenizer was erroneously doing the latter.
2020-01-07 10:14:59 +00:00
David Turner 2039cc813b Fix testDelayVariabilityAppliesToFutureTasks (#50667)
This test seems to be bogus as it was confusing a nominal execution time with a
delay (i.e. an elapsed time). This commit reworks the test to address this.

Fixes #50650
2020-01-07 10:01:03 +00:00
Alan Woodward a3ab7eb95d Correctly handle MSM for nested disjunctions (#50669)
With the rewrite of the percolator's QueryAnalyzer to use lucene's QueryVisitor API,
term queries that are direct children of a boolean query are handled separately from
other children. This works fine for conjunctions, but for disjunctions we need to
treat the extracted terms from these direct descendents along with extractions from
more deeply nested children to ensure that minimum-should-match requirements
are met correctly.

This commit changes the logic in QueryAnalyzer#getResult() to bundle child term
results with all other results before handling them.

Fixes #50305
2020-01-07 09:32:30 +00:00
Yannick Welsch a2ef0e8830 Check allocation id when failing shard on recovery (#50656)
A failure of a recovering shard can race with a new allocation of the shard, and cause the new
allocation to be failed as well. This can result in a shard being marked as initializing in the cluster
state, but not exist on the node anymore.

Closes #50508
2020-01-07 09:41:28 +01:00
Lee Hinman 552edd862e
[7.x] Add aditional logging for ILM history store tests (#5062… (#50678)
* Add aditional logging for ILM history store tests (#50624)

These tests use the same index name, making it hard to read logs when
diagnosing the failures. Additionally more information about the current
state of the index could be retrieved when failing.

This changes these two things in the hope of capturing more data about
why this fails on some CI nodes but not others.

Relates to #50353
2020-01-06 15:24:24 -07:00
Jay Modi e5191e77e3
Remove unused IndicesOptions#fromByte method (#50683)
This change removes a no longer used method, `fromByte`, in
IndicesOptions. This method was necessary for backwards compatibility
with versions prior to 6.4.0 and was used when talking to those
versions. However, the minimum wire compatibility version has changed
and we no longer use this code.

Backport of #50665
2020-01-06 14:57:10 -07:00
Nik Everett 7fd84a03a0
Drop references to deprecated logger (#50474) (#50681)
This drops all remaining references to `BaseRestHandler.logger` which
has been deprecated for something like a year now. I replaced all of the
references with locally declared loggers which is so much less spooky
action at a distance to me.
2020-01-06 16:34:07 -05:00
Benjamin Trent 06cea5136e
[ML] construct new random generator on each persistence call (#50657) (#50684)
Sharing a random generator may cause test failures as non-threadsafe random generators are periodically utilized in tests (see: https://github.com/elastic/elasticsearch/issues/50651)

This change constructs a calls `Randomness.get()` within the  `bulkIndexWithRetry` method so that the returned `Random` object is only used in a single thread. Before, the member variable could have been used between threads, which caused test failures.
2020-01-06 16:26:29 -05:00
Benjamin Trent 5ab9e75e28
[7.x] [ML][Inference] lang_ident model (#50292) (#50675)
* [ML][Inference] lang_ident model (#50292)

This PR contains a java port of Google's CLD3 compact NN model https://github.com/google/cld3

The ported model is formatted to fit within our inference model formatting and stored as a resource in the `:xpack:ml:` plugin and is under basic license.

The model is broken up into two major parts:
- Preprocessing through the custom embedding (based on CLD3's embedding layer)
- Pushing the embedded text through the two layers of fully connected shallow NN. 

Main differences between this port and CLD3:
- We take advantage of Java's internal Unicode handling where possible (i.e. codepoints, characters, decoders, etc.)
- We do not trim down input text by removing duplicated tokens
- We do not encode doubles/floats as longs/integers.
2020-01-06 16:24:03 -05:00
Nik Everett 76bb661023
Replace AggParseContext with a String (backport of #50625) (#50679)
We used to have a *ton* off stuff in the `AggParseContext` but now we
parse aggs entirely with named xcontent. So we don't need the context
any more.
2020-01-06 14:32:03 -05:00
Benjamin Trent f52af7977d
[ML][Inference] minor cleanup for inference (#50444) (#50676) 2020-01-06 14:05:04 -05:00
Nik Everett 1b28af489f
Fix bare warnings on RollupJobTests (#50633) (#50677)
Silences some ugly warnings.
2020-01-06 14:03:30 -05:00
Nhat Nguyen 926c0aa74c Fix testRelocationEstablishedPeerRecoveryRetentionLeases (#50673)
The redNodes are calculated incorrectly.

Closes #50660
2020-01-06 13:32:04 -05:00
Armin Braun 72a405fafb
Fix GCS Mock Broken Handling of some Blobs (#50666) (#50671)
* Fix GCS Mock Broken Handling of some Blobs

We were incorrectly handling blobs starting in `\r\n` which broke
tests randomly when blobs started on these.

Relates #49429
2020-01-06 19:27:57 +01:00
Nik Everett f576aefd0f
Replace bespoke parser for significance heuristics (#50623) (#50659)
This replaces the hand written xcontent parsers for significance
heristics with `ObjectParser` and parsing named xcontent.

As a happy accident, this was the last user of `ParseFieldRegistry` so
this PR entirely removes that class.

Closes #25519
2020-01-06 12:57:43 -05:00
Tim Brooks fa57813c6d
Remove races in ProxyConnectionStrategyTests (#50620)
Currently, we use delayed address resolution in the proxy strategy tests
to allow tests to connect to different addresses. Unfortunately, this
has the potential to introduce races as the address is resolved each
connection attempt. The number of connection attempts can vary based on
when connections are opening and closing. This commit modifies the test
be allowing them to specifically control which address is used.

Related to #50618
2020-01-06 10:20:53 -07:00
James Rodewig 074866256b [DOCS] Remove unneeded redirects (#50510)
The docs/reference/redirects.asciidoc file stores a list of relocated or
deleted pages for the Elasticsearch Reference documentation.

This prunes several older redirects that are no longer needed.
2020-01-06 09:11:48 -06:00
Martijn van Groningen 7be43e9f6d
Fix ingest stats test bug. (#50653)
This test code fixes a serialization test bug:
https://gradle-enterprise.elastic.co/s/7x2ct6yywkw3o

Rarely stats for the same processor are generated and
the production code then sums up these stats. However
the test code wasn't summing up in that case,
which caused inconsistencies between the actual and expected results.

Closes #50507
2020-01-06 15:37:47 +01:00
James Rodewig 1299dda437 [DOCS] Warn about using `geo_centroid` as sub-agg to `geohash_grid` (#50038)
If `geo_point fields` are multi-valued, using `geo_centroid` as a
sub-agg to `geohash_grid` could result in centroids outside of bucket
boundaries.

This adds a related warning to the geo_centroid agg docs.
2020-01-06 07:47:54 -06:00
Nhat Nguyen b71490b06b
Deprecate indices without soft-deletes (#50502) (#50634)
Soft-deletes will be enabled for all indices in 8.0. Hence, we should
deprecate new indices without soft-deletes in 7.x.

Backport of #50502
2020-01-06 08:44:30 -05:00
Henning Andersen ec0ec61881 Deleted docs disregarded for if_seq_no check (#50526)
Previously, as long as a deleted version value was kept as a tombstone,
another index or delete operation against the same id would leak that
the doc had existed (through seq_no info) or would allow the operation
if the client forged the seq_no. Fixed to disregard info on deleted docs
when doing seq_no based optimistic concurrency check.
2020-01-06 13:54:36 +01:00
Nikita Glashenko 5533e1172c Add tests for remaining IntervalsSourceProvider implementations (#50326)
This PR adds unit tests for wire and xContent serialization of remaining IntervalsSourceProvider
implementations.

Closes #50150
2020-01-06 13:18:53 +01:00
David Turner 66c690922c Collect shard sizes for closed indices (#50645)
Today the `InternalClusterInfoService` collects information on the sizes of
shards of open indices, but does not consider closed indices. This means that
shards of closed indices are treated as having zero size when they are being
allocated. This commit fixes this, obtaining the sizes of all shards.

Relates #33888
2020-01-06 11:44:19 +00:00
Henning Andersen 312bf44601 Workaround for JDK 14 EA FileChannel.map issue (#50523)
FileChannel.map provokes static initialization of ExtendedMapMode in
JDK14 EA, which needs elevated privileges.

Relates #50512
2020-01-06 12:18:49 +01:00
Albert Zaharovits 9ae3cd2a78
Add 'monitor_snapshot' cluster privilege (#50489) (#50647)
This adds a new cluster privilege `monitor_snapshot` which is a restricted
version of `create_snapshot`, granting the same privileges to view
snapshot and repository info and status but not granting the actual
privilege to create a snapshot.

Co-authored-by: j-bean <anton.shuvaev91@gmail.com>
2020-01-06 13:15:55 +02:00
Martijn van Groningen 0f2d26bdca
Unmute 'Test url escaping with url mustache function' webhook watcher test (#50439)
Some changes had to be made in order to make the test pass due to the removal or types.
Added some more assertions. The failure description in this comment [0] indicates that the rest handler couldn't be found. The test passes now.
I plan to merge this into master and see how CI reacts, if it handles this change well then I will also unmute this test in 7 dot x branch.

Also check watch count after stopping watcher in test teardown and
disabled slm in smoke test watcher qa test.

Relates to #41172

0: https://github.com/elastic/elasticsearch/issues/41172#issuecomment-496993976
2020-01-06 10:43:55 +01:00
Nik Everett 2362c430cd
Clean up wire test case a bit (#50627) (#50632)
* Adds JavaDoc to `AbstractWireTestCase` and
`AbstractWireSerializingTestCase` so it is more obvious you should prefer
the latter if you have a choice
* Moves the `instanceReader` method out of `AbstractWireTestCase` becaue
it is no longer used.
* Marks a bunch of methods final so it is more obvious which classes are
for what.
* Cleans up the side effects of the above.
2020-01-05 16:20:38 -05:00
Armin Braun 8819fa4ebe
Make EC2 Discovery Cache Empty Seed Hosts List (#50607) (#50626)
Follow up to #50550. Cache empty nodes lists (`fetchDynamicNodes` will return an empty list in case of failure)
now that the plugin properly retries requests to AWS EC2 APIs.
2020-01-03 21:32:36 +01:00