Commit Graph

43535 Commits

Author SHA1 Message Date
Ioannis Kakavas 3c12c372d0
[DOCS] Adjust role mapping docs for SAML (#37083)
Explicitly mention that file based role mappings cannot be used with
the SAML realm.
2019-01-04 08:26:41 +02:00
Julie Tibshirani ac1c6940d2
Stop automatically nesting mappings in index creation requests. (#36924)
Now that we unwrap mappings in DocumentMapperParser#extractMappings, it is not
necessary for the mapping definition to always be nested under the type. This
leniency around the mapping format was added in 2341825358.
2019-01-03 17:41:28 -08:00
Tal Levy eaeccd8401
[ILM] Add Freeze Action (#36910)
This commit adds a new ILM Action for
freezing indices in the cold phase.

Closes #34630.
2019-01-03 15:00:40 -08:00
Costin Leau 40a30c6f5f
SQL: Preserve original source for each expression (#36912)
Improve parsing to save the source for each token alongside the location 
 of each Node/Expression for accurate reproducibility of an expression
 name and source

Fix #36894
2019-01-04 00:57:50 +02:00
Armin Braun 7686ee7631
TESTS: Shutdown ThreadPool after TestNodes (#37123)
* If the threadpool gets shut down before the testnodes we run into an error => fixed by moving to single `After` method
* Relates #36976
2019-01-03 22:35:44 +01:00
Benjamin Trent 2c3ce5b9de
[ML] add min_version field to machine learning results index mapping (#37115)
* ML: add min_version field to mapping

* Adding min_version as reserved field name
2019-01-03 14:59:42 -06:00
Igor Motov d6acd8e15f
Docs: add clarification about geohash use in geohashgrid agg (#36901)
Adds an example on translating geohashes returned by geohashgrid 
agg as bucket keys into geo bounding box filters in elasticsearch as well
as 3rd party applications.

Closes #36413
2019-01-03 15:40:48 -05:00
Julie Tibshirani 54f53d2a51
Make sure to accept empty unnested mappings in create index requests. (#37089) 2019-01-03 11:53:08 -08:00
Nick Knize e613bcae43
Remove XLatLonShape classes (#37094)
This commit removes local XLatLonShape classes and replaces with current LatLonShape classes in latest lucene snapshot
2019-01-03 12:48:36 -06:00
Daniel Mitterdorfer e4ec51879e
Retry JDK download when building Docker image
With this commit we instruct curl to retry with a backoff when
downloading the JDK for the Elasticsearch Docker image. This avoids
build failures on transient network issues. Note that this option
requires curl 7.12.3 or better.

Relates #37103
Relates #37113
2019-01-03 19:27:18 +01:00
Nicholas Knize de962b2f39 Revert "Adjust Lucene version for 6.7"
This reverts commit b7f6ee72a6.
2019-01-03 11:52:31 -06:00
Lisa Cawley 366b6b5e11
[DOCS] Adds cluster setting to enable/disable config migration (#36948) 2019-01-03 09:26:57 -08:00
David Kyle fc49d88b92
[ML] Adjust migration full cluster restart tests based on old cluster version #37118 2019-01-03 17:15:35 +00:00
David Roberts 13649aa70a
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
The problem that caused the test to be muted was fixed in
https://github.com/elastic/ml-cpp/pull/332

Closes #36258
2019-01-03 16:23:18 +00:00
Armin Braun 99f13b90d3
SNAPSHOT: Speed up HDFS Repository Writes (#37069)
* There is no point in hsyncing after every individual write since there is only value in completely written blobs for restores, this is ensures by the `SYNC` flag already and there is no need for separately invoking `hsync` after individual writes
2019-01-03 16:16:05 +01:00
Armin Braun 10d9819f99
Implement Atomic Blob Writes for HDFS Repository (#37066)
* Implement atomic writes the same way we do for the FsBlobContainer via rename which is atomic
* Relates #37011
2019-01-03 15:51:47 +01:00
Armin Braun 675ea4c59c
TESTS: Remove Static Threadpool in TaskManagerTest (#36976)
* The static threadpool leaks a lot of memory in these tests because it prevents things like the connect listeners from `org.elasticsearch.transport.TcpTransport#initiateConnection`
to be GCed between tests (since they keep being referenced by the threadpool) which in turn reference channels and their underlying buffers
  * I could not find any slowdown in executing these tests from this change, if anything they are slightly faster now on my machine
* Relates #36906 (which may be caused by slowness from leaking memory and also becomes testable in a loop by this change)
2019-01-03 15:19:21 +01:00
Benjamin Trent cfc310748d
addressing (#36891)(#36888)(#36889) (#37080) 2019-01-03 07:25:57 -06:00
Dimitris Athanasiou b04b3173db
[ML][TEST] Clean up max_model_memory_limit cluster setting (#37101)
Removes the `xpack.ml.max_model_memory_limit` cluster setting
at the teardown of the `ml_info.yml` tests to ensure the setting
does not trip other tests.
2019-01-03 15:15:31 +02:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg (#36933)
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: #36903
2019-01-03 13:55:09 +02:00
Christoph Büscher 046f86f274
Deprecate use of type in reindex request body (#36823)
Types can be used both in the source and dest section of the body which will
be translated to search and index requests respectively. Adding a deprecation warning
for those cases and removing examples using more than one type in reindex since
support for this is going to be removed.
2019-01-03 10:29:14 +01:00
Christoph Büscher e21054d176
Remove two unused methods in Iterables (#37075)
These helper methods are unused in the rest of the codebase.
2019-01-03 10:28:47 +01:00
Nhat Nguyen b7f6ee72a6 Adjust Lucene version for 6.7
Relates #37088
2019-01-03 04:20:47 -05:00
Jim Ferenczi 78ba1889cf
Replace the TreeMap in the composite aggregation (#36675)
The `composite` aggregation uses a TreeMap to keep track of the best buckets.
This ensures a log(n) time cost to insert new buckets but also to retrieve buckets
that are already present in the map. In order to speed up the retrieval of buckets
this change replaces the TreeMap with a priority queue and a HashMap. The insertion
cost is still log(n) but the retrieval of buckets through the HashMap is now done in constant
time. This optimization can bring significant improvement since each document needs
to check if its associated buckets are already present in the current best buckets.
2019-01-03 09:51:35 +01:00
David Turner a40f0545e6
Make ILM examples larger (#37039)
There are a handful of examples in the ILM documentation that could result in
rolling over indices more quickly than we might normally recommend,
contributing to over-sharding in cases where the examples are copied without
modification. This change makes some numbers bigger to try and avoid this.
2019-01-03 08:28:49 +00:00
Daniel Mitterdorfer 75f3443c62
Rename setting to enable mmap
With this commit we rename `node.store.allow_mmapfs` to
`node.store.allow_mmap`. Previously this setting has controlled whether
`mmapfs` could be used as a store type. With the introduction of
`hybridfs` which also relies on memory-mapping,
`node.store.allow_mmapfs` also applies to `hybridfs` and thus we rename
it in order to convey that it is actually used to allow memory-mapping
but not a specific store type.

Relates #36668
Relates #37070
2019-01-03 07:10:34 +01:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Ioannis Kakavas f0cd19f39b [TEST] Lower deleteInterval in TokenAuthIntegTests
We run subsequent token invalidation requests and we still want to
trigger the deletion of expired tokens so we need to lower the
deleteInterval parameter significantly. Especially now that the
bwc expiration logic is removed and the invalidation process is
much shorter

Resolves #37063
2019-01-02 23:54:02 +02:00
David Findley d4e7660248 Fix weighted_avg parser not found for RestHighLevelClient (#37027)
Add integration test for weighted avg sub aggregation
Add weighted avg parser to DefaultNamedXContents

Fixes #36861
2019-01-02 15:53:21 -06:00
Ke Li 62ece69b92 Remove system property es.enforce_max_shards_per_node (#36968)
The system property es.enforce_max_shards_per_node was not needed any more,
because we always enforce cluster-wide shard limit now.
2019-01-02 14:01:12 -07:00
Alan Woodward 7a0047744d
`query_string` should use indexed prefixes (#36895)
The QueryStringQueryBuilder does not currently delegate to the field mapper's prefixQuery
method, so does not use indexed prefixes. This commit corrects this.

It also fixes a bug where a query a* would not match the word a if indexed prefixes were used with
a minchar setting of 2.
2019-01-02 20:12:24 +00:00
Julie Tibshirani 265fdce312 Fix FullClusterRestartIT#testRollupIDSchemeAfterRestart by adding a missing expected warning. 2019-01-02 10:48:32 -08:00
David Kyle 42bb2bae21
[ML] Order GET job stats response by job id (#36841) 2019-01-02 16:52:20 +00:00
Luca Cavanna 42ea644903
Remove single shard optimization when suggesting shard_size (#37041)
When executing terms aggregations we set the shard_size, meaning the
number of buckets to collect on each shard, to a value that's higher than
the number of requested buckets, to guarantee some basic level of
precision. We have an optimization in place so that we leave shard_size
set to size whenever we are searching against a single shard, in which
case maximum precision is guaranteed by definition.

Such optimization requires us access to the total number of shards that
the search is executing against. In the context of cross-cluster search,
once we will introduce multiple reduction steps (one per cluster) each
cluster will only know the number of local shards, which is problematic
as we should only optimize if we are searching against a single shard in a
single cluster. It could be that we are searching against one shard per cluster
in which case the current code would optimize number of terms causing
a loss of precision.

While discussing how to address the CCS scenario, we decided that we do
not want to introduce further complexity caused by this single shard
optimization, as it benefits only a minority of cases, especially when
the benefits are not so great.

This commit removes the single shard optimization, meaning that we will
always have heuristic enabled on how many number of buckets to collect
on the shards, even when searching against a single shard.

This will cause more buckets to be collected when searching against a single
shard compared to before. If that becomes a problem for some users, they
can work around that by setting the shard_size equal to the size.

Relates to #32125
2019-01-02 17:45:49 +01:00
Josh Soref e0a677b033 Clarify what to run for gradle idea (#37058) 2019-01-02 17:40:11 +01:00
Josh Soref 02aa9f51ee Spelling: correct wrong spellings of similar (#37054) 2019-01-02 17:38:43 +01:00
Josh Soref f6e4b9a014 Spelling: correct wrong spellings of likelihood (#37052) 2019-01-02 17:37:03 +01:00
Josh Soref 5d5d5c26bc Spelling: replace interruptable with interruptible (#37049) 2019-01-02 17:35:53 +01:00
Josh Soref c04d5c5468 Spelling: correct wrong spellings of "input" (#37048) 2019-01-02 17:34:37 +01:00
Josh Soref 1df66d21fe Spelling: replace uknown with unknown (#37056) 2019-01-02 17:33:02 +01:00
Przemyslaw Gomulka 15c4d5b184
Fix line length in org.elasticsearch.get (#37071)
Remove the line suppression for this package and fix offedning lines
relates #34844
2019-01-02 17:22:15 +01:00
Luca Cavanna 450d3014f6 [TEST] Fix testLimitConcurrentShardRequests failure
With #36221 we introduced shards counting to address a rare failure.
This caused a worse problem in this test when replicas were allocated
and shards failures were randomly returned. The latch has to take into
account additional attempts caused by the shard failures, which means
that in order for run to be called, performPhaseOnShard will be called
(numShards + numFailures) times.

To address this, we need to decide upfront which shard is going to fail,
making sure that at least one shards is successful otherwise the whole
request fails.

Closes #37074
2019-01-02 16:46:41 +01:00
Luca Cavanna b10a33054e Mute failing testLimitConcurrentShardRequests
relates to #37074
2019-01-02 16:35:24 +01:00
Christoph Büscher 8e6f68f6b0
Fix Eclipse specific type casting issue (#37072)
The eclipse compiler needs some help in infering the functions return values.
2019-01-02 15:58:20 +01:00
Josh Soref 51cb63f934 Spelling: replace substract with subtract (#37055) 2019-01-02 14:15:21 +01:00
Josh Soref 565a95c382 Spelling: replace respositories with repositories (#37053) 2019-01-02 14:13:54 +01:00
Josh Soref d3e98278c3 Spelling: replace cachable with cacheable (#37047) 2019-01-02 14:10:30 +01:00
Abdullah DURSUN 8a02bacf76 Fix typo in multi-search.asciidoc (#37060) 2019-01-02 10:32:42 +01:00
Daniel Mitterdorfer f0052b1a7a
Add hybridfs store type
With this commit we introduce a new store type `hybridfs` that is a
hybrid between `mmapfs` and `niofs`. This store type chooses different
strategies to read Lucene files based on the read access pattern (random
or linear) in order to optimize performance.

This store type has been available in earlier versions of Elasticsearch
as `default_fs`. We have chosen a different name now in order to convey
the intent of the store type instead of tying it to the fact whether it
is the default choice.

Relates #36668
2019-01-02 10:10:32 +01:00
Luca Cavanna 9e70696628
[TEST] Address rejected execution in SearchAsyncActionTests (#37028)
SearchAsyncActionTests may fail with RejectedExecutionException as InitialSearchPhase may try to execute a runnable after the test has successfully completed, and the corresponding executor was already shut down. The latch was located in getNextPhase that is almost correct, but does not cover for the last finishAndRunNext round that gets executed after onShardResult is invoked.

This commit moves the latch to count the number of shards, and allowing the test to count down later, after finishAndRunNext has been potentially forked. This way nothing else will be executed once the executor is shut down at the end of the tests.

Closes #36221
Closes #33699
2019-01-02 09:47:13 +01:00