Commit Graph

43721 Commits

Author SHA1 Message Date
Jason Tedor bf5bc88f50
Fix handling of fractional time value settings (#37171)
This commit addresses an issue when setting a time value setting using a
value that has a fractional component when converted to its string
representation. For example, trying to set a time value setting to a
value of 1500ms is problematic because internally this is converted to
the string "1.5s". When we go to get this setting, we try to parse
"1.5s" back to a time value, which does not support fractional
values. The problem is that internally we are relying on a method which
loses the unit when doing the string conversion. Instead, we are going
to use a method that does not lose the unit and therefore we can
roundtrip from the time value to the string and back to the time value.
2019-01-06 22:34:52 -08:00
Marios Trivyzas da3d8fb5b7
SQL: Fix issue with wrong NULL optimization (#37124)
Logical operators OR and AND as well as conditional functions
(COALESCE, LEAST, GREATEST, etc.) cannot be folded to NULL if one
of their children is NULL as is the case for most of the functions.
Therefore, their nullable() implementation cannot return true. On
the other hand they cannot return false as if they're wrapped within
an IS NULL or IS NOT NULL expression, the expression will be folded
to false and true respectively leading to wrong results.

Change the signature of nullable() method and add a third value UKNOWN
to handle these cases.

Fixes: #35872
2019-01-06 18:29:34 +02:00
Armin Braun b34e7d4f19
SNAPSHOT+TESTS: Relax Assertion in DisruptionIT (#37144)
* The retries on the failing master can lead to concurrently trying to create and delete a snapshot, catch this for now to fix this test
* closes #36779
2019-01-05 17:52:24 +01:00
Simon Willnauer 0cc877026f
Subclass NIOFSDirectory instead of using FileSwitchDirectory (#37140)
We don't want two FSDirectories manage pending deletes separately
and optimize file listing. This confuses IndexWriter and causes exceptions
when files are deleted twice but are pending for deletion. This change
move to using a NIOFS subclass that only delegates to MMAP for opening files
all metadata and pending deletes are managed on top.

Closes #37111
Relates to #36668
2019-01-05 10:15:33 +01:00
Julie Tibshirani 0bac64fbd3
Deprecate the _type field in aggregations. (#37131) 2019-01-04 13:05:52 -08:00
Michael Basnight e40193ae66
HLRC: Fix Reindex from remote query logic (#36908)
The query object was incorrectly added to the remote object in the
xcontent. This fix moves the query back into the source, if it was
passed in as part of the RemoteInfo. It also adds a IPv6 test for
reindex from remote such that we can properly validate this.
2019-01-04 13:37:59 -06:00
Jim Ferenczi e38cf1d0dc
Add the ability to set the number of hits to track accurately (#36357)
In Lucene 8 searches can skip non-competitive hits if the total hit count is not requested.
It is also possible to track the number of hits up to a certain threshold. This is a trade off to speed up searches while still being able to know a lower bound of the total hit count. This change adds the ability to set this threshold directly in the track_total_hits search option. A boolean value (true, false) indicates whether the total hit count should be tracked in the response. When set as an integer this option allows to compute a lower bound of the total hits while preserving the ability to skip non-competitive hits when enough matches have been collected.

Relates #33028
2019-01-04 20:36:49 +01:00
Adam Thomson ac4aecc92d [Docs] Update ingest-node.asciidoc (#37116) 2019-01-04 19:33:06 +01:00
Simon Willnauer b4f113d3ea
Don't block on peer recovery on the target side (#37076)
Today we block using the generic thread-pool on the target side
until the source side has fully executed the recovery. We still
block on the source side executing the recovery in a blocking fashion
but there is no reason to block on the target side. This will
release generic threads early if there are many concurrent recoveries
happen.

Relates to #36195
2019-01-04 13:51:06 +01:00
Simon Willnauer 41d7e3a2fe
Expose `search.throttled` on `_cat/indices` (#37073)
Today it's very difficult to see which indices are frozen or rather
throttled via the commonly used monitoring APIs. This change adds
a cell to the `_cat/indices` API to render if an index is `search.throttled`

Relates to #34352
2019-01-04 13:49:40 +01:00
David Roberts ff7df40b20
[ML] Uplift model memory limit on job migration (#37126)
When a 6.1-6.3 job is opened in a later version
we increase the model memory limit by 30% if it's
below 0.5GB. The migration of jobs from cluster
state to the config index changes the job version,
so we need to also do this uplift as part of that
config migration.

Relates #36961
2019-01-04 12:21:28 +00:00
Luca Cavanna 21d52f0dab
Ensure that local cluster alias is never treated as remote (#37121)
With #36997 we added support for providing a local cluster alias with a
`SearchRequest`. We intended to make sure that when provided as part of
a search request, the cluster alias would never be used for connection
lookups. Yet due to a bug we would still end up looking up the
connection from the remote ones.

This commit adds a test to make sure that whenever we set the cluster
alias to the `SearchRequest` (which can only be done at transport), such
alias is used as index prefix in the returned hits. No errors are thrown
despite no remote clusters are configured indicating that such alias is
never used for connection look-ups.

Also, we add explicit support for the empty cluster alias when printing
 out index names through `RemoteClusterAware#buildRemoteIndexName`.
In fact we don't want to print out `:index` when the cluster alias is
set to empty string, but rather `index`. Yet, the semantic of empty
string is different compared to `null` as it will still disable final
reduction. This will be used in CCS when searching against remote
clusters as well as the local one, the local one will have empty prefix
yet it will need to disable final reduction so that its results will be
properly merged with the ones coming from the remote clusters.
2019-01-04 12:19:31 +01:00
Dimitris Athanasiou 0fd27d4d6f [ML] Unused state remover should also account for jobs in index (#37119)
The unused state remover was never adjusted to account for jobs stored
in the config index. The result was that when triggered it removed
state for all jobs stored in the config index.
This commit fixes the issue.

Closes #37109
2019-01-04 12:43:44 +02:00
Luca Cavanna bfe6f091da [DOCS] fix typo in 7.0.0-alpha2 release notes 2019-01-04 10:38:34 +01:00
David Turner 3f7d6a989a
[Zen2] Elect freshest master in upgrade (#37122)
Today when electing a master in Zen2 we use the cluster state version to
determine whether a node has a fresh-enough cluster state to become master.
However the cluster state version is not a reliable measure of freshness in the
Zen1 world; furthermore in 6.x the cluster state version is not persisted. This
means that when upgrading from 6.x via a full cluster restart a cluster state
update may be lost if a stale master wins the initial election.

This change fixes this by using the metadata version as a measure of freshness
when in term 0, since this is persisted in 6.x and does more reliably indicate
the freshness of nodes.

It also makes changes parallel to elastic/elasticsearch-formal-models#40 to
support situations in which nodes accept cluster state versions in term 0: this
does not happen in a pure Zen2 cluster, but can happen in mixed clusters and
during upgrades.
2019-01-04 09:09:16 +00:00
Dimitris Athanasiou 586453fef1
[ML] Remove types from datafeed (#36538)
Closes #34265
2019-01-04 09:43:44 +02:00
lcawley 00997b4f60 [DOCS] Fixes broken links 2019-01-04 17:41:28 +10:00
Peter Dyson 7839cec301
subsequent fix to edit in recent cherry-pick 2019-01-04 17:34:24 +10:00
Peter Dyson 7cc9754d94
fix to edit in recent cherry-pick 2019-01-04 17:26:42 +10:00
Andrei Stefan 3578e69669
SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless (#37024)
* Handle the bwc Joda ZonedDateTime scripting class in Painless

* Integrated the types checking in the already existent method
2019-01-04 09:19:24 +02:00
Peter Dyson 0ff2707c9f Add Profile API to search speed tuning howto (#29489)
* Add Profile API to search speed tuning howto

Seemed useful to mention the Profile API in the context of tuning for search speed.
2019-01-04 16:49:12 +10:00
Ioannis Kakavas 3c12c372d0
[DOCS] Adjust role mapping docs for SAML (#37083)
Explicitly mention that file based role mappings cannot be used with
the SAML realm.
2019-01-04 08:26:41 +02:00
Julie Tibshirani ac1c6940d2
Stop automatically nesting mappings in index creation requests. (#36924)
Now that we unwrap mappings in DocumentMapperParser#extractMappings, it is not
necessary for the mapping definition to always be nested under the type. This
leniency around the mapping format was added in 2341825358.
2019-01-03 17:41:28 -08:00
Tal Levy eaeccd8401
[ILM] Add Freeze Action (#36910)
This commit adds a new ILM Action for
freezing indices in the cold phase.

Closes #34630.
2019-01-03 15:00:40 -08:00
Costin Leau 40a30c6f5f
SQL: Preserve original source for each expression (#36912)
Improve parsing to save the source for each token alongside the location 
 of each Node/Expression for accurate reproducibility of an expression
 name and source

Fix #36894
2019-01-04 00:57:50 +02:00
Armin Braun 7686ee7631
TESTS: Shutdown ThreadPool after TestNodes (#37123)
* If the threadpool gets shut down before the testnodes we run into an error => fixed by moving to single `After` method
* Relates #36976
2019-01-03 22:35:44 +01:00
Benjamin Trent 2c3ce5b9de
[ML] add min_version field to machine learning results index mapping (#37115)
* ML: add min_version field to mapping

* Adding min_version as reserved field name
2019-01-03 14:59:42 -06:00
Igor Motov d6acd8e15f
Docs: add clarification about geohash use in geohashgrid agg (#36901)
Adds an example on translating geohashes returned by geohashgrid 
agg as bucket keys into geo bounding box filters in elasticsearch as well
as 3rd party applications.

Closes #36413
2019-01-03 15:40:48 -05:00
Julie Tibshirani 54f53d2a51
Make sure to accept empty unnested mappings in create index requests. (#37089) 2019-01-03 11:53:08 -08:00
Nick Knize e613bcae43
Remove XLatLonShape classes (#37094)
This commit removes local XLatLonShape classes and replaces with current LatLonShape classes in latest lucene snapshot
2019-01-03 12:48:36 -06:00
Daniel Mitterdorfer e4ec51879e
Retry JDK download when building Docker image
With this commit we instruct curl to retry with a backoff when
downloading the JDK for the Elasticsearch Docker image. This avoids
build failures on transient network issues. Note that this option
requires curl 7.12.3 or better.

Relates #37103
Relates #37113
2019-01-03 19:27:18 +01:00
Nicholas Knize de962b2f39 Revert "Adjust Lucene version for 6.7"
This reverts commit b7f6ee72a6.
2019-01-03 11:52:31 -06:00
Lisa Cawley 366b6b5e11
[DOCS] Adds cluster setting to enable/disable config migration (#36948) 2019-01-03 09:26:57 -08:00
David Kyle fc49d88b92
[ML] Adjust migration full cluster restart tests based on old cluster version #37118 2019-01-03 17:15:35 +00:00
David Roberts 13649aa70a
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
The problem that caused the test to be muted was fixed in
https://github.com/elastic/ml-cpp/pull/332

Closes #36258
2019-01-03 16:23:18 +00:00
Armin Braun 99f13b90d3
SNAPSHOT: Speed up HDFS Repository Writes (#37069)
* There is no point in hsyncing after every individual write since there is only value in completely written blobs for restores, this is ensures by the `SYNC` flag already and there is no need for separately invoking `hsync` after individual writes
2019-01-03 16:16:05 +01:00
Armin Braun 10d9819f99
Implement Atomic Blob Writes for HDFS Repository (#37066)
* Implement atomic writes the same way we do for the FsBlobContainer via rename which is atomic
* Relates #37011
2019-01-03 15:51:47 +01:00
Armin Braun 675ea4c59c
TESTS: Remove Static Threadpool in TaskManagerTest (#36976)
* The static threadpool leaks a lot of memory in these tests because it prevents things like the connect listeners from `org.elasticsearch.transport.TcpTransport#initiateConnection`
to be GCed between tests (since they keep being referenced by the threadpool) which in turn reference channels and their underlying buffers
  * I could not find any slowdown in executing these tests from this change, if anything they are slightly faster now on my machine
* Relates #36906 (which may be caused by slowness from leaking memory and also becomes testable in a loop by this change)
2019-01-03 15:19:21 +01:00
Benjamin Trent cfc310748d
addressing (#36891)(#36888)(#36889) (#37080) 2019-01-03 07:25:57 -06:00
Dimitris Athanasiou b04b3173db
[ML][TEST] Clean up max_model_memory_limit cluster setting (#37101)
Removes the `xpack.ml.max_model_memory_limit` cluster setting
at the teardown of the `ml_info.yml` tests to ensure the setting
does not trip other tests.
2019-01-03 15:15:31 +02:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg (#36933)
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: #36903
2019-01-03 13:55:09 +02:00
Christoph Büscher 046f86f274
Deprecate use of type in reindex request body (#36823)
Types can be used both in the source and dest section of the body which will
be translated to search and index requests respectively. Adding a deprecation warning
for those cases and removing examples using more than one type in reindex since
support for this is going to be removed.
2019-01-03 10:29:14 +01:00
Christoph Büscher e21054d176
Remove two unused methods in Iterables (#37075)
These helper methods are unused in the rest of the codebase.
2019-01-03 10:28:47 +01:00
Nhat Nguyen b7f6ee72a6 Adjust Lucene version for 6.7
Relates #37088
2019-01-03 04:20:47 -05:00
Jim Ferenczi 78ba1889cf
Replace the TreeMap in the composite aggregation (#36675)
The `composite` aggregation uses a TreeMap to keep track of the best buckets.
This ensures a log(n) time cost to insert new buckets but also to retrieve buckets
that are already present in the map. In order to speed up the retrieval of buckets
this change replaces the TreeMap with a priority queue and a HashMap. The insertion
cost is still log(n) but the retrieval of buckets through the HashMap is now done in constant
time. This optimization can bring significant improvement since each document needs
to check if its associated buckets are already present in the current best buckets.
2019-01-03 09:51:35 +01:00
David Turner a40f0545e6
Make ILM examples larger (#37039)
There are a handful of examples in the ILM documentation that could result in
rolling over indices more quickly than we might normally recommend,
contributing to over-sharding in cases where the examples are copied without
modification. This change makes some numbers bigger to try and avoid this.
2019-01-03 08:28:49 +00:00
Daniel Mitterdorfer 75f3443c62
Rename setting to enable mmap
With this commit we rename `node.store.allow_mmapfs` to
`node.store.allow_mmap`. Previously this setting has controlled whether
`mmapfs` could be used as a store type. With the introduction of
`hybridfs` which also relies on memory-mapping,
`node.store.allow_mmapfs` also applies to `hybridfs` and thus we rename
it in order to convey that it is actually used to allow memory-mapping
but not a specific store type.

Relates #36668
Relates #37070
2019-01-03 07:10:34 +01:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Ioannis Kakavas f0cd19f39b [TEST] Lower deleteInterval in TokenAuthIntegTests
We run subsequent token invalidation requests and we still want to
trigger the deletion of expired tokens so we need to lower the
deleteInterval parameter significantly. Especially now that the
bwc expiration logic is removed and the invalidation process is
much shorter

Resolves #37063
2019-01-02 23:54:02 +02:00
David Findley d4e7660248 Fix weighted_avg parser not found for RestHighLevelClient (#37027)
Add integration test for weighted avg sub aggregation
Add weighted avg parser to DefaultNamedXContents

Fixes #36861
2019-01-02 15:53:21 -06:00