Commit Graph

27037 Commits

Author SHA1 Message Date
Andrew Selden f8b15abe9a Update reference docs for geocentroid aggregation. (#24141)
This includes a link to the Wikipedia page explaining what a centroid
is.

Closes #24140
2017-04-17 21:27:43 -04:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster (#24142)
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Ulugbek Baymuradov 9cb477d387 Update filter-aggregation.asciidoc (#24138)
Fix a discrepancy between the example and the prose.
2017-04-17 18:46:13 -04:00
Ryan Ernst 1629c9fd5c S3 Repository: Cleanup deprecated settings (#24097)
This commit removes all deprecated settings which start with
`cloud.aws`, `repositories.s3` and repository level client settings.
2017-04-17 14:18:49 -07:00
Nik Everett 718e332c64 Docs: Be ok with long recovery times
The _cat docs were asserting that an index took only some number of
milliseconds to recovery. In this build it took a whole second:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-intake/192/consoleFull

So this changes the assertion to be ok with a second.
2017-04-17 16:56:12 -04:00
Nik Everett 9653551072 Add package test to catch leaking default.data.path setting (#24029)
Adds a packaging test that would have detected #23981. The test configures path.data and validates that nothing ends up in default.path.data.
2017-04-17 16:36:32 -04:00
Suhas Karanth f97d8bc78d Update reference docs for Highlighter fragmenter (#23754)
Explain the fragmenter and add examples.
2017-04-17 14:00:24 -04:00
Nik Everett 25119a7e78 Harden painless test against "fun" caching (#24077)
The JVM caches `Integer` objects. This is known. A test in Painless
was relying on the JVM not caching the particular integer `1000`.
It turns out that when you provide `-XX:+AggressiveOpts` the JVM
*does* cache `1000`, causing the test to fail when that is
specified.

This replaces `1000` with a randomly selected integer that we test
to make sure *isn't* cached by the JVM. *Hopefully* this test is
good enough. It relies on the caching not changing in between when
we check that the value isn't cached and when we run the painless
code. The cache now is a simple array but there is nothing
preventing it from changing. If it does change in a way that thwarts
this test then the test fail fail again. At least when that happens
the next person can see the comment about how it is important
that the integer isn't cached and can follow that line of inquiry.

Closes #24041
2017-04-17 13:44:05 -04:00
Jason Tedor 34eda1a1a8 Do not set path.data in environment if not set
When preparing the final settings in the environment, we unconditionally
set path.data even if path.data was not explicitly set. This confounds
detection for whether or not path.data was explicitly set, and this is
trappy. This commit adds logic to only set path.data in the final
settings if path.data was explicitly set, and provides a test case that
fails without this logic.

Relates #24132
2017-04-17 10:43:13 -04:00
Jason Tedor f7ebe9d18f Preserve multiple translog generations
Today when a flush is performed, the translog is committed and if there
are no outstanding views, only the current translog generation is
preserved. Yet for the purpose of sequence numbers, we need stronger
guarantees than this. This commit migrates the preservation of translog
generations to keep the minimum generation that would be needed to
recover after the local checkpoint.

Relates #24015
2017-04-17 08:51:54 -04:00
Jason Tedor 8033c576b7 Detect remnants of path.data/default.path.data bug
In Elasticsearch 5.3.0 a bug was introduced in the merging of default
settings when the target setting existed as an array. When this bug
concerns path.data and default.path.data, we ended up in a situation
where the paths specified in both settings would be used to write index
data. Since our packaging sets default.path.data, users that configure
multiple data paths via an array and use the packaging are subject to
having shards land in paths in default.path.data when that is very
likely not what they intended.

This commit is an attempt to rectify this situation. If path.data and
default.path.data are configured, we check for the presence of indices
there. If we find any, we log messages explaining the situation and fail
the node.

Relates #24099
2017-04-17 07:03:46 -04:00
jaymode a8be0a5836
Cat APIs should not close the stream obtained from the channel
The cat APIs and rest tables would obtain a stream from the RestChannel, which happened to be a
ReleasableBytesStreamOutput. These APIs used the stream to write content to, closed the stream,
and then tried to send a response. After #23941 was merged, closing the stream meant that the bytes
were released for use elsewhere. This caused occasional corruption of the response when the bytes
were used prior to the response being sent.

This commit changes these two usages to wrap the stream obtained from the channel in a flush on
close stream so that the bytes are still reserved until the message is sent.
2017-04-15 14:57:00 -04:00
Jason Tedor 51db6ab7dc Revert "Ensure output for createClone task"
This reverts commit 2620200ff7.
2017-04-15 13:54:01 -04:00
Jason Tedor cd8e059885 Do not produce empty IDs in simple versioning test
Empty IDs are rejected during indexing, so we should not randomly
produce them during tests. This commit modifies the simple versioning
tests to no longer produce empty IDs.
2017-04-15 12:15:45 -04:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates #24118
2017-04-15 10:36:03 -04:00
Boaz Leskes ecf81688fb Use sequence numbers to identify out of order delivery in replicas & recovery (#24060)
Internal indexing requests in Elasticsearch may be processed out of order and repeatedly. This is important during recovery and due to concurrency in replicating requests between primary and replicas. As such, a replica/recovering shard needs to be able to identify that an incoming request contains information that is old and thus need not be processed. The current logic is based on external version. This is sadly not sufficient. This PR moves the logic to rely on sequences numbers and primary terms which give the semantics we need.

Relates to #10708
2017-04-14 21:46:17 +02:00
Lee Hinman 162ce85ff2 Remove customization of ES_USER and ES_GROUP
This removes the ability to configure Elasticsearch to use custom username
and/or group when it is run.

Resolves #23848
2017-04-14 13:24:21 -06:00
Jason Tedor 09efdc3151 Improve performance of extracting warning value
When building headers for a REST response, we de-duplicate the warning
headers based on the actual warning value. The current implementation of
this uses a capturing regular expression that is prone to excessive
backtracking. In cases a request involves a large number of warnings,
this extraction can be a severe performance penalty. An example where
this can arise is a bulk indexing request that utilizes a deprecated
feature (e.g., using deprecated forms of boolean values). This commit is
an attempt to address this performance regression. We already know the
format of the warning header, so we do not need to use a regular
expression to parse it but rather can parse it by hand to extract the
warning value. This gains back the vast majority of the performance lost
due to the usage of a deprecated feature. There is still a performance
loss due to logging the deprecation message but we do not address that
concern in this commit.

Relates #24114
2017-04-14 12:18:00 -04:00
Lee Hinman 989da585b2
Go back to 140 column limit in .dir-locals.el 2017-04-14 08:50:53 -06:00
Jay Modi 30ab8739a6 Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23941)
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only
release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use
the channel builder and in turn the bytes stream output that is managed by the channel.

Note, this commit differs from 6bfecdf921 in that it updates
ReleasableBytesStreamOutput to handle the case of the BigArray decreasing in size, which changes
the reference to the BigArray. When the reference is changed, the releasable needs to be updated
otherwise there could be a leak of bytes and corruption of data in unrelated streams.

This reverts commit afd45c1432, which reverted #23572.
2017-04-14 10:50:31 -04:00
Yannick Welsch e3aa2a89f9 [TEST] Wait in OldIndexBackwardsCompatibilityIT for cluster to be fully initialized
There are test failures that suggest that the import of dangling indices is happening too early, before the dangling indices are ready to be consumed.
This commit adds an ensureGreen() at the end of cluster initialization to make sure that no cluster state updates are happening while the dangling
indices are prepared on-disk.
2017-04-14 11:02:55 +02:00
Yannick Welsch 7e581187a7 [TEST] Increase node startup timeouts
Tests are periodically failing on CI as the wait conditions are not met. This commit increases the timeouts.
2017-04-14 10:36:46 +02:00
Jason Tedor d4947c26ae Use jvm.options where needed in packaging tests
We have some packaging tests where we use a custom jvm.options file. The
flags we use here are barebones, just enough to exercise that using a
custom jvm.options files actually works. Due to this, we are missing a
Log4j flag that prevents Log4j from trying to use JMX. If Log4j tries to
use JMX, it hits a security manager exception and tries to log
this. This attempt to log happens before we've configured
logging. Previously, Elasticsearch was lenient here so this was treated
as harmless and the test could march on. Now, we fail startup if we
detect an attempt to log before logging is configured so this prevents
Elasticsearch from starting if we do not have jvm.options files in place
that prevent these log messages from being written before logging is
being configured. This commit adds jvm.options files in the places need
to prevent this.
2017-04-13 23:19:07 -04:00
Ali Beyad 5e54c0261a [TEST] fixes InternalTopHitsTests test to initialize the SearchHits
maxScore to Float.NaN if there is no max score, as that is what Lucene's
TopDocs does
2017-04-13 18:27:42 -04:00
Ryan Ernst 13489ff3a7 Test: Fix repro line for platformTest (#24095)
This was accidentally being added to packagingTest, which then had two
repro lines.
2017-04-13 14:17:11 -07:00
Ali Beyad 0afcaf5627 [TEST] fix BytesReference tests to never have a negative slice offset 2017-04-13 16:16:53 -04:00
Igor Motov cce321a560 Task Management: Make TaskInfo parsing forwards compatible (#24073)
TaskInfo is stored as a part of TaskResult and therefore can be read by nodes with an older version. If we add any additional information to TaskInfo (for #23250, for example), nodes with an older version should be able to ignore it, otherwise they will not be able to read TaskResults stored by newer nodes.
2017-04-13 16:16:01 -04:00
Tim Brooks ffaac5a08a Simplify BulkProcessor handling and retry logic (#24051)
This commit collapses the SyncBulkRequestHandler and
AsyncBulkRequestHandler into a single BulkRequestHandler. The new
handler executes a bulk request and awaits for the completion if the
BulkProcessor was configured with a concurrentRequests setting of 0.
Otherwise the execution happens asynchronously.

As part of this change the Retry class has been refactored.
withSyncBackoff and withAsyncBackoff have been replaced with two
versions of withBackoff. One method takes a listener that will be
called on completion. The other method returns a future that will been
complete on request completion.
2017-04-13 14:48:52 -05:00
Jason Tedor 99e0268e0a Remove support for default settings
Today Elasticsearch allows default settings to be used only if the
actual setting is not set. These settings are trappy, and the complexity
invites bugs. This commit removes support for default settings with the
exception of default.path.data, default.path.conf, and default.path.logs
which are maintainted to support packaging. A follow-up will remove
support for these as well.

Relates #24093
2017-04-13 14:25:45 -04:00
Jason Tedor 52c9159fff Remove awaits fix from evil JNA native tests
These tests were marked as awaits fix due to JNA requiring a version of
glibc greater than or equal to version 2.14. Since we still support
systems that would not have this version, we have released our own JNA
dependency that is built to support earlier versions of glibc. This
commit removes some await fixes that were added to tests that failed as
a result of this situation.
2017-04-13 06:53:02 -04:00
Jason Tedor 32b2caad42 Correct handling of default and array settings
In Elasticsearch 5.3.0 a bug was introduced in the merging of default
settings when the target setting existed as an array. This arose due to
the fact that when a target setting is an array, the setting key is
broken into key.0, key.1, ..., key.n, one for each element of the
array. When settings are replaced by default.key, we are looking for the
target key but not the target key.0. This leads to key, and key.0, ...,
key.n being present in the constructed settings object. This commit
addresses two issues here. The first is that we fix the merging of the
keys so that when we try to merge default.key, we also check for the
presence of the flattened keys. The second is that when we try to get a
setting value as an array from a settings object, we check whether or
not the backing map contains the top-level key as well as the flattened
keys. This latter check would have caught the first bug. For kicks, we
add some tests.

Relates #24074
2017-04-13 06:34:58 -04:00
Ryan Ernst fb3a281755 Build: Switch jna dependency to an elastic version (#24081)
This new version of jna is rebuilt from the official release of jna, but
with native libs linked against older glibc in order to support all
platforms elasticsearch supports.

closes #23640
2017-04-13 00:17:50 -07:00
Boaz Leskes 215a9b2df9 fix CategoryContextMappingTests compilation bugs 2017-04-13 09:15:10 +02:00
Boaz Leskes 342e745fc7 testConcurrentGetAndSetOnPrimary - fix a race condition between indexing and updating value map
Currently the map can be lagging behind what's actually in lucene causes assertions about adding/removing values to fail
2017-04-13 09:03:09 +02:00
Nilabh Sagar ec421974b9 Allow different data types for category in Context suggester (#23491)
The "category" in context suggester could be String, Number or Boolean. However with the changes in version 5 this is failing and only accepting String. This will have problem for existing users of Elasticsearch if they choose to migrate to higher version; as their existing Mapping and query will fail as mentioned in a bug #22358

This PR fixes the above mentioned issue and allows user to migrate seamlessly.

Closes #22358
2017-04-12 23:43:29 -07:00
Ryan Ernst c19044ddf6 Restrict build info loading to ES jar, not any jar (#24049)
This change makes the build info initialization only try to load a jar
manifest if it is the elasticsearch jar. Anything else (eg a repackaged
ES for use of transport client in an uber jar) will contain "Unknown"
for the build info as it does for tests currently.

fixes #21955
2017-04-12 23:22:43 -07:00
Jason Tedor 12b46bdbc4 Remove more hidden file leniency from plugins
This commit removes one more instance of leniency from the plugin
service which skips hidden files in the plugins directory.

Relates #23982
2017-04-12 22:23:42 -04:00
Jason Tedor edd16fa27e Register error listener in evil logger tests
This test needs an error listener registered since we configure logging
here.
2017-04-12 21:23:05 -04:00
Jason Tedor a1c2fe9e3a Detect using logging before configuration
It can easily happen that we touch a logger before logging is configured
due to chains of static intializers and other such scenarios. This
commit adds detection for this mechanism that will fail startup if we
touch a logger before logging is configured. This is a bug that will
cause builds to fail.

Relates #24076
2017-04-12 21:13:08 -04:00
debadair 7e1903469e [DOCS] Added note about Elastic Cloud to improve 'elastic aws' SERP results. 2017-04-12 17:57:03 -07:00
Nik Everett 31c8903492 Add version constant for 5.5 (#24075)
This is required in master now that #24071 is in or else we fail during BWC testing because the 5.x branch contains 5.5 but the build thinks it should contain 5.4.
2017-04-12 16:29:47 -04:00
Zachary Tong 1fd50bc54d Add unit tests for NestedAggregator (#24054)
Add unit tests for NestedAggregator, change class visibilities

Relates to #22278
2017-04-12 15:59:51 -04:00
Nik Everett e99f90fb46 Add more debugging information to rethrottles
I'm still trying to track down failures like:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/1180/console

It looks like a task is hanging but I'm not sure why. So this
adds more logging for next time.
2017-04-12 08:37:31 -04:00
Christoph Büscher 1847bbac4d Tests: Use random analyzer only on string fields in Match/MultiMatchBuilderTests
Currently we can run into test errors by accidently using e.g. a "simple"
analyzer on a numeric field which might lead to number parsing errors. While
these errors are correct, we should avoid these combinations in our regular
tests.
2017-04-12 11:32:48 +02:00
Ryan Ernst 7cebc0fc93 Cleanup outdated comments for fixing up pom dependencies (#24056) 2017-04-11 23:15:30 -07:00
Ryan Ernst 1207103b6d S3 Repository: Eagerly load static settings (#23910)
The S3 repostiory has many levels of settings it looks at to create a
repository, and these settings were read at repository creation time.
This meant secure settings like access and secret keys had to be
available after node construction. This change makes setting loading for
every except repository level settings eager, so that secure settings
can be stashed, and the keystore can once again be closed after
bootstrapping the node is complete.
2017-04-11 15:42:56 -07:00
Jason Tedor b4c3bb5d21 Reject duplicate settings on the command line
Today Elasticsearch and other CLI tools that rely on environment aware
command leniently accept duplicate settings with the last one
winning. This commit removes this leniency.

Relates #24053
2017-04-11 18:30:05 -04:00
Tim Brooks cf6b03c8f4 Wildcard cluster names for cross cluster search (#23985)
This is related to #23893. This commit allows users to use wilcards for
cluster names when executing a cross cluster search.

So instead of defining every cluster such as:

GET one:*,two:*,three:*/_search

A user could just search:

GET *:*/_search

As ":" characters are currently allowed in index names, if the text
up to the first ":" does not match a defined cluster name, the entire
string is treated as an index name.
2017-04-11 13:56:26 -05:00
Nik Everett c17bee571f Update scripts/security docs for sandboxed world (#23977)
Drops any mention of non-sandboxed scripting languages other than a
brief "we don't support them and we shouldn't because A and B"
statement.

Relates to #23930
2017-04-11 14:55:57 -04:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00