Commit Graph

27020 Commits

Author SHA1 Message Date
Jason Tedor 09efdc3151 Improve performance of extracting warning value
When building headers for a REST response, we de-duplicate the warning
headers based on the actual warning value. The current implementation of
this uses a capturing regular expression that is prone to excessive
backtracking. In cases a request involves a large number of warnings,
this extraction can be a severe performance penalty. An example where
this can arise is a bulk indexing request that utilizes a deprecated
feature (e.g., using deprecated forms of boolean values). This commit is
an attempt to address this performance regression. We already know the
format of the warning header, so we do not need to use a regular
expression to parse it but rather can parse it by hand to extract the
warning value. This gains back the vast majority of the performance lost
due to the usage of a deprecated feature. There is still a performance
loss due to logging the deprecation message but we do not address that
concern in this commit.

Relates #24114
2017-04-14 12:18:00 -04:00
Lee Hinman 989da585b2
Go back to 140 column limit in .dir-locals.el 2017-04-14 08:50:53 -06:00
Jay Modi 30ab8739a6 Closing a ReleasableBytesStreamOutput closes the underlying BigArray (#23941)
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only
release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use
the channel builder and in turn the bytes stream output that is managed by the channel.

Note, this commit differs from 6bfecdf921 in that it updates
ReleasableBytesStreamOutput to handle the case of the BigArray decreasing in size, which changes
the reference to the BigArray. When the reference is changed, the releasable needs to be updated
otherwise there could be a leak of bytes and corruption of data in unrelated streams.

This reverts commit afd45c1432, which reverted #23572.
2017-04-14 10:50:31 -04:00
Yannick Welsch e3aa2a89f9 [TEST] Wait in OldIndexBackwardsCompatibilityIT for cluster to be fully initialized
There are test failures that suggest that the import of dangling indices is happening too early, before the dangling indices are ready to be consumed.
This commit adds an ensureGreen() at the end of cluster initialization to make sure that no cluster state updates are happening while the dangling
indices are prepared on-disk.
2017-04-14 11:02:55 +02:00
Yannick Welsch 7e581187a7 [TEST] Increase node startup timeouts
Tests are periodically failing on CI as the wait conditions are not met. This commit increases the timeouts.
2017-04-14 10:36:46 +02:00
Jason Tedor d4947c26ae Use jvm.options where needed in packaging tests
We have some packaging tests where we use a custom jvm.options file. The
flags we use here are barebones, just enough to exercise that using a
custom jvm.options files actually works. Due to this, we are missing a
Log4j flag that prevents Log4j from trying to use JMX. If Log4j tries to
use JMX, it hits a security manager exception and tries to log
this. This attempt to log happens before we've configured
logging. Previously, Elasticsearch was lenient here so this was treated
as harmless and the test could march on. Now, we fail startup if we
detect an attempt to log before logging is configured so this prevents
Elasticsearch from starting if we do not have jvm.options files in place
that prevent these log messages from being written before logging is
being configured. This commit adds jvm.options files in the places need
to prevent this.
2017-04-13 23:19:07 -04:00
Ali Beyad 5e54c0261a [TEST] fixes InternalTopHitsTests test to initialize the SearchHits
maxScore to Float.NaN if there is no max score, as that is what Lucene's
TopDocs does
2017-04-13 18:27:42 -04:00
Ryan Ernst 13489ff3a7 Test: Fix repro line for platformTest (#24095)
This was accidentally being added to packagingTest, which then had two
repro lines.
2017-04-13 14:17:11 -07:00
Ali Beyad 0afcaf5627 [TEST] fix BytesReference tests to never have a negative slice offset 2017-04-13 16:16:53 -04:00
Igor Motov cce321a560 Task Management: Make TaskInfo parsing forwards compatible (#24073)
TaskInfo is stored as a part of TaskResult and therefore can be read by nodes with an older version. If we add any additional information to TaskInfo (for #23250, for example), nodes with an older version should be able to ignore it, otherwise they will not be able to read TaskResults stored by newer nodes.
2017-04-13 16:16:01 -04:00
Tim Brooks ffaac5a08a Simplify BulkProcessor handling and retry logic (#24051)
This commit collapses the SyncBulkRequestHandler and
AsyncBulkRequestHandler into a single BulkRequestHandler. The new
handler executes a bulk request and awaits for the completion if the
BulkProcessor was configured with a concurrentRequests setting of 0.
Otherwise the execution happens asynchronously.

As part of this change the Retry class has been refactored.
withSyncBackoff and withAsyncBackoff have been replaced with two
versions of withBackoff. One method takes a listener that will be
called on completion. The other method returns a future that will been
complete on request completion.
2017-04-13 14:48:52 -05:00
Jason Tedor 99e0268e0a Remove support for default settings
Today Elasticsearch allows default settings to be used only if the
actual setting is not set. These settings are trappy, and the complexity
invites bugs. This commit removes support for default settings with the
exception of default.path.data, default.path.conf, and default.path.logs
which are maintainted to support packaging. A follow-up will remove
support for these as well.

Relates #24093
2017-04-13 14:25:45 -04:00
Jason Tedor 52c9159fff Remove awaits fix from evil JNA native tests
These tests were marked as awaits fix due to JNA requiring a version of
glibc greater than or equal to version 2.14. Since we still support
systems that would not have this version, we have released our own JNA
dependency that is built to support earlier versions of glibc. This
commit removes some await fixes that were added to tests that failed as
a result of this situation.
2017-04-13 06:53:02 -04:00
Jason Tedor 32b2caad42 Correct handling of default and array settings
In Elasticsearch 5.3.0 a bug was introduced in the merging of default
settings when the target setting existed as an array. This arose due to
the fact that when a target setting is an array, the setting key is
broken into key.0, key.1, ..., key.n, one for each element of the
array. When settings are replaced by default.key, we are looking for the
target key but not the target key.0. This leads to key, and key.0, ...,
key.n being present in the constructed settings object. This commit
addresses two issues here. The first is that we fix the merging of the
keys so that when we try to merge default.key, we also check for the
presence of the flattened keys. The second is that when we try to get a
setting value as an array from a settings object, we check whether or
not the backing map contains the top-level key as well as the flattened
keys. This latter check would have caught the first bug. For kicks, we
add some tests.

Relates #24074
2017-04-13 06:34:58 -04:00
Ryan Ernst fb3a281755 Build: Switch jna dependency to an elastic version (#24081)
This new version of jna is rebuilt from the official release of jna, but
with native libs linked against older glibc in order to support all
platforms elasticsearch supports.

closes #23640
2017-04-13 00:17:50 -07:00
Boaz Leskes 215a9b2df9 fix CategoryContextMappingTests compilation bugs 2017-04-13 09:15:10 +02:00
Boaz Leskes 342e745fc7 testConcurrentGetAndSetOnPrimary - fix a race condition between indexing and updating value map
Currently the map can be lagging behind what's actually in lucene causes assertions about adding/removing values to fail
2017-04-13 09:03:09 +02:00
Nilabh Sagar ec421974b9 Allow different data types for category in Context suggester (#23491)
The "category" in context suggester could be String, Number or Boolean. However with the changes in version 5 this is failing and only accepting String. This will have problem for existing users of Elasticsearch if they choose to migrate to higher version; as their existing Mapping and query will fail as mentioned in a bug #22358

This PR fixes the above mentioned issue and allows user to migrate seamlessly.

Closes #22358
2017-04-12 23:43:29 -07:00
Ryan Ernst c19044ddf6 Restrict build info loading to ES jar, not any jar (#24049)
This change makes the build info initialization only try to load a jar
manifest if it is the elasticsearch jar. Anything else (eg a repackaged
ES for use of transport client in an uber jar) will contain "Unknown"
for the build info as it does for tests currently.

fixes #21955
2017-04-12 23:22:43 -07:00
Jason Tedor 12b46bdbc4 Remove more hidden file leniency from plugins
This commit removes one more instance of leniency from the plugin
service which skips hidden files in the plugins directory.

Relates #23982
2017-04-12 22:23:42 -04:00
Jason Tedor edd16fa27e Register error listener in evil logger tests
This test needs an error listener registered since we configure logging
here.
2017-04-12 21:23:05 -04:00
Jason Tedor a1c2fe9e3a Detect using logging before configuration
It can easily happen that we touch a logger before logging is configured
due to chains of static intializers and other such scenarios. This
commit adds detection for this mechanism that will fail startup if we
touch a logger before logging is configured. This is a bug that will
cause builds to fail.

Relates #24076
2017-04-12 21:13:08 -04:00
debadair 7e1903469e [DOCS] Added note about Elastic Cloud to improve 'elastic aws' SERP results. 2017-04-12 17:57:03 -07:00
Nik Everett 31c8903492 Add version constant for 5.5 (#24075)
This is required in master now that #24071 is in or else we fail during BWC testing because the 5.x branch contains 5.5 but the build thinks it should contain 5.4.
2017-04-12 16:29:47 -04:00
Zachary Tong 1fd50bc54d Add unit tests for NestedAggregator (#24054)
Add unit tests for NestedAggregator, change class visibilities

Relates to #22278
2017-04-12 15:59:51 -04:00
Nik Everett e99f90fb46 Add more debugging information to rethrottles
I'm still trying to track down failures like:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/1180/console

It looks like a task is hanging but I'm not sure why. So this
adds more logging for next time.
2017-04-12 08:37:31 -04:00
Christoph Büscher 1847bbac4d Tests: Use random analyzer only on string fields in Match/MultiMatchBuilderTests
Currently we can run into test errors by accidently using e.g. a "simple"
analyzer on a numeric field which might lead to number parsing errors. While
these errors are correct, we should avoid these combinations in our regular
tests.
2017-04-12 11:32:48 +02:00
Ryan Ernst 7cebc0fc93 Cleanup outdated comments for fixing up pom dependencies (#24056) 2017-04-11 23:15:30 -07:00
Ryan Ernst 1207103b6d S3 Repository: Eagerly load static settings (#23910)
The S3 repostiory has many levels of settings it looks at to create a
repository, and these settings were read at repository creation time.
This meant secure settings like access and secret keys had to be
available after node construction. This change makes setting loading for
every except repository level settings eager, so that secure settings
can be stashed, and the keystore can once again be closed after
bootstrapping the node is complete.
2017-04-11 15:42:56 -07:00
Jason Tedor b4c3bb5d21 Reject duplicate settings on the command line
Today Elasticsearch and other CLI tools that rely on environment aware
command leniently accept duplicate settings with the last one
winning. This commit removes this leniency.

Relates #24053
2017-04-11 18:30:05 -04:00
Tim Brooks cf6b03c8f4 Wildcard cluster names for cross cluster search (#23985)
This is related to #23893. This commit allows users to use wilcards for
cluster names when executing a cross cluster search.

So instead of defining every cluster such as:

GET one:*,two:*,three:*/_search

A user could just search:

GET *:*/_search

As ":" characters are currently allowed in index names, if the text
up to the first ":" does not match a defined cluster name, the entire
string is treated as an index name.
2017-04-11 13:56:26 -05:00
Nik Everett c17bee571f Update scripts/security docs for sandboxed world (#23977)
Drops any mention of non-sandboxed scripting languages other than a
brief "we don't support them and we shouldn't because A and B"
statement.

Relates to #23930
2017-04-11 14:55:57 -04:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Nik Everett 76603d4413 Move line split in eclipse back to 140 characters
Now that we've moved checkstyle back to 140 we should move the
split.
2017-04-11 13:23:03 -04:00
Boaz Leskes d5512d6319 Packaging: Get more info when `[INIT.D] don't mkdir when it contains a comma` fails 2017-04-11 15:15:51 +02:00
Dmitriy Troyan 1ea4c285b6 Update aggregation.asciidoc (#24042)
There are four kinds of aggregations now, not three.
2017-04-11 09:02:38 -04:00
Simon Willnauer e30a275bfe Add a dedicated TransportRemoteInfoAction for consistency (#24040)
All our actions that are invoked from rest actions have corresponding
transport actions. This adds the transport action for RestRemoteClusterInfoAction
for consistency.

Relates to #23969
2017-04-11 14:40:37 +02:00
Simon Willnauer 42e0b4f5e9 [TEST] Enable new REST test for 5.4 and BWC against 5.4.x 2017-04-11 13:30:45 +02:00
Jason Tedor 653619079c Skip two Painless branch tests on Windows
This commit skips the two Painless tests
EqualsTests#testBranchEqualsDefAndPrimitive and
EqualsTests#testBranchNotEqualsDefAndPrimitive on Windows as the tests
are repeatedly failing there.
2017-04-11 06:19:42 -04:00
Yannick Welsch 88a54f14c7 Trigger replica recovery restarts by master when primary relocation completes (#23926)
When a primary relocation completes while there are ongoing replica recoveries, the recoveries for these replicas need to be restarted (as a new primary is in charge of replicating changes). Before this commit, the need for a recovery restart was detected by the data nodes that had the replicas, by checking on each cluster state update if the recovery process had completed before the recovery source changed. That code had a race, however, which could lead to a not-fully recovered shard exposing itself as started (see #23904).

This commit takes a different approach: When the primary relocation completes and the master updates the cluster state to move the primary shard from relocating to started, it will reinitialize all initializing replica shards, by giving them a fresh allocation id. Data nodes that have the replica shard will simply detect that the allocation id changed and restart the recovery process (instead of trying to determine the need to restart based on ongoing recoveries).

Note: Removal of the code in IndicesClusterStateService that checks whether the recovery source has changed will not be backported to the 5.x branch. This ensures backward compatibility for the situation where the master node is older and does not have the code changes that have been introduced in this PR.

Closes #23904
2017-04-11 11:21:57 +02:00
Colin Goodheart-Smithe 0114f0061c Removes version 2.x constants from Version (#24011)
* Removes version 2.x constants from Version

Closes #21887

* Addresses review comments
2017-04-11 08:31:22 +01:00
Simon Willnauer f22e0dc30b Add cross-cluster search remote cluster info API (#23969)
This commit adds an API to discover information like seed nodes,
http addresses and connection status of a configured remote cluster.

Closes #23925
2017-04-11 09:24:40 +02:00
Ryan Ernst b0003edcf4 Build: Support offline build of bwc zip (#24037)
The bwc checkout for backcompat tests currently always tries to fetch
the latest from the upstream remote. This change makes fetching from
upstream conditional on not running an offline build.
2017-04-10 20:04:18 -07:00
Nik Everett 16a2048416 Remove real time from tests (#24025)
The `AsyncBulkByScrollActionTests` were brittle because they used the
current time. That was a mistake. This removes the current time from
the test, instead adding it to the parameters passed in to the
appropriate methods. This means that we take the current time slightly
earlier in all cases, but that shouldn't make a difference.

Closes #24005

Example failure:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+nfs/161/consoleFull
2017-04-10 17:55:02 -04:00
Ryan Ernst dd3c1137a4 Repository S3: Simplify client method (#24034)
This commit removes passing the repository metadata object through to
s3 client creation. It is not needed, and in fact in tests was confusing
because you could create the metadata but have it contain different
settings than were passed in as repository settings.
2017-04-10 14:43:34 -07:00
Ryan Ernst 65f7a76630 Settings: Add secure file setting to keystore (#24001)
Some systems like GCE rely on a plaintext file containing credentials.
Rather than extract the information out of that credentials file and
store each peace individually in the keystore, it is cleaner to just
store the entire file.

This commit adds support to the keystore wrapper for secure file
settings. These are settings that contain an entire file that would
normally be stored on the local filesystem. Retrieving the file returns
an input stream to the file contents. This also adds a `add-file`
command to the keystore cli.

In order to support both strings and files as values for settings, the
metadata format of the keystore has also been updated (with backcompat)
to keep a map of setting name to type.
2017-04-10 13:10:42 -07:00
Simon Willnauer a61fb3f708 Remote support for lucene versions without checksums (#24021)
We are still carrying some legacy code that deals with lucene indices
that don't have checksums. Yet, we do not support these indices
for a while now, in fact since version 5.0 such an index is not supported
anymore. This commit removes all the special handling and leniency involved.
2017-04-10 18:16:34 +02:00
Jay Modi 42b0b05af1 Test: add support for replacing stashed values within headers of REST tests (#24014)
This commit adds support for replacing a stashed value within a header of a REST test. This is
useful for requests that may want to use a value previously obtained within a header.
2017-04-10 12:10:01 -04:00
Martijn van Groningen 887f3ed8dc
inner_hits: Replace `NestedChildrenQuery` with `ParentChildrenBlockJoinQuery`.
Closes #24009
2017-04-10 17:36:45 +02:00
Lee Hinman 53d4d747a6 Mark IndexWithShadowReplicasIT as AwaitsFix
Relates to #24007 and #23906
2017-04-10 09:32:20 -06:00