Commit Graph

47229 Commits

Author SHA1 Message Date
Przemyslaw Gomulka fe20e217a4
Deprecation messages with the same key but different x-opaque-id are allowed backport(#44587) #44682
Deprecation logger was filtering log entries by key, that means that if two log messages with the same key are logged from different users, then the second log messages will be filtered.
This change allows to log deprecation message with the same key by different users.

relates #41354
backport #44587
2019-07-22 11:38:11 +02:00
Armin Braun a6adcecd20 Fix Tring to Mutate Immutable Collections
Fixes two spots where #44665 caused a previously mutable collection to now be read as an immutable one, leading to errors
2019-07-22 11:04:05 +02:00
Armin Braun b9067ba1ba
Remove Needless Synchronization in FollowersChecker (#44631) (#44680)
* It seems redundant to synchronize here and check that the map hasn't checked via the `isRunning` under the mutex
* The map won't change if under the mutex that locks on all the updates to it
* Without the mutex it's very unlikely to change inside the method call relative to the likelihood of changing until the generic pool where we check for `isRunning` again anyway

-> just remove the synchronization (it's on the IO loop) and check since we do check the running state on the generic pool under the mutex anyway when we actually fail it
2019-07-22 10:57:30 +02:00
Jason Tedor ff76b0af8b
Copy field names in stored fields context
We have to copy the field names otherwise we either have a handle of a
list that a caller might mutate or we might mutate when they aren't
expecting it, or worse, a handle of a list that is not mutable (and we
end up mutating the list).

Relates #44665
2019-07-22 17:40:07 +09:00
Alpar Torok b34ac66d96
Mute multiple tests on Windows (7.x) (#44676)
* Mute failing test

tracked in #44552

* mute EvilSecurityTests

tracking in #44558

* Fix line endings in ESJsonLayoutTests

* Mute failing ForecastIT  test on windows

Tracking in #44609

* mute BasicRenormalizationIT.testDefaultRenormalization

tracked in #44613

* fix mute testDefaultRenormalization

* Increase busyWait timeout windows is slow

* Mute failure unconfigured node name

* mute x-pack internal cluster test windows

tracking #44610

* Mute JvmErgonomicsTests on windows

Tracking #44669

* mute SharedClusterSnapshotRestoreIT testParallelRestoreOperationsFromSingleSnapshot

Tracking #44671

* Mute NodeTests on Windows

Tracking #44256
2019-07-22 11:32:29 +03:00
Jason Tedor d82a570a2a
Reomve debugging loging statements from Azure tests
This commit removes some unneeded debugging logging statements from the
Azure storage tests.

Relates #44672
2019-07-22 16:55:55 +09:00
Jason Tedor a493a34143
Use debug logging instead for Azure tests (#44672)
These Azure tests have hard println statements which means we always see
these messages during configuration. Yet, there are unnecessary most of
the time. This commit changes them to use debug logging.
2019-07-22 16:46:13 +09:00
Armin Braun 0e2e83f591
More Efficient Deserialization of Empty Collections in StreamInput (#44665) (#44674)
* We only had the `size == 0` optimization in some but not all spots of deserializing collections in this class, fixed the remaining spots.
* Also fixed the a similar spot when deserializing `ThreadContextStruct` that could now be simplified (it was apparently doing it's own version of this optimization for the first map it deserialized before ... but not for the second map -> made it not instantiate anything if both maps are empty since it's always the same object here anyway)
2019-07-22 09:31:12 +02:00
Armin Braun 0ac137a9a1
Optimize some StreamOutput Operations (#44660) (#44668)
* Optimize some StreamOutput Operations

* Writing numbers byte by byte adds a lot of unnecessary bounds checks to serialization
* Serializing to a threadlocal `byte[]` instead and bulk writing gives about a 50% speedup on `long` and `vlong` (for large numbers) writes and 30% for `int`, `vint` on Linux on an i9
* Using a threadlocal of the maximum string buffer size we used to allocate before also removes allocations when writing strings in general since we now never have to allocate a `byte[]` for that
   * And don't have to GC one either resolving the TODO removed here
2019-07-22 07:09:32 +02:00
Ryan Ernst 226a753e93
Restore setting up temp dir for windows service (#44541) (#44661)
In https://github.com/elastic/elasticsearch/pull/41913 setting up the
temp dir for ES was moved from the env script to individual cli scripts.
However, moving it to the windows service cli was missed. This commit
restores setting up the temp dir for the windows service control script.
2019-07-21 13:54:46 -07:00
Tal Levy 1a9cfe9110
Removal Streamable (#44647) (#44655)
This commit ends the grand adventure that was the
refactoring effort to migrate all usages of
Streamable to Writeable.

Closes #34389.
2019-07-20 19:10:49 -07:00
Jason Tedor cdd06d40d2
Do not checksum all bytes at once in plugin install (#44649)
Today when checksumming a plugin zip during plugin install, we read all
of the bytes of the zip into memory at once. When trying to run the
plugin installer on a small heap (say, 64 MiB), this can lead to the
plugin installer running out of memory when checksumming large
plugins. This commit addresses this by reading the plugin bytes in 8 KiB
chunks, thus using a constant amount of memory independent of the size
of the plugin.
2019-07-21 07:24:23 +09:00
Ryan Ernst 4c05d25ec7
Convert Transport Request/Response to Writeable (#44636) (#44654)
This commit converts all remaining TransportRequest and
TransportResponse classes to implement Writeable, and disallows
Streamable implementations.

relates #34389
2019-07-20 11:25:58 -07:00
Ryan Ernst f4ee2e9e91
Convert direct implementations of Streamable to Writeable (#44605) (#44646)
This commit converts Streamable to Writeable for direct implementations.

relates #34389
2019-07-20 08:32:29 -07:00
Jason Tedor 1f7fc1b497
Add default CLI JVM options (#44545)
This commit adds some default CLI JVM options to control the heap size
and the garbage collector used for the CLI tools. We do this because
otherwise the JVM will default to large initial and max heap sizes based
on the RAM visible to the JVM (which could be all the physical RAM on
the machine if not run in a container-aware JVM). This commit therefore
sets the initial heap size to 4m, the max heap size to 64m, the garbage
collector to the serial collector, and leaves this user-configurable by
honoring ES_JAVA_OPTS last.
2019-07-20 09:30:13 +09:00
Tal Levy 7c84636029
Remove StreamOutput #writeOptionalStreamable and #writeStreamableList (#44602) (#44643)
remove usages of writeOptionalStreamable and writeStreambaleList

relates #34389.
2019-07-19 15:55:53 -07:00
Benjamin Trent 2e303fc5f7
[ML][Data Frame] adding dynamic cluster setting for failure retries (#44577) (#44639)
This adds a new dynamic cluster setting `xpack.data_frame.num_transform_failure_retries`.

This setting indicates how many times non-critical failures should be retried before a data frame transform is marked as failed and should stop executing. At the time of this commit; Min: 0, Max: 100, Default: 10
2019-07-19 16:17:39 -05:00
Ryan Ernst f193d14764
Convert remaining Action Response/Request to writeable.reader (#44528) (#44607)
This commit converts readFrom to ctor with StreamInput on the remaining
ActionResponse and ActionRequest classes.

relates #34389
2019-07-19 13:33:38 -07:00
James Rodewig a63f60b776 [DOCS] Remove heading offsets for REST APIs (#44568)
Several files in the REST APIs nav section are included using
:leveloffset: tags. This increments headings (h2 -> h3, h3 -> h4, etc.)
in those files and removes the :leveloffset: tags.

Other supporting changes:
* Alphabetizes top-level REST API nav items.
* Change 'indices APIs' heading to 'index APIs.'
* Changes 'Snapshot lifecycle management' heading to sentence case.
2019-07-19 14:36:06 -04:00
Armin Braun f028ab43ad
Don't Swallow Interrupt in TransportService#onRequestReceived (#44622) (#44627)
* We shouldn't just swallow the interrupt here quietly and keep going on the IO thread
   * Currently interrupt continues here just the same way an invocation of `acceptIncomingRequests` woudl have made things continue
* Relates #44610
2019-07-19 20:35:29 +02:00
Christoph Büscher eafe54c81c Fix AnalysisMode propagation in NamedAnalyzer (#44626)
NamedAnalyzer should return the same AnalysisMode than any custom analyzer it
wraps, otherwise AnalysisMode.ALL. This used to be only CustomAnalyzer in the
past, but with the introduction of the ReloadableCustomAnalyzer this needs to be
added as an option where the analysis mode gets propagated.

Closes #44625
2019-07-19 18:18:43 +02:00
David Turner c3b6ab4dec Fix docs line length
Followup to #44617 that fixes some unexpected wrapping.
2019-07-19 17:04:42 +01:00
David Turner 87f246f6bd Clarify repository-s3 defaulting to instance role (#44617)
Specifies more clearly that you can leave the client credential settings unset
if you want to use the instance or container role.
2019-07-19 16:11:23 +01:00
James Rodewig 1f5a2ac3ad [DOCS] Reformat `parent_id` query docs (#44449) 2019-07-19 11:03:45 -04:00
Nikita Glashenko 804476c35d Remove support for old translog checkpoint formats (#44280)
This commit removes support for the translog checkpoint format from versions
before 6.0.0 since 7.x versions are incompatible with indices from these
versions.

Relates #44720
Fixes #44210
2019-07-19 16:01:47 +01:00
James Rodewig 51aefbdd3d [DOCS] Reformat `has_parent` query docs (#44443) 2019-07-19 10:51:11 -04:00
James Rodewig d46545f729 [DOCS] Update anchors and links for Elasticsearch API relocation (#44500) 2019-07-19 09:18:23 -04:00
James Rodewig 76c7e3a05f [DOCS] Replace `_meta` with `metadata` for snapshot APIs. (#44596)
elastic/elasticsearch#41281 added custom metadata parameter to
snapshots. During review, the parameter name was changed from '_meta' to
'metadata,' but the documentation wasn't updated. This corrects the
documentation to use the 'metadata' name.
2019-07-19 08:40:57 -04:00
Przemyslaw Gomulka 597d2dfaf5
Add types field to slow logs in 7.x (#44592)
By mistake in 7.x types field was removed from slow logs. Types are
still present in that version, so this have to be present as a JSON
field
relates #41354
backport that was causing this #44178
2019-07-19 08:31:00 +02:00
Ryan Ernst 60785a9fa8
Convert several direct uses of Streamable to Writeable (#44586) (#44604)
This commit converts several utility classes that implement Streamable
to have StreamInput constructors. It also adds a default version of
readFrom to Streamable so that overriding to throw UOE is not necessary.

relates #34389
2019-07-18 21:25:44 -07:00
Julie Tibshirani 336364fefe
Convert more classes in 'server' to Writeable. (#44600)
* Convert GetTask*.
* Convert RemoteInfo*.
* Convert GetFieldMappings*.
* Convert ValidateQueryRequest*.
* Convert MainResponse*.
* Convert MultiGet*.
* Convert Update*.
* Add a missing call to parent constructors.

Relates to #34389.
2019-07-18 18:45:10 -07:00
Ryan Ernst 13f46aa801
Convert index and persistent actions/response to writeable (#44582) (#44601)
This commit converts several more classes from streamable to writeable
in server, mostly within the o.e.index and o.e.persistent packages.

relates #34389
2019-07-18 18:32:09 -07:00
Tal Levy 03f5084ac7
remove usages of #readOptionalStreamable, #readStreamableList. (#44578) (#44598)
This commit removes references to Streamable from StreamInput.

This is all a part of the effort to remove Streamable usage.

relates #34389.
2019-07-18 16:19:02 -07:00
Lisa Cawley 5a05bd76b4 [DOCS] Reformat info and deprecation APIs to use new API format (#44599) 2019-07-18 16:11:48 -07:00
Lee Hinman 3001f7941f Allow empty configuration for SLM policies (#44465)
* Allow empty configuration for SLM policies

When putting or updating a snapshot lifecycle policy it was not possible
to elide the `config` map. This commit makes the configuration optional,
the same way that it is when taking a snapshot.

Relates to #38461

* Add Objects.requireNonNull for required parts of the policy
2019-07-18 16:20:31 -06:00
Lee Hinman fe2ef66e45 Expose index age in ILM explain output (#44457)
* Expose index age in ILM explain output

This adds the index's age to the ILM explain output, for example:

```
{
  "indices" : {
    "ilm-000001" : {
      "index" : "ilm-000001",
      "managed" : true,
      "policy" : "full-lifecycle",
      "lifecycle_date" : "2019-07-16T19:48:22.294Z",
      "lifecycle_date_millis" : 1563306502294,
      "age" : "1.34m",
      "phase" : "hot",
      "phase_time" : "2019-07-16T19:48:22.487Z",
      ... etc ...
    }
  }
}
```

This age can be used to tell when ILM will transition the index to the
next phase, based on that phase's `min_age`.

Resolves #38988

* Expose age in getters and in HLRC
2019-07-18 15:33:45 -06:00
Ryan Ernst af093a4095
Convert ShardOperationFailedException to Writeable (#44532) (#44580)
This commit converts subclasses of ShardOperationFailedException to
implement ctors with StreamInput instead of readFrom. It also simplifies
IndicesShardStoresResponse.Failure to serialize its shardId after the
super data.

relates #34389
2019-07-18 13:29:19 -07:00
Benjamin Trent 3477f5ae04
muting test testBulkIndexFailuresCauseTaskToFail (#44594) 2019-07-18 15:03:50 -05:00
Benjamin Trent d5ca72740e
[ML][Data Frame] adjust onFinish audit frequency (#44450) (#44508) 2019-07-18 14:28:34 -05:00
Marios Trivyzas 8745d38dc8
Compile subset of build-tools classes with Java 8 target compatibility (#44560)
Backport of #43177 so that VersionProperties is Java 8 compatible and
can be used by https://github.com/elastic/elasticsearch-hadoop
to retrieve snapshot versions for Lucene.

(cherry picked from commit ec3ac9b62452f04ce44dea0a904a6e2b31dd8076)
2019-07-18 21:15:56 +03:00
Armin Braun 3b5038b837
Implement Eventually Consistent Mock Repository for SnapshotResiliencyTests (#40893) (#44570)
* Add eventually consistent mock repository for reproducing and testing AWS S3 blob store behavior
* Relates #38941
2019-07-18 17:54:54 +02:00
Lisa Cawley 8445c41004
[DOCS] Moves content to ML anomaly-detection folder (#44520) (#44530) 2019-07-18 08:44:52 -07:00
Tal Levy c8a8915b27
migrate rollup/monitoring/graph/watcher actions to Writeable (#44464) (#44538)
this commit migrates leftover actions from a few x-pack plugins
to the new Writeable.Reader infrastructure.

relates #34389.
2019-07-18 08:42:56 -07:00
Dimitrios Liappis 651b7a001f
Support log file rolling in jvm presence packaging tests (#44548)
Make JVM presence related packaging tests more resilient, by
also checking for entries in rolled over log files.

Closes #41436
Backport of #44423
2019-07-18 18:39:55 +03:00
Andrey Ershov ef6ddd15c6 Revert "Snapshot tool: S3 orphaned files
cleanup (#44551)"

This reverts commit 09edeeb3
2019-07-18 17:21:45 +02:00
Lisa Cawley 213af8411f [DOCS] Fixes query default value (#44572) 2019-07-18 08:18:58 -07:00
Nhat Nguyen 47cfc25fa0 Skip update if leader and follower settings identical (#44535)
If the setting on the follower and the leader are identical after
filtering out private and internal settings, then we should not call
update setting (on the follower) as there's nothing to change. Moreover,
this makes the ShardFollowTask abort as it considers
ActionRequestValidationException (caused by an empty update setting
request) as a fatal error.

Closes #44521
2019-07-18 11:08:45 -04:00
Lisa Cawley fe23dd1073 [DOCS] Clarifies data transform source definition (#44504) 2019-07-18 08:04:56 -07:00
Andrey Ershov 6f5327ba45 Fix BlobStoreTestUtil 2019-07-18 17:00:23 +02:00
Armin Braun 07cf2cb665
Add disable_chunked_encoding Setting to S3 Repo (#44052) (#44562)
* Add disable_chunked_encoding setting to S3 repo plugin to support S3 implementations that don't support chunked encoding
2019-07-18 16:57:56 +02:00