Commit Graph

47110 Commits

Author SHA1 Message Date
James Rodewig 1c96e8a6af [DOCS] Reformat cat health API (#45218) 2019-08-06 08:41:18 -04:00
James Rodewig 2489508e1f [DOCS] Reformat cat count API (#45160) 2019-08-06 08:37:09 -04:00
Alpar Torok bc60eebfc8 Add retries to vault access in init scripts 2019-08-06 15:39:36 +03:00
Yannick Welsch 17846212bd Fix tests after backport of #44055 2019-08-06 14:19:20 +02:00
Alpar Torok b3076adae6 Fix reaper build failures on Windows (#45205)
We configure the service ID as the node's toString but this containes
characters that Windows doesn't like.
This PR fixes it by allowing only alphanumeric characters
2019-08-06 14:36:36 +03:00
Yannick Welsch a453cd489e Run testExtendedSocketOptions only on JDK11+ (#44055)
This functionality only works on JDK 11 or higher
2019-08-06 13:15:17 +02:00
Jason Tedor 5b1b146099
Normalize environment paths (#45179)
This commit applies a normalization process to environment paths, both
in how they are stored internally, also their settings values. This
normalization is done via two means:
 - we make the paths absolute
 - we remove redundant name elements from the path (what Java calls
   "normalization")

This change ensures that when we compare and refer to these paths within
the system, we are using a common ground. For example, prior to the
change if the data path was relative, we would not compare it correctly
to paths from disk usage. This is because the paths in disk usage were
being made absolute.
2019-08-06 06:04:30 -04:00
Yannick Welsch 7aeb2fe73c Add per-socket keepalive options (#44055)
Uses JDK 11's per-socket configuration of TCP keepalive (supported on Linux and Mac), see
https://bugs.openjdk.java.net/browse/JDK-8194298, and exposes these as transport settings.
By default, these options are disabled for now (i.e. fall-back to OS behavior), but we would like
to explore whether we can enable them by default, in particular to force keepalive configurations
that are better tuned for running ES.
2019-08-06 10:45:44 +02:00
Alpar Torok 81ea08b8ca Update ci bwc versions 2019-08-06 11:45:28 +03:00
David Turner f91487baff Relax recovery duration in docs test (#45186)
Sometimes the recovery in this docs test takes long enough that it is expressed
in `s` rather than `ms`. This commit relaxes the assertion to account for this.
2019-08-06 09:30:53 +01:00
Hendrik Muhs 6b5a2513a9 [ML-DataFrame] introduce an abstraction for checkpointing (#44900)
introduces an abstraction for how checkpointing and synchronization works, covering

 - retrieval of checkpoints
 - check for updates
 - retrieving stats information
2019-08-06 07:38:59 +02:00
Gordon Brown d0d808776f
Add guide for using ILM with existing indices (#43667)
In many cases, including migration from previous versions of data
shippers (e.g. Beats), it is useful to use ILM to manage historical
indices, which are no longer being written to. This commit adds a guide
which gives an example of how to do that.
2019-08-05 17:28:59 -06:00
Mark Vieira 87fb487299
Fix failed dependency resolution with external build-tools users (#45107) (#45219) 2019-08-05 15:52:46 -07:00
Igor Motov b5f88120b5 Geo: add Geometry-based query builders to QueryBuilders (#45058)
Add Geometry-based method for creation of query builders in
QueryBuilder

Relates to #44715
2019-08-05 13:34:48 -04:00
Zachary Tong 3df1c76f9b Allow pipeline aggs to select specific buckets from multi-bucket aggs (#44179)
This adjusts the `buckets_path` parser so that pipeline aggs can
select specific buckets (via their bucket keys) instead of fetching
the entire set of buckets.  This is useful for bucket_script in
particular, which might want specific buckets for calculations.

It's possible to workaround this with `filter` aggs, but the workaround
is hacky and probably less performant.

- Adjusts documentation
- Adds a barebones AggregatorTestCase for bucket_script
- Tweaks AggTestCase to use getMockScriptService() for reductions and
pipelines.  Previously pipelines could just pass in a script service
for testing, but this didnt work for regular aggs.  The new
getMockScriptService() method fixes that issue, but needs to be used
for pipelines too.  This had a knock-on effect of touching MovFn,
AvgBucket and ScriptedMetric
2019-08-05 12:18:40 -04:00
Zachary Tong e5079ac288
[7.x backport] Add more flexibility to MovingFunction window alignment (#45159)
Introduce shift field to MovingFunction aggregation.

By default, shift = 0. Behavior, in this case, is the same as before.
Increasing shift by 1 moves starting window position by 1 to the right.

    To simply include current bucket to the window, use shift = 1
    For center alignment (n/2 values before and after the current bucket), use shift = window / 2
    For right alignment (n values after the current bucket), use shift = window.
2019-08-05 11:56:52 -04:00
James Rodewig adaa54c066 [DOCS] Reformats cat allocation API (#45158) 2019-08-05 11:24:27 -04:00
Alpar Torok f3570aa27b CI specific init script updates (#45016)
- Add a vault integration so that we don't need Jenkins to do that for
us
   - This will make it easier to enable for windows too
- Move everything to a single file so we can read other secrets in the
same way
2019-08-05 18:27:10 +03:00
Nhat Nguyen 56083ba1ff Remove assertion after locally recover replica (#45181)
If the disk becomes broken after we have locally recovered shard up to
the global checkpoint, then the assertion won't hold.
2019-08-05 10:48:02 -04:00
Benjamin Trent 7bfaba98c2
[ML][Data Frame] cleaning up and adjusting failure tests (#45101) (#45144) 2019-08-05 09:12:11 -05:00
James Rodewig 2f4c342fff [DOCS] Reformat cat alias API (#45119) 2019-08-05 08:29:55 -04:00
István Zoltán Szabó dae648eb32 [DOCS] Makes clearer the note under freq_rare. (#45193) 2019-08-05 13:29:43 +02:00
David Turner 13a167051f
Remove fileBasedRecovery flag (#45146)
Today `RecoveryTarget#prepareForTranslogOperations` takes a boolean flag
indicating whether the recovery is file-based or not. This was used in 6.x to
bootstrap some commit data that were missing in indices created in 5.x:

b506955f8d/server/src/main/java/org/elasticsearch/indices/recovery/RecoveryTarget.java (L298-L300)

This flag no longer has any effect, so this commit removes it.

Backport of #45131 to 7.x.
2019-08-05 08:17:40 +01:00
Armin Braun 41815ed614
Optimize StreamInput#readString (#44930) (#45180)
* Resolve TODO in `readString` by moving to reading chunks of `byte[]` instead of going byte by byte
* Motivated by `readString` showing up as a significant user of CPU time on the IO thread in Rally PMC benchmark
* Benchmarking this:
  * Could not reproduce a slowdown in the potential worst case (one or two non-ascii chars) since in this case the cost of creating the string itself exceeds the read times anyway
  * Speedup for 50%+ for reading 200 char ascii strings from `ByteBuf` or pages bytes backed streams
  * Longer strings obviously get bigger speedups
  * More ascii chars -> more speedup
2019-08-05 07:22:42 +02:00
Jason Tedor 89b21102e5
Upgrade to JDK 12.0.2 (#45172)
This commit bumps the embedded JDK from 12.0.1 to 12.0.2.
2019-08-04 19:22:31 -04:00
Jason Tedor 872ae4d6c2
Add OCI annotations and adjust existing annotations (#45167)
The org.label-schema labels on Docker images have been superseded by
pre-defined OCI annotations. However, there is still a lot of tooling in
use that relies on the org.label-schema, so we do not want to drop
them. This commit adds values for the org.opencontainers.image
pre-defined annotation keys. Additionally, we correct an issue with the
label used to represent the license, to use the org.label-schema.license
label. While this label was never accepted into the org.label-schema
specfication (because this specification was superseded, it's not that
it was explicitly rejected) there are containers out there using this
label. In particular, our base image is and so we need to override
otherwise we inherit, and end up mis-reporting the license.
2019-08-04 13:52:13 -04:00
Jason Tedor e708710f38
Simplify handling of git revision (#45175)
This commit simplifies the handling of git revision in the build. In
particular we remove pushing git revision through the generate build
info and print build info tasks as the git revision does not need to be
cached.
2019-08-04 11:09:11 -04:00
Jason Tedor 78aaf1b2d5
Lift build date to global build info (#45166)
This commit adds the build date to global build info mainly so that it
is accessible as an extension property.
2019-08-03 14:49:46 -04:00
Jason Tedor d78ecd9c09
Use the full hash in build info (#45163)
This commit switches to using the full hash to build into the JAR
manifest, which is used in node startup and the REST main action to
display the build hash.
2019-08-03 11:27:53 -04:00
Ryan Ernst 925bd4faa1
Use reaper process instead of shutdown hooks for testclusters (#44927)
Testclusters currently provides protection from clusters living past the
life of a build by adding a shutdown hook to java. While this works in
some cases, it does not cover all cases like where the daemon is killed
with SIGKILL.

To handle these other cases, this commit replaces the shutdown hooks with
a separate process (one per build) that manages reaping external services
if gradle dies.
2019-08-02 18:58:04 -07:00
Jason Tedor 5dc22e29c6
Add build hash to global build info (#45162)
This commit adds the commit hash to the global build info, and adds the
git revision as an extension. There are a couple motivations for this
change:
 - the current mechanism of getting the build hash does not work with
   git worktrees (because jgit does not understand them)
 - a follow-up will want to use the git revision when building the
   Docker images, so we want it available as an extension
 - it allows us to simplify our usage around the build hash as we no
   longer have to hack around silliness in the info-scm plugin

A follow-up will also stop using the short hash in the product build, so
that we use the full hash there. We already know that short hashes in
our codebase do collide, so we should move to the full hash to avoid
this problem.
2019-08-02 19:02:08 -04:00
Tim Brooks 984ba82251
Move nio channel initialization to event loop (#45155)
Currently in the transport-nio work we connect and bind channels on the
a thread before the channel is registered with a selector. Additionally,
it is at this point that we set all the socket options. This commit
moves these operations onto the event-loop after the channel has been
registered with a selector. It attempts to set the socket options for a
non-server channel at registration time. If that fails, it will attempt
to set the options after the channel is connected. This should fix
#41071.
2019-08-02 17:31:31 -04:00
Zachary Tong ffbe047c32 Revert "Add more flexibility to MovingFunction window alignment (#44360)"
This reverts commit 1a58a487f0.
2019-08-02 15:16:04 -04:00
Nikita Glashenko 1a58a487f0 Add more flexibility to MovingFunction window alignment (#44360)
Introduce shift field to MovingFunction aggregation.

By default, shift = 0. Behavior, in this case, is the same as before.
Increasing shift by 1 moves starting window position by 1 to the right.

    To simply include current bucket to the window, use shift = 1
    For center alignment (n/2 values before and after the current bucket), use shift = window / 2
    For right alignment (n values after the current bucket), use shift = window.
2019-08-02 15:10:21 -04:00
Jack Conradson 54552edaf6 Whitelist randomUUID in Painless (#45148)
This whitelists randomUUID with the understanding that it's possible for 
/dev/random to cause blocking on *nix systems. Users that need 
randomUUID should switch their random generator source to /dev/urandom 
if this is a concern for them.
2019-08-02 11:53:56 -07:00
Lisa Cawley 00235bbecd [DOCS] Reformats the security APIs (#45124) 2019-08-02 11:32:47 -07:00
James Rodewig 4b8ae97e9d [DOCS] Update relevance score cross-references (#45092) 2019-08-02 14:15:34 -04:00
James Rodewig 8dd74dfe0b Rename "indices APIs" to "index APIs" (#44863) 2019-08-02 14:10:09 -04:00
Alison Goryachev b607148ae9 [DOCS] Fix watcher email action docs (#44877) 2019-08-02 14:02:08 -04:00
Lisa Cawley 09bd6c4692 [DOCS] Clarifies bucket span in overall buckets API (#45110) 2019-08-02 08:42:39 -07:00
Christoph Büscher 47386d4f0a [Docs] Correct README example snippet (#45133)
The example tries to create a new index but the create index API requires the
"settings" parameter to be there.

Closes #45129
2019-08-02 16:53:49 +02:00
David Roberts a1f0285f0e [TEST] Only test US locale in day/month order test in FIPS JVM (#45141)
In the FIPS JVM the JVM default locale seems to leak into places
where it should be overridden. This change skips assertions
in TimestampFormatFinderTests.testGuessIsDayFirstFromLocale
that may be impacted.

Fixes #45140
2019-08-02 15:04:47 +01:00
David Turner 9ff320d967
Use index for peer recovery instead of translog (#45137)
Today we recover a replica by copying operations from the primary's translog.
However we also retain some historical operations in the index itself, as long
as soft-deletes are enabled. This commit adjusts peer recovery to use the
operations in the index for recovery rather than those in the translog, and
ensures that the replication group retains enough history for use in peer
recovery by means of retention leases.

Reverts #38904 and #42211
Relates #41536
Backport of #45136 to 7.x.
2019-08-02 15:00:43 +01:00
Christoph Büscher 3366726ad1 Enable reloading of synonym_graph filters (#45135)
Reloading of synonym_graph filter doesn't work currently because the search time
AnalysisMode doesn't get propagated to the TokenFilterFactory emitted by the
graph filters getChainAwareTokenFilterFactory() method. This change fixes that.

Closes #45127
2019-08-02 15:33:42 +02:00
James Rodewig f657f3914d [DOCS] Create common parameters file for REST APIs (#45117) 2019-08-02 08:43:12 -04:00
Tomas Della Vedova 6b71621afc
Updated slm API spec parameters and URL (#44797) (#45102) 2019-08-02 11:39:52 +02:00
David Roberts f617585dbd [ML] Improve CSV header row detection in find_file_structure (#45099)
When doing a fieldwise Levenshtein distance comparison
between CSV rows, this change ignores all fields that
have long values, not just the longest field.

This approach works better for CSV formats that have
multiple freeform text fields rather than just a single
"message" field.

Fixes #45047
2019-08-02 09:08:21 +01:00
Armin Braun 9450505d5b
Stop Passing Around REST Request in Multiple Spots (#44949) (#45109)
* Stop Passing Around REST Request in Multiple Spots

* Motivated by #44564
  * We are currently passing the REST request object around to a large number of places. This works fine since we simply copy the full request content before we handle the rest itself which is needlessly hard on GC and heap.
  * This PR removes a number of spots where the request is passed around needlessly. There are many more spots to optimize in follow-ups to this, but this one would already enable bypassing the request copying for some error paths in a follow up.
2019-08-02 07:31:38 +02:00
Tim Vernum e21d58541a
Improve errors when TLS files cannot be read (#45122)
This change improves the exception messages that are thrown when the
system cannot read TLS resources such as keystores, truststores,
certificates, keys or certificate-chains (CAs).

This change specifically handles:

- Files that do not exist
- Files that cannot be read due to file-system permissions
- Files that cannot be read due to the ES security-manager

Backport of: #44787
2019-08-02 12:29:43 +10:00
Tim Vernum 590777150f
Explicitly fail if a realm only exists in keystore (#45091)
There are no realms that can be configured exclusively with secure
settings. Every realm that supports secure settings also requires one
or more non-secure settings.
However, sometimes a node will be configured with entries in the
keystore for which there is nothing in elasticsearch.yml - this may be
because the realm we removed from the yml, but not deleted from the
keystore, or it could be because there was a typo in the realm name
which has accidentially orphaned the keystore entry.

In these cases the realm building would fail, but the error would not
always be clear or point to the root cause (orphaned keystore
entries). RealmSettings would act as though the realm existed, but
then fail because an incorrect combination of settings was provided.

This change causes realm building to fail early, with an explicit
message about incorrect keystore entries.

Backport of: #44471
2019-08-02 12:28:59 +10:00