Commit Graph

49285 Commits

Author SHA1 Message Date
James Rodewig 726c35dfd0 [DOCS] Add identifier mapping tip to numeric and keyword datatype docs (#49933)
Users often mistakenly map numeric IDs to numeric datatypes. However,
this is often slow for the `term` and other term-level queries.

The "Tune for search speed" docs includes advice for mapping numeric
IDs to `keyword` fields. However, this tip is not included in the
`numeric` or `keyword` field datatype doc pages.

This rewords the tip in the "Tune for search speed" docs, relocates it
to the `numeric` field docs, and reuses it using tagged regions.
2019-12-17 09:34:32 -05:00
Armin Braun 55cc5432d6
Fix S3 Repo Tests Incomplete Reads (#50268) (#50275)
We need to read in a loop here. A single read to a huge byte array will
only read 16k max with the S3 SDK so if the blob we're trying to fully
read is larger we close early and fail the size comparison.
Also, drain streams fully when checking existence to avoid S3 SDK warnings.
2019-12-17 15:33:09 +01:00
James Rodewig 99fdea50dd
[DOCS] Correct cat snapshots API request example (#50274)
For 7.x and earlier branches, `_cat/repositories` API requests require a
repository name.

This removes an erroneous request example without a repository name
added with a8e0275.
2019-12-17 09:23:38 -05:00
Yannick Welsch 37b8c139b3 Omit loading IndexMetaData when inspecting shards (#50214)
Loading shard state information during shard allocation sometimes runs into a situation where a
data node does not know yet how to look up the shard on disk if custom data paths are used.
The current implementation loads the index metadata from disk to determine what the custom
data path looks like. This PR removes this dependency, simplifying the lookup.

Relates #48701
2019-12-17 14:33:02 +01:00
Yannick Welsch 1f981580aa Simplify InternalTestCluster.fullRestart (#50218)
With node ordinals gone, there's no longer a need for such a complicated full cluster restart
procedure (as we can now uniquely associate nodes to data folders).

Follow-up to #41652
2019-12-17 14:33:01 +01:00
David Kyle 098f540f9d
[ML] Remove usage of base action logger in ml actions (#50074) (#50236) 2019-12-17 13:03:27 +00:00
Martijn van Groningen 2079f1cbeb
Backport: Fix ingest simulate response document order if processor executes async (#50269)
Backport #50244 to 7.x branch.

If a processor executes asynchronously and the ingest simulate api simulates with
multiple documents then the order of the documents in the response may not match
the order of the documents in the request.

Alexander Reelsen discovered this issue with the enrich processor with the following reproduction:

```
PUT cities/_doc/munich
{"zip":"80331","city":"Munich"}

PUT cities/_doc/berlin
{"zip":"10965","city":"Berlin"}

PUT /_enrich/policy/zip-policy
{
  "match": {
    "indices": "cities",
    "match_field": "zip",
    "enrich_fields": [ "city" ]
  }
}

POST /_enrich/policy/zip-policy/_execute

GET _cat/indices/.enrich-*

POST /_ingest/pipeline/_simulate
{
  "pipeline": {
  "processors" : [
    {
      "enrich" : {
        "policy_name": "zip-policy",
        "field" : "zip",
        "target_field": "city",
        "max_matches": "1"
      }
    }
  ]
  },
  "docs": [
    { "_id": "first", "_source" : { "zip" : "80331" } } ,
    { "_id": "second", "_source" : { "zip" : "50667" } }
  ]
}
```

* fixed test compile error
2019-12-17 12:27:07 +01:00
Armin Braun 4f24739fbe
Fix Index Deletion During Partial Snapshot Create (#50234) (#50266)
We can simply filter out shard generation updates for indices
that were removed from the cluster state concurrently to fix
index deletes during partial snapshots as that completely removes
any reference to those shards from the snapshot.

Follow up to #50202
Closes #50200
2019-12-17 10:58:15 +01:00
Armin Braun 2e7b1ab375
Use ClusterState as Consistency Source for Snapshot Repositories (#49060) (#50267)
Follow up to #49729

This change removes falling back to listing out the repository contents to find the latest `index-N` in write-mounted blob store repositories.
This saves 2-3 list operations on each snapshot create and delete operation. Also it makes all the snapshot status APIs cheaper (and faster) by saving one list operation there as well in many cases.
This removes the resiliency to concurrent modifications of the repository as a result and puts a repository in a `corrupted` state in case loading `RepositoryData` failed from the assumed generation.
2019-12-17 10:55:13 +01:00
Armin Braun ce294e1564
Better Logging S3 Bulk Delete Failures (#50203) (#50262)
Unfortunately bulk delete exceptions don't show the individual delete
errors when a bulk delete fails when you log them outright so I added this work-around
to get the individual details to get useful logging.
2019-12-17 09:42:39 +01:00
Andrei Stefan c6fdf9ed8a Handle NULL in ResultSet's getDate() method (#50184)
(cherry picked from commit 08214eb1338fef5c8082c3f8b84c24dd53224ebe)
2019-12-17 10:03:23 +02:00
Tim Vernum ce2aab3f2f
Add setting to restrict license types (#50252)
This adds a new "xpack.license.upload.types" setting that restricts
which license types may be uploaded to a cluster.

By default all types are allowed (excluding basic, which can only be
generated and never uploaded).
This setting does not restrict APIs that generate licenses such as the
start trial API.

This setting is not documented as it is intended to be set by
orchestrators and not end users.

Backport of: #49418
2019-12-17 14:58:58 +11:00
Przemko Robakowski 0efb241b3c
Fix flakiness in CsvProcessorTests (#50254) (#50256)
There's flakiness in CsvProcesorTests, where tests fail if random document generator add field that should not be present. This change cleans generated document from these problematic fields.

Closes #50209
2019-12-17 01:15:15 +01:00
Henning Andersen 8391b974c5 Recovery buffer size 16B smaller (#50100)
G1GC will use humongous allocations when an allocation exceeds half the
chosen region size, which is minimum 1MB. By reducing the recovery
buffer size by 16 bytes we ensure that the recovery buffer is never
allocated as a humongous allocation.
2019-12-16 22:00:22 +01:00
Jason Tedor 620b84c84d
Respect ES_PATH_CONF during upgrades too (#50246)
A previous commit taught Elasticsearch packages to respect ES_PATH_CONF
during installs. Missed in that commit was respecting ES_PATH_CONF on
upgrades. This commit does that. Additionally, while ES_PATH_CONF is not
currently used in pre-install, this commit adds respect to the preinst
script in case we do in the future.
2019-12-16 15:08:23 -05:00
Julie Tibshirani 463cd414aa Bump the scroll keep-alive time in cluster upgrade tests. (#50195)
In the yaml cluster upgrade tests, we start a scroll in a mixed-version cluster,
then attempt to continue the scroll after the upgrade is complete. This test
occasionally fails because the scroll can expire before the cluster is done
upgrading.

The current scroll keep-alive time 5m. This PR bumps it to 10m, which gives a
good buffer since in failing tests the time was only exceeded by ~30 seconds.

Addresses #46529.
2019-12-16 10:58:31 -08:00
Nhat Nguyen 731bfa6614 Account trimAboveSeqNo in committed translog generation (#50205)
Today we do not consider trimAboveSeqNo when calculating the translog 
generation of an index commit. If there is no new indexing after the
primary promotion, then we won't be able to clean up the translog.
2019-12-16 11:40:16 -05:00
Zachary Tong be78d5cc74 Migrate MinAggregator integration tests to AggregatorTestCase (#50053)
Also renames MinTests to MinAggregationBuilderTests
2019-12-16 11:15:50 -05:00
Rory Hunter 2bd3a05892
Refactor environment variable processing for Docker (#50221)
Backport of #49612.

The current Docker entrypoint script picks up environment variables and
translates them into -E command line arguments. However, since any tool
executes via `docker exec` doesn't run the entrypoint, it results in
a poorer user experience.

Therefore, refactor the env var handling so that the -E options are
generated in `elasticsearch-env`. These have to be appended to any
existing command arguments, since some CLI tools have subcommands and
-E arguments must come after the subcommand.

Also extract the support for `_FILE` env vars into a separate script, so
that it can be called from more than once place (the behaviour is
idempotent).

Finally, add noop -E handling to CronEvalTool for parity, and support
`-E` in MultiCommand before subcommands.
2019-12-16 15:39:28 +00:00
David Kyle 5542686283 [ML] Wait for green after opening job in NetworkDisruptionIT (#50232)
Closes #49908
2019-12-16 14:55:58 +00:00
David Roberts 32b2445744
Change process kill order for testclusters shutdown (#50215)
The testclusters shutdown code was killing child processes
of the ES JVM before the ES JVM.  This causes any running
ML jobs to be recorded as failed, as the ES JVM notices that
they have disconnected from it without being told to stop,
as they would if they crashed.  In many test suites this
doesn't matter because the test cluster will never be
restarted, but in the case of upgrade tests it makes it
impossible to test what happens when an ML job is running
at the time of the upgrade.

This change reverses the order of killing the ES process
tree such that the parent processes are killed before their
children.  A list of children is stored before killing the
parent so that they can subsequently be killed (if they
don't exit by themselves as a side effect of the parent
dying).

Backport of #50175
2019-12-16 14:12:36 +00:00
Andrew Odendaal d934f8ff10 [Docs] Fix typo in README (#50229) 2019-12-16 14:37:23 +01:00
Armin Braun afcdc27c02
Fix Index Deletion during Snapshot Finalization (#50202) (#50227)
With #45689 making it so that index metadata is written
after all shards have been snapshotted we can't delete indices
that are part of the upcoming snapshot finalization any longer
and it is not sufficient to check if all shards of an index have been
snapshotted before deciding that it is safe to delete it.
This change forbids deleting any index that is in the process of being
snapshot to avoid issues during snapshot finalization.

Relates #50200 (doesn't fully fix yet because we're not fixing the `partial=true`
snapshot case here
2019-12-16 13:30:05 +01:00
Henning Andersen 4ced237a7f Disk threshold decider is enabled by default (#50222)
An old comment had survived after the default was flipped.

Relates #6204
2019-12-16 12:43:34 +01:00
Dimitris Athanasiou 73add726d7
[7.x][ML] Fix exception when field is not included and excluded at the same time (#50192) (#50223)
Executing the data frame analytics _explain API with a config that contains
a field that is not in the includes list but at the same time is the excludes
list results to trying to remove the field twice from the iterator. That causes
an `IllegalStateException`. This commit fixes this issue and adds a test that
captures the scenario.

Backport of #50192
2019-12-16 11:30:06 +00:00
Armin Braun 761d6e8e4b
Remove BlobContainer Tests against Mocks (#50194) (#50220)
* Remove BlobContainer Tests against Mocks

Removing all these weird mocks as asked for by #30424.
All these tests are now part of real repository ITs and otherwise left unchanged if they had
independent tests that didn't call the `createBlobStore` method previously.
The HDFS tests also get added coverage as a side-effect because they did not have an implementation
of the abstract repository ITs.

Closes #30424
2019-12-16 11:37:09 +01:00
Ignacio Vera 3717c733ff
"CONTAINS" support for BKD-backed geo_shape and shape fields (#50141) (#50213)
Lucene 8.4 added support for "CONTAINS", therefore in this commit those
changes are integrated in Elasticsearch. This commit contains as well a
bug fix when querying with a geometry collection with "DISJOINT" relation.
2019-12-16 09:17:51 +01:00
Nhat Nguyen c732d9923d Fix doc type in FullClusterRestartIT
"_doc" is not accepted in 6.x version.
2019-12-15 21:54:57 -05:00
Tim Vernum a9d16ee895
Skip enterprise license tests in release build (#50182)
The release builds use a production license key, and our rest test load
licenses that are signed by the dev license key.

This change adds the new enterprise license Rest tests to the
blacklist on release builds.

Backport of: #50163
2019-12-16 10:11:21 +11:00
Nhat Nguyen 4d22e3cd15 Skip bwc versions without retention leases 2019-12-15 13:18:13 -05:00
Nhat Nguyen 6f1098cceb Fix version in testTurnOffTranslogRetentionAfterAllShardStarted
Soft-deletes requires 6.5 or later.
2019-12-15 12:58:28 -05:00
Henning Andersen a0044df78a
Reindex source types disregarded in 7.x (#49580)
Clarify that types in source index are disregarded.
2019-12-15 17:30:28 +01:00
Nhat Nguyen df46848fb0 Migrate peer recovery from translog to retention lease (#49448)
Since 7.4, we switch from translog to Lucene as the source of history
for peer recoveries. However, we reduce the likelihood of
operation-based recoveries when performing a full cluster restart from
pre-7.4 because existing copies do not have PPRL.

To remedy this issue, we fallback using translog in peer recoveries if
the recovering replica does not have a peer recovery retention lease,
and the replication group hasn't fully migrated to PRRL.

Relates #45136
2019-12-15 10:24:39 -05:00
Nhat Nguyen c151a75dfe Use retention lease in peer recovery of closed indices (#48430)
Today we do not use retention leases in peer recovery for closed indices
because we can't sync retention leases on closed indices. This change
allows that ability and adjusts peer recovery to use retention leases
for all indices with soft-deletes enabled.

Relates #45136

Co-authored-by: David Turner <david.turner@elastic.co>
2019-12-15 10:24:34 -05:00
James Rodewig bbb872a022 Revert "[DOCS] Add `index-extra-title-page.html` for direct HTML migration (#50189)"
This reverts commit 79c40728f7.
2019-12-13 12:58:11 -05:00
James Rodewig 79c40728f7 [DOCS] Add `index-extra-title-page.html` for direct HTML migration (#50189) 2019-12-13 12:47:13 -05:00
Benjamin Trent 4805d8ac7d
[ML][Inference] Adding a warning_field for warning msgs. (#49838) (#50183)
This adds a new field for the inference processor.

`warning_field` is a place for us to write warnings provided from the inference call. When there are warnings we are not going to write an inference result. The goal of this is to indicate that the data provided was too poor or too different for the model to make an accurate prediction.

The user could optionally include the `warning_field`. When it is not provided, it is assumed no warnings were desired to be written.

The first of these warnings is when ALL of the input fields are missing. If none of the trained fields are present, we don't bother inferencing against the model and instead provide a warning stating that the fields were missing.

Also, this adds checks to not allow duplicated fields during processor creation.
2019-12-13 10:39:51 -05:00
Benjamin Trent 41736dd6c3
[ML] retry bulk indexing of state docs (#50149) (#50185)
This exchanges the direct use of the `Client` for `ResultsPersisterService`. State doc persistence will now retry. Failures to persist state will still not throw, but will be audited and logged.
2019-12-13 10:39:34 -05:00
Dimitris Athanasiou fe3c9e71d1
[7.x][ML] Fix DFA explain API timeout when source index is missing (#50176) (#50180)
This commit fixes a bug that caused the data frame analytics
_explain API to time out in a multi-node setup when the source
index was missing. When we try to create the extracted fields detector,
we check the index settings. If the index is missing that responds
with a failure that could be wrapped as a remote exception.
While we unwrapped correctly to check if the cause was an
`IndexNotFoundException`, we then proceeded to cast the original
exception instead of the cause.

Backport of #50176
2019-12-13 17:00:55 +02:00
James Rodewig cd04021961 [DOCS] Reformat token count limit filter docs (#49835) 2019-12-13 08:44:39 -05:00
Ioannis Kakavas 46376100b1
Fix testMalformedToken (#50164) (#50170)
This test was fixed as part of #49736 so that it used a
TokenService mock instance that was enabled, so that token
verification fails because the token is invalid and not because
the token service is not enabled.
When the randomly generated token we send, decodes to being of
version > 7.2 , we need to have mocked a GetResponse for the call
that TokenService#getUserTokenFromId will make, otherwise this
hangs and times out.
2019-12-13 13:46:44 +02:00
István Zoltán Szabó 8f36bfa37f
[7.x][DOCS] Changes hyperparam optimization section ID (#50173) 2019-12-13 12:22:50 +01:00
István Zoltán Szabó 7611b3c9be
[7.x][DOCS] Moves data frame analytics job resource definitions into APIs (#50165)
* [7.x][DOCS] Moves data frame analytics job resource definitions into APIs.
2019-12-13 11:48:21 +01:00
Ioannis Kakavas ac83e45a6b
Disable TLS diagnostics in FIPS mode (#49900)
This commit sets xpack.security.ssl.diagnose.trust to false in all
the nodes of our TestClusters when running integTest. This is needed
in 7.x because setting xpack.security.ssl.diagnose.trust to true
wraps SunJSSE TrustManager with our own DiagnosticTrustManager and
this is not allowed when SunJSSE is in FIPS mode.
An alternative would be to set `xpack.security.fips.enabled` to
true which would also implicitly disable
xpack.security.ssl.diagnose.trust but would have additional effects
(would require that we set PBKDF2 for password hashing algorithm in
all test clusters, would prohibit using JKS keystores in nodes even
if relevant tests have been muted in FIPS mode etc.)
2019-12-13 12:47:53 +02:00
Christoph Büscher c0216f9a06 Improve DateFieldMapper `ignore_malformed` handling (#50090)
A recent change around date parsing (#46675) made it stricter, so we should now also
catch DateTimeExceptions in DateFieldMapper and ignore those when the
`ignore_malformed` option is set.

Closes #50081
2019-12-13 10:00:10 +01:00
Dimitris Athanasiou e6cbcf7f7c
[7.x] [ML] Persist/restore state for DFA classification (#50040) (#50147)
This commit adds state persist/restore for data frame analytics classification jobs.

Backport of #50040
2019-12-13 10:33:19 +02:00
Hendrik Muhs 1c3ce110bd [Transform] add actual timeout in message (#50140)
add the timeout to the message if stopping a transform times out
2019-12-13 08:10:25 +01:00
Jason Tedor efa6ef3c06
Respect ES_PATH_CONF on package install (#50158)
We respect ES_PATH_CONF everywhere except package install. This commit
addresses this by respecting ES_PATH_CONF when installing the RPM/Debian
packages.
2019-12-12 22:28:24 -05:00
Jason Tedor 29526d0dfe
Validate exporter type is HTTP for HTTP exporter (#49992)
Today the HTTP exporter settings without the exporter type having been
configured to HTTP. When it is time to initialize the exporter, we can
blow up. Since this initialization happens on the cluster state applier
thread, it is quite problematic that we do not reject settings updates
where the type is not configured to HTTP, but there are HTTP exporter
settings configured. This commit addresses this by validating that the
exporter type is not only set, but is set to HTTP.
2019-12-12 20:01:04 -05:00
Mark Vieira 5b9fce48c0
Trim minimumGradleVersion whitespace before parsing 2019-12-12 15:26:05 -08:00