Commit Graph

49405 Commits

Author SHA1 Message Date
Dimitris Athanasiou 422422a2bc
[7.x][ML] Reuse SourceDestValidator for data frame analytics (#50841) (#50850)
This commit removes validation logic of source and dest indices
for data frame analytics and replaces it with using the common
`SourceDestValidator` class which is already used by transforms.
This way the validations and their messages become consistent
while we reduce code.

This means that where these validations fail the error messages
will be slightly different for data frame analytics.

Backport of #50841
2020-01-10 14:24:13 +02:00
Armin Braun 7e68989dae
Fix Snapshot Shard Status Request Deduplication (#50788) (#50840)
* Fix Snapshot Shard Status Request Deduplication

The request deduplication didn't actually work for these requests
since they had no `equals` and `hashCode` so the deduplicator wouldn't
actually recognize equal requests.
2020-01-10 11:49:52 +01:00
Christoph Büscher 75cb4e0b69 Muting InternalAggregationsTests.testSerialization 2020-01-10 09:24:09 +01:00
debadair 83d961391b
[DOCS] Move snapshot-restore out of modules. (#49618) (#50829)
* [DOCS] Move snapshot-restore docs out of modules.

* [DOCS] Incorporates comments from @jrodewig.

* [DOCS] Fix snippet tests
2020-01-09 16:55:46 -08:00
Mark Vieira 2ab3e49dec
Skip test suite entirely for non-applicable distribution types (#50824) 2020-01-09 15:44:08 -08:00
Julie Tibshirani a9c4db4134 Correct the skip version in tests for field collapsing on an alias. 2020-01-09 15:30:12 -08:00
Nik Everett d021071ab9
Move scripted metric to ObjectParser (#50708) (#50811)
This replaces the hand rolled parsing code for scripted metric with
`ObjectParser` which is simpler to work with because it is declarative.
2020-01-09 16:09:21 -05:00
Nik Everett ae40e22452
Drop "funny" functions building parsers (#50715) (#50814)
Replaces the "funny"
`Function<String, ConstructingObjectParser<T, Void>>` with a much
simpler `ConstructingObjectParser<T, String>`. This makes pretty much
all of our object parsers static.
2020-01-09 15:53:03 -05:00
Armin Braun f70e8f6ab5
Fix Snapshot Repository Corruption in Downgrade Scenarios (#50692) (#50797)
* Fix Snapshot Repository Corruption in Downgrade Scenarios (#50692)

This PR introduces test infrastructure for downgrading a cluster while interacting with a given repository.
It fixes the fact that repository metadata in the new format could be written while there's still older snapshots in the repository that require the old-format metadata to be restorable.
2020-01-09 21:21:13 +01:00
Matt Braymer-Hayes 344c21813b [DOCS] Fix typo in refresh API docs (#50759) 2020-01-09 14:41:19 -05:00
Lisa Cawley ef1c14ad01 [DOCS] Update license expiry links (#50812) 2020-01-09 11:28:43 -08:00
Jake Landis de6f132887
[7.x] Foreach processor - fork recursive call (#50514) (#50773)
A very large number of recursive calls can cause a stack overflow
exception. This commit forks the recursive calls for non-async
processors. Once forked, each thread will handle at most 10
recursive calls to help keep the stack size and thread count
down to a reasonable size.
2020-01-09 13:21:18 -06:00
Nik Everett 1d8e51f89d
Support offset in composite aggs (#50609) (#50808)
Adds support for the `offset` parameter to the `date_histogram` source
of composite aggs. The `offset` parameter is supported by the normal
`date_histogram` aggregation and is useful for folks that need to
measure things from, say, 6am one day to 6am the next day.

This is implemented by creating a new `Rounding` that knows how to
handle offsets and delegates to other rounding implementations. That
implementation doesn't fully implement the `Rounding` contract, namely
`nextRoundingValue`. That method isn't used by composite aggs so I can't
be sure that any implementation that I add will be correct. I propose to
leave it throwing `UnsupportedOperationException` until I need it.

Closes #48757
2020-01-09 14:11:24 -05:00
Sean Story c51303d051 Typo of ' instead of ` (#50767) 2020-01-09 09:41:41 -08:00
lcawl 8a5de4f56f [DOCS] Clarify detector_index property in ML APIs (#50723) 2020-01-09 08:34:34 -08:00
Benjamin Trent cc0e64572a
[ML][Inference][HLRC] Add necessary lang ident classes (#50705) (#50794)
This adds the necessary named XContent classes to the HLRC for the lang ident model. This is so the HLRC can call `GET _ml/inference/lang_ident_model_1?include_definition=true` without XContent parsing errors.

The constructors are package private as since this classes are used exclusively within the pre-packaged model (and require the specific weights, etc. to be of any use).
2020-01-09 10:33:38 -05:00
Benjamin Trent 3e014d39c2
[Transform] fail to start/put on missing pipeline (#50701) (#50795)
If a pipeline referenced by a transform does not exist, we should not allow the transform to be created. 

We do allow the pipeline existence check to be skipped with defer_validations, but if the pipeline still does not exist on `_start`, the pipeline will fail to start.

relates:  #50135
2020-01-09 10:33:22 -05:00
István Zoltán Szabó 4f150e4961
[7.x][DOCS] Moves analysis resources to PUT DFA API docs (#50793) 2020-01-09 16:21:35 +01:00
István Zoltán Szabó 71afeec7d0 Revert "[DOCS] Moves analysis resources to PUT DFA API docs (#50704)"
This reverts commit 4e1107d5d7.
2020-01-09 14:31:35 +01:00
István Zoltán Szabó 4e1107d5d7 [DOCS] Moves analysis resources to PUT DFA API docs (#50704)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-09 14:13:37 +01:00
István Zoltán Szabó acd73dda1c [DOCS] Improves find_file_structure documentation (#50743)
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
2020-01-09 11:20:29 +01:00
Armin Braun 4a7e09f624
Enforce Logging of Errors in GCS Rest RetriesTests (#50761) (#50783)
It's impossible to tell why #50754 fails without this change.
We're failing to close the `exchange` somewhere and there is no
write timeout in the GCS SDK (something to look into separately)
only a read timeout on the socket so if we're failing on an assertion without
reading the full request body (at least into the read-buffer) we're locking up
waiting forever on `write0`.

This change ensure the `exchange` is closed in the tests where we could lock up
on a write and logs the failure so we can find out what broke #50754.
2020-01-09 10:46:07 +01:00
István Zoltán Szabó 0ac6786f41 [DOCS] Forms role and privilege requirements as bulleted lists in DFA API docs (#50732)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-01-09 10:45:18 +01:00
István Zoltán Szabó 8a1bb440e2 [DOCS] Clarifies model_size_stats.total_xxx_field_count objects and removes notes in GET job stats API docs. (#50728) 2020-01-09 09:45:37 +01:00
István Zoltán Szabó d7bb5d7531 [DOCS] Improves description for forecast_stats (#50729)
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2020-01-09 09:35:47 +01:00
Martijn van Groningen f75d99149b
Wrap triggering of a watch inside an assertBusy(...) invocation
This test replaces the watch index after watcher got started.
This triggers watches being reloaded and while this happens the
trigger engine is paused, which disallows watches from being
triggered. At this time there are no watches in the .watches
index and I think this is just unlucky timing.

Reloading of watches happens in the background and
the watch state can be started when that happens.
For normal schedule trigger engines this is not an issue,
because watches that are meant to be triggered are triggered
when the engine triggers the next time. However for the
mock scheduled trigger engine this is different,
because watches are triggered programatically and
there is no retry in this test.

I think just adding `timeWarp().trigger("mywatch");` inside
a `assertBusy(...)`` is the right fix here.  If it fails
because the mock schedule trigger engine is paused then
the test will try again. In the mean time the the watches
can be reloaded, which then resumes the mock scheduled trigger engine.

Closes #50658
2020-01-09 09:05:20 +01:00
Ioannis Kakavas d2189b9d80
Mute SamlAuthenticatorTests in Azulu Zulu (#50779)
See #49742
2020-01-09 09:41:04 +02:00
Jason Tedor 8d9ec1c66d
Remove UBI-based Docker images (#50747)
This commit removes the UBI-based Docker images as we are not using this
work for now.
2020-01-08 20:30:12 -05:00
Julie Tibshirani 3273c366b5 Adjust the skip version in tests for field collapsing with aliases.
Follow-up to ##50766, which was also backported to 7.5.
2020-01-08 16:13:03 -08:00
Julie Tibshirani a299aba2f8
Ensure that field collapsing works with field aliases. (#50766)
Previously, the following situation would throw an error:
* A search contains a `collapse` on a particular field.
* The search spans multiple indices, and in one index the field is mapped as a
  concrete field, but in another it is a field alias.

The error occurs when we attempt to merge `CollapseTopFieldDocs` across shards.
When merging, we validate that the name of the collapse field is the same across
shards. But the name has already been resolved to the concrete field name, so it
will be different on shards where the field was mapped as an alias vs. shards
where it was a concrete field.

This PR updates the collapse field name in `CollapseTopFieldDocs` to the
original requested field, so that it will always be consistent across shards.

Note that in #32648, we already made a fix around collapsing on field aliases.
However, we didn't test this specific scenario where the field was mapped as an
alias in only one of the indices being searched.
2020-01-08 14:51:15 -08:00
Christoph Büscher b1b4282273 Make Multiplexer inherit filter chains analysis mode (#50662)
Currently, if an updateable synonym filter is included in a multiplexer filter,
it is not reloaded via the _reload_search_analyzers because the multiplexer
itself doesn't pass on the analysis mode of the filters it contains, so its not
recognized as "updateable" in itself. Instead we can check and merge the
AnalysisMode settings of all filters in the multiplexer and use the resulting
mode (e.g. search-time only) for the multiplexer itself, thus making any synonym
filters contained in it reloadable.  This, of course, will also make the
analyzers using the multiplexer be usable at search-time only.

Closes #50554
2020-01-08 22:12:01 +01:00
James Rodewig 78c9eee5ea [DOCS] Add section ID to analysis overview page 2020-01-08 14:43:41 -06:00
Lee Hinman 8dc6e98819
[7.x] Make InitializePolicyContextStep retryable (#50685) (#50760)
This commits makes the "init" ILM step retryable. It also adds a test
where an index is created with a non-parsable index name and then fails.

Related to #48183
2020-01-08 13:13:57 -07:00
Przemyslaw Gomulka e95b0c447f
Allow parsing timezone without fully provided time backport(#50178) (#50740)
strict_date_optional_time changes to have optional minute part.
It already allowed optional second and fraction of second part.
This allows parsing 2018-01-01T00+01 , 2018-01-01T00:00+01 , 2018-01-01T00:00:00+01 , 2018-01-01T00:00:00.000+01
It won't allow parsing a timezone without an hour part as this is not allowed by iso8601 spec

closes #49351
2020-01-08 20:04:57 +01:00
James Rodewig 9d1567b13b [DOCS] Add overview page to analysis topic (#50515)
Adds a 'text analysis overview' page to the analysis topic docs.

The goals of this page are:

* Concisely summarize the analysis process while avoiding in-depth concepts, tutorials, or API examples
* Explain why analysis is important, largely through highlighting problems with full-text searches missing analysis
* Highlight how analysis can be used to improve search results
2020-01-08 12:54:00 -06:00
Nhat Nguyen 90e66a7b97 Mute testPolicyCRUD
Tracked at #44997
2020-01-08 13:25:40 -05:00
Henning Andersen 125feecabc
Guess root cause support unwrap (#50525) (#50742)
ElasticsearchException.guessRootCauses would return wrapper exception if
inner exception was not an ElasticsearchException. Fixed to never return
wrapper exceptions.

At least following APIs change root_cause.0.type as a result:

_update with bad script
_index with bad pipeline

Relates #50417
2020-01-08 19:09:14 +01:00
István Zoltán Szabó 0444da944e [DOCS] Adds DFA resources as deleted page to redirects. (#50756) 2020-01-08 19:01:16 +01:00
Adrien Grand 4f2299c714
Upgrade to Lucene 8.4.0. (#50518) (#50750) 2020-01-08 18:53:59 +01:00
Lee Hinman 615532b4f8 Mute TimeSeriesLifecycleActionsIT.testHistoryIsWritten* (#50755)
Related to #50353
2020-01-08 10:35:44 -07:00
James Rodewig f87e61ec30 [DOCS] Add default index-time analyzer example (#50501)
The Analysis docs mention including a default analyzer in the index settings. However, no example snippet is included.

This adds an example snippet that users can easily copy and adjust.
2020-01-08 11:07:49 -06:00
Valentin Crettaz f3ddd4066a [DOCS] Fixed typos (_op => op) in Painless context docs (#50301) 2020-01-08 10:54:06 -06:00
blueSky1825821 5ff6eafb4b [Docs] Update similarity.asciidoc (#50719)
DFRSimilarity -> DFR similarity
2020-01-08 17:48:26 +01:00
Armin Braun a725896c92
Fix and Reenable SnapshotTool Minio Tests (#50736) (#50745)
This solves half of the problem in #46813 by moving the S3
tests to using the shared minio fixture so we at least have
some non-3rd-party, constantly running coverage on these tests.
2020-01-08 16:33:36 +01:00
Adrien Grand 31158ab3d5
Add per-field metadata. (#50333)
This PR adds per-field metadata that can be set in the mappings and is later
returned by the field capabilities API. This metadata is completely opaque to
Elasticsearch but may be used by tools that index data in Elasticsearch to
communicate metadata about fields with tools that then search this data. A
typical example that has been requested in the past is the ability to attach
a unit to a numeric field.

In order to not bloat the cluster state, Elasticsearch requires that this
metadata be small:
 - keys can't be longer than 20 chars,
 - values can only be numbers or strings of no more than 50 chars - no inner
   arrays or objects,
 - the metadata can't have more than 5 keys in total.

Given that metadata is opaque to Elasticsearch, field capabilities don't try to
do anything smart when merging metadata about multiple indices, the union of
all field metadatas is returned.

Here is how the meta might look like in mappings:

```json
{
  "properties": {
    "latency": {
      "type": "long",
      "meta": {
        "unit": "ms"
      }
    }
  }
}
```

And then in the field capabilities response:

```json
{
  "latency": {
    "long": {
      "searchable": true,
      "aggreggatable": true,
      "meta": {
        "unit": [ "ms" ]
      }
    }
  }
}
```

When there are no conflicts, values are arrays of size 1, but when there are
conflicts, Elasticsearch includes all unique values in this array, without
giving ways to know which index has which metadata value:

```json
{
  "latency": {
    "long": {
      "searchable": true,
      "aggreggatable": true,
      "meta": {
        "unit": [ "ms", "ns" ]
      }
    }
  }
}
```

Closes #33267
2020-01-08 16:21:18 +01:00
Alexander Reelsen 71054d269b Sync grok patterns with logstash patterns (#50381)
In order to ensure that logstash and Elasticsearch are able to understand
the same patterns, this commit adapts to changes in logstash, adds a few
patterns and changes a few.
2020-01-08 14:59:34 +01:00
James Rodewig d3094f9d23 [DOCS] Fix typo in mapping date format docs 2020-01-08 07:55:51 -06:00
Yannick Welsch f203c2b39d Import replicated closed dangling indices (#50649)
Dangling replicated closed indices are not imported properly (they miss their routing table when imported).
2020-01-08 13:39:20 +01:00
Rory Hunter b1ff74f652 New setting to prevent automatically importing dangling indices (#49174)
Introduce a new static setting, `gateway.auto_import_dangling_indices`, which prevents dangling indices from being automatically imported. Part of #48366.
2020-01-08 13:39:20 +01:00
Andrei Dan 3915d4c055
Make the UpdateRolloverLifecycleDateStep retryable (#50702) (#50730)
This makes the "update-rollover-lifecycle-date" step, which is part of the
rollover action, retryable. It also adds an integration test to check the
step is retried and it eventually succeeds.

(cherry picked from commit 5bf068522deb2b6cd2563bcf80f34fdbf459c9f2)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-01-08 11:45:26 +01:00