Commit Graph

48187 Commits

Author SHA1 Message Date
Martijn van Groningen 63b169b600
Upgrade joni from 2.1.6 to 2.1.29 (#47570)
Backport of #47374

Changed the Grok class to use searchInterruptible(...) instead of search(...)
otherwise we can't interrupt long running matching via the thread watch
dog.

Joni now also provides another way to interrupt long running matches.
By invoking the interrupt() method on the Matcher. We need then to refactor
the watch thread dog to keep track of Matchers instead of Threads, but
it is a better way of doing this, since interrupting would be more direct
(not every 30k iterations) and efficient (checking a volatile field).
This work needs to be done in a follow up.
2019-10-04 12:54:49 -05:00
James Rodewig 0b4fb05540
[DOCS] Reformat refresh API docs (#46667) (#47589) 2019-10-04 13:50:09 -04:00
Przemysław Witek ee952da2e2
[7.x] Implement evaluation API for multiclass classification problem (#47126) (#47343) 2019-10-04 17:54:51 +02:00
Jack Conradson e3aab1295e Add a ScriptRoot to consolidate global data necessary for multiple passes (#47532)
This PR is to get plumbing in for a ScriptRoot class that will consolidate 
several pieces of state required by potentially multiple passes including 
PainlessLookup, CompilerSettings, FunctionTable, the root class node, and a 
synthetic counter. It's possible more may be added to this as we move 
forward and slowly make the the nodes have less mutable state.
2019-10-04 08:37:19 -07:00
Lisa Cawley 9b3e5409c1
[7.x][DOCS] Copies security source files from stack-docs (#47534) 2019-10-04 08:19:10 -07:00
James Rodewig 3cc8081274 [DOCS] Correct headings for split index API docs 2019-10-04 11:07:41 -04:00
James Rodewig bb82addf35
[DOCS] Reformat split index API docs (#46713) (#47578) 2019-10-04 10:41:31 -04:00
Andrei Stefan a46f312ded SQL: fix multi full-text functions usage with aggregate functions (#47444)
* Skip functions involving full-text predicates when replacing multiple
aggregate functions with "stats" or "matrix_stats" aggregations.

(cherry picked from commit bb14ba83128dfb7a70f825ea08b1524072fb9ad0)
2019-10-04 16:27:22 +03:00
Alpar Torok 2b16d7bcf8
Backport testclusters all (#47565)
* Bwc testclusters all (#46265)

Convert all bwc projects to testclusters

* Fix bwc versions config

* WIP fix rolling upgrade

* Fix bwc tests on old versions

* Fix rolling upgrade
2019-10-04 16:12:53 +03:00
Przemysław Witek 8c180a77f0
[7.x] Fix serialization of evaluation response. (#47557) (#47566) 2019-10-04 15:12:18 +02:00
Armin Braun c1be7a802c
Simplify Snapshot Delete Process (#47439) (#47533)
We don't need to read the SnapshotInfo for a
snapshot to determine the indices that need to
be updated when it is deleted as the `RepositoryData`
contains that information already.
This PR makes it so the `RepositoryData` is used to
determine which indices to update and also removes
the special handling for deleting snapshot metadata
and the CS snapshot blob and has those simply be
deleted as part of the deleting of other unreferenced
blobs in the last step of the delete.

This makes the snapshot delete a little faster and
more resilient by removing two RPC calls
(the separate delete and the get).

Also, this shortens the diff with #46250 as a
side-effect.
2019-10-04 13:55:16 +02:00
Przemysław Witek ec9b77deaa
[7.x] Implement new analysis type: classification (#46537) (#47559) 2019-10-04 13:47:19 +02:00
Alpar Torok 65c473bd4b Fix windows packaging tests (#47554)
On windows, it happens that the process we called terminates but some
other process it creates still has the same output strems and thus the
files open, so we can't clean it up.

This PR makes the cleanup a best effort.
2019-10-04 14:02:57 +03:00
David Roberts 31a5e1c7ee [ML] More accurate job memory overhead (#47516)
When an ML job runs the memory required can be
broken down into:

1. Memory required to load the executable code
2. Instrumented model memory
3. Other memory used by the job's main process or
   ancilliary processes that is not instrumented

Previously we added a simple fixed overhead to
account for 1 and 3. This was 100MB for anomaly
detection jobs (large because of the completely
uninstrumented categorization function and
normalize process), and 20MB for data frame
analytics jobs.

However, this was an oversimplification because
the executable code only needs to be loaded once
per machine.  Also the 100MB overhead for anomaly
detection jobs was probably too high in most cases
because categorization and normalization don't use
_that_ much memory.

This PR therefore changes the calculation of memory
requirements as follows:

1. A per-node overhead of 30MB for _only_ the first
   job of any type to be run on a given node - this
   is to account for loading the executable code
2. The established model memory (if applicable) or
   model memory limit of the job
3. A per-job overhead of 10MB for anomaly detection
   jobs and 5MB for data frame analytics jobs, to
   account for the uninstrumented memory usage

This change will enable more jobs to be run on the
same node.  It will be particularly beneficial when
there are a large number of small jobs.  It will
have less of an effect when there are a small number
of large jobs.
2019-10-04 09:57:31 +01:00
David Roberts defc97a300 Remove fallback for controller location (#47104)
This change removes the temporary controller
location fallback introduced in #47013.

Relates elastic/ml-cpp#593
2019-10-04 09:50:26 +01:00
István Zoltán Szabó a57cb5843f [DOCS] Fixes an attribute in the update datafeed API docs. (#47551) 2019-10-04 08:46:16 +02:00
Alpar Torok 67bf3a4f51 Fix default distro bwc tests 2019-10-04 09:44:17 +03:00
Alpar Torok 97a0b7dcbc Make All OS tests run on GCP instances (#46924)
This PR makes the necesary adaptations to the tests and adds a power shell script to
invoke the OS tests on GCP instances connected as CI workers.

Also noticed that logs were not being produced by the tests and that theses were not using log4j so fixed that too.

One of the difficulties in working on theses tests was that the tests just stalled with no indication where the problem is.
To ease with the debugging, after process explorer suggested that the tests are running some commands, we now have multiple timeouts: one for the tests ( which will generate a thread dump ) and one for individual commands ( that bails with the command being ran and output and error so far ) to make it easier to see what went wrong.

The tests were blocking because apparently the pipes to the sub-process were not closing, thus the threads were blocking on them and we were blocking indefinitely on the join. I'm not sure why this doesn't happen in vagrant, but we now properly deal with it.
2019-10-04 08:46:52 +03:00
Ryan Ernst f32692208e
Add explanations to script score queries (#46693) (#47548)
While function scores using scripts do allow explanations, they are only
creatable with an expert plugin. This commit improves the situation for
the newer script score query by adding the ability to set the
explanation from the script itself.

To set the explanation, a user would check for `explanation != null` to
indicate an explanation is needed, and then call
`explanation.set("some description")`.
2019-10-03 21:05:05 -07:00
Yogesh Gaikwad d371f9d44d
Fix for ApiKeyIntegTests related to Expired API keys remover (#43477) (#47546)
When API key is invalidated we do two things first it tries to trigger `ExpiredApiKeysRemover` task
and second, we do index the invalidation for the API key. The index invalidation may happen
before the `ExpiredApiKeysRemover` task is run and in that case, the API key
invalidated will also get deleted. If the `ExpiredApiKeysRemover` runs before the
API key invalidation is indexed then the API key is not deleted and will be
deleted in the future run.
This behavior was not captured in the tests related to `ExpiredApiKeysRemover`
causing intermittent failures.
This commit fixes those tests by checking if the API key invalidated is reported
back when we get API keys after invalidation and perform the checks based on that.

Closes #41747
2019-10-04 13:17:52 +10:00
Nhat Nguyen 5e4732f2bb Limit number of retaining translog files for peer recovery (#47414)
Today we control the extra translog (when soft-deletes is disabled) for
peer recoveries by size and age. If users manually (force) flush many
times within a short period, we can keep many small (or empty) translog
files as neither the size or age condition is reached. We can protect
the cluster from running out of the file descriptors in such a situation
by limiting the number of retaining translog files.
2019-10-03 20:45:29 -04:00
Lisa Cawley f81d9a4aa6
Update link to ls monitoring settings (#47529) (#47542) 2019-10-03 17:06:26 -07:00
James Rodewig b8e7f5069d
[DOCS] Reformat index recovery API docs (#46493) (#47527) 2019-10-03 16:43:26 -04:00
James Rodewig 56e7668c78 [DOCS] Reformat index shard stores API docs (#46504) 2019-10-03 15:57:35 -04:00
Francois-Clement Brossard dc02868671 [DOCS] Update painless statements with if/else example (#47485) 2019-10-03 15:23:54 -04:00
Alpar Torok 392013cf05 Fix double start of test cluster 2019-10-03 18:14:58 +03:00
Lisa Cawley 9c7b58900c [DOCS] Fixes missing link title (#47481) 2019-10-03 08:06:31 -07:00
Ioannis Kakavas fd6a585009
Fix ADRealmTests in FIPS 140 JVMs (#47437) (#47506)
The changes introduced in #47179 made it so that we could try to
build an SSLContext with verification mode set to None, which is
not allowed in FIPS 140 JVMs. This commit address that
2019-10-03 17:14:26 +03:00
Alpar Torok 510d7d84b2 bwc tests: make sure first node is only started once 2019-10-03 16:34:20 +03:00
Paweł Krześniak edbc375916 [DOCS] Change index name in rollover ILM example (#47492)
The warning section above the example tells that index name has to end with the digits but the example itself uses index name without digits which is confusing.
2019-10-03 09:26:10 -04:00
AndyHunt66 4a516f4ae2 [DOCS] Remove duplicated half-sentence from secure settings docs (#47498) 2019-10-03 08:41:06 -04:00
Alpar Torok 93032c9dea Fix bwcTests for versions < 6.5.0 2019-10-03 15:22:01 +03:00
István Zoltán Szabó 9dcdbea228 [DOCS] Amends update datafeed API docs (#47448) 2019-10-03 13:13:18 +02:00
Armin Braun bac119f672
Fix getSnapshotIndexMetaData Exception Behavior (#47488) (#47496)
If we fail to read the global metadata in a snapshot
we would throw `SnapshotMissingException` but wouldn't
do so for the index metadata.
This is breaking SLM tests at a low rate because they
use `SnapshotMissingException` thrown from snapshot status APIs
to wait for a snapshot being gone.
Also, we should be consistent here in general and not leak the
`NoSuchFileException` to the transport layer for index meta.

Closes #46508
2019-10-03 12:47:50 +02:00
Armin Braun 7549be4489
Fix es.http.cname_in_publish_address Deprecation Logging (#47451)
Since the property defaulted to `true` this deprecation logging
runs every time unless its set to `false` manually (in which case
it should've also logged but didn't).
I didn't add a tests and removed the tests we had in `7.x` that
covered this logging. I did move the check out of the `if (InetAddresses.isInetAddress(hostString) == false) {` condition so this is sort-of covered by the REST tests.
IMO, any unit-test of this would be somewhat redundant and would've forced
adding a field that just indicates that the deprecated property was used to
every instance which seemed pointless.

Closes #47436
2019-10-03 11:10:48 +02:00
Russ Cam 3bb17a0c28 Remove string interning from release notes (#47489)
This commit removes the string interning issue and PR from the 7.4.0 release notes. 

The PR https://github.com/elastic/elasticsearch/pull/41039 was closed and not merged.
2019-10-03 11:01:03 +02:00
Alpar Torok 0a14bb174f Remove eclipse conditionals (#44075)
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Ioannis Kakavas d55378e8e7
Add realm uniqueness enforcement in 7.4 migration docs (#47487)
Add the change to the migration docs as it was omitted in #46253
2019-10-03 11:03:29 +03:00
Jim Ferenczi 5a3fa4a479
Add client jar for mapper-extras (#47430)
The rest high level client has a dependency on mapper-extras but the jar
is not published so this commit adds a client jar for this module.

Closes #47413
2019-10-03 01:23:45 +02:00
Nhat Nguyen 44fdf2020a Always flush in FullClusterRestartIT#testRecovery (#47465)
The pattern in the latest failure is similar to the source fixed in #46956
but relates to synced-flush. If peer recovery happens after indexing,
and indexing flushes some shard at the end, then a synced flush in the
test will not roll or commit translog.

Closes #46712
2019-10-02 18:03:22 -04:00
Armin Braun 0beb5263b4
Fix Snapshot Finalization not Waiting for Index Metadata (#47445) (#47459)
* Fix Snapshot Finalization not Waiting for Index Metadata

We were mixing up the listeners here which led to the final listener
that should be called after all the metadata has been written
to be called before that.
I fixed this by removing the one redundant listener and flattening
the logic out.

* Closes #47425
2019-10-02 23:26:18 +02:00
James Rodewig 6b96f53ea0 [DOCS] Change xrefs to external links in 7.4 rls highlights (#47474) 2019-10-02 17:03:21 -04:00
Jason Tedor 52b97ec539
Allow setting validation against arbitrary types (#47264)
Today when settings validate, they can only validate against settings
that are of the same type. While this strong-type is convenient from a
development perspective, it is too limiting in that some settings need
to validate against settings of a different type. For example, the list
setting xpack.monitoring.exporters.<namespace>.host wants to validate
that it is non-empty if and only if the string setting
xpack.monitoring.exporters.<namespace>.type is "http". Today this is
impossible since the settings validation framework only allows that
setting to validate against other list settings. This commit increases
the flexibility here to validate against settings of arbitrary type, at
the expense of losing strong-typing during development.
2019-10-02 16:31:06 -04:00
Lee Hinman 2e3eb4b24e
Add API to execute SLM retention on-demand (#47405) (#47463)
* Add API to execute SLM retention on-demand (#47405)

This is a backport of #47405

This commit adds the `/_slm/_execute_retention` API endpoint. This
endpoint kicks off SLM retention and then returns immediately.

This in particular allows us to run retention without scheduling it
(for entirely manual invocation) or perform a one-off cleanup.

This commit also includes HLRC for the new API, and fixes an issue
in SLMSnapshotBlockingIntegTests where retention invoked prior to the
test completing could resurrect an index the internal test cluster
cleanup had already deleted.

Resolves #46508
Relates to #43663
2019-10-02 12:29:04 -06:00
Mark Vieira 6acc5ca8d1
Remove groovy test code from buildSrc (#47416) 2019-10-02 11:05:04 -07:00
Lee Hinman 013d87d716 Fix AllocationRoutedStepTests.testConditionMetOnlyOneCopyAlloc… (#47313)
* Fix AllocationRoutedStepTests.testConditionMetOnlyOneCopyAllocated

These tests were using randomly generated includes/excludes/requires for
routing, however, it was possible to generate mutually exclusive
allocation settings (about 1 out of 50,000 times for my runs).

This splits the test into three different tests, and removes the
randomization (it doesn't add anything to the testing here) to fix the
issue.

Resolves #47142
2019-10-02 10:01:23 -06:00
Jim Ferenczi c340814b34 Fix highlighting of overlapping terms in the unified highlighter (#47227)
The passage formatter that the unified highlighter use doesn't handle terms with overlapping offsets.
For tokenizer that provides multiple segmentation of the same terms (edge ngram for instance) the formatter
should select the largest span in order to highlight the term only once. This change implements this logic.
2019-10-02 16:34:12 +02:00
Ioannis Kakavas 4f722f0f53
Fix Active Directory tests (#47358) (#47440)
Fixes multiple Active Directory related tests that run against the
samba fixture. Some were failing since we changed the realm settings
format in 7.0 and a few were slightly broken in other ways.
We can move to cleanup the tests in a follow up but this work fits
better to be done with or after we move the tests from a Samba
based fixture to a real(-ish) Microsoft Active Directory based
fixture.

Resolves: #33425, #35738
2019-10-02 17:18:12 +03:00
Benjamin Trent 2228a7dd8d
[ML][Inference] adding ensemble model objects (#47241) (#47438)
* [ML][Inference] adding ensemble model objects

* addressing PR comments

* Update TreeTests.java

* addressing PR comments

* fixing test
2019-10-02 09:49:46 -04:00
Dimitris Athanasiou b9541eb3af
[7.x][ML] Make PUT data frame analytics action a master node action (… (#47433)
While it seemed like the PUT data frame analytics action did not
have to be a master node action as the config is stored in an index
rather than the cluster state, there are other subtle nuances which
make it worthwhile to convert it. In particular, it helps maintain
order of execution for put actions which are anyhow user driven and
are expected to have low volume.

This commit converts `TransportPutDataFrameAnalyticsAction` from
a handled transport action to a master node action.

Note this means that the action might fail in a mixed cluster
but as the API is still experimental and not widely used there will
be few moments more suitable to make this change than now.
2019-10-02 16:24:21 +03:00