Commit Graph

2323 Commits

Author SHA1 Message Date
Dimitris Athanasiou 586453fef1
[ML] Remove types from datafeed (#36538)
Closes #34265
2019-01-04 09:43:44 +02:00
Andrei Stefan 3578e69669
SQL: Handle the bwc Joda ZonedDateTime scripting class in Painless (#37024)
* Handle the bwc Joda ZonedDateTime scripting class in Painless

* Integrated the types checking in the already existent method
2019-01-04 09:19:24 +02:00
Ioannis Kakavas 3c12c372d0
[DOCS] Adjust role mapping docs for SAML (#37083)
Explicitly mention that file based role mappings cannot be used with
the SAML realm.
2019-01-04 08:26:41 +02:00
Tal Levy eaeccd8401
[ILM] Add Freeze Action (#36910)
This commit adds a new ILM Action for
freezing indices in the cold phase.

Closes #34630.
2019-01-03 15:00:40 -08:00
Costin Leau 40a30c6f5f
SQL: Preserve original source for each expression (#36912)
Improve parsing to save the source for each token alongside the location 
 of each Node/Expression for accurate reproducibility of an expression
 name and source

Fix #36894
2019-01-04 00:57:50 +02:00
Benjamin Trent 2c3ce5b9de
[ML] add min_version field to machine learning results index mapping (#37115)
* ML: add min_version field to mapping

* Adding min_version as reserved field name
2019-01-03 14:59:42 -06:00
David Kyle fc49d88b92
[ML] Adjust migration full cluster restart tests based on old cluster version #37118 2019-01-03 17:15:35 +00:00
David Roberts 13649aa70a
[TEST] Revert "Mute ForecastIT.testSingleSeries" (#37110)
The problem that caused the test to be muted was fixed in
https://github.com/elastic/ml-cpp/pull/332

Closes #36258
2019-01-03 16:23:18 +00:00
Benjamin Trent cfc310748d
addressing (#36891)(#36888)(#36889) (#37080) 2019-01-03 07:25:57 -06:00
Dimitris Athanasiou b04b3173db
[ML][TEST] Clean up max_model_memory_limit cluster setting (#37101)
Removes the `xpack.ml.max_model_memory_limit` cluster setting
at the teardown of the `ml_info.yml` tests to ensure the setting
does not trip other tests.
2019-01-03 15:15:31 +02:00
Marios Trivyzas 33137907cf
SQL: Enhance message for PERCENTILE[_RANK] with field as 2nd arg (#36933)
Enhance error message for the case that the 2nd argument of PERCENTILE
and PERCENTILE_RANK is not a foldable, as it doesn't make sense to have
a dynamic value coming from a field.

Fixes: #36903
2019-01-03 13:55:09 +02:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Ioannis Kakavas f0cd19f39b [TEST] Lower deleteInterval in TokenAuthIntegTests
We run subsequent token invalidation requests and we still want to
trigger the deletion of expired tokens so we need to lower the
deleteInterval parameter significantly. Especially now that the
bwc expiration logic is removed and the invalidation process is
much shorter

Resolves #37063
2019-01-02 23:54:02 +02:00
Julie Tibshirani 265fdce312 Fix FullClusterRestartIT#testRollupIDSchemeAfterRestart by adding a missing expected warning. 2019-01-02 10:48:32 -08:00
David Kyle 42bb2bae21
[ML] Order GET job stats response by job id (#36841) 2019-01-02 16:52:20 +00:00
Josh Soref 02aa9f51ee Spelling: correct wrong spellings of similar (#37054) 2019-01-02 17:38:43 +01:00
Josh Soref c04d5c5468 Spelling: correct wrong spellings of "input" (#37048) 2019-01-02 17:34:37 +01:00
Josh Soref 1df66d21fe Spelling: replace uknown with unknown (#37056) 2019-01-02 17:33:02 +01:00
Christoph Büscher 8e6f68f6b0
Fix Eclipse specific type casting issue (#37072)
The eclipse compiler needs some help in infering the functions return values.
2019-01-02 15:58:20 +01:00
Josh Soref 51cb63f934 Spelling: replace substract with subtract (#37055) 2019-01-02 14:15:21 +01:00
Andrei Stefan 9a6c61015c
Use the same format when comparing the error messages containing date fields (#37006)
Test fix: make sure the format in which we check the date is the same as the one we used to format the initial JDBC date field.
2018-12-31 13:41:54 +02:00
Hendrik Muhs 632c7fbed2
[ML] fix x-pack usage regression caused by index migration (#36936)
Changes the feature usage retrieval to use the job manager rather than
directly talking to the cluster state, because jobs can now be either in
cluster state or stored in an index

This is a follow-up of #36702 / #36698
2018-12-31 08:30:08 +01:00
Luca Cavanna c1beb95aa1 Mute LocalIndexFollowingIT#testRemoveRemoteConnection
Relates to #37014
2018-12-28 16:39:36 +01:00
Ioannis Kakavas 0cae979dfe
Remove bwc logic for token invalidation (#36893)
- Removes bwc invalidation logic from the TokenService
- Removes bwc serialization for InvalidateTokenResponse objects as
    old nodes in supported mixed clusters during upgrade will be 6.7 and
    thus will know of the new format
- Removes the created field from the TokensInvalidationResult and the
    InvalidateTokenResponse as it is no longer useful in > 7.0
2018-12-28 13:09:42 +02:00
Ioannis Kakavas 44bd7db59e [TEST] Pass a clientSSLContext in LdapTestCase
If we don't explicitly sett the client SSLSocketFactory when
creating an InMemoryDirectoryServer and setting its SSL config, it
will result in using a TrustAllTrustManager(that extends
X509TrustManager) which is not allowed in a FIPS 140 JVM.
Instead, we get the SSLSocketFactory from the existing SSLContext
and pass that to be used.

Resolves #37013
2018-12-28 11:19:58 +02:00
Tim Vernum e16fd4ebd6
Change missing authn message to not mention tokens (#36750)
The phrase "missing authentication token" is historic and is based
around the use of "AuthenticationToken" objects inside the Realm code.

However, now that we have a TokenService and token API, this message
would sometimes lead people in the wrong direction and they would try
and generate a "token" for authentication purposes when they would
typically just need a username:password Basic Auth header.

This change replaces the word "token" with "credentials".
2018-12-28 15:26:46 +11:00
Tim Vernum 51c18c0e76
Reload SSL context on file change for LDAP (#36937)
In #30509 we changed the way SSL configuration is reloaded when the
content of a file changes. As a consequence of that implementation
change the LDAP realm ceased to pick up changes to CA files (or other
certificate material) if they changed.

This commit repairs the reloading behaviour for LDAP realms, and adds
a test for this functionality.

Resolves: #36923
2018-12-28 11:31:34 +11:00
Nhat Nguyen 7580d9d925
Make SourceToParse immutable (#36971)
Today the routing of a SourceToParse is assigned in a separate step
after the object is created. We can easily forget to set the routing.
With this commit, the routing must be provided in the constructor of
SourceToParse.

Relates #36921
2018-12-24 14:06:50 -05:00
Martijn van Groningen 561b704129
[CCR] AutoFollowCoordinator and follower index already created (#36540)
The AutoFollowCoordinator should be resilient to the fact that the follower
index has already been created and in that case it should only update
the auto follow metadata with the fact that the follower index was created.

Relates to #33007
2018-12-24 10:16:38 +01:00
Martijn van Groningen 44fe265d82
[CCR] Added auto_follow_exception.timestamp field to auto follow stats (#36947)
Currently auto follow stats users are unable to see whether an auto follow
error was recent or old. The new timestamp field will help user distinguish
between old and new errors.
2018-12-24 07:53:51 +01:00
Martijn van Groningen 4fb62fcba6
Make CCR resilient against missing remote cluster connections (#36682)
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behavior similar how CCR
deals with authorization errors.

Closes #36667
Closes #36255
2018-12-24 07:28:34 +01:00
Jason Tedor 7e2bb9c1aa
Fix NPE in CachingUsernamePasswordRealm (#36953)
This commit fixes an NPE in the CachingUsernamePasswordRealm when the
cache is disabled.
2018-12-22 07:34:14 -05:00
Andrei Stefan 09fa827adc
SQL: documentation improvements and updates (#36918)
* Added Limitations page
* Made the aggregations page follow the common template for functions
* Modified all tables to have the first row's cells content centered
* Polishing in other various sections
2018-12-21 23:25:54 +02:00
Dimitrios Liappis 8e97bf9d06
Fix typo in unitTest task (#36930)
Fix the typo introduced in #36311 causing CI failures with the
FipsJvm.
2018-12-21 15:18:43 +02:00
Tanguy Leroux bd2af2c400 Merge branch 'master' into close-index-api-refactoring 2018-12-21 12:22:24 +01:00
Martijn van Groningen 4ded4717fe
[CCR] Add `ccr.auto_follow_coordinator.wait_for_timeout` setting (#36714)
This setting controls the wait for timeout the autofollow coordinator
should use when setting cluster state requests to a remote cluster.
2018-12-21 09:36:40 +01:00
Michael Basnight 7cbf03c001
Scripting: Remove deprecated params.ctx (#36848)
When the script contexts were created in 6, the use of params.ctx was
deprecated. This commit cleans up that code and ensures that params.ctx
is null in both watcher script contexts.

Relates: #34059
2018-12-20 21:30:24 -06:00
Tim Vernum 59da7c3cc4
Improve error message for 6.x style realm settings (#36876)
Realm settings were changed in #30241 in a non-BWC way.
If you try and start a 7.x node using a 6.x config style, then the
default error messages do not adequately describe the cause of
the problem, or the solution.

This change detects the when realms are using the 6.x style and fails
with a specific error message.

This detection is a best-effort, and will detect issues when the
realms have not been modified to use the 7.x style, but may not detect
situations where the configuration was partially changed.

e.g. We can detect this:

    xpack.security.authc:
      realms.pki1.type: pki
      realms.pki1.order: 3
      realms.pki1.ssl.certificate_authorities: [ "ca.crt" ]

But this (where the "order" has been updated, but the "ssl.*" has not)
will fall back to the standard "unknown setting" check

    xpack.security.authc:
      realms.pki.pki1.order: 3
      realms.pki1.ssl.certificate_authorities: [ "ca.crt" ]

Closes: #36026
2018-12-21 12:02:02 +11:00
Tim Brooks d9b2ed6135
Send clear session as routable remote request (#36805)
This commit adds a RemoteClusterAwareRequest interface that allows a
request to specify which remote node it should be routed to. The remote
cluster aware client will attempt to route the request directly to this
node. Otherwise it will send it as a proxy action to eventually end up
on the requested node.

It implements the ccr clean_session action with this client.
2018-12-20 17:43:12 -07:00
Costin Leau ac032a0b9d
SQL: Fix bug regarding histograms usage in scripting (#36866)
Allow scripts to correctly reference grouping functions
Fix bug in translation of date/time functions mixed with histograms.
Enhance Verifier to prevent histograms being nested inside other
 functions inside GROUP BY (as it implies double grouping)
Extend Histogram docs
2018-12-20 23:11:56 +02:00
Tim Brooks 4cd570593d
Update index mappings when ccr restore complete (#36879)
This is related to #35975. When the shard restore process is complete,
the index mappings need to be updated to ensure that the data in the
files restores is compatible with the follower mappings. This commit
implements a mapping update as the final step in a shard restore.
2018-12-20 13:53:04 -07:00
Michael Basnight d00780d00c
Watcher: Remove unused local variable in doExecute (#36655) 2018-12-20 13:43:10 -06:00
lcawl 32bed098bb [DOCS] Synchs titles of X-Pack APIs 2018-12-20 10:27:24 -08:00
Tal Levy f6c1e3f14f
[ILM][TEST] increase assertBusy timeout (#36864)
the testFullPolicy and testMoveToRolloverStep tests
are very important tests, but they sometimes timeout
beyond the default 10sec wait for shrink to occur.
This commit increases one of the assertBusys to
20 seconds
2018-12-20 08:55:02 -08:00
Martijn van Groningen b42074c1cc
[CCR] Report error if auto follower tries auto follow a leader index with soft deletes disabled (#36886)
Currently if a leader index with soft deletes disabled is auto followed then this index is silently ignored.
This commit changes this behavior to mark these indices as auto followed and report an error, which is visible in auto follow stats. Marking the index as auto follow is important, because otherwise the auto follower will continuously try to auto follow and fail.

Relates to #33007
2018-12-20 15:21:52 +01:00
Martijn van Groningen 7b1dfeff2e
Renamed `WHITE_LISTED_SETTINGS` to `NON_REPLICATED_SETTINGS`
because the latter better describes the purpose of this field.
2018-12-20 15:08:04 +01:00
Dimitris Athanasiou 08bcd83757
[ML] Reduce persistent tasks periodic reassignment interval in ... (#36845)
... MlDistributedFailureIT.testLoseDedicatedMasterNode.

An intermittent failure has been observed in
`MlDistributedFailureIT. testLoseDedicatedMasterNode`.
The test launches a cluster comprised by a dedicated master node
and a data and ML node. It creates a job and datafeed and starts them.
It then shuts down and restarts the master node. Finally, the test asserts
that the two tasks have been reassigned within 10s.

The intermittent failure is due to the assertions that the tasks have been
reassigned failing. Investigating the failure revealed that the `assertBusy`
that performs that assertion times out. Furthermore, it appears that the
job task is not reassigned because the memory tracking info is stale.

Memory tracking info is refreshed asynchronously when a job is attempted
to be reassigned. Tasks are attempted to be reassigned either due to a relevant
cluster state change or periodically. The periodic interval is controlled by a cluster
setting called `cluster.persistent_tasks.allocation.recheck_interval` and defaults to 30s.

What seems to be happening in this test is that if all cluster state changes after the
master node is restarted come through before the async memory info refresh completes,
then the job might take up to 30s until it is attempted to reassigned. Thus the `assertBusy`
times out.

This commit changes the test to reduce the periodic check that reassigns persistent
tasks to `200ms`. If the above theory is correct, this should eradicate those failures.

Closes #36760
2018-12-20 14:53:36 +02:00
David Roberts 0f2f00a20a
[ML] Resolve 7.0.0 TODOs in ML code (#36842)
This change cleans up a number of ugly BWC
workarounds in the ML code.

7.0 cannot run in a mixed version cluster with
versions prior to 6.7, so code that deals with
these old versions is no longer required.

Closes #29963
2018-12-20 12:49:57 +00:00
Andrei Stefan 1236461e3e
SQL: Make sure now() always uses milliseconds precision (#36877)
* This change is to account for different system clock implementations
or different Java versions (for Java 8, milliseconds precision is used;
for Java 9+ a system specific clock implementation is used which can
have greater precision than what we need here).
2018-12-20 13:39:55 +02:00
Martijn van Groningen 9c2980a55e
Remote cluster license checker and no license info. (#36837)
Fail with a descriptive error when the xpack info returns no license info.

Relates to #36815
2018-12-20 09:58:04 +01:00
Ryan Ernst cfc0a47232
Core: Deprecate negative epoch timestamps (#36793)
Negative timestamps are currently supported in joda time. These are
dates before epoch. However, it doesn't really make sense to have a
negative timestamp, since this is a modern format. Any dates before
epoch can be represented with normal date formats, like ISO8601.
Additionally, implementing negative epoch timestamp parsing in java time
has an edge case which would more than double the code required. This
commit deprecates use of negative epoch timestamps.
2018-12-20 00:17:06 -08:00
Marios Trivyzas 6221d6be49
SQL: Fix issue with always false filter involving functions (#36830)
When a filter is evaluated to false then it becomes a LocalRelation
with an EmptyExecutable. The LocalRelation in turn, becomes a
LocalExec and the the SkipQueryIfFoldingProjection was wrongly
converting it to a SingletonExecutable. Moreover made a change, so
that the queries without FROM clause, which are supposed to return a
single row, to become a LocalRelation with a SingletonExecutable
instead of EmptyExecutable to avoid mixing up with the ones operating
on a table but with a filter that evaluates to false.

Fixes: #35980
2018-12-20 09:56:07 +02:00
lcawl 8b8121279a [DOCS] Fix typos 2018-12-19 15:28:25 -08:00
Ioannis Kakavas c4e12f06d7
Link to the SAML troubleshooting doc from guide (#36754)
Explicitly call out the existence of the troubleshooting guide so
that hopefully users can solve common and easy problems with their
initial configuration
2018-12-20 01:11:55 +02:00
Lisa Cawley 4140b9eede
[DOCS] Update X-Pack terminology in security docs (#36564) 2018-12-19 14:53:37 -08:00
Julie Tibshirani 5185d61ad1
Make sure to use _doc for the type in SQL REST tests. (#36806) 2018-12-19 11:25:45 -08:00
Gordon Brown d39956c65c
Remove `indexing_complete` when removing policy (#36620)
Leaving `index.lifecycle.indexing_complete` in place when removing the
lifecycle policy from an index can cause confusion, as if a new policy
is associated with the policy, rollover will be silently skipped.
Removing that setting when removing the policy from an index makes
associating a new policy with the index more involved, but allows ILM to
fail loudly, rather than silently skipping operations which the user may
assume are being performed.

* Adjust order of checks in WaitForRolloverReadyStep

This allows ILM to error out properly for indices that have a valid
alias, but are not the write index, while still handling
`indexing_complete` on old-style aliases and rollover (that is, those
which only point to a single index at a time with no explicit write
index)
2018-12-19 12:11:30 -07:00
Gordon Brown f99975bc29
Rename confusing variable holding name of Logstash index template (#36840)
The previous name was confusing, so this commit clarifies it.
2018-12-19 12:09:37 -07:00
Tanguy Leroux fb24469fe7 Merge branch 'master' into close-index-api-refactoring 2018-12-19 16:17:26 +01:00
Costin Leau 9584adf9d9
SQL: Enhance Verifier to prevent aggregate or grouping functions from (#36799)
Improve Verifier to prevent aggregate or grouping functions from
 being used in a WHERE clause.

Fix #36798
2018-12-19 17:14:01 +02:00
Andrei Stefan d31eaf7313
SQL: protocol returns ISO 8601 String formatted dates instead of Long for JDBC/ODBC requests (#36800)
* Change the way the protocol returns date fields from Long values in case
of JDBC/ODBC, to ISO 8601 with millis String.
2018-12-19 16:36:16 +02:00
David Kyle d43cbdab97
[ML] ensure the ml-config index (#36792) (#36832) 2018-12-19 13:43:43 +00:00
David Roberts ad20d6bb83
[ML] Followup to annotations index creation (#36824)
Fixes two minor problems reported after merge of #36731:

1. Name the creation method to make clear it only creates
   if necessary
2. Avoid multiple simultaneous in-flight creation requests
2018-12-19 13:06:24 +00:00
Martijn van Groningen 18691daebe
[TEST] Renamed ccr qa module. 2018-12-19 13:57:12 +01:00
Albert Zaharovits 63aa8756b2
Add X-Forwarded-For to the logfile audit (#36427)
Extracts the value of the X-Forwarded-For HTTP request header and
places it in the audit entries from the logfile output.
2018-12-19 14:56:40 +02:00
Martijn van Groningen 3cc0cf03c6
[TEST] No need to specifically check licensesMetaData on master node. 2018-12-19 13:51:24 +01:00
Martijn van Groningen a6af33ef0b
[TEST] Wait for license metadata to be installed 2018-12-19 13:03:45 +01:00
Tanguy Leroux c99fd6a53b Merge branch 'master' into close-index-api-refactoring 2018-12-19 09:34:59 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Tim Vernum 06b175dd69 [TEST] Mute MlMigrationFullClusterRestartIT
AwaitsFix: https://github.com/elastic/elasticsearch/issues/36816

This test fails quite reliably.
2018-12-19 17:02:26 +11:00
Tim Brooks aaf466ff5e
Revert transport.port change for tests (#36809)
Commit #36786 updated docs and strings to reference transport.port instead of
transport.tcp.port. However, this breaks backwards compatibility tests
as the tests rely on string configurations and transport.port does not
exist prior to 6.6. This commit reverts the places were we reference
transport.tcp.port for tests. This work will need to be reintroduced in
a backwards compatible way.
2018-12-18 19:01:13 -07:00
Benjamin Trent 1d429cf1c9
ML having delayed data detection create annotations (#36796)
* ML having delayed data detection create annotations

* adding upsertAsDoc, audit, and changing user

* changing update to just index the doc with the id set
2018-12-18 18:40:38 -06:00
Nick Knize 20b58f0b0f
[GEO] Fork Lucene's LatLonShape Classes to local lucene package (#36794)
Lucene 7.6 uses a smaller encoding for LatLonShape. This commit forks the LatLonShape classes to Elasticsearch's local lucene package. These classes will be removed on the release of Lucene 7.6.
2018-12-18 16:48:30 -06:00
lcawl a8387592db [DOCS] Fixes broken links to tcp_tranport_profiles 2018-12-18 14:43:58 -08:00
Gordon Brown 7bd7f022ef
Rename logstash management index template (#36626)
The logstash management template was named in such a way as to confuse
users, who misunderstood it to be a template for indices created by
logstash. It is now renamed to more clearly communicate its purpose and
match the format of the other templates for system indices.
2018-12-18 15:37:54 -07:00
Zachary Tong 6d49873ab7
Fix Rollup's metadata parser (#36791)
The parser used for rollup configs in _meta fields was not able to
handle unrelated data in the meta field.  If an unrelated object
was encountered, it would half-consume the JSON object, realize it
wasn'ta rollup config, then stop parsing.  This would leave the object
halfway consumed and the parsing framework would throw an exception.

This commit replaces the parsing logic with a set of minimal parsers,
each for the specific component we care about (`_doc`, `_meta`,
`_rollup`) and configured to ignore unknown fields where applicable.

More verbose, but less hacky than before and should be more robust.

Also adds tests (randomized and explicit) to make sure this doesn't
break in the future.
2018-12-18 16:35:39 -05:00
Tim Brooks 47a9a8de49
Update transport docs and settings for changes (#36786)
This is related to #36652. In 7.0 we plan to deprecate a number of
settings that make reference to the concept of a tcp transport. We
mostly just have a single transport type now (based on tcp). Settings
should only reference tcp if they are referring to socket options. This
commit updates the settings in the docs. And removes string usages of
the old settings. Additionally it adds a missing remote compress setting
to the docs.
2018-12-18 13:09:58 -07:00
Tim Brooks 1fa105658e
Add CcrRestoreSourceService to track sessions (#36578)
This commit is related to #36127. It adds a CcrRestoreSourceService to
track Engine.IndexCommitRef need for in-process file restores. When a
follower starts restoring a shard through the CcrRepository it opens a
session with the leader through the PutCcrRestoreSessionAction. The
leader responds to the request by telling the follower what files it
needs to fetch for a restore. This is not yet implemented.

Once, the restore is complete, the follower closes the session with the
DeleteCcrRestoreSessionAction action.
2018-12-18 11:23:13 -07:00
David Kyle e294056bbf
[ML] Merge the Jindex master feature branch (#36702)
* [ML] Job and datafeed mappings with index template (#32719)

Index mappings for the configuration documents

* [ML] Job config document CRUD operations (#32738)

* [ML] Datafeed config CRUD operations (#32854)

* [ML] Change JobManager to work with Job config in index  (#33064)

* [ML] Change Datafeed actions to read config from the config index (#33273)

* [ML] Allocate jobs based on JobParams rather than cluster state config (#33994)

* [ML] Return missing job error when .ml-config is does not exist (#34177)

* [ML] Close job in index (#34217)

* [ML] Adjust finalize job action to work with documents (#34226)

* [ML] Job in index: Datafeed node selector (#34218)

* [ML] Job in Index: Stop and preview datafeed (#34605)

* [ML] Delete job document (#34595)

* [ML] Convert job data remover to work with index configs (#34532)

* [ML] Job in index: Get datafeed and job stats from index (#34645)

* [ML] Job in Index: Convert get calendar events to index docs (#34710)

* [ML] Job in index: delete filter action (#34642)

This changes the delete filter action to search
for jobs using the filter to be deleted in the index
rather than the cluster state.

* [ML] Job in Index: Enable integ tests (#34851)

Enables the ml integration tests excluding the rolling upgrade tests and a lot of fixes to
make the tests pass again.

* [ML] Reimplement established model memory (#35500)

This is the 7.0 implementation of a master node service to
keep track of the native process memory requirement of each ML
job with an associated native process.

The new ML memory tracker service works when the whole cluster
is upgraded to at least version 6.6. For mixed version clusters
the old mechanism of established model memory stored on the job
in cluster state was used. This means that the old (and complex)
code to keep established model memory up to date on the job object
has been removed in 7.0.

Forward port of #35263

* [ML] Need to wait for shards to replicate in distributed test (#35541)

Because the cluster was expanded from 1 node to 3 indices would
initially start off with 0 replicas.  If the original node was
killed before auto-expansion to 1 replica was complete then
the test would fail because the indices would be unavailable.

* [ML] DelayedDataCheckConfig index mappings (#35646)

* [ML] JIndex: Restore finalize job action (#35939)

* [ML] Replace Version.CURRENT in streaming functions (#36118)

* [ML] Use 'anomaly-detector' in job config doc name (#36254)

* [ML] Job In Index: Migrate config from the clusterstate (#35834)

Migrate ML configuration from clusterstate to index for closed jobs
only once all nodes are v6.6.0 or higher

* [ML] Check groups against job Ids on update (#36317)

* [ML] Adapt to periodic persistent task refresh (#36633)

* [ML] Adapt to periodic persistent task refresh

If https://github.com/elastic/elasticsearch/pull/36069/files is
merged then the approach for reallocating ML persistent tasks
after refreshing job memory requirements can be simplified.
This change begins the simplification process.

* Remove AwaitsFix and implement TODO

* [ML] Default search size for configs

* Fix TooManyJobsIT.testMultipleNodes

Two problems:

1. Stack overflow during async iteration when lots of
   jobs on same machine
2. Not effectively setting search size in all cases

* Use execute() instead of submit() in MlMemoryTracker

We don't need a Future to wait for completion

* [ML][TEST] Fix NPE in JobManagerTests

* [ML] JIindex: Limit the size of bulk migrations (#36481)

* [ML] Prevent updates and upgrade tests (#36649)

* [FEATURE][ML] Add cluster setting that enables/disables config  migration (#36700)

This commit adds a cluster settings called `xpack.ml.enable_config_migration`.
The setting is `true` by default. When set to `false`, no config migration will
be attempted and non-migrated resources (e.g. jobs, datafeeds) will be able
to be updated normally.

Relates #32905

* [ML] Snapshot ml configs before migrating (#36645)

* [FEATURE][ML] Split in batches and migrate all jobs and datafeeds (#36716)

Relates #32905

* SQL: Fix translation of LIKE/RLIKE keywords (#36672)

* SQL: Fix translation of LIKE/RLIKE keywords

Refactor Like/RLike functions to simplify internals and improve query
 translation when chained or within a script context.

Fix #36039
Fix #36584

* Fixing line length for EnvironmentTests and RecoveryTests (#36657)

Relates #34884

* Add back one line removed by mistake regarding java version check and
COMPAT jvm parameter existence

* Do not resolve addresses in remote connection info (#36671)

The remote connection info API leads to resolving addresses of seed
nodes when invoked. This is problematic because if a hostname fails to
resolve, we would not display any remote connection info. Yet, a
hostname not resolving can happen across remote clusters, especially in
the modern world of cloud services with dynamically chaning
IPs. Instead, the remote connection info API should be providing the
configured seed nodes. This commit changes the remote connection info to
display the configured seed nodes, avoiding a hostname resolution. Note
that care was taken to preserve backwards compatibility with previous
versions that expect the remote connection info to serialize a transport
address instead of a string representing the hostname.

* [Painless] Add boxed type to boxed type casts for method/return (#36571)

This adds implicit boxed type to boxed types casts for non-def types to create asymmetric casting relative to the def type when calling methods or returning values. This means that a user calling a method taking an Integer can call it with a Byte, Short, etc. legally which matches the way def works. This creates consistency in the casting model that did not previously exist.

* SNAPSHOTS: Adjust BwC Versions in Restore Logic (#36718)

* Re-enables bwc tests with adjusted version conditions now that #36397 enables concurrent snapshots in 6.6+

* ingest: fix on_failure with Drop processor (#36686)

This commit allows a document to be dropped when a Drop processor
is used in the on_failure fork of the processor chain.

Fixes #36151

* Initialize startup `CcrRepositories` (#36730)

Currently, the CcrRepositoryManger only listens for settings updates
and installs new repositories. It does not install the repositories that
are in the initial settings. This commit, modifies the manager to
install the initial repositories. Additionally, it modifies the ccr
integration test to configure the remote leader node at startup, instead
of using a settings update.

* [TEST] fix float comparison in RandomObjects#getExpectedParsedValue

This commit fixes a test bug introduced with #36597. This caused some
test failure as stored field values comparisons would not work when CBOR
xcontent type was used.

Closes #29080

* [Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)

This commit  exposes lucene's LatLonShape field as the
default type in GeoShapeFieldMapper. To use the new 
indexing approach, simply set "type" : "geo_shape" in 
the mappings without setting any of the strategy, precision, 
tree_levels, or distance_error_pct parameters. Note the 
following when using the new indexing approach:

* geo_shape query does not support querying by 
MULTIPOINT.
* LINESTRING and MULTILINESTRING queries do not 
yet support WITHIN relation.
* CONTAINS relation is not yet supported.
The tree, precision, tree_levels, distance_error_pct, 
and points_only parameters are deprecated.

* TESTS:Debug Log. IndexStatsIT#testFilterCacheStats

* ingest: support default pipelines + bulk upserts (#36618)

This commit adds support to enable bulk upserts to use an index's
default pipeline. Bulk upsert, doc_as_upsert, and script_as_upsert
are all supported.

However, bulk script_as_upsert has slightly surprising behavior since
the pipeline is executed _before_ the script is evaluated. This means
that the pipeline only has access the data found in the upsert field
of the script_as_upsert. The non-bulk script_as_upsert (existing behavior)
runs the pipeline _after_ the script is executed. This commit
does _not_ attempt to consolidate the bulk and non-bulk behavior for
script_as_upsert.

This commit also adds additional testing for the non-bulk behavior,
which remains unchanged with this commit.

fixes #36219

* Fix duplicate phrase in shrink/split error message (#36734)

This commit removes a duplicate "must be a" from the shrink/split error
messages.

* Deprecate types in get_source and exist_source (#36426)

This change adds a new untyped endpoint `{index}/_source/{id}` for both the
GET and the HEAD methods to get the source of a document or check for its
existance. It also adds deprecation warnings to RestGetSourceAction that emit
a warning when the old deprecated "type" parameter is still used. Also updating
documentation and tests where appropriate.

Relates to #35190

* Revert "[Geo] Integrate Lucene's LatLonShape (BKD Backed GeoShapes) as default `geo_shape` indexing approach (#35320)"

This reverts commit 5bc7822562.

* Enhance Invalidate Token API (#35388)

This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556

* Add raw sort values to SearchSortValues transport serialization (#36617)

In order for CCS alternate execution mode (see #32125) to be able to do the final reduction step on the CCS coordinating node, we need to serialize additional info in the transport layer as part of each `SearchHit`. Sort values are already present but they are formatted according to the provided `DocValueFormat` provided. The CCS node needs to be able to reconstruct the lucene `FieldDoc` to include in the `TopFieldDocs` and `CollapseTopFieldDocs` which will feed the `mergeTopDocs` method used to reduce multiple search responses (one per cluster) into one.

This commit adds such information to the `SearchSortValues` and exposes it through a new getter method added to `SearchHit` for retrieval. This info is only serialized at transport and never printed out at REST.

* Watcher: Ensure all internal search requests count hits (#36697)

In previous commits only the stored toXContent version of a search
request was using the old format. However an executed search request was
already disabling hit counts. In 7.0 hit counts will stay enabled by
default to allow for proper migration.

Closes #36177

* [TEST] Ensure shard follow tasks have really stopped.

Relates to #36696

* Ensure MapperService#getAllMetaFields elements order is deterministic (#36739)

MapperService#getAllMetaFields returns an array, which is created out of
an `ObjectHashSet`. Such set does not guarantee deterministic hash
ordering. The array returned by its toArray may be sorted differently
at each run. This caused some repeatability issues in our tests (see #29080)
as we pick random fields from the array of possible metadata fields,
but that won't be repeatable if the input array is sorted differently at
every run. Once setting the tests seed, hppc picks that up and the sorting is
deterministic, but failures don't repeat with the seed that gets printed out
originally (as a seed was not originally set).
See also https://issues.carrot2.org/projects/HPPC/issues/HPPC-173.

With this commit, we simply create a static sorted array that is used for
`getAllMetaFields`. The change is in production code but really affects
only testing as the only production usage of this method was to iterate
through all values when parsing fields in the high-level REST client code.
Anyways, this seems like a good change as returning an array would imply
that it's deterministically sorted.

* Expose Sequence Number based Optimistic Concurrency Control in the rest layer (#36721)

Relates #36148 
Relates #10708

* [ML] Mute MlDistributedFailureIT
2018-12-18 17:45:31 +00:00
Jim Ferenczi f05c404934 Fix rollup search statistics (#36674)
This change fixes the rollup statistics regarding search times. Search times are
computed from the first query and never updated. This commit adds the missing
calls to the subsequent search.
2018-12-18 10:16:33 -05:00
Marios Trivyzas 74502d7f4b
SQL: Fix wrong appliance of StackOverflow limit for IN (#36724)
Fix grammar so that each element inside the list of values for IN
is a valueExpression and not a more generic expression. Introduce a
mapping for context names as some rules in the grammar are exited with
a different rule from the one they entered.This helps so that the decrement
of depth counts in the Parser's CircuitBreakerListener works correctly.

For the list of values for IN, don't count the
PrimaryExpressionContext as this is not visited on exitRule() due to
the peculiarity in our gramamr with the predicate and predicated.

Fixes: #36592
2018-12-18 17:07:27 +02:00
Martijn van Groningen 1afcfc97bd
[TEST] Added more logging
Relates to #36761
2018-12-18 16:01:02 +01:00
Tanguy Leroux cd3a1af77b Merge branch 'master' into close-index-api-refactoring 2018-12-18 15:52:39 +01:00
Mayya Sharipova f884b2b1cd
Deprecate types in index API (#36575)
* Deprecate types in index API

- deprecate type-based constructors of IndexRequest
- update tests to use typeless IndexRequest constructors
- no yaml tests as they have been already added in #35790

Relates to #35190
2018-12-18 08:53:49 -05:00
Boaz Leskes 5f76f39386
Rename seq# powered optimistic concurrency control parameters to ifSeqNo/ifPrimaryTerm (#36757)
This PR renames the parameters previously introduce to the following:

### URL Parameters
```
PUT twitter/_doc/1?if_seq_no=501&if_primary_term=1
{
    "user" : "kimchy",
    "post_date" : "2009-11-15T14:12:12",
    "message" : "trying out Elasticsearch"
}

DELETE twitter/_doc/1?if_seq_no=501&if_primary_term=1
```

### Bulk API
```
POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1", "if_seq_no": 501, "if_primary_term": 1 } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2", "if_seq_no": 501, "if_primary_term": 1 } }
```

### Java API
```
IndexRequest.ifSeqNo(long seqNo)
IndexRequest.ifPrimaryTerm(long primaryTerm)
DeleteRequest.ifSeqNo(long seqNo)
DeleteRequest.ifPrimaryTerm(long primaryTerm)
```

Relates #36148
Relates #10708
2018-12-18 14:35:18 +01:00
Martijn van Groningen 0ff1f1fa18
Muted tests.
Relates to #36764
2018-12-18 13:39:01 +01:00
David Roberts 624307410e
[ML] Create the ML annotations index (#36731)
The ML UI now provides the ability for users to annotate
time periods with arbitrary text to add insight to what
happened.

This change makes the backend create the index for these
annotations, together with read and write aliases to
make future upgrades possible without adding complexity
to the UI.

It also adds read and write permission to the index for
all ML users (not just admins).

The spec for the index is in
https://github.com/elastic/kibana/pull/26034/files#diff-c5c6ac3dbb0e7c91b6d127aa06121b2cR7

Relates #33376
Relates elastic/kibana#26034
2018-12-18 12:18:29 +00:00
Andrei Stefan 4bc9bffb4d
SQL: Extend the ODBC metric by differentiating between 32 and 64bit platforms (#36753)
* The "overall" ODBC metric will now track a total of requests between 32bit and 64bit ODBC metrics, being calculated passively, upon request.
2018-12-18 13:29:49 +02:00
Martijn van Groningen 57e1a4bc9f
[TEST] Ensure shard follow tasks have really stopped.
Relates to #36696
2018-12-18 10:43:27 +01:00
Alexander Reelsen 00521a5b36
Watcher: Ensure all internal search requests count hits (#36697)
In previous commits only the stored toXContent version of a search
request was using the old format. However an executed search request was
already disabling hit counts. In 7.0 hit counts will stay enabled by
default to allow for proper migration.

Closes #36177
2018-12-18 10:11:39 +01:00
Tanguy Leroux 0a0c969517 Merge branch 'master' into close-index-api-refactoring 2018-12-18 09:27:35 +01:00
Ioannis Kakavas 7b9ca62174
Enhance Invalidate Token API (#35388)
This change:

- Adds functionality to invalidate all (refresh+access) tokens for all users of a realm
- Adds functionality to invalidate all (refresh+access)tokens for a user in all realms
- Adds functionality to invalidate all (refresh+access) tokens for a user in a specific realm
- Changes the response format for the invalidate token API to contain information about the 
   number of the invalidated tokens and possible errors that were encountered.
- Updates the API Documentation

After back-porting to 6.x, the `created` field will be removed from master as a field in the 
response

Resolves: #35115
Relates: #34556
2018-12-18 10:05:50 +02:00
Tim Brooks 3dd5a5a3c5
Initialize startup `CcrRepositories` (#36730)
Currently, the CcrRepositoryManger only listens for settings updates
and installs new repositories. It does not install the repositories that
are in the initial settings. This commit, modifies the manager to
install the initial repositories. Additionally, it modifies the ccr
integration test to configure the remote leader node at startup, instead
of using a settings update.
2018-12-17 13:19:32 -07:00
Andrei Stefan 1f80c80b3a Add back one line removed by mistake regarding java version check and
COMPAT jvm parameter existence
2018-12-17 19:06:23 +02:00
Costin Leau 6d9d5e397b
SQL: Fix translation of LIKE/RLIKE keywords (#36672)
* SQL: Fix translation of LIKE/RLIKE keywords

Refactor Like/RLike functions to simplify internals and improve query
 translation when chained or within a script context.

Fix #36039
Fix #36584
2018-12-17 18:58:06 +02:00
Boaz Leskes e356b8cb95
Add doc's sequence number + primary term to GetResult and use it for updates (#36680)
This commit adds the last sequence number and primary term of the last operation that have
modified a document to `GetResult` and uses it to power the Update API.

Relates #36148 
Relates #10708
2018-12-17 15:22:13 +01:00
Martijn van Groningen a181a25226
[CCR] Add time since last auto follow fetch to auto follow stats (#36542)
For each remote cluster the auto follow coordinator, starts an auto
follower that checks the remote cluster state and determines whether an
index needs to be auto followed. The time since last auto follow is
reported per remote cluster and gives insight whether the auto follow
process is alive.

Relates to #33007
Originates from #35895
2018-12-17 14:14:56 +01:00
Albert Zaharovits 6f038997e1
Watcher accounts constructed lazily (#36656)
This fixes two bugs about watcher notifications:
* registering accounts that had only secure settings was not possible before;
these accounts are very much practical for Slack and PagerDuty integrations.
* removes the limitation that, for an account with both secure and cluster settings,
the admin had to first change/add the secure settings and only then add the
dependent dynamic cluster settings. The reverse order would trigger a
SettingsException for an incomplete account.

The workaround is to lazily instantiate account objects, hoping that when accounts
are instantiated all the required settings are in place. Previously, the approach
was to greedily validate all the account settings by constructing the account objects,
even if they would not ever be used by actions. This made sense in a world where
all the settings were set by a single API. But given that accounts have dependent
settings (that must be used together) that have to be changed using different APIs
(POST _nodes/reload_secure_settings and PUT _cluster/settings), the settings group
would technically be in an invalid state in between the calls.
This fix builds account objects, and validates the settings, when they are
needed by actions.
2018-12-17 14:45:55 +02:00
Andrei Stefan 2ed6ab9648
SQL: Concat should be always not nullable (#36601) 2018-12-17 14:00:56 +02:00
Costin Leau b376edf269
SQL: Move internals from TimeZone to ZoneId (#36651)
As the internals have moved to java.time, the usage of TimeZone itself
 should be minimized as it creates issues when being converted to ZoneId
Protocol wise the two are mostly identical so consumer should not see
 any difference.
Note that terminology wise, inside the docs, the public API and inside
 the protocol timeZone will continue to be used as it's more widely
 understood as oppose to zoneId which is an implementation detail
 specific to the JVM

Fix #36535
2018-12-17 13:55:44 +02:00