Commit Graph

1065 Commits

Author SHA1 Message Date
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Simon Willnauer 833e0f8ecf
Prevent throttled indices to be searched through wildcards by default (#34354)
Today if a wildcard, date-math expression or alias expands/resolves
to an index that is search-throttled we still search it. This is likely
not the desired behavior since it can unexpectedly slow down searches
significantly.

This change adds a new indices option that allows `search`, `count`
and `msearch` to ignore throttled indices by default. Users can
force expansion to throttled indices by using `ignore_throttled=true`
on the rest request to expand also to throttled indices.

Relates to #34352
2018-11-06 09:45:30 +01:00
Tim Vernum 574ec6686e
Include realm type in Security Realm setting keys (#30241)
This moves all Realm settings to an Affix definition.
However, because different realm types define different settings
(potentially conflicting settings) this requires that the realm type
become part of the setting key.

Thus, we now need to define realm settings as:

    xpack.security.authc.realms:
      file.file1:
        order: 0

      native.native1:
        order: 1

- This is a breaking change to realm config
- This is also a breaking change to custom security realms (SecurityExtension)
2018-11-06 14:56:50 +11:00
Lee Hinman 3ee004c7b9
Add dedicated step for checking shrink allocation status (#35161)
This adds a new step for checking whether an index is allocated correctly based
on the rules added prior to running the shrink step. It also fixes a bug where
for shrink we are not allowed to have the shards relocating for the shrink step.

This also allows us to simplify AllocationRoutedStep and provide better
feedback in the step info for why either the allocation or the shrink checks
have failed.

Resolves #34938
2018-11-05 12:02:59 -07:00
Gordon Brown 0fbb8a16bc
Skip Rollover step if next index already exists (#35168)
If the Rollover step would fail due to the next index in sequence
already existing, just skip to the next step instead of going to the
Error step.

This prevents spurious `ResourceAlreadyExistsException`s created by
simultaneous RolloverStep executions from causing ILM to error out
unnecessarily.
2018-11-05 09:20:43 -07:00
Lee Hinman 3473217563
Remove Joda usage from ILM (#35220)
This commit removes the Joda time usage from ILM and the HLRC components of ILM.
It also fixes an issue where using the `?human=true` flag could have caused the
parser not to work. These millisecond fields now follow the standard we use
elsewhere in the code, with additional fields added iff the `human` flag is
specified.

This is a breaking change for ILM, but since ILM has not yet been released, no
compatibility shim is needed.
2018-11-05 08:17:15 -07:00
Alexander Reelsen 409050e8de
Refactor: Remove settings from transport action CTOR (#35208)
As settings are not used in the transport action constructor, this
removes the passing of the settings in all the transport actions.
2018-11-05 13:08:18 +01:00
Jay Modi ec1c2c4a76
Remove AbstractComponent usage in x-pack core (#35187)
This change removes the use of AbstractComponent in the security
module. The classes now declare their own loggers.

Relates #34488
2018-11-02 14:30:43 -06:00
Alan Woodward e2af849f70
Move ObjectPath and XContentUtils to libs/x-content (#34803)
These are generally useful utility classes that do not need to live in the Watcher code
2018-11-02 15:12:09 +00:00
Benjamin Trent 7fb95be8bd
Moving xpack.plugin.core ml response to use new base class (#35157)
* Moving xpack.plugin.core ml response to use new base class

* changing back to streamable
2018-11-02 07:46:53 -05:00
Colin Goodheart-Smithe fc6e1f7f3f
Merge branch 'master' into index-lifecycle 2018-11-02 10:56:35 +00:00
Martijn van Groningen 19d6cf1b9e
[CCR] Change response classes to not use from Streamable. (#35085)
Only the response classes of get auto follow pattern, the follow and stats APIs
were moved away from Streamable.  The other APIs use `AcknowledgedResponse`
or `BaseTasksResponse` as response class and
moving that class away from Streamable is a bigger change.
2018-11-02 08:02:17 +01:00
Tal Levy b260406ecf
ILM: use node _id attribute when allocating to one node (#35061)
ILM's Shrink Action was using a nodes "_name" attribute to
allocate to prepare for the shrink step. Since the name is
configurable by a user and may use the same name for
multiple nodes on one machine, _id is safer since it is guaranteed
to be unique.

closes #35043.
2018-11-01 14:38:00 -07:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Benjamin Trent 2fadec5c3d
ML: Add support for rollup Indexes in Datafeeds (#34654)
* Adding rollup support for datafeeds

* Fixing tests and adjusting formatting

* minor formatting chagne

* fixing some syntax and removing redundancies

* Refactoring and fixing failing test

* Refactoring, adding paranoid null check

* Moving rollup into the aggregation package

* making AggregationToJsonProcessor package private again

* Addressing test failure

* Fixing validations, chunking

* Addressing failing test

* rolling back RollupJobCaps changes

* Adding comment and cleaning up test

* Addressing review comments and test failures

* Moving builder logic into separate methods

* Addressing PR comments, adding test for rollup permissions

* Fixing test failure

* Adding rollup priv check on datafeed put

* Handling missing index when getting caps

* Fixing unused import
2018-11-01 10:02:24 -05:00
Nik Everett e28509fbfe
Core: Less settings to AbstractComponent (#35140)
Stop passing `Settings` to `AbstractComponent`'s ctor. This allows us to
stop passing around `Settings` in a *ton* of places. While this change
touches many files, it touches them all in fairly small, mechanical
ways, doing a few things per file:
1. Drop the `super(settings);` line on everything that extends
`AbstractComponent`.
2. Drop the `settings` argument to the ctor if it is no longer used.
3. If the file doesn't use `logger` then drop `extends
AbstractComponent` from it.
4. Clean up all compilation failure caused by the `settings` removal
and drop any now unused `settings` isntances and method arguments.

I've intentionally *not* removed the `settings` argument from a few
files:
1. TransportAction
2. AbstractLifecycleComponent
3. BaseRestHandler

These files don't *need* `settings` either, but this change is large
enough as is.

Relates to #34488
2018-10-31 21:23:20 -04:00
Tal Levy d5d28420b6 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-31 10:47:07 -07:00
Vladimir Dolzhenko 28bb1a3ac0
XPack/HLRC request/response compatibility tests (#34547)
Relates #34451 , #29827
2018-10-31 13:48:35 +01:00
Tal Levy fdb0db30c4
throw exception when ilm-rollover-step runs on index without rollover alias (#35065)
The ILM Rollover Step can execute on the incorrect index if the rollover alias
exists on another valid index, but not the one the step is executing against. This
is a problem and is now guarded against
2018-10-30 16:42:12 -07:00
Nik Everett 086ada4c08
Core: Drop settings member from AbstractComponent (#35083)
Drops the `Settings` member from `AbstractComponent`, moving it from the
base class on to the classes that use it. For the most part this is a
mechanical change that doesn't drop `Settings` accesses. The one
exception to this is naming threads where it switches from an invocation
that passes `Settings` and extracts the node name to one that explicitly
passes the node name.

This change doesn't drop the `Settings` argument from
`AbstractComponent`'s ctor because this change is big enough as is.
We'll do that in a follow up change.
2018-10-30 16:10:38 -04:00
Tal Levy f5323cd86e
rename ILM Phase "minimum_age" to "min_age" (#35058)
closes #34927.
2018-10-30 12:54:02 -07:00
Tal Levy 18c72e86c5 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-30 08:09:57 -07:00
Martijn van Groningen 6da2fb7d5b
Change CCR API request classes to use Writeable serialization instead of Streamable (#34911)
Only the follow stats request couldn't be changed to use Writeable serialization,
because that requires changes in `TransportTasksAction` and `BaseTasksRequest` base classes.
2018-10-30 11:03:30 +01:00
Tal Levy c9e4d26a53 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-29 14:03:55 -07:00
Pratik Sanglikar f1135ef0ce Core: Replace deprecated Loggers calls with LogManager. (#34691)
Replace deprecated Loggers calls with LogManager.

Relates to #32174
2018-10-29 15:52:30 -04:00
Jason Tedor 26f5c509af
Fix CCR API specification (#34963)
This commit fixes two issues with the CCR API specification:
 - remove the CCR stats endpoint, it is not currently implemented
 - fix the documentation links
2018-10-29 09:37:13 -04:00
David Roberts c455be7bc2
[ML] Rename the json file structure to ndjson (#34901)
The file structure finder endpoint can find the NDJSON
(newline-delimited JSON) file format, but called it
`json`.  This change renames the `format` for this file
structure to `ndjson`, which is more precise and will
hopefully avoid confusion.
2018-10-29 10:06:12 +01:00
Martijn van Groningen 1801518527
[CCR] Refactor stats APIs (#34912)
* Changed the auto follow stats to also include follow stats.
* Renamed the auto follow stats api to stats api and changed its url path
  from `/_ccr/auto_follow/stats` `/_ccr/stats`.
* Removed `/_ccr/stats` url path for the follow stats api, which makes
  the index parameter a required parameter.
* Fixed docs.
2018-10-29 07:45:27 +01:00
Martijn van Groningen bad5972f62
[CCR] Fix request serialization bug (#34917)
and some parameters that were not set in tests.
2018-10-29 07:38:55 +01:00
Tal Levy d8322ca069 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-26 12:46:21 -07:00
Tal Levy e737ea7d4a
remove old doc placeholder and migrate ilm docs to top-level (#34615)
we are restructuring the docs, this migrates ILM docs outside of the x-pack doc structure.
2018-10-26 12:19:52 -07:00
Lee Hinman 6880e2ad79 [TEST] Fix RolloverActionTests.mutateInstance
In some cases it wasn't mutating and caused test failures
2018-10-26 09:10:55 -06:00
Jason Tedor 43f6ba1c63
Fix put/resume follow request parsing (#34913)
This commit adds some fields that were missing from put follow, and
fixes a bug in resume follow.
2018-10-26 11:09:55 -04:00
Dimitris Athanasiou a39a67cd38
[ML] Extract common native process base class (#34856)
We currently have two different native processes:
autodetect & normalizer. There are plans for introducing
a new process. All these share many things in common.
This commit refactors the processes to extend an
`AbstractNativeProcess` class that encapsulates those
commonalities with the purpose of reusing the code
for new processes in the future.
2018-10-26 15:34:48 +01:00
Tal Levy 4c4afef9cc fix PolicyStatsTests: mutate kept same indicesManaged 2018-10-25 16:13:09 -07:00
Lee Hinman 3e7042832a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-25 11:00:36 -06:00
Tal Levy 41eaa586e8
remove index.lifecycle.skip setting (#34823)
With the introduction of _ilm/stop and _ilm/start APIs, the
use cases where one would only target a select group
of indices to start/stop has been reduced. Since there is no
strong use-case for skipping specific indices, it is best to
remove this functionality and only adding if later desired, with the
hopes of keeping things more simple.
2018-10-25 07:27:04 -07:00
Tal Levy 21b9b024c7
fix PolicyStatsTests mutateInstance (#34835)
through randomization, there is a chance that the mutateInstance
for PolicyStatsTests does not actually mutate the original object.
This PR aims to fix this
2018-10-25 07:24:54 -07:00
Costin Leau b97546a5a9
SQL: Introduce ODBC mode, similar to JDBC (#34825)
Close #34720
2018-10-25 12:40:00 +03:00
Nhat Nguyen ff49e79d40 CCR: Rename follow-task parameters and stats (#34836)
* CCR: Rename follow parameters and stats

This commit renames the follow-task parameters and its stats.
Below are the changes:

## Params
- remote_cluster (unchanged)
- leader_index (unchanged)
- max_read_request_operation_count -> max_read_request_operation_count
- max_batch_size -> max_read_request_size
- max_write_request_operation_count (new)
- max_write_request_size (new)
- max_concurrent_read_batches -> max_outstanding_read_requests
- max_concurrent_write_batches -> max_outstanding_write_requests
- max_write_buffer_size (unchanged)
- max_write_buffer_count (unchanged)
- max_retry_delay (unchanged)
- poll_timeout -> read_poll_timeout

## Stats
- remote_cluster (unchanged)
- leader_index (unchanged)
- follower_index (unchanged)
- shard_id (unchanged)
- leader_global_checkpoint (unchanged)
- leader_max_seq_no (unchanged)
- follower_global_checkpoint (unchanged)
- follower_max_seq_no (unchanged)
- last_requested_seq_no (unchanged)
- number_of_concurrent_reads -> outstanding_read_requests
- number_of_concurrent_writes -> outstanding_write_requests
- buffer_size_in_bytes -> write_buffer_size_in_bytes (new)
- number_of_queued_writes -> write_buffer_operation_count
- mapping_version -> follower_mapping_version
- total_fetch_time_millis -> total_read_time_millis
- total_fetch_remote_time_millis -> total_read_remote_exec_time_millis
- number_of_successful_fetches -> successful_read_requests
- number_of_failed_fetches -> failed_read_requests
- operation_received -> operations_read
- total_transferred_bytes -> bytes_read
- total_index_time_millis -> total_write_time_millis [?]
- number_of_successful_bulk_operations -> successful_write_requests
- number_of_failed_bulk_operations -> failed_write_requests
- number_of_operations_indexed -> operations_written
- fetch_exception -> read_exceptions
- time_since_last_read_millis -> time_since_last_read_millis

* add test for max_write_request_(operation_count|size)
2018-10-25 10:36:15 +02:00
Armin Braun e7ced94a65
NETWORKING: Add SSL Handler before other Handlers (#34636)
* NETWORKING: Add SSL Handler before other Handlers

* The only way to run into the issue in #33998 is for `Netty4MessageChannelHandler`
to be in the pipeline while the SslHandler is not. Adding the SslHandler before any
other handlers should ensure correct ordering here even when we handle upstream events
in our own thread pool
* Ensure that channels that were closed concurrently don't trip the assertion
* Closes #33998
2018-10-25 07:51:14 +02:00
Ryan Ernst 687dc1eb11
Scripting: Remove SearchScript (#34730)
This commit removes the last non context based script class.
2018-10-24 15:03:38 -07:00
Martijn van Groningen 6fe0e62b7a
[CCR] Added write buffer size limit (#34797)
This limit is based on the size in bytes of the operations in the write buffer. If this limit is exceeded then no more read operations will be coordinated until the size in bytes of the write buffer has dropped below the configured write buffer size limit.

Renamed existing `max_write_buffer_size` to ``max_write_buffer_count` to indicate that limit is count based.

Closes #34705
2018-10-24 23:48:49 +02:00
Benjamin Trent cd27b0b996
Revert "Rollup add default metrics to histo groups (#34534)" (#34815)
This reverts commit 4236358f5d.
2018-10-24 14:25:10 -05:00
Colin Goodheart-Smithe e7fddb5c93
Adds usage data for ILM (#33377)
* Adds usage data for ILM

* Adds tests for IndexLifecycleFeatureSetUsage and friends

* Adds tests for IndexLifecycleFeatureSet

* Fixes merge errors

* Add number of indices managed to usage stats

Also adds more tests

* Addresses Review comments
2018-10-24 18:28:46 +01:00
Colin Goodheart-Smithe c7fe87e43f
Removes Set Policy API in favour of setting index.lifecycle.name directly (#34304)
* Removes Set Policy API in favour of setting index.lifecycle.name
directly

* Reinstates matcher that will still be used

* Cleans up code after rebase

* Adds test to check changing policy with ndex settings works

* Fixes TimeseriesLifecycleActionsIT after API removal

* Fixes docs tests

* Fixes case on close where lifecycle service was never created
2018-10-24 16:14:59 +01:00
Shaunak Kashyap 0f1a5ecf61
Adding stack_monitoring_agent role (#34369)
* Adding stack_monitoring_agent role

* Fixing checkstyle issues

* Adding tests for new role

* Tighten up privileges around index templates

* s/stack_monitoring_user/remote_monitoring_collector/ + remote_monitoring_user

* Fixing checkstyle violation

* Fix test

* Removing unused field

* Adding missed code

* Fixing data type

* Update Integration Test for new builtin user
2018-10-24 07:20:46 -07:00
Martijn van Groningen 76240e6bbe
[CCR] Renamed leader_cluster to remote_cluster (#34776)
and also some occurrences of clusterAlias to remoteCluster.

Closes #34682
2018-10-24 13:39:36 +02:00
Martijn van Groningen 18e004c3ea
[CCR] Removed unused static parse field. 2018-10-24 09:27:08 +02:00
Martijn van Groningen abf8cb6706
[CCR] Cleanup pause follow action (#34183)
* Change the `TransportPauseFollowAction` to extend from `TransportMasterNodeAction`
  instead of `HandledAction`, this removes a sync cluster state api call.
* Introduced `ResponseHandler` that removes duplicated code in `TransportPauseFollowAction` and
  `TransportResumeFollowAction`.
* Changed `PauseFollowAction.Request` to not use `readFrom()`.
2018-10-24 08:12:39 +02:00
Martijn van Groningen ed817fb265
[CCR] Move leader_index and leader_cluster parameters from resume follow to put follow api (#34638)
As part of this change the leader index name and leader cluster name are
stored in the CCR metadata in the follow index. The resume follow api
will read that when a resume follow request is executed.
2018-10-23 19:37:45 +02:00
Tal Levy 62ac2fa5ec Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-23 09:43:46 -07:00
Zachary Tong 4dbf498721
[Rollup] Job deletion should be invoked on the allocated task (#34574)
We should delete a job by directly talking to the allocated 
task and telling it to shutdown. Today we shut down a job 
via the persistent task framework. This is not ideal because, 
while the job has been removed from the persistent task 
CS, the allocated task continues to live until it gets the 
shutdown message.

This means a user can delete a job, immediately delete 
the rollup index, and then see new documents appear in
 the just-deleted index. This happens because the indexer
 in the allocated task is still running and indexes a few 
more documents before getting the shutdown command.

In this PR, the transport action is changed to a TransportTasksAction, 
and we invoke onCancelled() directly on the matching job. 
The race condition still exists after this PR (albeit less likely), 
but this was a precursor to fixing the issue and a self-contained
chunk of code. A second PR will followup to fix the race itself.
2018-10-23 12:23:22 -04:00
Gordon Brown 59b28da251
[ILM] Add null check to CopyExecutionStateStep (#34619)
Adds a null check for IndexMetaData similar to those in #33455.
2018-10-23 08:46:27 -06:00
Martijn van Groningen e6d87cc09f
[CCR] Add total fetch time leader stat (#34577)
Add total fetch time leader stat, that
keeps track how much time was spent on fetches
from the leader cluster perspective.
2018-10-23 16:41:06 +02:00
Martijn van Groningen 36baf3823d
[CCR] Auto follow pattern APIs adjustments (#34518)
* Changed the resource id of auto follow patterns to be a user defined name
instead of being the leader cluster alias name.
* Fail when an unfollowed leader index matches with two or more auto follow patterns.
2018-10-23 15:48:51 +02:00
Tal Levy 67bfdb16ad Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-22 13:09:37 -07:00
Jason Tedor 7af19b8f81
Migrate wait for pending tasks helper to server (#34675)
In some of our X-Pack REST tests we have to wait for pending tasks to
complete. We are now needing this functionality in ESRestTestCase for
the docs tests where we run against X-Pack features. This commit moves
the helper method that we have in X-Pack to ESRestTestCase, and removes
duplicate logic from waiting for rollup tasks to complete.
2018-10-22 11:14:02 -04:00
David Kyle c476f91811
[TEST][ML] Reinstate date parsing test (#34634)
The test was muted due to a bug in an early access build of Java 11
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8206980
2018-10-19 16:29:16 +01:00
Colin Goodheart-Smithe 84ef91529c
Merge branch 'master' into index-lifecycle 2018-10-19 13:24:04 +01:00
Benjamin Trent 4236358f5d
Rollup add default metrics to histo groups (#34534)
* Rollup: Adding default metrics for histo group timefield (#34379)

* Rollup: Adding default histo metrics and tests

* fixing failing client side test with new default values

* Adding HLRC docs for default values

* Addressing PR comments

* Removing value_count default agg

* Updating docs for rollups

* Minor interval change
2018-10-19 07:23:25 -05:00
Albert Zaharovits aeb3cda585 [TEST] Fix GetUserPrivilegesResponse#Indices(StreamInput) constr
Closes #34631
2018-10-19 14:40:03 +03:00
Martijn van Groningen 44b461aff2
[CCR] Make leader cluster a required argument. (#34580)
This change makes it no longer possible to follow / auto follow without
specifying a leader cluster. If a local index needs to be followed
then `cluster.remote.*.seeds` should point to nodes in the local cluster.

Closes #34258
2018-10-19 07:41:46 +02:00
Tim Vernum 2283a337b9 [TEST] Fix bit indexing in test mutator
There are 2 hard problems in computer science: cache invalidation,
naming things, and off-by-1 errors.
 --  Leon Bambrick (@secretGeek)

Fixes: #34610
2018-10-19 11:40:53 +11:00
Zachary Tong 45546e71c2
Add GetRollupCaps API to high level rest client (#32880)
Adds GetRollupCaps API to the HLRC, and tweaks some of the
Caps objects to be immutable.  Also various style tweaks
2018-10-18 17:12:38 -04:00
Martijn van Groningen 0d62f6102c
[CCR] Split cluster alias from leader index field into its own field in follow APIs (#34366) 2018-10-18 12:11:48 +02:00
Tim Vernum e41c5cb47a Fix compilation issue due to 47e9082 and 9200e15
47e9082 removed a method that 9200e15 made use of.
2018-10-18 18:03:54 +11:00
Tim Vernum 47e9082bb4
Replace Streamable w/ Writeable in RoleDescriptor (#34544)
This commit replaces usage of Streamable with Writeable within the
RoleDescriptor class (and inner classes).

Relates: #34389
2018-10-18 17:28:43 +11:00
Tim Vernum 9200e15b74
Add get-user-privileges API (#33928)
This API is intended as a companion to the _has_privileges API.
It returns the list of privileges that are held by the current user.

This information is difficult to reason about, and consumers should
avoid making direct security decisions based solely on this data.
For example, each of the following index privileges (as well as many
more) would grant a user access to index a new document into the
"metrics-2018-08-30" index, but clients should not try and deduce
that information from this API.
- "all" on "*"
- "all" on "metrics-*"
- "write" on "metrics-2018-*"
- "write" on "metrics-2018-08-30"

Rather, if a client wished to know if a user had "index" access to
_any_ index, it would be possible to use this API to determine whether
the user has any index privileges, and on which index patterns, and
then feed those index patterns into _has_privileges in order to
determine whether the "index" privilege had been granted.

The result JSON is modelled on the Role API, with a few small changes
to reflect how privileges are modelled when multiple roles are merged
together (multiple DLS queries, multiple FLS grants, multiple global
conditions, etc).
2018-10-18 14:09:04 +11:00
Tal Levy 09067c8942 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-17 15:37:11 -07:00
Armin Braun 08d4bf6e84
TESTS: Remove Dead Code in Test Infra. (#34548)
* None of this infrastructure is used
* Some redundant throws and resulting catch code removed
2018-10-17 20:08:39 +01:00
Colin Goodheart-Smithe 90f7cec7a5
Merge branch 'master' into index-lifecycle 2018-10-17 18:22:23 +01:00
Nik Everett 139bbc3f03
Rollup: Consolidate rollup cleanup for http tests (#34342)
This moves the rollup cleanup code for http tests from the high level rest
client into the test framework and then entirely removes the rollup cleanup
code for http tests that lived in x-pack. This is nice because it
consolidates the cleanup into one spot, automatically invokes the cleanup
without the test having to know that it is "about rollup", and should allow
us to run the rollup docs tests.

Part of #34530
2018-10-17 09:32:16 -04:00
Shaunak Kashyap e0cab14c6e
Cleanup: removing unused class (#34541)
* Cleanup: removing unused field in other similar classes

* Removing unused class
2018-10-17 10:15:12 +01:00
Tal Levy fbe8dc014c Merge branch 'master' into index-lifecycle 2018-10-16 13:58:53 -07:00
Tal Levy 2846effc74
initial api reference docs for ILM (#33866)
This adds the bulk of the ILM APIs in docs with examples
2018-10-16 12:18:54 -07:00
Jack Conradson 80474e138f
HLRC: Add remove index lifecycle policy (#34204)
This change adds the command RemoveIndexLifecyclePolicy to the HLRC. This uses the 
new TimeRequest as a base class for RemoveIndexLifecyclePolicyRequest on the client side.
2018-10-16 08:12:06 -07:00
Benjamin Trent af2877cb7b
Rollup adding support for date field metrics (#34185) (#34200)
* Rollup adding support for date field metrics (#34185)

* Restricting supported metrics for `date` field rollup

* fixing expected error message for yaml test

* Addressing PR comments
2018-10-16 08:55:49 -05:00
Martijn van Groningen a1ec91395c
Changed CCR internal integration tests to use a leader and follower cluster instead of a single cluster (#34344)
The `AutoFollowTests` needs to restart the clusters between each tests, because
it is using auto follow stats in assertions. Auto follow stats are only reset
by stopping the elected master node.

Extracted the `testGetOperationsBasedOnGlobalSequenceId()` test to its own test, because it just tests the shard changes api.

* Renamed AutoFollowTests to AutoFollowIT, because it is an integration test.
Renamed ShardChangesIT to IndexFollowingIT, because shard changes it the name
of an internal api and isn't a good name for an integration test.

* move creation of NodeConfigurationSource to a seperate method

* Fixes issues after merge, moved assertSeqNos() and assertSameDocIdsOnShards() methods from ESIntegTestCase to InternalTestCluster, so that ccr tests can use these methods too.
2018-10-16 14:45:46 +02:00
Jason Tedor 4b2052c683
Introduce index settings version (#34429)
This commit introduces settings version to index metadata. This value is
monotonically increasing and is updated on settings updates. This will
be useful in cross-cluster replication so that we can request settings
updates from the leader only when there is a settings update.
2018-10-16 06:22:20 -04:00
markharwood 75c973f57e
Test fix - GraphExploreResponse HLRC xContent ordering was unreliable(#34473)
xContent ordering is unreliable when derived from map insertions but the parsed objects’ .equals() methods have the sort logic required to prove connections and vertices are correct. Disabled the xContent equivalence checks. 

Closes #33686
2018-10-16 09:10:48 +01:00
Martijn van Groningen f7df8718b9
[CCR] Don't fail shard follow tasks in case of a non-retryable error (#34404) 2018-10-16 07:44:15 +02:00
Daniel Mitterdorfer 92b2e1a209
Remove lenient boolean handling
With this commit we remove some leftovers from #26389 which cleaned up
lenient boolean handling.

Relates #26389
Relates #22298
Relates #34467
2018-10-16 06:30:00 +02:00
Jay Modi 0cd03d3581
Use RoleRetrievalResult for better caching (#34197)
Security caches the result of role lookups and negative lookups are
cached indefinitely. In the case of transient failures this leads to a
bad experience as the roles could truly exist. The CompositeRolesStore
needs to know if a failure occurred in one of the roles stores in order
to make the appropriate decision as it relates to caching. In order to
provide this information to the CompositeRolesStore, the return type of
methods to retrieve roles has changed to a new class,
RoleRetrievalResult. This class provides the ability to pass back an
exception to the roles store. This exception does not mean that a
request should be failed but instead serves as a signal to the roles
store that missing roles should not be cached and neither should the
combined role if there are missing roles.

As part of this, the negative lookup cache was also changed from an
unbounded cache to a cache with a configurable limit.

Relates #33205
2018-10-15 20:52:54 +01:00
Chris Roberson fb31236188
[Monitoring] Add additional necessary mappings for apm-server (#34392)
* Add additional necessary mappings for apm-server

* Add open handles for beats

* Add mappings missing for es
2018-10-15 12:37:52 -04:00
Colin Goodheart-Smithe 0b42eda0e3
Merge branch 'master' into index-lifecycle 2018-10-15 16:03:37 +01:00
Tim Vernum 8d83688328
Allow an AuthenticationResult to return metadata (#34382)
PR #34290 made it impossible to use thread-context values to pass
authentication metadata out of a realm. The SAML realm used this
technique to allow the SamlAuthenticateAction to process the parsed
SAML token, and apply them to the access token that was generated.

This new method adds metadata to the AuthenticationResult itself, and
then the authentication service makes this result available on the
thread context.

Closes: #34332
2018-10-12 14:58:04 +01:00
David Roberts 21c759af0e
[ML] Add an ingest pipeline definition to structure finder (#34350)
The ingest pipeline that is produced is very simple.  It
contains a grok processor if the format is semi-structured
text, a date processor if the format contains a timestamp,
and a remove processor if required to remove the interim
timestamp field parsed out of semi-structured text.

Eventually the UI should offer the option to customize the
pipeline with additional processors to perform other data
preparation steps before ingesting data to an index.
2018-10-12 07:56:35 +01:00
Lee Hinman 9ad2a7fa77
Fix expected next step being incorrect when executing async action (#34313)
This fixes an issue where an incorrect expected next step is used when checking
to execute `AsyncActionStep`s after a cluster state step.

It fixes this scenario:

- `ExecuteStepsUpdateTask` executes a `ClusterStateWaitStep` or
  `ClusterStateActionStep` successfully
- The next step is also a `ClusterStateWaitStep`, so it loops
- The `ClusterStateWaitStep` has a next stepkey (which gets set to the
  `nextStepKey` in the code)
- The `ClusterStateWaitStep` fails the condition, meaning that it will have to
  wait longer
- The `nextStepKey` is now incorrect though, because we did not advance the
  index's step, and it's not `null` (which is another safe value if there is no
  step after the `ClusterStateWaitStep`)

This fixes the problem by resetting the nextStepKey to null if the condition is
not met, since we are not going to advance the step metadata in this
case (thereby skipping the `maybeRunAsyncAction` invocation).

This commit also tightens up and enhances much of the ILM logging. A lot of
logging was missing the index name (making it hard to debug in the presence of
multiple indices) and a lot was using the wrong logging level (DEBUG is now
actually readable without being a wall of text).

Resolves #34297
2018-10-08 11:25:18 -06:00
Nik Everett 06993e0c35
Logging: Make ESLoggerFactory package private (#34199)
Since all calls to `ESLoggerFactory` outside of the logging package were
deprecated, it seemed like it'd simplify things to migrate all of the
deprecated calls and declare `ESLoggerFactory` to be package private.
This does that.
2018-10-06 09:54:08 -04:00
Martijn van Groningen 899e48395b
[CCR] Change unfollow API's privilege scheme. (#34175)
Unfollow should be allowed / disallowed on a per index level instead of
cluster level.

Also renamed `create_follow_index` index privilege to
`manage_follow_index` privilege and include unfollow and close APIs.
2018-10-06 07:38:28 -04:00
Nathan Delhaye 1bd8ff520b Typo in x-pack template for thread_pool.management (#34224)
Typo in x-pack template for the node_stats.thread_pool.management
2018-10-05 13:17:55 -04:00
Jason Tedor 7d57bdb3a0
Follow stats structure (#34301)
This commit modifies the follow stats API response structure to more
clearly highlight meaning of the higher level fields. In particular,
previously the response had a top-level key for each index. Instead, we
nest the indices under an "indices" field which is now an array. The
values in this array are objects containing two fields: "index" which is
the name of the follower index, and "shards" which is an array where
each value in the array is the follower stats for that shard. That is,
we have gone from:

{
  "bar": [
    {
      "shard_id": 0...
    }...
  ]...
}

to

{
  "indices": [
    {
      "index": "bar",
      "shards": [
        {
          "shard_id": 0...
        }...
      ]
   }...
}
2018-10-05 06:38:20 -04:00
Jason Tedor 7478167d60
Rename CCR stats implementation (#34300)
In the CCR docs we want to refer to the endpoint that returns following
stats as the follow stats API. This commit renames the internal
implementation of this endpoint to reflect this usage.
2018-10-05 06:25:24 -04:00
Tim Vernum 63dbd1dce0
Allow User/Password realms to disable authc (#34033)
The "lookupUser" method on a realm facilitates the "run-as" and
"authorization_realms" features.
This commit allows a realm to be used for "lookup only", in which
case the "authenticate" method (and associated token methods) are
disabled.
It does this through the introduction of a new
"authentication.enabled" setting, which defaults to true.
2018-10-05 12:10:42 +10:00
Tim Vernum 6608992523
Enable security automaton caching (#34028)
Building automatons can be costly. For the most part we cache things
that use automatons so the cost is limited.
However:
- We don't (currently) do that everywhere (e.g. we don't cache role
  mappings)
- It is sometimes necessary to clear some of those caches which can
  cause significant CPU overhead and processing delays.

This commit introduces a new cache in the Automatons class to avoid
unnecesarily recomputing automatons.
2018-10-05 12:09:36 +10:00
Dimitris Athanasiou 4dacfa95d2
[ML] Allow asynchronous job deletion (#34058)
This changes the delete job API by adding
the choice to delete a job asynchronously.
The commit adds a `wait_for_completion` parameter
to the delete job request. When set to `false`,
the action returns immediately and the response
contains the task id.

This also changes the handling of subsequent
delete requests for a job that is already being
deleted. It now uses the task framework to check
if the job is being deleted instead of the cluster
state. This is a beneficial for it is going to also
be working once the job configs are moved out of the
cluster state and into an index. Also, force delete
requests that are waiting for the job to be deleted
will not proceed with the deletion if the first task
fails. This will prevent overloading the cluster. Instead,
the failure is communicated better via notifications
so that the user may retry.

Finally, this makes the `deleting` property of the job
visible (also it was renamed from `deleted`). This allows
a client to render a deleting job differently.

Closes #32836
2018-10-05 02:41:28 +03:00
Nik Everett ab8a5563f2
Logging: Drop remaining Settings log ctor (#34149)
Drops the last logging constructor that takes `Settings` because it is
no longer needed.

Watcher goes through a lot of effort to pass `Settings` to `Logger`
constructors and dropping `Settings` from all of those calls allowed us
to remove quite a bit of log-based ceremony from watcher.
2018-10-04 09:18:04 -04:00
David Kyle ef5007b6d8
[ML] Remove unused last_data_time member from Job (#34262) 2018-10-04 13:16:14 +01:00
Ed Savage 577261ee57
[ML] Label anomalies with multi_bucket_impact (#34233)
* [ML] Label anomalies with  multi_bucket_impact

Add the multi_bucket_impact field to record results.
2018-10-04 09:08:21 +01:00
Ioannis Kakavas 2c82b80b85
Support PKCS#11 tokens as keystores and truststores (#34063)
This enables Elasticsearch to use the JVM-wide configured
PKCS#11 token as a keystore or a truststore for its TLS configuration.
The JVM is assumed to be configured accordingly with the appropriate
Security Provider implementation that supports PKCS#11 tokens.
For the PKCS#11 token to be used as a keystore or a truststore for an
SSLConfiguration, the .keystore.type or .truststore.type must be
explicitly set to pkcs11 in the configuration.
The fact that the PKCS#11 token configuration is JVM wide implies that
there is only one available keystore and truststore that can be used by TLS
configurations in Elasticsearch.
The PIN for the PKCS#11 token can be set as a truststore parameter in
Elasticsearch or as a JVM parameter ( -Djavax.net.ssl.trustStorePassword).

The basic goal of enabling PKCS#11 token support is to allow PKCS#11-NSS in
FIPS mode to be used as a FIPS 140-2 enabled Security Provider.
2018-10-04 10:51:58 +03:00
Yogesh Gaikwad 81227dc389
[Authz] Allow update settings action for system user (#34030)
When the cluster.routing.allocation.disk.watermark.flood_stage watermark
is breached, DiskThresholdMonitor marks the indices as read-only. This
failed when x-pack security was present as system user does not have the privilege
for update settings action("indices:admin/settings/update").
This commit adds the required privilege for the system user. Also added missing
debug logs when access is denied to help future debugging.
An assert statement is added to catch any missed privileges required for
system user.

Closes #33119
2018-10-04 11:31:51 +10:00
Jay Modi a21a99da18
Security: upgrade unboundid ldapsdk to 4.0.8 (#34247)
This commit upgrades the unboundid ldapsdk to version 4.0.8. The
primary driver for upgrading is a fix that prevents this library from
rewrapping Error instances that would normally bubble up to the
UncaughtExceptionHandler and terminate the JVM. Other notable changes
include some fixes related to connection handling in the library's
connection pool implementation.

Closes #33175
2018-10-03 12:31:19 -06:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Lee Hinman 388f754a8e
Change step execution flow to be deliberate about type (#34126)
This commit changes the way that step execution flows. Rather than have any step
run when the cluster state changes or the periodic scheduler fires, this now
runs the different types of steps at different times.

`AsyncWaitStep` is run at a periodic manner, ie, every 10 minutes by default
`ClusterStateActionStep` and `ClusterStateWaitStep` are run every time the
cluster state changes.
`AsyncActionStep` is now run only after the cluster state has been transitioned
into a new step. This prevents these non-idempotent steps from running at the
same time. It addition to being run when transitioned into, this is also run
when a node is newly elected master (only if set as the current step) so that
master failover does not fail to run the step.

This also changes the `RolloverStep` from an `AsyncActionStep` to an
`AsyncWaitStep` so that it can run periodically.

Relates to #29823
2018-10-02 20:02:50 -06:00
Gordon Brown fb907706ec Merge branch 'master' into index-lifecycle 2018-10-02 13:43:46 -06:00
jaymode 306e178d83
Test: remove awaitsfix incorrectly added in #34148 2018-10-02 10:02:20 -06:00
Jay Modi 8539fb68d9
Test: Revert pinning MockWebServer to TLSv1.2 (#34148)
Revert "[TESTS] Pin MockWebServer to TLS1.2 (#33127)" (commit
214652d4af) and "Pin TLS1.2 in
SSLConfigurationReloaderTests" (commit
d9f5e4fd2e), which pinned the
MockWebServer used in the SSLConfigurationReloaderTests to TLSv1.2 in
order to prevent failures with JDK 11 related to ssl session
invalidation. We no longer need this pinning as the problematic code
was fixed in #34130.
2018-10-02 09:54:21 -06:00
Nik Everett f904c41506
HLRC: Add get rollup job (#33921)
Adds support for the get rollup job to the High Level REST Client. I had
to do three interesting and unexpected things:
1. I ported the rollup state wiping code into the high level client
tests. I'll move this into the test framework in a followup and remove
the x-pack version.
2. The `timeout` in the rollup config was serialized using the
`toString` representation of `TimeValue` which produces fractional time
values which are more human readable but aren't supported by parsing. So
I switched it to `getStringRep`.
3. Refactor the xcontent round trip testing utilities so we can test
parsing of classes that don't implements `ToXContent`.
2018-10-02 09:11:29 -04:00
Lee Hinman 2d9cb21490 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-01 14:10:09 -06:00
Jason Tedor e2bd2028d8
Allow specifying shard changes batch sizes in bytes (#34168)
This commit changes the shard changes requests from using a raw byte
value to being able to be specified using bytes units (e.g., 4mb).
2018-09-30 14:22:22 -04:00
Martijn van Groningen b1a27b2e6b
[CCR] Add unfollow API (#34132)
The unfollow API changes a follower index into a regular index, so that it will accept write requests from clients.

For the unfollow api to work the index follow needs to be stopped and the index needs to be closed.

Closes #33931
2018-09-30 19:19:34 +02:00
Ryan Ernst 47cbae9b26
Scripting: Remove ExecutableScript (#34154)
This commit removes the legacy ExecutableScript, which was no longer
used except in tests. All uses have previously been converted to script
contexts.
2018-09-28 17:13:08 -07:00
Lee Hinman 6ea396a476 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-28 15:40:12 -06:00
David Roberts f709c2f694
[ML] Add a timeout option to file structure finder (#34117)
This can be used to restrict the amount of CPU a single
structure finder request can use.

The timeout is not implemented precisely, so requests
may run for slightly longer than the timeout before
aborting.

The default is 25 seconds, which is a little below
Kibana's default timeout of 30 seconds for calls to
Elasticsearch APIs.
2018-09-28 17:32:35 +01:00
Martijn van Groningen a984f8afb3
[CCR] Validate index privileges prior to following an index (#33758)
Prior to following an index in the follow API, check whether current
user has sufficient privileges in the leader cluster to read and
monitor the leader index.

Also check this in the create and follow API prior to creating the
follow index.

Also introduced READ_CCR cluster privilege that include the minimal
cluster level actions that are required for ccr in the leader cluster.
So a user can follow indices in a cluster, but not use the ccr admin APIs.

Closes #33553

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-09-28 17:51:23 +02:00
Jay Modi 14d841ef21
Handle null SSLSessions during invalidation (#34130)
The SSLService invalidates SSLSessions when there is a change to any of
the underlying key or trust material. However, this invalidation code
did not check for a null SSLSession being returned from the context and
assumed that the context would always return a non-null object. The
return of a null object is possible in all versions, but JDK11 seems to
return them more often due to changes for TLS 1.3. There are a number
of reasons that we get a id of a session but the context returns null
when the session with that id is requested. Some of the reasons for
this are:

* Session was evicted by session cache
* Session has timed out
* Session has been invalidated by another caller

To handle this, the SSLService now checks if the value is null before
calling invalidate on the SSLSession.

Closes #32124
2018-09-28 09:03:35 -06:00
Hendrik Muhs e2f310b56c
Fix AggregationFactories.Builder equality and hash regarding order (#34005)
Fixes the equals and hash function to ignore the order of aggregations to ensure equality after serialization
and deserialization. This ensures storing configs with aggregation works properly.

This also addresses a potential issue in caching when the same query contains aggregations but in 
different order. 1st it will not hit in the cache, 2nd cache objects which shall be equal might end up twice in 
the cache.
2018-09-28 13:30:50 +02:00
Martijn van Groningen 9129948f60
Rename CCR APIs (#34027)
* Renamed CCR APIs

Renamed:
* `/{index}/_ccr/create_and_follow` to `/{index}/_ccr/follow`
* `/{index}/_ccr/unfollow` to `/{index}/_ccr/pause_follow`
* `/{index}/_ccr/follow` to `/{index}/_ccr/resume_follow`

Relates to #33931
2018-09-28 08:02:20 +02:00
Martijn van Groningen 17b3b97899
Fixed CCR stats api serialization issues and (#33983)
always use `IndicesOptions.strictExpand()` for indices options.

The follow index may be closed and we still want to get stats from
shard follow task and the whether the provided index name matches with
follow index name is checked when locating the task itself in the ccr
stats transport action.
2018-09-28 07:45:32 +02:00
Nik Everett 33a264a408
Logging: Drop Settings from security logger get calls (#33940)
`Settings` is no longer required to get a `Logger` and we went to quite
a bit of effort to pass it to the `Logger` getters. This removes the
`Settings` from all of the logger fetches in security and x-pack:core.
2018-09-27 17:35:04 -04:00
Lee Hinman a26cc1a242 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-27 11:00:37 -06:00
Jay Modi a48b86e7c6
Security: use default scroll keepalive (#33639)
Security previously hardcoded a default scroll keepalive of 10 seconds,
but in some cases this is not enough time as there can be network
issues or overloading of host machines. After this change, security
will now use the default keepalive timeout, which is controllable using
a setting and the default value is 5 minutes.
2018-09-26 14:42:22 -06:00
Nik Everett ddce9704d4
Logging: Drop two deprecated methods (#34055)
This drops two deprecated methods from `ESLoggerFactory`, switching all
calls to those methods to calls to methods of the same name on
`LogManager`.
2018-09-26 11:20:52 -04:00
Ryan Ernst 7800b4fa91
Core: Abstract DateMathParser in an interface (#33905)
This commits creates a DateMathParser interface, which is already
implemented for both joda and java time. While currently the java time
DateMathParser is not used, this change will allow a followup which will
create a DateMathParser from a DateFormatter, so the caller does not
need to know the internals of the DateFormatter they have.
2018-09-26 07:56:25 -07:00
Christoph Büscher ba3ceeaccf
Clean up "unused variable" warnings (#31876)
This change cleans up "unused variable" warnings. There are several cases were we 
most likely want to suppress the warnings (especially in the client documentation test
where the snippets contain many unused variables). In a lot of cases the unused
variables can just be deleted though.
2018-09-26 14:09:32 +02:00
Gordon Brown c0bfc07f53
Only make indexes read-only on Shrink and ForceMerge actions (#33907)
ILM now only forces indices to become read only in the case of Shrink
and Force Merge actions, as these are most useful in cases where the
index is no longer being written to.
2018-09-25 10:16:01 -06:00
David Roberts dfe5af0411
[ML] Return both Joda and Java formats from structure finder (#33900)
Previously the timestamp_formats field in the response
from the find_file_structure endpoint contained Joda
timestamp formats.  This change makes that clear by
renaming the field to joda_timestamp_formats, and also
adds a java_timestamp_formats field containing the
equivalent Java time format strings.
2018-09-25 12:52:51 +01:00
David Kyle 7bc7624caf
Leniently parse ml persistent task parameters (#33950) 2018-09-25 10:18:30 +01:00
Martijn van Groningen 793b2a94b4
[CCR] Expose auto follow stats to monitoring (#33886) 2018-09-25 07:19:46 +02:00
Nhat Nguyen 6ec36b1273
CCR: Make AutoFollowMetadata immutable (#33977)
We should make AutoFollowMetadata immutable to avoid being inconsistent
when one thread modifies it while other reads it.
2018-09-24 17:47:10 -04:00
Martijn van Groningen 2795ef561f
[CCR] Add get auto follow pattern api (#33849)
Relates to #33007
2018-09-24 20:26:13 +02:00
Lee Hinman 243e863f6e Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-24 10:33:51 -06:00
Daniel Mitterdorfer 5a53193db0 Remove public modifier in SourceOnlySnapshotTests
The public modifier is reported as redundant by checkstyle. With this
commit we remove it to avoid the checkstyle error.
2018-09-24 10:27:43 +02:00
Simon Willnauer 738d9f1a3a Drop all fully deleted segments for comparison
The source only snapshot drops fully deleted segments before snapshotting
them. In order to compare them we need to drop all fully deleted segments
in the test as well.

Closes #33755
2018-09-24 08:32:32 +02:00
David Roberts b89551c452
[ML] Display integers without .0 in file structure field stats (#33947)
Previously numeric values in the field_stats created by the
find_file_structure endpoint were always output with a
decimal point.  This looked unfriendly and unnatural for
fields that clearly store integer values.  This change
converts integer values to type Integer before output in
the file structure field stats.
2018-09-22 15:48:59 +01:00
Martijn van Groningen e1e5f40727
[CCR] Move headers from auto follow pattern to auto follow metadata (#33846)
This ensures that we will not serialize the headers as part of the
auto follow pattern in the to be added get auto follow api.
2018-09-21 18:08:29 +02:00
Christoph Büscher b654d986d7
Add OneStatementPerLineCheck to Checkstyle rules (#33682)
This change adds the OneStatementPerLineCheck to our checkstyle precommit
checks. This rule restricts the number of statements per line to one. The
resoning behind this is that it is very difficult to read multiple statements on
one line. People seem to mostly use it in short lambdas and switch statements in
our code base, but just going through the changes already uncovered some actual
problems in randomization in test code, so I think its worth it.
2018-09-21 11:52:31 +02:00
Dimitris Athanasiou 8e3a0fad9d
[ML] Refactor job deletion logic into the transport action (#33891)
The job deletion logic was scattered around a few places:
the transport action, the job manager and the deletion task.
Overloading the task with deletion logic also meant extra
dependencies in the core package which should be unnecessary.

This commit consolidates all this logic into the transport action
and replaces the deletion task with a plain one that needs not be
aware of deletion logic.
2018-09-20 15:48:42 +01:00
Martijn van Groningen 44c7c4b166
[CCR] Add auto follow stats api (#33801)
GET /_ccr/auto_follow/stats

Returns:

```
{
   "number_of_successful_follow_indices": ...
   "number_of_failed_follow_indices": ...
   "number_of_failed_remote_cluster_state_requests": ...
   "recent_auto_follow_errors": [
      ...
   ]
}
```

Relates to #33007
2018-09-20 07:16:20 +02:00
Gordon Brown 90de436e55
Use custom index metadata for ILM state (#33783)
Using index settings for ILM state is fragile and exposes too much
information that doesn't need to be exposed. Using custom index metadata
is more resilient and allows more controlled access to internal
information.

As part of these changes, moves away from using defaults for ILM-related
values, in favor of using null values to clearly indicate that the value is not
present.
2018-09-19 14:50:48 -06:00
Benjamin Trent 4767a016a5
Adding node_count to ML Usage (#33850) (#33863) 2018-09-19 13:35:09 -07:00
Lee Hinman 81e9150c7a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-19 09:43:26 -06:00
Benjamin Trent 4190a9f1e9
Delete custom index if the only contained job is deleted (#33788)
* Delete custom index if the only contained job is deleted
2018-09-19 07:42:26 -07:00
markharwood c118581617 Test fix - Graph connections could appear in different orders
Graph connections could appear in different orders based on insertion sequence

Closes #33686
2018-09-19 15:16:14 +01:00
Martijn van Groningen d9947c631a
[CCR] Rename idle_shard_retry_delay to poll_timout in auto follow patterns (#33821) 2018-09-19 13:13:20 +02:00
Simon Willnauer 251489d59a
Cut over to unwrap segment reader (#33843)
The fix in #33757 introduces some workaround since FilterCodecReader didn't
support unwrapping. This cuts over to a more elegant fix to access the readers
segment infos.
2018-09-19 10:18:03 +02:00
Martijn van Groningen 013b64a07c
[CCR] Change FollowIndexAction.Request class to be more user friendly (#33810)
Instead of having one constructor that accepts all arguments, all parameters
should be provided via setters. Only leader and follower index are required
arguments. This makes using this class in tests and transport client easier.
2018-09-19 07:18:24 +02:00
Lee Hinman 27dd25857b
Rebuild step on PolicyStepsRegistry.getStep (#33780)
This moves away from caching a list of steps for a current phase, instead
rebuilding the necessary step from the phase JSON stored in the index's
metadata.

Relates to #29823
2018-09-18 17:07:57 -06:00
Lee Hinman c87cff22b4 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-18 13:57:41 -06:00
Martijn van Groningen 47b86d6e6a
[CCR] Changed AutoFollowCoordinator to keep track of certain statistics (#33684)
The following stats are being kept track of:
1) The total number of times that auto following a leader index succeed.
2) The total number of times that auto following a leader index failed.
3) The total number of times that fetching a remote cluster state failed.
4) The most recent 256 auto follow failures per auto leader index
   (e.g. create_and_follow api call fails) or cluster alias
   (e.g. fetching remote cluster state fails).

Each auto follow run now produces a result that is being used to update
the stats being kept track of in AutoFollowCoordinator.

Relates to #33007
2018-09-18 09:43:50 +02:00
Tal Levy 94a66c556d
add phase execution info to ILM Explain API (#33488)
adds a section for phase execution to the Explain API.

This contains

- phase definition
- policy name
- policy version
- modified date
2018-09-17 17:00:00 -07:00
Martijn van Groningen 15f30d689b
[CCR] Do not unnecessarily wrap fetch exception in a ElasticSearch exception and (#33777)
* [CCR] Do not unnecessarily wrap fetch exception in a ElasticSearch exception and
properly map fetch_exception.exception field as object.

The extra caused by level is not necessary here:

```
"fetch_exceptions": [
              {
                "from_seq_no": 1,
                "retries": 106,
                "exception": {
                  "type": "exception",
                  "reason": "[index1] IndexNotFoundException[no such index]",
                  "caused_by": {
                    "type": "index_not_found_exception",
                    "reason": "no such index",
                    "index_uuid": "_na_",
                    "index": "index1"
                  }
                }
              }
            ],
```
2018-09-17 22:33:37 +02:00
Lee Hinman 7ff11b4ae1 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-17 10:41:10 -06:00
Simon Willnauer 48a5b45d28
Ensure fully deleted segments are accounted for correctly (#33757)
We can't rely on the leaf reader ordinal in a wrapped reader since
it might not correspond to the ordinal in the SegmentInfos for it's
SegmentCommitInfo.

Relates to #32844
Closes #33689
Closes #33755
2018-09-17 18:18:58 +02:00
Martijn van Groningen 481f8a9a07
[CCR] Make auto follow patterns work with security (#33501)
Relates to #33007
2018-09-17 07:29:00 +02:00
Jason Tedor 770ad53978
Introduce long polling for changes (#33683)
Rather than scheduling pings to the leader index when we are caught up
to the leader, this commit introduces long polling for changes. We will
fire off a request to the leader which if we are already caught up will
enter a poll on the leader side to listen for global checkpoint
changes. These polls will timeout after a default of one minute, but can
also be specified when creating the following task. We use these time
outs as a way to keep statistics up to date, to not exaggerate time
since last fetches, and to avoid pipes being broken.
2018-09-16 10:35:23 -04:00
Jason Tedor 069605bd91
Do not count shard changes tasks against REST tests (#33738)
When executing CCR REST tests it is going to be expected after global
checkpoint polling goes in that shard changes tasks can still be pending
at the end of the test. One way to deal with this is to set a low
timeout on these polls, but then that means we are not executing our
REST tests with our default production settings and instead would be
using an unrealistic low timeout. Alternatively, since we expect these
tasks to be there, we can not count them against the test. That is what
this commit does.
2018-09-16 07:32:12 -04:00
Lee Hinman e6cbaa5a78 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-14 16:27:37 -06:00
Martijn van Groningen 82a6ae1dae
[CCR] Move ccr tests in core module back to ccr module (#33711)
When developing ccr it is not ideal if tests are in multiple modules.
Even the classes these tests test are in the core module, it is easier
if these tests are in ccr module in order to avoid running the test task
in core module. This results in running many non ccr tests.

This way when developing ccr we can run locally:
./gradlew x-pack:plugin:core:precommit x-pack:plugin:ccr:check

before pushing to PR branches and be confident that the PR build passes,
without running x-pack:plugin:core:check task.
2018-09-14 17:18:00 +02:00
Ioannis Kakavas d9f5e4fd2e Pin TLS1.2 in SSLConfigurationReloaderTests
Ensure that the SSLConfigurationReloaderTests can run with JDK 11
by pinning the HttpClient to TLS version to TLS1.2. This is necessary
becase even if the MockWebServer is set to user TLS1.2, we don't
set its enabled protocols, so if it receives a TLS1.3 request (which
is the default behavior for HttpClient in JDK11), it will use TLS1.3
and the original issue will manifest again.

Relates  #33127
Resolves #32124
2018-09-14 16:39:20 +03:00
Jason Tedor 2282150f34
Expose retries for CCR fetch failures (#33694)
This commit exposes the number of times that a fetch has been tried to
the CCR stats endpoint, and to CCR monitoring.
2018-09-14 08:52:46 -04:00
Albert Zaharovits c86e2d5211
Structured audit logging (#31931)
Changes the format of log events in the audit logfile.
It also changes the filename suffix from `_access` to `_audit`.
The new entry format is consistent with Elastic Common Schema.
Entries are formatted as JSON with no nested objects and field
names have a dotted syntax. Moreover, log entries themselves
are not spaced by commas and there is exactly one entry per line.
In addition, entry fields are ordered, unlike a typical JSON doc,
such that a human would not strain his eyes over jumbled 
fields from one line to the other; the order is defined in the log4j2
properties file.
The implementation utilizes the log4j2's `StringMapMessage`.
This means that the application builds the log event as a map
and the log4j logic (the appender's layout) handle the format
internally. The layout, such as the set of printed fields and their
order, can be changed at runtime without restarting the node.
2018-09-14 15:25:53 +03:00
David Roberts 568ac10ca6
[ML] Allow overrides for some file structure detection decisions (#33630)
This change modifies the file structure detection functionality
such that some of the decisions can be overridden with user
supplied values.

The fields that can be overridden are:

- charset
- format
- has_header_row
- column_names
- delimiter
- quote
- should_trim_fields
- grok_pattern
- timestamp_field
- timestamp_format

If an override makes finding the file structure impossible then
the endpoint will return an exception.
2018-09-14 09:29:11 +01:00
Ioannis Kakavas 8ae1eeb303
[TESTS] Disable specific locales for RestrictedTrustManagerTest (#33299)
Disable specific Thai and Japanese locales as Certificate expiration
validation fails due to the date parsing of BouncyCastle (that manifests
in a FIPS 140 JVM as this is the only place we use BouncyCastle).
Added the locale switching logic here instead of subclassing
ESTestCase as these are the only tests that fail for these locales and
JVM combination.

Resolves #33081
2018-09-14 09:42:03 +03:00
Nhat Nguyen 189aaceecf AwaitsFix testRestoreMinmal
Tracked at #33689
2018-09-13 22:15:21 -04:00
Jay Modi 3914a980f7
Security: remove wrapping in put user response (#33512)
This change removes the wrapping of the created field in the put user
response. The created field was added as a top level field in #32332,
while also still being wrapped within the `user` object of the
response. Since the value is available in both formats in 6.x, we can
remove the wrapped version for 7.0.
2018-09-13 14:40:36 -06:00
Martijn van Groningen 53ba253aa4
[CCR] Add validation for max_retry_delay (#33648) 2018-09-13 20:52:00 +02:00
Colin Goodheart-Smithe 8e59de3eb2
Merge branch 'master' into index-lifecycle 2018-09-13 09:46:14 +01:00
Jason Tedor eb715d5290
Add follower index to CCR monitoring and status (#33645)
This commit adds the follower index to CCR shard follow task status, and
to monitoring.
2018-09-12 17:35:06 -04:00
Martijn van Groningen b5d8495789
[CCR] Add auto follow pattern APIs to transport client. (#33629) 2018-09-12 21:50:22 +02:00
Jay Modi 20c6c9c542
Address license state update/read thread safety (#33396)
This change addresses some issues regarding thread safety around
updates and method calls on the XPackLicenseState object. There exists
a possibility that there could be a concurrent update to the
XPackLicenseState when there is a scheduled check to see if the license
is expired and a cluster state update. In order to address this, the
update method now has a synchronized block where member variables are
updated. Each method that reads these variables is now also
synchronized.

Along with the above change, there was a consistency issue around
security calls to the license state. The majority of security checks
make two calls to the license state, which could result in incorrect
behavior due to the checks being made against different license states.
The majority of this behavior was introduced for 6.3 with the inclusion
of x-pack in the default distribution. In order to resolve the majority
of these cases, the `isSecurityEnabled` method is no longer public and
the logic is also included in individual methods about security such as
`isAuthAllowed`. There were a few cases where this did not remove
multiple calls on the license state, so a new method has been added
which creates a copy of the current license state that will not change.
Callers can use this copy of the license state to make decisions based
on a consistent view of the license state.
2018-09-12 13:08:09 -06:00
Martijn van Groningen 901d8035d9
[CCR] Update es monitoring mapping and (#33635)
* [CCR] Update es monitoring mapping and
change qa tests to query based on leader index.


Co-authored-by: Jason Tedor <jason@tedor.me>
2018-09-12 19:36:17 +02:00
Simon Willnauer c783488e97
Add `_source`-only snapshot repository (#32844)
This change adds a `_source` only snapshot repository that allows to wrap
any existing repository as a _backend_ to snapshot only the `_source` part
including live docs markers. Snapshots taken with the `source` repository
won't include any indices,  doc-values or points. The snapshot will be reduced in size and
functionality such that it requires full re-indexing after it's successfully restored.

The restore process will copy the `_source` data locally starts a special shard and engine
to allow `match_all` scrolls and searches. Any other query, or get call will fail with and unsupported operation exception.  The restored index is also marked as read-only.

This feature aims mainly for disaster recovery use-cases where snapshot size is
a concern or where time to restore is less of an issue.

**NOTE**: The snapshot produced by this repository is still a valid lucene index. This change doesn't allow for any longer retention policies which is out of scope for this change.
2018-09-12 17:47:10 +02:00
Jason Tedor 23f12e42c1
Expose CCR stats to monitoring (#33617)
This commit exposes the CCR stats endpoint to monitoring collection.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-09-12 09:13:07 -04:00
Martijn van Groningen 96c49e5ed0
[CCR] Improve shard follow task's retryable error handling (#33371)
Improve failure handling of retryable errors by retrying remote calls in
a exponential backoff like manner. The delay between a retry would not be
longer than the configured max retry delay. Also retryable errors will be
retried indefinitely.

Relates to #30086
2018-09-12 12:49:51 +02:00
Jason Tedor eca37e6e0a
Expose CCR to the transport client (#33608)
This commit exposes CCR to the transport client.
2018-09-11 16:37:52 -04:00
David Roberts 8e05ce567f
[ML] Rename input_fields to column_names in file structure (#33568)
This change tightens up the meaning of the "input_fields" field
in the file structure finder output.  Previously it was permitted
but not calculated for JSON and XML files.  Following this change
the field is called "column_names" and is only permitted for
delimited files.

Additionally the way the column names are set for headerless
delimited files is refactored to encapsulate the way they're
named to one line of the code rather than having the same
logic in two places.
2018-09-11 08:46:26 +01:00
Colin Goodheart-Smithe cdc4f57a77
Merge branch 'master' into index-lifecycle 2018-09-10 21:30:44 +01:00
Chris Roberson 369db8a9d6
Update beats template to include apm-server metrics (#33286) 2018-09-10 08:50:07 -05:00
Martijn van Groningen c4adcee3ea
[CCR] Add create_follow_index privilege (#33559)
This is a new index privilege that the user needs to have in the follow cluster.
This privilege is required in addition to the `manage_ccr` cluster privilege in
order to execute the create and follow api.

Closes #33555
2018-09-10 13:08:20 +02:00
Ioannis Kakavas 77aeeda275
Correctly handle PKCS#11 tokens for system keystore (#33460)
* Correctly handle NONE keyword for system keystore

As defined in the PKCS#11 reference guide
https://docs.oracle.com/javase/8/docs/technotes/guides/security/p11guide.html
PKCS#11 tokens can be used as the JSSE keystore and truststore and
the way to indicate this is to set `javax.net.ssl.keyStore` and
`javax.net.ssl.trustStore` to `NONE` (case sensitive).

This commits ensures that we honor this convention and do not
attempt to load the keystore or truststore if the system property is
set to NONE.

* Handle password protected system truststore

When a PKCS#11 token is used as the system truststore, we need to
pass a password when loading it, even if only for reading
certificate entries. This commit ensures that if
`javax.net.ssl.trustStoreType` is set to `PKCS#11` (as it would
when a PKCS#11 token is in use) the password specified in
`javax.net.ssl.trustStorePassword` is passed when attempting to
load the truststore.

Relates #33459
2018-09-10 11:18:44 +03:00
Jason Tedor 6bb817004b
Add infrastructure to upgrade settings (#33536)
In some cases we want to deprecate a setting, and then automatically
upgrade uses of that setting to a replacement setting. This commit adds
infrastructure for this so that we can upgrade settings when recovering
the cluster state, as well as when such settings are dynamically applied
on cluster update settings requests. This commit only focuses on cluster
settings, index settings can build on this infrastructure in a
follow-up.
2018-09-09 20:49:19 -04:00
Dimitris Athanasiou fcb15b0ce3
[ML] Get job stats request should filter non-ML job tasks (#33516)
When requesting job stats for `_all`, all ES tasks are accepted
resulting to loads of cluster traffic and a memory overhead.
This commit correctly filters out non ML job tasks.

Closes #33515
2018-09-09 22:53:03 +01:00
Lee Hinman b7649fce0c
Rename "after" to "minimum_age" in lifecycle definition (#33530)
This renames the "after" field to better reflect what the meaning is.

Supercedes #32624
2018-09-08 21:40:55 -06:00
Gordon Brown 008e64b395
HLRC: Add Get Lifecycle Policy API to HLRC (#33323)
Adds Request and Reponse classes for accessing lifecycle policies.
Changes existing tests to use these classes where appropriate.
Sets up SPI configuration to allow parsing *Actions from XContent.
2018-09-07 17:41:34 -06:00
David Roberts e42cc5cd8c
[ML] Add a file structure determination endpoint (#33471)
This endpoint accepts an arbitrary file in the request body and
attempts to determine the structure.  If successful it also
proposes mappings that could be used when indexing the file's
contents, and calculates simple statistics for each of the fields
that are useful in the data preparation step prior to configuring
machine learning jobs.
2018-09-07 17:41:57 +01:00
Colin Goodheart-Smithe f83641346f
Adds checks to ensure index metadata exists when we try to use it (#33455)
* Adds checks to ensure index metadata exists when we try to use it

* Fixes failing test
2018-09-07 13:06:51 +01:00
Colin Goodheart-Smithe 017ffe5d12
Merge branch 'master' into index-lifecycle 2018-09-07 10:59:10 +01:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
Yogesh Gaikwad ee73bc2f3f
[SECURITY] Set Auth-scheme preference (#33156)
Some browsers (eg. Firefox) behave differently when presented with
multiple auth schemes in 'WWW-Authenticate' header. The expected
behavior is that browser select the most secure auth-scheme before
trying others, but Firefox selects the first presented auth scheme and
tries the next ones sequentially. As the browser interpretation is
something that we do not control, we can at least present the auth
schemes in most to least secure order as the server's preference.

This commit modifies the code to collect and sort the auth schemes
presented by most to least secure. The priority of the auth schemes is
fixed, the lower number denoting more secure auth-scheme.
The current order of schemes based on the ES supported auth-scheme is
[Negotiate, Bearer,Basic] and when we add future support for
other schemes we will need to update the code. If need be we will make
this configuration customizable in future.

Unit test to verify the WWW-Authenticate header values are sorted by
server preference as more secure to least secure auth schemes.
Tested with Firefox, Chrome, Internet Explorer 11.

Closes#32699
2018-09-07 08:46:49 +10:00
Tal Levy 21bb4720a2
add notion of version and modified_date to LifecyclePolicyMetadata (#33450)
It is useful to keep track of which version of a policy is currently
being executed by a specific index. For management purposes, it would
also be useful to know at which time the latest version was inserted
so that an audit trail is left for reconciling changes happening in ILM.
2018-09-06 13:32:24 -07:00
Jim Ferenczi 7ad71f906a
Upgrade to a Lucene 8 snapshot (#33310)
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309

Closes #32899
2018-09-06 14:42:06 +02:00
David Roberts 0849b98f60
[ML] Rename log structure to file structure (#33421)
Many files supplied to the upcoming ML data preparation
functionality will not be "log" files.  For example,
CSV files are generally not "log" files.  Therefore it
makes sense to rename library that determines the
structure of these files.

Although "file structure" could be considered too broad,
as the library currently only works with a few text
formats, in the future it may be extended to work with
more formats.
2018-09-06 09:13:08 +01:00
Alan Woodward e134f9b5f3
Fix generics in ScriptPlugin#getContexts() (#33426)
Changes the return value from List<ScriptContext> to List<ScriptContext<?>> to remove raw-types warnings.
2018-09-06 09:04:22 +01:00
Colin Goodheart-Smithe b1257d873b
Merge branch 'master' into index-lifecycle 2018-09-06 08:17:40 +01:00
Martijn van Groningen a721d09c81
[CCR] Added auto follow patterns feature (#33118)
Auto Following Patterns is a cross cluster replication feature that
keeps track whether in the leader cluster indices are being created with
names that match with a specific pattern and if so automatically let
the follower cluster follow these newly created indices.

This change adds an `AutoFollowCoordinator` component that is only active
on the elected master node. Periodically this component checks the
 the cluster state of remote clusters if there new leader indices that
match with configured auto follow patterns that have been defined in
`AutoFollowMetadata` custom metadata.

This change also adds two new APIs to manage auto follow patterns. A put
auto follow pattern api:

```
PUT /_ccr/_autofollow/{{remote_cluster}}
{
   "leader_index_pattern": ["logs-*", ...],
   "follow_index_pattern": "{{leader_index}}-copy",
   "max_concurrent_read_batches": 2
   ... // other optional parameters
}
```

and delete auto follow pattern api:

```
DELETE /_ccr/_autofollow/{{remote_cluster_alias}}
```

The auto follow patterns are directly tied to the remote cluster aliases
configured in the follow cluster.

Relates to #33007


Co-authored-by: Jason Tedor jason@tedor.me
2018-09-06 08:01:58 +02:00
Lee Hinman 96d515e3f5
Replace PhaseAfterStep with PhaseCompleteStep (#33398)
This removes `PhaseAfterStep` in favor of a new `PhaseCompleteStep`. This step
in only a marker that the `LifecyclePolicyRunner` needs to halt until the time
indicated for entering the next phase.

This also fixes a bug where phase times were encapsulated into the policy
instead of dynamically adjusting to policy changes.

Supersedes #33140, which it replaces
Relates to #29823
2018-09-05 16:37:45 -06:00
Tim Brooks 88c178dca6
Add sni name to SSLEngine in netty transport (#33144)
This commit is related to #32517. It allows an "server_name"
attribute on a DiscoveryNode to be propagated to the server using
the TLS SNI extentsion. This functionality is only implemented for
the netty security transport.
2018-09-05 16:12:10 -06:00
Tal Levy b5f7fb6882 Merge branch 'master' into index-lifecycle 2018-09-05 12:56:58 -07:00
Tal Levy 0f8bc10bcf
add new phase definition setting used for retrieving phase to execute (#33289)
Since policies can be updated independent of execution plans for the current
phase being executed, it would be nice to know what the phase that is executing
looks like in JSON. This PR does just that, while also using that index setting
to recontruct the phase steps to execute (for consistency)
2018-09-05 11:35:20 -07:00
Jay Modi ea52277a1e
HLRest: add put user API (#32332)
This commit adds a security client to the high level rest client, which
includes an implementation for the put user api. As part of these
changes, a new request and response class have been added that are
specific to the high level rest client. One change here is that the response
was previously wrapped inside a user object. The plan is to remove this
wrapping and this PR adds an unwrapped response outside of the user
object so we can remove the user object later on.

See #29827
2018-09-05 10:56:30 -06:00
Colin Goodheart-Smithe a7b5f2c07d
Adds ILMClient for use with transport client (#33357)
* Adds ILMClient for use with transport client

* add sync methods

* adds ILMClient to XPackClient
2018-09-05 12:49:56 +01:00
Colin Goodheart-Smithe f00a28a909
Merge branch 'master' into index-lifecycle 2018-09-05 09:48:48 +01:00
Colin Goodheart-Smithe 30544cbe7a
Fixes Rendering of after in Phase (#33304)
Closes #33282
2018-09-04 18:03:07 +01:00
Sohaib Iftikhar 761e8c461f HLRC: Add delete by query API (#32782)
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou 1457b07a06
[ML] The sort field on get records should default to the record_score (#33358)
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
Colin Goodheart-Smithe 92ab442aee
Merge branch 'master' into index-lifecycle 2018-09-04 10:34:49 +01:00
Benjamin Trent 767d8e0801
[ML] Delete forecast API (#31134) (#33218)
* Delete forecast API (#31134)
2018-09-03 19:06:18 -05:00
David Kyle ccb2ad25cc
Prevent NPE parsing the stop datafeed request. (#33347)
The issue depends on the request parameters being passed in the request
body rather than as query parameters.
2018-09-03 13:35:04 +01:00
Colin Goodheart-Smithe e2c1beb1be
Merge branch 'master' into index-lifecycle 2018-09-03 10:01:16 +01:00
Zachary Tong 90ce3a6224 [Rollup] Fix Caps Comparator to handle calendar/fixed time (#33336)
The comparator used TimeValue parsing, which meant it couldn't handle
calendar time.  This fixes the comparator to handle either (and potentially
mixed).  The mixing shouldn't be an issue since the validation code
upstream will prevent it, but was simplest to allow the comparator
to handle both.
2018-09-03 10:49:19 +02:00
Jim Ferenczi f0a61b6dec
Security for _field_names field should not override field statistics (#33261)
In Lucene 8 the statistics for a field (doc_count, sum_doc_count, ...) are
checked and invalid values (v < 0) are rejected. Though for the _field_names
field we hide the statistics of the field if security is enabled since
some terms (field names) may be filtered. However this statistics are never
used, this field is not used for ranking and cannot be used to generate
term vectors. For these reasons this commit restores the original statistics
for the field in order to be compliant with Lucene 8.
2018-09-03 09:36:39 +02:00
Colin Goodheart-Smithe 3eef74d5d5
Merge branch 'master' into index-lifecycle 2018-08-31 14:45:22 +01:00
Colin Goodheart-Smithe ada3e710f6
Renames XPackField.INDEX _LIFCYCLE value to "ilm" (#33270)
This brings the name in line with everywhere else and means that name
seen on the feature usage and `GET _xpack` APIs will match the plugin
name.

This change also removes `IndexLifcycle.NAME` since this was only used
to name the scheduler job and that can be done using
`XPackField.INDEX_LIFECYCLE` instead
2018-08-31 08:29:44 +01:00
Nhat Nguyen 5330067033 Merge branch 'master' into ccr
* master:
  Integrates soft-deletes into Elasticsearch (#33222)
  Revert "Integrates soft-deletes into Elasticsearch (#33222)"
  Add support for "authorization_realms" (#33262)
2018-08-31 00:07:21 -04:00
Tim Vernum 273c82d7c9
Add support for "authorization_realms" (#33262)
Authorization Realms allow an authenticating realm to delegate the task
of constructing a User object (with name, roles, etc) to one or more
other realms.

E.g. A client could authenticate using PKI, but then delegate to an LDAP
realm. The LDAP realm performs a "lookup" by principal, and then does
regular role-mapping from the discovered user.

This commit includes:
- authorization_realm support in the pki, ldap, saml & kerberos realms
- docs for authorization_realms
- checks that there are no "authorization chains"
   (whereby "realm-a" delegates to "realm-b", but "realm-b" delegates to "realm-c")

Authorization realms is a platinum feature.
2018-08-31 13:25:27 +10:00
Nhat Nguyen d3f32273eb Merge branch 'master' into ccr 2018-08-30 23:22:58 -04:00
Tal Levy 4a958ee2bc fix shrunken index source name metadata usage 2018-08-30 15:29:04 -07:00
Tal Levy 13a0d822d0 Merge branch 'master' into index-lifecycle 2018-08-30 15:04:17 -07:00
Ioannis Kakavas 214652d4af
[TESTS] Pin MockWebServer to TLS1.2 (#33127)
Ensure that the SSLConfigurationReloaderTests can run with JDK 11
by pinning the Server TLS version to TLS1.2. This can be revisited
while tackling the effort to full support TLSv1.3 in
https://github.com/elastic/elasticsearch/issues/32276

Resolves #32124
2018-08-30 09:13:28 +03:00
Hendrik Muhs cfc003d485 [Rollup] Re-factor Rollup Indexer into a generic indexer for re-usability (#32743)
This extracts a super class out of the rollup indexer called the AsyncTwoPhaseIterator. 
The implementor of it can define the query, transformation of the response, 
indexing and the object to persist the position/state of the indexer.

The stats object used by the indexer to record progress is also now abstract, allowing
the implementation provide custom stats beyond what the indexer provides.  It also
allows the implementation to decide how the stats are presented (leaves toXContent()
up to the implementation).

This should allow new projects to reuse the search-then-index persistent task that Rollup
uses, but without the restrictions/baggage of how Rollup has to work internally to
satisfy time-based rollups.
2018-08-29 14:28:21 -04:00
Nhat Nguyen 5632e31c74 Merge branch 'master' into ccr
* master:
  Painless: Add Bindings (#33042)
  Update version after client credentials backport
  Fix forbidden apis on FIPS (#33202)
  Remote 6.x transport BWC Layer for `_shrink` (#33236)
  Test fix - Graph HLRC tests needed another field adding to randomisation exception list
  HLRC: Add ML Get Records API (#33085)
  [ML] Fix character set finder bug with unencodable charsets (#33234)
  TESTS: Fix overly long lines (#33240)
  Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic
  Remove unsupported group_shard_failures parameter (#33208)
  Update BucketUtils#suggestShardSideQueueSize signature (#33210)
  Parse PEM Key files leniantly (#33173)
  INGEST: Add Pipeline Processor (#32473)
  Core: Add java time xcontent serializers (#33120)
  Consider multi release jars when running third party audit (#33206)
  Update MSI documentation (#31950)
  HLRC: create base timed request class (#33216)
  [DOCS] Fixes command page titles
  HLRC: Move ML protocol classes into client ml package (#33203)
  Scroll queries asking for rescore are considered invalid (#32918)
  Painless: Fix Semicolon Regression (#33212)
  ingest: minor - update test to include dissect (#33211)
  Switch remaining LLREST usage to new style Requests (#33171)
  HLREST: add reindex API (#32679)
2018-08-29 12:30:24 -04:00
jaymode 6daf8115d6
Update version after client credentials backport
This commit changes the serialization version from V_7_0_0_alpha1 to
V_6_5_0 for the create token request and response with a client
credentials grant type. The client credentials work has now been
backported to 6.x.

Relates #33106
2018-08-29 09:02:32 -06:00
Gordon Brown 454ce99b01 Merge branch 'master' into index-lifecycle 2018-08-29 08:28:23 -06:00
markharwood e95c2afe3c Test fix - Graph HLRC tests needed another field adding to randomisation exception list
Related to #33231
2018-08-29 15:19:26 +01:00
Armin Braun dd1956cf19
TESTS: Fix overly long lines (#33240) 2018-08-29 15:49:35 +02:00
markharwood 63b2db1d84 Test fix - Graph HLRC test was missing field name to be excluded from randomisation logic
Closes #33231
2018-08-29 14:13:30 +01:00
Ioannis Kakavas 8c57d4af6a
Parse PEM Key files leniantly (#33173)
Allow for extra non-whitespace before the Header of PEM encoded
key files.

Resolves #33168
2018-08-29 12:35:31 +03:00
Nhat Nguyen c42dc77896 Merge branch 'master' into ccr
* master:
  [Rollup] Better error message when trying to set non-rollup index (#32965)
  HLRC: Use Optional in validation logic (#33104)
  Remove unused User class from protocol (#33137)
  ingest: Introduce the dissect processor (#32884)
  [Docs] Add link to es-kotlin-wrapper-client (#32618)
  [Docs] Remove repeating words (#33087)
  Minor spelling and grammar fix (#32931)
  Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979)
  Watcher: Simplify finding next date in cron schedule (#33015)
  Run Third party audit with forbidden APIs CLI  (part3/3) (#33052)
  Fix plugin build test on Windows (#33078)
  HLRC+MINOR: Remove Unused Private Method (#33165)
  Remove old unused test script files (#32970)
  Build analysis-icu client JAR (#33184)
  Ensure to generate identical NoOp for the same failure (#33141)
  ShardSearchFailure#readFrom to set index and shardId (#33161)
2018-08-28 13:56:38 -04:00
Jay Modi 1e11b05b58
Remove unused User class from protocol (#33137)
This commit removes the unused User class from the protocol project.
This class was originally moved into protocol in preparation for moving
more request and response classes, but given the change in direction
for the HLRC this is no longer needed. Additionally, this change also
changes the package name for the User object in x-pack/plugin/core to
its original name.
2018-08-28 08:55:29 -06:00
Gordon Brown 50368656ee Merge branch 'master' into index-lifecycle 2018-08-27 15:35:19 -06:00
Jason Tedor 0e5d42ca38
Merge branch 'master' into ccr
* master:
  Adjust BWC version on mapping version
  Token API supports the client_credentials grant (#33106)
  Build: forked compiler max memory matches jvmArgs (#33138)
  Introduce mapping version to index metadata (#33147)
  SQL: Enable aggregations to create a separate bucket for missing values (#32832)
  Fix grammar in contributing docs
  SECURITY: Fix Compile Error in ReservedRealmTests (#33166)
  APM server monitoring (#32515)
  Support only string `format` in date, root object & date range (#28117)
  [Rollup] Move toBuilders() methods out of rollup config objects (#32585)
  Fix forbiddenapis on java 11  (#33116)
  Apply publishing to genreate pom (#33094)
  Have circuit breaker succeed on unknown mem usage
  Do not lose default mapper on metadata updates (#33153)
  Fix a mappings update test (#33146)
  Reload Secure Settings REST specs & docs (#32990)
  Refactor CachingUsernamePassword realm (#32646)
2018-08-27 13:49:59 -04:00
Jay Modi 5d9c270608
Token API supports the client_credentials grant (#33106)
This change adds support for the client credentials grant type to the
token api. The client credentials grant allows for a client to
authenticate with the authorization server and obtain a token to access
as itself. Per RFC 6749, a refresh token should not be included with
the access token and as such a refresh token is not issued when the
client credentials grant is used.

The addition of the client credentials grant will allow users
authenticated with mechanisms such as kerberos or PKI to obtain a token
that can be used for subsequent access.
2018-08-27 10:56:21 -06:00
Tal Levy 5783545222 Merge branch 'master' into index-lifecycle 2018-08-27 08:19:05 -07:00
Shaunak Kashyap 1779d3376a
APM server monitoring (#32515)
* Adding new MonitoredSystem for APM server

* Teaching Monitoring template utils about APM server monitoring indices

* Documenting new monitoring index for APM server

* Adding monitoring index template for APM server

* Copy pasta typo

* Removing metrics.libbeat.config section from mapping

* Adding built-in user and role for APM server user

* Actually define the role :)

* Adding missing import

* Removing index template and system ID for apm server

* Shortening line lengths

* Updating expected number of built-in users in integration test

* Removing "system" from role and user names

* Rearranging users to make tests pass
2018-08-27 08:42:40 -04:00
Tanguy Leroux e1e8cf382f
[Rollup] Move toBuilders() methods out of rollup config objects (#32585) 2018-08-27 09:18:26 +02:00
Nhat Nguyen 75304f405b Merge branch 'master' into ccr
* master:
  Add proxy support to RemoteClusterConnection (#33062)
  TEST: Skip assertSeqNos for closed shards (#33130)
  TEST: resync operation on replica should acquire shard permit (#33103)
  Switch remaining x-pack tests to new style Requests (#33108)
  Switch remaining tests to new style Requests (#33109)
  Switch remaining ml tests to new style Requests (#33107)
  Build: Line up IDE detection logic
  Security index expands to a single replica (#33131)
  HLRC: request/response homogeneity and JavaDoc improvements (#33133)
  Checkstyle!
  [Test] Fix sporadic failure in MembershipActionTests
  Revert "Do NOT allow termvectors on nested fields (#32728)"
  [Rollup] Move toAggCap() methods out of rollup config objects (#32583)
  Fix race condition in scheduler engine test
2018-08-25 21:41:53 -04:00
Tal Levy 74312be0ea Merge branch 'master' into index-lifecycle 2018-08-24 12:41:12 -07:00
Jay Modi b86dad22ce
Security index expands to a single replica (#33131)
This change removes the use of 0-all for auto expand replicas for the
security index. The use of 0-all causes some unexpected behavior with
certain allocation settings. This change allows us to avoid these with
a default install. If necessary, the number of replicas can be tuned by
the user.

Closes #29933
Closes #29712
2018-08-24 12:51:22 -06:00
Tanguy Leroux 7e5efad929
[Rollup] Move toAggCap() methods out of rollup config objects (#32583) 2018-08-24 15:31:41 +02:00
Jason Tedor 9fddf7e6a3
Fix race condition in scheduler engine test
This commit addresses a race condition in the scheduler engine test that
a listener that throws an exception does not cause other listeners to be
skipped. The race here is that we were counting down a latch, and then
throwing an exception yet an assertion that expected the exception to
have been thrown already could execute after the latch was counted down
for the final time but before the exception was thrown and acted upon by
the scheduler engine. This commit addresses this by moving the counting
down of the latch to definitely be after the exception was acted upon by
the scheduler engine.
2018-08-24 07:45:16 -04:00
Jason Tedor 91a052b617
Merge branch 'master' into ccr
* master:
  Add hook to skip asserting x-content equivalence (#33114)
  Muted testListenersThrowingExceptionsDoNotCauseOtherListenersToBeSkipped
  [Rollup] Move getMetadata() methods out of rollup config objects (#32579)
  Muted testEmptyAuthorizedIndicesSearchForAllDisallowNoIndices
  Update Google Cloud Storage Library for Java (#32940)
  Remove unsupported Version.V_5_* (#32937)
2018-08-24 06:55:10 -04:00
Andrei Stefan a2f0a1a0cb Merge branch 'master' of https://github.com/elastic/elasticsearch 2018-08-24 13:14:37 +03:00
Andrei Stefan 1d8745036f Muted testListenersThrowingExceptionsDoNotCauseOtherListenersToBeSkipped 2018-08-24 13:14:03 +03:00
Tanguy Leroux 879a90b999
[Rollup] Move getMetadata() methods out of rollup config objects (#32579)
This committ removes the getMetadata() methods from the DateHistoGroupConfig 
and HistoGroupConfig objects. This way the configuration objects do not rely on RollupField.formatMetaField() anymore and do not expose a getMetadata() 
method that is tighlty coupled to the rollup indexer.
2018-08-24 11:57:46 +02:00
Jim Ferenczi f4e9729d64
Remove unsupported Version.V_5_* (#32937)
This change removes the es 5x version constants and their usages.
2018-08-24 09:51:21 +02:00
Martijn van Groningen 82592dda5a
Merge remote-tracking branch 'es/master' into ccr
* es/master: (62 commits)
  [DOCS] Add docs for Application Privileges (#32635)
  Add versions 5.6.12 and 6.4.1
  Do NOT allow termvectors on nested fields (#32728)
  [Rollup] Return empty response when aggs are missing (#32796)
  [TEST] Add some ACL yaml tests for Rollup (#33035)
  Move non duplicated actions back into xpack core (#32952)
  Test fix - GraphExploreResponseTests should not randomise array elements Closes #33086
  Use `addIfAbsent` instead of checking if an element is contained
  TESTS: Fix Random Fail in MockTcpTransportTests (#33061)
  HLRC: Fix Compile Error From Missing Throws (#33083)
  [DOCS] Remove reload password from docs cf. #32889
  HLRC: Add ML Get Buckets API (#33056)
  Watcher: Improve error messages for CronEvalTool (#32800)
  Search: Support of wildcard on docvalue_fields (#32980)
  Change query field expansion (#33020)
  INGEST: Cleanup Redundant Put Method (#33034)
  SQL: skip uppercasing/lowercasing function tests for AZ locales as well (#32910)
  Fix the default pom file name (#33063)
  Switch ml basic tests to new style Requests (#32483)
  Switch some watcher tests to new style Requests (#33044)
  ...
2018-08-24 12:22:11 +07:00
Gordon Brown 650f12af1e Duplicate Protocol classes into Core
This is needed as with recent changes to master (see #32952), protocol
is no longer accessible from core, so these classes need to be
duplicated in both places.
2018-08-23 13:50:15 -06:00
Gordon Brown 1f13c77b49 Merge branch 'master' into index-lifecycle 2018-08-23 11:52:59 -06:00
Colin Goodheart-Smithe fd88ab8c75
Fixes shrink action to remove single ndoe allocation (#33091)
This change fixes the shrink action so when the shrink is performed we
remove the single node allocation fromt eh shard allocation filtering
settings. Without this fix replicas cannot be allocated after we have
performed the shrink and we cannot make progress with the rest of the
shink aciton.

This change also fixes a bug in the explain API where the maste node
timeout was being set to null if it wasn't provided instead of using
its default value causing a NPE
2018-08-23 16:36:57 +01:00
Michael Basnight 644c0de5ec
Move non duplicated actions back into xpack core (#32952)
Most actions' request and response were moved from xpack core into
protocol. We have decided to instead duplicate the actions in the HLRC
instead of trying to reuse them. This commit moves the non duplicated
actions back into xpack core and severs the tie between xpack core and
protocol so no other actions can be moved and not duplicated.
2018-08-23 09:48:53 -05:00
Tal Levy dfc70ddcc7
rename pre-phase/pre-action to new/init (#32996)
this will keep things more consistent with the initial PhaseAfterStep, which has a phase name of `new`
2018-08-22 11:41:09 -07:00
Jason Tedor 528e727999
Fix method reference in comment in SchedulerEngine
This commit fixes the name of a method reference in a comment in
SchedulerEngine.
2018-08-22 10:20:20 -04:00
Jason Tedor 67bfb765ee
Refactor Netty4Utils#maybeDie (#33021)
In our Netty layer we have had to take extra precautions against Netty
catching throwables which prevents them from reaching the uncaught
exception handler. This code has taken on additional uses in NIO layer
and now in the scheduler engine because there are other components in
stack traces that could catch throwables and suppress them from reaching
the uncaught exception handler. This commit is a simple cleanup of the
iterative evolution of this code to refactor all uses into a single
method in ExceptionsHelper.
2018-08-22 10:18:07 -04:00
Tal Levy 55cb08a352
move ESLoggerFactory usage to LogManager (#33043)
Work done in #32513 has deprecated the old constructor in favor of
log4j2's LogManager
2018-08-21 17:59:32 -07:00
Nik Everett 2c81d7f77e
Build: Rework shadow plugin configuration (#32409)
This reworks how we configure the `shadow` plugin in the build. The major
change is that we no longer bundle dependencies in the `compile` configuration,
instead we bundle dependencies in the new `bundle` configuration. This feels
more right because it is a little more "opt in" rather than "opt out" and the
name of the `bundle` configuration is a little more obvious.

As an neat side effect of this, the `runtimeElements` configuration used when
one project depends on another now contains exactly the dependencies needed
to run the project so you no longer need to reference projects that use the
shadow plugin like this:

```
testCompile project(path: ':client:rest-high-level', configuration: 'shadow')
```

You can instead use the much more normal:

```
testCompile "org.elasticsearch.client:elasticsearch-rest-high-level-client:${version}"
```
2018-08-21 20:03:28 -04:00
markharwood 38bdf9ce32
HLRC GraphClient and associated tests (#32366)
GraphClient for the high level REST client and associated tests.
Part of #29827 work
2018-08-21 13:29:18 +01:00
Colin Goodheart-Smithe 10c60fae93
Merge branch 'master' into index-lifecycle 2018-08-21 11:54:06 +01:00
Tim Vernum b595b1a20c
Handle 6.4.0+ BWC for Application Privileges (#32929)
When the application privileges feature was backported to 6.x/6.4 the
BWC version checks on the backport were updated to 6.4.0, but master
was not updated.

This commit updates all relevant version checks, and adds tests.
2018-08-21 17:58:37 +10:00
Jason Tedor b08d02e3b7
Implement CCR licensing (#33002)
This commit implements licensing for CCR. CCR will require a platinum
license, and administrative endpoints will be disabled when a license is
non-compliant.
2018-08-20 23:33:18 -04:00
Jason Tedor 6d62d6755a
Fix typo in comment in scheduler engine
This commit fixes a minor typo in a big block comment in
SchedulerEngine.java.
2018-08-20 22:58:07 -04:00
Jason Tedor ad0a965db9
Protect scheduler engine against throwing listeners (#32998)
There are two problems with the scheduler engine today. Both relate to
listeners that throw.

The first problem is that any triggered listener that throws a plain old
exception will cause no additional listeners to be triggered for the
event, and will also cause the scheduler to never be invoked again. This
leads to lost events and is bad.

The second problem is that any triggered listener that throws an error
of the fatal kind will not lead to that error because caught by the
uncaught exception handler. This is because the triggered listener is
executed as a future task under a scheduled thread pool executor. A
throwable there goes caught by the JDK framework and set as the outcome
on the future task. Since we never inspect these tasks for their
outcomes, nor is there a good place to do this, we have to handle these
errors ourselves. To do this, we catch them and dispatch them to the
uncaught exception handler via a forked thread. This is similar to our
handling in Netty.
2018-08-20 22:07:16 -04:00
Jason Tedor 853eb1c51c
Merge branch 'master' into ccr
* master:
  Generalize remote license checker (#32971)
  Trim translog when safe commit advanced (#32967)
  Fix an inaccuracy in the dynamic templates documentation. (#32890)
  Logging: Use settings when building daemon threads (#32751)
  All Translog inner closes should happen after tragedy exception is set (#32674)
  HLREST: AwaitsFix ML Test
  Pass DiscoveryNode to initiateChannel (#32958)
  Add mzn and dz to unsupported locales (#32957)
  Use settings from the context in BootstrapChecks (#32908)
  Update docs for node specifications (#30468)
  HLRC: Forbid all Elasticsearch logging infra (#32784)
  Only configure publishing if it's applied externally (#32351)
  Fixes libs:dissect when in eclipse
  Protect ScriptedMetricIT test cases against failures on 0-doc shards (#32959) (#32968)
  [Kerberos] Add documentation for Kerberos realm (#32662)
  Watcher: Properly find next valid date in cron expressions (#32734)
  Fix some small issues in the getting started docs (#30346)
  Set forbidden APIs target compatibility to compiler java version   (#32935)
  Move connection listener to ConnectionManager (#32956)
2018-08-20 15:49:31 -04:00
Jason Tedor 9050c7e846
Generalize remote license checker (#32971)
Machine learning has baked a remote license checker for use in checking
license compatibility of a remote license. This remote license checker
has general usage for any feature that relies on a remote cluster. For
example, cross-cluster replication will pull changes from a remote
cluster and require that the local and remote clusters have platinum
licenses. This commit generalizes the remote cluster license check for
use in cross-cluster replication.
2018-08-20 15:33:29 -04:00
Nik Everett 462e91d362
Logging: Use settings when building daemon threads (#32751)
Subclasses of `EsIntegTestCase` run multiple Elasticsearch nodes in the
same JVM and when we log we look at the name of the thread to figure out
the node name. This makes sure that all calls to `daemonThreadFactory`
include the node name.

Closes #32574

I'd like to follow this up with more drastic changes that make it
impossible to do this incorrectly but that change is much larger than
this and I'd like to get these log lines fixed up sooner rather than
later.
2018-08-20 13:53:15 -04:00
Tal Levy 5ce082cd0a
copy LifecyclePolicy to protocol.xpack (#32915)
This is the final PR for copying over the necessary components for
clients to parse/render LifecyclePolicy. Changes include:

- move of named-x-content server objects away from client
- move validation into the client copy of LifecyclePolicy
- move LifecycleAction into an interface with `getName`
2018-08-20 08:32:22 -07:00
Jason Tedor ac75968c0b
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master: (46 commits)
  NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (#32764)
  [DOCS] Splits the users API documentation into multiple pages (#32825)
  [DOCS] Splits the token APIs into separate pages (#32865)
  [DOCS] Creates redirects for role management APIs page
  Bypassing failing test PainlessDomainSplitIT#testHRDSplit (#32966)
  TEST: Mute testRetentionPolicyChangeDuringRecovery
  [DOCS] Fixes more broken links to role management APIs
  [Docs] Tweaks and fixes to rollup docs
  [DOCS] Fixes links to role management APIs
  [ML][TEST] Fix BasicRenormalizationIT after adding multibucket feature
  [DOCS] Splits the roles API documentation into multiple pages (#32794)
  [TEST]  Run pre 6.4 nodes in non-FIPS JVMs (#32901)
  Make Geo Context Mapping Parsing More Strict (#32821)
  [ML] fix updating opened jobs scheduled events (#31651) (#32881)
  Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
  Tests: Fix timezone conversion in DateTimeUnitTests
  Enable FIPS140LicenseBootstrapCheck (#32903)
  Fix InternalAutoDateHistogram reproducible failure (#32723)
  Remove assertion in testDocStats on deletedDocs counter (#32914)
  HLRC: Move ML request converters into their own class (#32906)
  ...
2018-08-18 09:48:55 -04:00
Tal Levy a26e108590 Merge branch 'master' into index-lifecycle 2018-08-17 13:57:28 -07:00
Benjamin Trent 9cec4aa14b
[ML] fix updating opened jobs scheduled events (#31651) (#32881)
* ML: fix updating opened jobs scheduled events (#31651)

* Adding UpdateParamsTests license header

* Adding integration test and addressing PR comments

* addressing test and job names
2018-08-17 07:21:17 -05:00
Jay Modi 1136a95837
Security: remove put privilege API (#32879)
This commit removes the put privilege API in favor of having a single API to
create and update privileges. If we see the need to have an API like this in
the future we can always add it back.
2018-08-16 21:16:06 -06:00
Tal Levy 711ac402b6 Merge branch 'master' into index-lifecycle 2018-08-16 15:08:22 -07:00
Ed Savage 62559d2b3c
Re enable ml bwc tests (#32916)
[ML] Re-enabling BWC tests

Re-enable BWC tests for ML now that #32816 has been backported to 6.x
2018-08-16 18:23:26 +01:00
Tal Levy 62819826c7
copy more actions to protocol.xpack (#32892)
this PR creates client-side instances of the following actions:

- DeleteAction
- ForceMergeAction
- ReadOnlyAction
- RolloverAction
- ForceMergeAction

AllocateAction was done separately
2018-08-16 08:48:26 -07:00
Tal Levy c9de707f58 Merge branch 'master' into index-lifecycle 2018-08-16 08:41:57 -07:00
Tal Levy 238eb93893
migrate allocate action pojo/xcontent to xpack.protocol (#32853) 2018-08-16 08:41:07 -07:00
David Kyle 7f6802cb51 [ML] Choose seconds to fix intermittent DatafeeedConfigTest failure 2018-08-16 10:48:56 +01:00
Jim Ferenczi 996ed73d73 Test: Fix unpredictive merges in DocumentSubsetReaderTests
The merge policy that was used could lead to unpredictive merges due to the
randomization of `setDeletesPctAllowed`.

Closes #32457
2018-08-16 11:44:57 +02:00
Tal Levy 9406aa5af9
remove extraneous CreateIndexRequest context in ShrinkAction (#32893) 2018-08-15 21:55:18 -07:00
Tal Levy 4baa721459
remove `type` config from LifecyclePolicy JSON (#32660)
Since there is only one production policy, Timeseries, there
is no reason to expose the `type` argument to the user.
2018-08-15 14:47:22 -07:00
Tal Levy d1ab5dd650 fix merge with master 2018-08-15 14:46:31 -07:00
Jay Modi 1a45b27d8b
Move CharArrays to core lib (#32851)
This change cleans up some methods in the CharArrays class from x-pack, which
includes the unification of char[] to utf8 and utf8 to char[] conversions that
intentionally do not use strings. There was previously an implementation in
x-pack and in the reloading of secure settings. The method from the reloading
of secure settings was adopted as it handled more scenarios related to the
backing byte and char buffers that were used to perform the conversions. The
cleaned up class is moved into libs/core to allow it to be used by requests
that will be migrated to the high level rest client.

Relates #32332
2018-08-15 15:26:00 -06:00
Tal Levy ec93756600 Merge branch 'master' into index-lifecycle 2018-08-15 12:56:01 -07:00
Jason Tedor aa147cca44
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master:
  Revert "cluster formation DSL - Gradle integration -  part 2 (#32028)" (#32876)
  cluster formation DSL - Gradle integration -  part 2 (#32028)
  Introduce global checkpoint listeners (#32696)
  Move connection profile into connection manager (#32858)
  [ML] Temporarily disabling rolling-upgrade tests
  Use generic AcknowledgedResponse instead of extended classes (#32859)
  [ML] Removing old per-partition normalization code (#32816)
  Use JDK 10 for 6.4 BWC builds (#32866)
  Removed flaky test. Looks like randomisation makes these assertions unreliable.
  [test] mute IndexShardTests.testDocStats
  Introduce the dissect library (#32297)
  Security: remove password hash bootstrap check (#32440)
  Move validation to server for put user requests (#32471)
  [ML] Add high level REST client docs for ML put job endpoint (#32843)
  Test: Fix forbidden uses in test framework (#32824)
  Painless: Change fqn_only to no_import (#32817)
  [test] mute testSearchWithSignificantTermsAgg
  Watcher: Remove unused hipchat render method (#32211)
  Watcher: Remove extraneous auth classes (#32300)
  Watcher: migrate PagerDuty v1 events API to v2 API (#32285)
2018-08-15 12:30:35 -04:00
Lee Hinman 48281ac5bc
Use generic AcknowledgedResponse instead of extended classes (#32859)
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.

While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
2018-08-15 08:06:14 -06:00
Tal Levy 92ecd1d271 Merge branch 'master' into index-lifecycle 2018-08-15 06:11:25 -07:00
Ed Savage 8ce1ab3ed9
[ML] Removing old per-partition normalization code (#32816)
[ML] Removing old per-partition normalization code

Per-partition normalization is an old, undocumented feature that was
never used by clients. It has been superseded by per-partition maximum
scoring.

To maintain communication compatibility with nodes prior to 6.5 it is
necessary to maintain/cope with the old wire format
2018-08-15 13:13:32 +01:00
Tal Levy b218b1c68d
introduce random timeseries lifecycle policy util method (#32852)
It is useful to have a random TimeseriesLifecycleType-backed LifecyclePolicy
for testing. This PR exposes a helper method to create one and use it for serialization tests
in LifecyclePolicyTests
2018-08-14 12:57:43 -07:00
Jay Modi ac5ef8c389
Security: remove password hash bootstrap check (#32440)
This change removes the PasswordHashingBootstrapCheck and replaces it
with validation on the setting itself. This ensures we always get a
valid value from the setting when it is used.
2018-08-14 13:34:22 -06:00
Jay Modi 7d8a64d703
Move validation to server for put user requests (#32471)
This change moves the validation for values of usernames and passwords
from the request to the transport action. This is done to prevent
the need to move more classes into protocol once we add this API to the
high level rest client. Additionally, this resolves an issue where
validation depends on settings and we always pass empty settings
instead of the actual settings.

Relates #32332
2018-08-14 13:18:07 -06:00
Nhat Nguyen 6556186d9a Merge branch 'master' into ccr 2018-08-14 12:11:35 -04:00
Colin Goodheart-Smithe a84b3239c3
Merge branch 'master' into index-lifecycle
client/rest-high-level/src/main/java/org/elasticsearch/client/RequestCon
verters.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

client/rest-high-level/src/main/java/org/elasticsearch/client/LicenseCli
ent.java
client/rest-high-level/src/main/java/org/elasticsearch/client/RequestCon
verters.java
client/rest-high-level/src/test/java/org/elasticsearch/client/SearchIT.j
ava
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/LicensingDocumentationIT.java
docs/java-rest/high-level/licensing/delete-license.asciidoc
server/src/main/java/org/elasticsearch/action/bulk/BulkPrimaryExecutionC
ontext.java
server/src/main/java/org/elasticsearch/action/bulk/TransportBulkAction.j
ava
server/src/main/java/org/elasticsearch/common/Rounding.java
server/src/main/java/org/elasticsearch/common/rounding/Rounding.java
server/src/main/java/org/elasticsearch/search/aggregations/bucket/signif
icant/ParsedSignificantTerms.java
server/src/test/java/org/elasticsearch/action/IndicesRequestIT.java
server/src/test/java/org/elasticsearch/action/bulk/TransportBulkActionIn
gestTests.java
server/src/test/java/org/elasticsearch/action/bulk/TransportShardBulkAct
ionTests.java
server/src/test/java/org/elasticsearch/common/RoundingTests.java
server/src/test/java/org/elasticsearch/common/rounding/DateTimeUnitTests
.java
server/src/test/java/org/elasticsearch/common/rounding/RoundingDuelTests
.java
server/src/test/java/org/elasticsearch/search/aggregations/bucket/signif
icant/SignificantLongTermsTests.java
server/src/test/java/org/elasticsearch/search/aggregations/bucket/signif
icant/SignificantStringTermsTests.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/DeleteLicense
Action.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/DeleteLicense
Request.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/DeleteLicense
RequestBuilder.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/DeleteLicense
Response.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/LicenseServic
e.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/LicensingClie
nt.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/RestDeleteLic
enseAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/TransportDele
teLicenseAction.java
x-pack/plugin/core/src/test/java/org/elasticsearch/license/LicensesManag
erServiceTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/license/LicensesTrans
portTests.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/D
eleteLicenseRequest.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/D
eleteLicenseResponse.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/license/D
eleteLicenseResponseTests.java
2018-08-14 13:07:26 +01:00
Yannick Welsch 00b006f033
HLRC: Add Delete License API (#32586)
Relates to #29827
2018-08-14 11:55:54 +02:00
Tal Levy a771478940 Merge branch 'master' into index-lifecycle 2018-08-13 09:14:00 -07:00
Luca Cavanna 3e692c3f3d
HLRC: migration get assistance API (#32744)
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.

Relates to #29827
2018-08-13 11:05:27 +02:00
Tim Brooks 38ec0ff6ca
Make x-pack core pull transport-nio (#32757)
The security nio transports depend on transport-nio. This commit
modifies x-pack core to include the transport-nio jar into the x-pack
core module.
2018-08-10 17:36:29 -06:00
Colin Goodheart-Smithe 8750d622fc
Adds REST client support for starting and stopping ILM (#32609)
* Adds REST client support for PutOperationMode in ILM

* Corrects licence headers

* iter

* add request converter test

* Fixes tests

* Creates start and stop actions for controlling ILM operation

* Addresses review comments
2018-08-09 20:39:06 +01:00
Colin Goodheart-Smithe 2305cb8a6b
Fixes test after replicas removal 2018-08-09 17:45:29 +01:00
Colin Goodheart-Smithe 0fe21136db
Merge branch 'master' into index-lifecycle 2018-08-09 12:47:26 +01:00
Colin Goodheart-Smithe 5ff4f9347f
Adds explain lifecycle API to the Rest Client (#32606) 2018-08-09 10:18:45 +01:00
Tal Levy 2fc3f1d04c
move replicas action functionality into AllocateAction (#32523)
Since replica counts and allocation rules are set separately, it is not always clear how many replicas are to be allocated in the allocate action. Moving the replicas action to occur at the same time as the allocate action, resolves this confusion that could end an undesired state. This means that the ReplicasAction is removed, and a new optional replicas parameter is added to AllocateAction.
2018-08-08 11:43:29 -07:00
Jim Ferenczi f1869cca35
Fix role query that can match nested documents (#32705)
This change makes sure that the role query excludes nested documents
when it is ran to select the parent documents allowed by the role.
2018-08-08 16:36:51 +02:00
Tal Levy 2d925c9a9a Merge branch 'master' into index-lifecycle 2018-08-08 07:21:01 -07:00
Tanguy Leroux 2e65bac5dd
[Rollup] Remove builders from RollupJobConfig (#32669) 2018-08-07 18:54:42 +02:00
Colin Goodheart-Smithe b9c04adb29
Merge branch 'master' into index-lifecycle 2018-08-07 12:35:32 +01:00
Tanguy Leroux 1122314b3b
[Rollup] Remove builders from GroupConfig (#32614) 2018-08-07 09:39:24 +02:00
Tal Levy 0ad252d502
change default indices.lifecycle.poll_interval to something sane (#32521)
This was originally set to a few seconds while prototyping things.
This interval is for the scheduled trigger of policies. Policies
have this extra trigger beyond just on cluster-state changes because
cluster-state changes may not be happeneing in a cluster for
whatever reason, and we need to continue making progress. Updating
this value to be larger is reasonable since not all operations
are expected to be completed in the span of seconds, but instead in
minutes and hours. 10 minutes is sane.
2018-08-06 14:41:27 -07:00
Lee Hinman 0a9c3ae8bc
Remove UpdateSettingsTestHelper class (#32557)
* Remove UpdateSettingsTestHelper class

By making the `settings()` method public on `UpdateSettingsRequest` (I think it
should have been in the first place) we can get rid of this class entirely. Mock
response objects are now constructed by parsing JSON without making the
constructor public.

Relates to #29823
2018-08-06 08:53:44 -06:00
Lee Hinman 7ea7dd8018
Remove RolloverIndexTestHelper (#32559)
* Remove RolloverIndexTestHelper

This removes the `RolloverIndexTestHelper` class in favor of making a couple of
getters publically accessible as well as custom building a response object using
JSON parsing.

Relates to #29823
2018-08-06 08:53:28 -06:00
Lee Hinman aed466d5b6
Remove ILM constructor hacks (#32597)
This commit removes the hacks associated with mocking Response objects. Rather
than parse a wrapped byte array, the constructors for `IndicesAliasesResponse`
and `ResizeResponse` are made public

Relates to #29823
2018-08-06 08:53:12 -06:00
Nhat Nguyen 5881322b3f Merge branch 'master' into ccr
* master:
  Cross-cluster search: preserve cluster alias in shard failures (#32608)
  Handle AlreadyClosedException when bumping primary term
  [TEST] Allow to run in FIPS JVM (#32607)
  [Test] Add ckb to the list of unsupported languages (#32611)
  SCRIPTING: Move Aggregation Scripts to their own context (#32068)
  Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
  [TEST] Enhance failure message when bulk updates have failures
  [ML] Add ML result classes to protocol library (#32587)
  Suppress LicensingDocumentationIT.testPutLicense in release builds (#32613)
  [Rollup] Update wire version check after backport
  Suppress Wildfly test in FIPS JVMs (#32543)
  [Rollup] Improve ID scheme for rollup documents (#32558)
  ingest: doc: move Dot Expander Processor doc to correct position (#31743)
  [ML] Add some ML config classes to protocol library (#32502)
  [TEST]Split transport verification mode none tests (#32488)
  Core: Move helper date formatters over to java time (#32504)
  [Rollup] Remove builders from DateHistogramGroupConfig (#32555)
  [TEST} unmutes SearchAsyncActionTests and adds debugging info
  [ML] Add Detector config classes to protocol library (#32495)
  [Rollup] Remove builders from MetricConfig (#32536)
  Tests: Add rolling upgrade tests for watcher (#32428)
  Fix race between replica reset and primary promotion (#32442)
2018-08-06 10:27:18 -04:00
Igor Motov e641fccfe3
Rest HL client: Add get license action (#32438)
Rest HL client: Add get license action

Continues to use String instead of a more complex License class to
hold the license text similarly to put license.

Relates #29827
2018-08-06 07:15:40 -07:00
Lee Hinman 47a3e980b2 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-08-03 13:18:34 -06:00
Zachary Tong 992ec4be5d [Rollup] Update wire version check after backport
Bumping down the version to 6.4 since the backport is complete.  Also
adds some missing version checks to the bwc tests to make sure it
only runs on the correct versions
2018-08-03 14:09:01 -04:00
Tal Levy 78e94e1b00 Merge branch 'master' into index-lifecycle 2018-08-03 11:06:29 -07:00
Zachary Tong fc9fb64ad5
[Rollup] Improve ID scheme for rollup documents (#32558)
Previously, we were using a simple CRC32 for the IDs of rollup documents.
This is a very poor choice however, since 32bit IDs leads to collisions
between documents very quickly.

This commit moves Rollups over to a 128bit ID.  The ID is a concatenation
of all the keys in the document (similar to the rolling CRC before),
hashed with 128bit Murmur3, then base64 encoded.  Finally, the job
ID and a delimiter (`$`) are prepended to the ID.

This gurantees that there are 128bits per-job.  128bits should
essentially remove all chances of collisions, and the prepended
job ID means that _if_ there is a collision, it stays "within"
the job.

BWC notes:

We can only upgrade the ID scheme after we know there has been a good
checkpoint during indexing.  We don't rely on a STARTED/STOPPED
status since we can't guarantee that resulted from a real checkpoint,
or other state.  So we only upgrade the ID after we have reached
a checkpoint state during an active index run, and only after the
checkpoint has been confirmed.

Once a job has been upgraded and checkpointed, the version increments
and the new ID is used in the future.  All new jobs use the
new ID from the start
2018-08-03 11:13:25 -04:00
Ioannis Kakavas 1ee6393117
[TEST]Split transport verification mode none tests (#32488)
This commit splits SecurityNetty4TransportTests in two methods
one handling verification mode certificate and full and one
handling verification mode none. This is done so that the second
method can be muted in a FIPS 140 JVM where verification mode none
cannot be used.
2018-08-03 14:44:40 +03:00
Colin Goodheart-Smithe 20915a9baf
Merge branch 'master' into index-lifecycle
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLe
velClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
client/rest-high-level/src/test/java/org/elasticsearch/client/IndexLifec
ycleIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/RestHighLe
velClientTests.java
client/rest-high-level/src/test/java/org/elasticsearch/client/WatcherIT.
java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/LicensingDocumentationIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/WatcherDocumentationIT.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/Compiler.
java
modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/FunctionR
ef.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessE
xplainError.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/ScriptCla
ssInfo.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/Enh
ancedPainlessLexer.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/Pa
inlessLookup.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EExp
licit.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EFun
ctionRef.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EIns
tanceof.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELam
bda.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELis
tInit.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EMap
Init.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENew
Array.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENew
Obj.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ESta
tic.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PCal
lInvoke.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PFie
ld.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSub
ListShortcut.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSub
MapShortcut.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SCat
ch.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SDec
laration.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SEac
h.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SFun
ction.java
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SSub
EachIterable.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/OverloadT
ests.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/PainlessD
ocGenerator.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/RegexTest
s.java
modules/lang-painless/src/test/java/org/elasticsearch/painless/node/Node
ToStringTests.java
rest-api-spec/src/main/resources/rest-api-spec/test/cluster.stats/10_bas
ic.yml
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/Cluste
rStatsResponse.java
server/src/main/java/org/elasticsearch/action/admin/cluster/stats/Transp
ortClusterStatsAction.java
server/src/main/java/org/elasticsearch/action/bulk/TransportShardBulkAct
ion.java
server/src/main/java/org/elasticsearch/action/support/replication/Transp
ortReplicationAction.java
server/src/main/java/org/elasticsearch/index/engine/Engine.java
server/src/main/java/org/elasticsearch/index/engine/InternalEngine.java
server/src/main/java/org/elasticsearch/index/seqno/ReplicationTracker.ja
va
server/src/main/java/org/elasticsearch/index/shard/IndexShard.java
server/src/main/java/org/elasticsearch/index/shard/IndexShardOperationPe
rmits.java
server/src/main/java/org/elasticsearch/index/shard/PrimaryReplicaSyncer.
java
server/src/main/java/org/elasticsearch/index/shard/StoreRecovery.java
server/src/main/java/org/elasticsearch/index/translog/Translog.java
server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHa
ndler.java
server/src/main/java/org/elasticsearch/indices/recovery/RecoveryTarget.j
ava
server/src/test/java/org/elasticsearch/action/bulk/TransportShardBulkAct
ionTests.java
server/src/test/java/org/elasticsearch/action/search/SearchAsyncActionTe
sts.java
server/src/test/java/org/elasticsearch/action/support/replication/Transp
ortReplicationActionTests.java
server/src/test/java/org/elasticsearch/action/support/replication/Transp
ortWriteActionTests.java
server/src/test/java/org/elasticsearch/cluster/routing/allocation/ShardS
tateIT.java
server/src/test/java/org/elasticsearch/index/replication/IndexLevelRepli
cationTests.java
server/src/test/java/org/elasticsearch/index/replication/RecoveryDuringR
eplicationTests.java
server/src/test/java/org/elasticsearch/index/seqno/ReplicationTrackerTes
ts.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardOperationPe
rmitsTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexingOperationList
enerTests.java
server/src/test/java/org/elasticsearch/index/shard/PrimaryReplicaSyncerT
ests.java
server/src/test/java/org/elasticsearch/index/translog/TranslogTests.java
server/src/test/java/org/elasticsearch/indices/recovery/RecoverySourceHa
ndlerTests.java
server/src/test/java/org/elasticsearch/indices/recovery/RecoveryTests.ja
va
server/src/test/java/org/elasticsearch/search/profile/query/QueryProfile
rIT.java
test/framework/src/main/java/org/elasticsearch/index/replication/ESIndex
LevelReplicationTestCase.java
test/framework/src/main/java/org/elasticsearch/index/shard/IndexShardTes
tCase.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/job
/MetricConfig.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/job
/RollupJobConfig.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/S
ecurityContext.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/U
serSettings.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/AuthenticateResponse.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/ChangePasswordRequestBuilder.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/GetUsersResponse.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
ction/user/PutUserRequestBuilder.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/Authentication.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/AuthenticationResult.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthc/Realm.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/RoleDescriptor.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/accesscontrol/SecurityIndexSearcherWrapper.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/s
upport/MetadataUtils.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/AnonymousUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/BeatsSystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/ElasticUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/InternalUserSerializationHelper.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/KibanaUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/LogstashSystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/SystemUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/XPackSecurityUser.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/XPackUser.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/Con
figTestHelpers.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/job
/MetricsConfigSerializingTests.java ->
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/rollup/job
/MetricConfigSerializingTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/a
uthz/accesscontrol/SecurityIndexSearcherWrapperUnitTests.java
x-pack/plugin/monitoring/src/test/java/org/elasticsearch/xpack/monitorin
g/collector/cluster/ClusterStatsMonitoringDocTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/Rollup
JobIdentifierUtilTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/Rollup
RequestTranslationTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/action
/SearchActionTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/config
/ConfigTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/job/In
dexerUtilsTests.java
x-pack/plugin/rollup/src/test/java/org/elasticsearch/xpack/rollup/job/Ro
llupIndexerIndexingTests.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlLogoutAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportAuthenticateAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportGetUsersAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/ac
tion/user/TransportHasPrivilegesAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/AuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/AuditTrailService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrail.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/AuthenticationService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeUsersStore.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/ReservedRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/esnative/UserAndPassword.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/file/FileRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/file/FileUserPasswdStore.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/ldap/LdapRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/pki/PkiRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/saml/SamlRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thc/support/CachingUsernamePasswordRealm.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/AuthorizationService.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/AuthorizedIndices.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/in
gest/SetSecurityUserProcessor.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/RestAuthenticateAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/user/RestChangePasswordAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/re
st/action/user/RestGetUsersAction.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/tr
ansport/ServerTransportFilter.java
x-pack/plugin/security/src/test/java/org/elasticsearch/integration/Clear
RealmsCacheTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/Se
curityContextTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/filter/SecurityActionFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/interceptor/IndicesAliasesRequestInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/interceptor/ResizeRequestInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlInvalidateSessionActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/saml/TransportSamlLogoutActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportAuthenticateActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportChangePasswordActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportDeleteUserActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportGetUsersActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportHasPrivilegesActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportPutUserActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/ac
tion/user/TransportSetEnabledActionTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/AuditTrailServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrailMutedTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/index/IndexAuditTrailTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrailFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
dit/logfile/LoggingAuditTrailTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/AuthenticationServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/RealmsTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/TokenServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/UserTokenTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeRealmIntegTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/NativeUsersStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/esnative/ReservedRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/file/FileRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/file/FileUserPasswdStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmAuthenticateFailedTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmCacheTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmTestCase.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/kerberos/KerberosRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/ActiveDirectoryRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/CancellableLdapRunnableTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/ldap/LdapRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/pki/PkiRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/support/CachingUsernamePasswordRealmTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thc/support/mapper/NativeRoleMappingStoreTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizationServiceTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizationUtilsTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/AuthorizedIndicesTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/IndicesAndAliasesResolverTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/au
thz/SecuritySearchOperationListenerTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/in
gest/SetSecurityUserProcessorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/tr
ansport/SecurityServerTransportInterceptorTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/tr
ansport/ServerTransportFilterTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/AnonymousUserTests.java
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/UserTests.java ->
x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/us
er/UserSerializationTests.java
x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/expression/f
unction/scalar/string/StringFunctionUtils.java
x-pack/plugin/watcher/src/test/java/org/elasticsearch/xpack/watcher/exec
ution/ExecutionServiceTests.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DefaultDetectorDescription.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectionRule.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/Detector.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectorFunction.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/FilterRef.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/MlFilter.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/Operator.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleAction.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleCondition.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleScope.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/ml/packag
e-info.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/u
ser/User.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/security/
User.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectionRuleTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/DetectorTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/FilterRefTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/MlFilterTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleConditionTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/ml/job/co
nfig/RuleScopeTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/security/
UserTests.java
x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/Watch
erRestartIT.java
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/mixed_cl
uster/60_watcher.yml
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/old_clus
ter/60_watcher.yml
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/upgraded
_cluster/60_watcher.yml
x-pack/qa/security-example-spi-extension/src/main/java/org/elasticsearch
/example/realm/CustomRealm.java
x-pack/qa/security-example-spi-extension/src/test/java/org/elasticsearch
/example/realm/CustomRealmTests.java
x-pack/qa/security-migrate-tests/src/test/java/org/elasticsearch/xpack/s
ecurity/MigrateToolIT.java
2018-08-03 12:25:38 +01:00
Tanguy Leroux 21f660d801
[Rollup] Remove builders from DateHistogramGroupConfig (#32555)
Same motivation as #32507 but for the DateHistogramGroupConfig
configuration object. This pull request also changes the format of the
time zone from a Joda's DateTimeZone to a simple String.

It should help to port the API to the high level rest client and allows
clients to not be forced to use the Joda Time library. Serialization is
impacted but does not need a backward compatibility layer as
DateTimeZone are serialized as String anyway. XContent also expects
a String for timezone, so I found it easier to move everything to String.

Related to #29827
2018-08-03 13:11:00 +02:00
Tanguy Leroux 937dcfd716
[Rollup] Remove builders from MetricConfig (#32536)
Related to #29827
2018-08-03 10:01:20 +02:00
Nhat Nguyen 6eeb628d6d Merge branch 'master' into ccr
* master:
  HLRC: Move commercial clients from XPackClient (#32596)
  Add cluster UUID to Cluster Stats API response (#32206)
  Security: move User to protocol project (#32367)
  [TEST] Test for shard failures, add debug to testProfileMatchesRegular
  Minor fix for javadoc (applicable for java 11). (#32573)
  Painless: Move Some Lookup Logic to PainlessLookup (#32565)
  TEST: Avoid merges in testSeqNoAndCheckpoints
  [Rollup] Remove builders from HistoGroupConfig (#32533)
  Mutes failing SQL string function tests due to #32589
  fixed elements in array of produced terms (#32519)
  INGEST: Enable default pipelines (#32286)
  Remove cluster state initial customs (#32501)
  Mutes LicensingDocumentationIT due to #32580
  [ML] Remove multiple_bucket_spans (#32496)
  [ML] Rename JobProvider to JobResultsProvider (#32551)
  Correct minor typo in explain.asciidoc for HLRC
  Build: Add elastic maven to repos used by BuildPlugin (#32549)
  Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
  Revert "[test] turn on host io cache for opensuse (#32053)"
  Enable packaging tests on suse boxes
  [ML] Improve error when no available field exists for rule scope (#32550)
  [ML] Improve error for functions with limited rule condition support (#32548)
  Painless: Clean Up PainlessField (#32525)
  Add @AwaitsFix for #32554
  Remove broken @link in Javadoc
  Scripting: Conditionally use java time api in scripting (#31441)
  [ML] Fix thread leak when waiting for job flush (#32196) (#32541)
  Add AwaitsFix to failing test - see #32546
  Core: Minor size reduction for AbstractComponent (#32509)
  SQL: Added support for string manipulating functions with more than one parameter (#32356)
  [DOCS] Reloadable Secure Settings (#31713)
  Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
  [Rollup] Remove builders from TermsGroupConfig (#32507)
  Use hostname instead of IP with SPNEGO test (#32514)
  Switch x-pack rolling restart to new style Requests (#32339)
  NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
  [DOCS] Small fixes in rule configuration page (#32516)
  Painless: Clean up PainlessMethod (#32476)
  Build: Remove shadowing from benchmarks (#32475)
  Docs: Add all JDKs to CONTRIBUTING.md
  Add licensing enforcement for FIPS mode (#32437)
  SQL: Add test for handling of partial results (#32474)
  Mute testFilterCacheStats
  [ML][DOCS] Fix typo applied_to => applies_to
  Scripting: Fix painless compiler loader to know about context classes (#32385)
2018-08-02 23:14:37 -04:00
Jay Modi eb3accb721 Security: move User to protocol project (#32367)
The User class has been moved to the protocol project for upcoming work
to add more security APIs to the high level rest client. As part of
this change, the toString method no longer uses a custom output method
from MetadataUtils and instead just relies on Java's toString
implementation.
2018-08-03 00:21:34 +03:00
Tal Levy f7162588df Merge branch 'master' into index-lifecycle 2018-08-02 10:22:46 -07:00
Jason Tedor 5de236e1e7
Rename ILM, ILM endpoints and drop _xpack (#32564)
This commit does the following:
 - renames index-lifecycle plugin to ilm
 - modifies the endpoints to ilm instead of index_lifecycle
 - drops _xpack from the endpoints
 - drops a few duplicate endpoints
2018-08-02 13:05:11 -04:00
Tanguy Leroux 08e4f4be42
[Rollup] Remove builders from HistoGroupConfig (#32533)
Related to #29827
2018-08-02 17:55:00 +02:00
Lee Hinman 1bb7b53f26
Make index.lifecycle.name setting internal (#32518)
This commit makes the `index.lifecycle.name` setting internal an index, this
means that the policy can only be set on the index creation, or with the
specialized `RestSetIndexLifecyclePolicy` action.

Relates to #29823
2018-08-02 09:51:08 -06:00
Yannick Welsch db6e8c736d
Remove cluster state initial customs (#32501)
This infrastructure was introduced in #26144 and made obsolete in #30743
2018-08-02 15:49:59 +02:00
Dimitris Athanasiou f30bb0ebf8
[ML] Remove multiple_bucket_spans (#32496)
This commit removes the never released multiple_bucket_spans
configuration parameter. This is now replaced with the new
multibucket feature that requires no configuration.
2018-08-02 11:25:56 +01:00
Dimitris Athanasiou 8bf83647f5
[ML] Improve error when no available field exists for rule scope (#32550)
Closes #32542
2018-08-01 18:58:20 +01:00
Dimitris Athanasiou c5140170f7
[ML] Improve error for functions with limited rule condition support (#32548)
Closes #32545
2018-08-01 18:15:46 +01:00
Tal Levy ea1e26ca27 Merge branch 'master' into index-lifecycle 2018-08-01 10:15:18 -07:00
Tanguy Leroux 82fe67b225
[Rollup] Remove builders from TermsGroupConfig (#32507)
While working on adding the Create Rollup Job API to the 
high level REST client (#29827), I noticed that the configuration 
objects like TermsGroupConfig rely on the Builder pattern in 
order to create or parse instances. These builders are doing 
some validation but the same validation could be done within 
the constructor itself or on the server side when appropriate.

This commit removes the builder for TermsGroupConfig, 
removes some other methods that I consider not really usefull 
once the TermsGroupConfig object will be exposed in the 
high level REST client. It also simplifies the parsing logic.

Related to #29827
2018-08-01 09:43:32 +02:00
Armin Braun 4b199dde8d
NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
* Upgrade to `4.1.28` since the problem reported in #32487 is a bug in Netty itself (see https://github.com/netty/netty/issues/7337)
* Fixed other leaks in test code that now showed up due to fixes improvements in leak reporting in the newer version
* Needed to extend permissions for netty common package because it now sets a classloader at runtime after changes in 63bae0956a
* Adjusted forbidden APIs check accordingly
* Closes #32487
2018-08-01 02:34:58 +02:00
Tal Levy 9a0f6b2c38 remove mention of old maintenance mode setting 2018-07-31 15:34:34 -07:00
Tal Levy 73cdfaf07b Merge branch 'master' into index-lifecycle 2018-07-31 11:50:33 -07:00
Jay Modi 0788188574
Add licensing enforcement for FIPS mode (#32437)
This commit adds licensing enforcement for FIPS mode through the use of
a bootstrap check, a node join validator, and a check in the license
service. The work done here is based on the current implementation of
the TLS enforcement with a production license.

The bootstrap check is always enforced since we need to enforce the
licensing and this is the best option to do so at the present time.
2018-07-31 12:16:22 -06:00
Nhat Nguyen 036cb3f864 Merge branch 'master' into ccr
* master:
  Logging: Make node name consistent in logger (#31588)
  Mute SSLTrustRestrictionsTests on JDK 11
  Increase max chunk size to 256Mb for repo-azure (#32101)
  Docs: Fix README upgrade mention (#32313)
  Changed ReindexRequest to use Writeable.Reader (#32401)
  Mute KerberosAuthenticationIT
  Fix AutoIntervalDateHistogram.testReduce random failures (#32301)
  fix no=>not typo (#32463)
  Mute QueryProfilerIT#testProfileMatchesRegular()
  HLRC: Add delete watch action (#32337)
  High-level client: fix clusterAlias parsing in SearchHit (#32465)
  Fix calculation of orientation of polygons (#27967)
  [Kerberos] Add missing javadocs (#32469)
  [Kerberos] Remove Kerberos bootstrap checks (#32451)
  Make get all app privs requires "*" permission (#32460)
  Switch security to new style Requests (#32290)
  Switch security spi example to new style Requests (#32341)
  Painless: Add PainlessConstructor (#32447)
  update rollover to leverage write-alias semantics (#32216)
  Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
  INGEST: Clean up Java8 Stream Usage (#32059)
  Ensure KeyStoreWrapper decryption exceptions are handled (#32464)
2018-07-31 10:56:10 -04:00
Yannick Welsch 11f6983d69
HLRC: Add delete watch action (#32337)
Adds the "delete watch" API to the High-Level Rest Client.

Relates #29827
2018-07-31 10:29:22 +02:00
Tim Vernum d75efbcf68
Make get all app privs requires "*" permission (#32460)
The default behaviour for "GetPrivileges" is to get all application
privileges. This should only be allowed if the user has access to
the "*" application.
2018-07-31 09:07:47 +10:00
Lee Hinman a314efc920
Add high level rest client support for SetIndexLifecyclePolicy (#32443)
This adds HLRC support for the ILM operation of setting an index's lifecycle
policy.

It also includes extracting and renaming a number of classes (like the request
and response objects) as well as the addition of a new `IndexLifecycleClient`
for the HLRC. This is a prerequisite to making the `index.lifecycle.name`
setting internal only, because we require a dedicated REST endpoint to change
the policy, and our tests currently set this setting with the REST client
multiple places. A subsequent PR will change the setting to be internal and move
those uses over to this new API.

This misses some links to the documentation because I don't think ILM has any
documentation available yet.

Relates to #29827 and #29823
2018-07-30 16:57:25 -06:00
Lee Hinman dc97051dd5
Make various LifecycleSettings Settings internal (#32381)
These are only ever set internally during regular ILM execution, they don't need
to be set otherwise.

A subsequent PR will work on adding a dedicated endpoint for the
`LIFECYCLE_NAME` setting so it can be changed by a user (and then marked as
`InternalIndex` as well)

Relates to #29823
2018-07-30 14:23:04 -06:00
Nhat Nguyen 2245812ef7 Merge branch 'master' into ccr
* master:
  Tests: Fix convert error tests to use fixed value (#32415)
  IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374)
  REST high-level client: parse back _ignored meta field (#32362)
  [CI] Mute DocumentSubsetReaderTests testSearch
2018-07-30 14:02:58 -04:00
Colin Goodheart-Smithe 59bb37b5da
Merge branch 'master' into index-lifecycle 2018-07-30 11:48:54 +01:00
Colin Goodheart-Smithe 0682b4bbc9
Removes redundent NORELEASES and ObjectParserUtils (#32427) 2018-07-30 09:08:30 +01:00
David Roberts 14a4a740ac [CI] Mute DocumentSubsetReaderTests testSearch
Relates #32457
2018-07-30 09:04:17 +01:00
Nhat Nguyen 2f756b00f6 Merge branch 'master' into ccr
* master:
  Remove reference to non-existent store type (#32418)
  [TEST] Mute failing FlushIT test
  Fix ordering of bootstrap checks in docs (#32417)
  [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
  [TEST] Mute failing testConvertLongHexError
  bump lucene version after backport
  Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
  [Kerberos] Avoid vagrant update on precommit (#32416)
  TESTS: Move netty leak detection to paranoid level (#32354)
  [DOCS] Fixes formatting of scope object in job resource
  Copy missing segment attributes in getSegmentInfo (#32396)
  AbstractQueryTestCase should run without type less often (#28936)
  INGEST: Fix Deprecation Warning in Script Proc. (#32407)
  Switch x-pack/plugin to new style Requests (#32327)
  Docs: Correcting a typo in tophits (#32359)
  Build: Stop double generating buildSrc pom (#32408)
  TEST: Avoid triggering merges in FlushIT
  Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
  Switch x-pack full restart to new style Requests (#32294)
  Release requests in cors handler (#32364)
  Painless: Clean Up PainlessClass Variables (#32380)
  Docs: Fix callouts in put license HL REST docs (#32363)
  [ML] Consistent pattern for strict/lenient parser names (#32399)
  Update update-settings.asciidoc (#31378)
  Remove some dead code (#31993)
  Introduce index store plugins (#32375)
  Rank-Eval: Reduce scope of an unchecked supression
  Make sure _forcemerge respects `max_num_segments`. (#32291)
  TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
  Only enforce password hashing check if FIPS enabled (#32383)
2018-07-27 16:24:03 -04:00
Colin Goodheart-Smithe 783e427d36
Skips to next available action on missing step (#32283)
if policy update on index means current step no longer exists

This change only updates the setPolicy for index to add this
functionality. The update policy API will be changed in a follow up PR.
2018-07-27 13:40:10 +01:00
Colin Goodheart-Smithe b209f95626
Fix serialisation bug 2018-07-27 13:20:38 +01:00
Colin Goodheart-Smithe dfb277cfb7
Merge branch 'master' into index-lifecycle 2018-07-27 10:51:09 +01:00
David Roberts 0afa265ac9
[ML] Consistent pattern for strict/lenient parser names (#32399)
Previously we had two patterns for naming of strict
and lenient parsers.

Some classes had CONFIG_PARSER and METADATA_PARSER,
and used an enum to pass the parser type to nested
parsers.

Other classes had STRICT_PARSER and LENIENT_PARSER
and used ternary operators to pass the parser type
to nested parsers.

This change makes all ML classes use the second of
the patterns described above.
2018-07-26 16:55:40 +01:00
Nhat Nguyen caefed4d33 Merge branch 'master' into ccr
* master:
  [DOCS] Fix formatting error in Slack action
  Painless: Fix documentation links to use existing refs (#32335)
  Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
  [DOCS] Adds recommendation for xpack.security.enabled (#32345)
  [TEST] Mute ConvertProcessortTests.testConvertIntHexError
  [TEST] Fix failure due to exception message in java11 (#32321)
  [DOCS] Fixes typo in ML aggregations page
  [DOCS] Adds link from bucket_span property to common time units
  [ML][DOCS] Add documentation for detector rules and filters (#32013)
  Add opaque_id to index audit logging (#32260)
  Add 6.5.0 version to master
  fixes broken build for third-party-tests (#32353)
2018-07-25 20:12:13 -04:00
Lee Hinman 3daefe66af
Remove "best_compression" option from the ForceMergeAction (#32373)
This option is only settable while the index is closed, and doesn't make sense
for a force merge.

Relates to #29823
2018-07-25 14:11:14 -06:00
Tal Levy 86928e7d24 Merge branch 'master' into index-lifecycle 2018-07-25 09:01:35 -07:00
Igor Motov e12e2e0cdd
Add opaque_id to index audit logging (#32260)
Logs opaque_id if it is available with all audit log messages using
index-based audit log.

Closes #31521
2018-07-25 08:07:09 -07:00
Colin Goodheart-Smithe 81faa2fe6b
Merge branch 'master' into index-lifecycle 2018-07-25 10:39:14 +01:00
Nhat Nguyen ab4deefbe0 Merge branch 'master' into ccr
* master:
  Security: revert to old way of merging automata (#32254)
  Networking: Fix test leaking buffer (#32296)
  Undo a debugging change that snuck in during the field aliases merge.
  Painless: Update More Methods to New Naming Scheme (#32305)
  [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests
  Ingest: Support integer and long hex values in convert (#32213)
  Introduce fips_mode setting and associated checks (#32326)
  Add V_6_3_3 version constant
  [DOCS] Removed extraneous callout number.
  Rest HL client: Add put license action (#32214)
  Add ERR to ranking evaluation documentation (#32314)
  Introduce Application Privileges with support for Kibana RBAC (#32309)
  Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
  [Kerberos] Add Kerberos authentication support (#32263)
  [ML] Extract persistent task methods from MlMetadata (#32319)
  Add Restore Snapshot High Level REST API
  Register ERR metric with NamedXContentRegistry (#32320)
  fixes broken build for third-party-tests (#32315)
  Allow Integ Tests to run in a FIPS-140 JVM (#31989)
  [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280)
  awaitsfix testRandomClusterStateUpdates
  [TEST] add version skip to weighted_avg tests
  Consistent encoder names (#29492)
  Add WeightedAvg metric aggregation (#31037)
  Switch monitoring to new style Requests (#32255)
  Rename ranking evaluation `quality_level` to `metric_score` (#32168)
  Fix a test bug around nested aggregations and field aliases. (#32287)
  Add new permission for JDK11 to load JAAS libraries (#32132)
  Silence SSL reload test that fails on JDK 11
  [test] package pre-install java check (#32259)
  specify subdirs of lib, bin, modules in package (#32253)
  Switch x-pack:core to new style Requests (#32252)
  awaitsfix SSLConfigurationReloaderTests
  Painless: Clean up add methods in PainlessLookup (#32258)
  Fail shard if IndexShard#storeStats runs into an IOException (#32241)
  AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
  Remove unnecessary warning supressions (#32250)
  CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185)
  Add new fields to monitoring template for Beats state (#32085)
2018-07-24 21:33:50 -04:00
Jay Modi e43375bf9a
Security: revert to old way of merging automata (#32254)
This commit reverts to the pre-6.3 way of merging automata as the
change in 6.3 significantly impacts the performance for roles with a
large number of concrete indices. In addition, the maximum number of
states for security automata has been increased to 100,000 in order
to allow users to use roles that caused problems pre-6.3 and 6.3 fixed.

As an escape hatch, the maximum number of states is configurable with
a setting so that users with complex patterns in roles can increase
the states with the knowledge that there is more memory usage.
2018-07-24 16:26:50 -06:00
Tal Levy e523030670 Merge branch 'master' into index-lifecycle 2018-07-24 11:23:44 -07:00
Igor Motov 29c802f88e
Rest HL client: Add put license action (#32214)
In the HL REST client we replace the License object with a string, because of 
complexity of this class. It is also not really needed on the client side since 
end-users are not interacting with the license besides passing it as a string 
to the server.

Relates #29827
2018-07-24 11:19:07 -07:00
Tim Vernum 387c3c7f1d Introduce Application Privileges with support for Kibana RBAC (#32309)
This commit introduces "Application Privileges" to the X-Pack security
model.

Application Privileges are managed within Elasticsearch, and can be
tested with the _has_privileges API, but do not grant access to any
actions or resources within Elasticsearch. Their purpose is to allow
applications outside of Elasticsearch to represent and store their own
privileges model within Elasticsearch roles.

Access to manage application privileges is handled in a new way that
grants permission to specific application names only. This lays the
foundation for more OLS on cluster privileges, which is implemented by
allowing a cluster permission to inspect not just the action being
executed, but also the request to which the action is applied.
To support this, a "conditional cluster privilege" is introduced, which
is like the existing cluster privilege, except that it has a Predicate
over the request as well as over the action name.

Specifically, this adds
- GET/PUT/DELETE actions for defining application level privileges
- application privileges in role definitions
- application privileges in the has_privileges API
- changes to the cluster permission class to support checking of request
  objects
- a new "global" element on role definition to provide cluster object
  level security (only for manage application privileges)
- changes to `kibana_user`, `kibana_dashboard_only_user` and
  `kibana_system` roles to use and manage application privileges

Closes #29820
Closes #31559
2018-07-24 10:34:46 -06:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Yogesh Gaikwad a525c36c60 [Kerberos] Add Kerberos authentication support (#32263)
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
`WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide
a Kerberos ticket in the `Authorization` header. The tickets are
validated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.

In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
`WWW-Authenticate: Negotiate` header as its default and another could
provide `WWW-Authenticate: Basic` to indicate to the client that basic
authentication can be used in place of SPNEGO.

In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.

Closes #30243
2018-07-24 08:44:26 -06:00
David Kyle 99426eb4f8
[ML] Extract persistent task methods from MlMetadata (#32319)
Move ML persistent task helper functions to the new class MlTasks
and remove MLMetadataField after moving the string constant to 
MlMetadata.
2018-07-24 15:22:57 +01:00
Colin Goodheart-Smithe a2abc86595
Merge branch 'master' into index-lifecycle 2018-07-24 13:13:52 +01:00
Nik Everett 4804da06bf
Switch x-pack:core to new style Requests (#32252)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:core` project to use the new versions.
2018-07-23 14:07:18 -04:00
Andy Bristol d43c3a18b9 awaitsfix SSLConfigurationReloaderTests 2018-07-23 10:57:40 -07:00
Colin Goodheart-Smithe 5f696e15ea
Merge branch 'master' into index-lifecycle 2018-07-23 14:49:40 +01:00
Christoph Büscher ff87b7aba4
Remove unnecessary warning supressions (#32250) 2018-07-23 11:31:04 +02:00
Nicolas Ruflin 8f1d15ce61
Add new fields to monitoring template for Beats state (#32085)
New data is reported from Beats to the monitoring endpoint. This PR adds the template change necessary for it. See https://github.com/elastic/beats/issues/7521 for more details.

Queue data is skipped for now as implementation is not finished yet.
2018-07-23 07:38:19 +02:00
Martijn van Groningen 8ed3624f47
Merge remote-tracking branch 'es/master' into ccr
* es/master: (23 commits)
  Switch full-cluster-restart to new style Requests (#32140)
  [DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016)
  Remove BouncyCastle dependency from runtime (#32193)
  INGEST: Extend KV Processor (#31789) (#32232)
  INGEST: Make a few Processors callable by Painless (#32170)
  Add region ISO code to GeoIP Ingest plugin (#31669)
  [Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236)
  Make sure that field aliases count towards the total fields limit. (#32222)
  Switch rolling restart to new style Requests (#32147)
  muting failing test for internal auto date histogram to avoid failure before fix is merged
  MINOR: Remove unused `IndexDynamicSettings` (#32237)
  Fix multi level nested sort (#32204)
  Enhance Parent circuit breaker error message (#32056)
  [ML] Use default request durability for .ml-state index (#32233)
  Remove indices stats timeout from monitoring docs
  Rename ranking evaluation response section (#32166)
  Dependencies: Upgrade to joda time 2.10 (#32160)
  Remove aliases resolution limitations when security is enabled (#31952)
  Ensure that field aliases cannot be used in multi-fields. (#32219)
  TESTS: Check for Netty resource leaks (#31861)
  ...
2018-07-21 09:06:13 +02:00
Ioannis Kakavas aaa8f842d6
Remove BouncyCastle dependency from runtime (#32193)
* Remove BouncyCastle dependency from runtime

This commit introduces a new gradle  project that contains
 the classes that have a dependency on BouncyCastle. For 
the default distribution, It builds  a jar from those and
 in puts it in a subdirectory of lib
 (/tools/security-cli) along with the BouncyCastle jars. 
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools 
that use these classes.

BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
2018-07-21 00:03:58 +03:00
Colin Goodheart-Smithe 573a9e6443
Merge branch 'master' into index-lifecycle 2018-07-20 07:56:57 +01:00
Nhat Nguyen d0f3ed5abd Merge branch 'master' into ccr
* master:
  Painless: Simplify Naming in Lookup Package (#32177)
  Handle missing values in painless (#32207)
  add support for write index resolution when creating/updating documents (#31520)
  ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
  Remove indication of future multi-homing support (#32187)
  Rest test - allow for snapshots to take 0 milliseconds
  Make x-pack-core generate a pom file
  Rest HL client: Add put watch action (#32026)
  Build: Remove pom generation for plugin zip files (#32180)
  Fix comments causing errors with Java 11
  Fix rollup on date fields that don't support epoch_millis (#31890)
  Detect and prevent configuration that triggers a Gradle bug (#31912)
  [test] port linux package packaging tests (#31943)
  Revert "Introduce a Hashing Processor (#31087)" (#32178)
  Remove empty @return from JavaDoc
  Adjust SSLDriver behavior for JDK11 changes (#32145)
  [test] use randomized runner in packaging tests (#32109)
  Add support for field aliases. (#32172)
  Painless: Fix caching bug and clean up addPainlessClass. (#32142)
  Call setReferences() on custom referring tokenfilters in _analyze (#32157)
  Fix BwC Tests looking for UUID Pre 6.4 (#32158)
  Improve docs for search preferences (#32159)
  use before instead of onOrBefore
  Add more contexts to painless execute api (#30511)
  Add EC2 credential test for repository-s3 (#31918)
  A replica can be promoted and started in one cluster state update (#32042)
  Fix Java 11 javadoc compile problem
  Fix CP for namingConventions when gradle home has spaces (#31914)
  Fix `range` queries on `_type` field for singe type indices (#31756)
  [DOCS] Update TLS on Docker for 6.3 (#32114)
  ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
  Remove versionType from translog (#31945)
  Switch distribution to new style Requests (#30595)
  Build: Skip jar tests if jar disabled
  Painless: Add PainlessClassBuilder (#32141)
  Build: Make additional test deps of check (#32015)
  Disable C2 from using AVX-512 on JDK 10 (#32138)
  Build: Move shadow customizations into common code (#32014)
  Painless: Fix Bug with Duplicate PainlessClasses (#32110)
  Remove empty @param from Javadoc
  Re-disable packaging tests on suse boxes
  Docs: Fix missing example script quote (#32010)
  [ML] Wait for aliases in multi-node tests (#32086)
  [ML] Move analyzer dependencies out of categorization config (#32123)
  Ensure to release translog snapshot in primary-replica resync (#32045)
  Handle TokenizerFactory  TODOs (#32063)
  Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
  Updates the build to gradle 4.9 (#32087)
  Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
  Check that client methods match API defined in the REST spec (#31825)
  Enable testing in FIPS140 JVM (#31666)
  Fix put mappings java API documentation (#31955)
  Add exclusion option to `keep_types` token filter (#32012)
  [Test] Modify assert statement for ssl handshake (#32072)
2018-07-19 23:03:01 -04:00
Tim Vernum c32981db6b
Detect old trial licenses and mimic behaviour (#32209)
Prior to 6.3 a trial license default to security enabled. Since 6.3
they default to security disabled. If a cluster is upgraded from <6.3
to >6.3, then we detect this and mimic the old behaviour with respect
to security.
2018-07-20 10:09:28 +10:00
Tal Levy 1acdd37968 Merge branch 'master' into index-lifecycle 2018-07-19 12:33:10 -07:00
Ryan Ernst 9aa562d3dd Make x-pack-core generate a pom file
This was a forgotten part of #32180
2018-07-19 01:53:08 -07:00
Alexander Reelsen 202894b832
Rest HL client: Add put watch action (#32026)
Relates #29827

This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.

This commit also changes the license type to trial, so that watcher is available in high level rest client tests.

/cc @hub-cap
2018-07-19 10:40:54 +02:00
Colin Goodheart-Smithe 6c77449a87
Merge branch 'master' into index-lifecycle 2018-07-18 11:41:23 +01:00
David Roberts 99c2a82c04
[ML] Move analyzer dependencies out of categorization config (#32123)
The ML config classes will shortly be moved to the X-Pack protocol
library to allow the ML APIs to be moved to the high level REST
client.  Dependencies on server functionality should be removed
from the config classes before this is done.

This change is entirely about moving code between packages.  It
does not add or remove any functionality or tests.
2018-07-17 15:01:12 +01:00
Ioannis Kakavas 9e529d9d58
Enable testing in FIPS140 JVM (#31666)
Ensure our tests can run in a FIPS JVM

JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.

IT will be covered in a subsequent PR
2018-07-17 10:54:10 +03:00
Martijn van Groningen d0c9cf26a9
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Add Index UUID to `/_stats` Response (#31871)
  Painless: Move and Rename Several Methods in the lookup package (#32105)
  Bypass highlight query terms extraction on empty fields (#32090)
  Switch non-x-pack to new style requests (#32106)
  [Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
  Revert "[test] disable packaging tests for suse boxes"
  SQL: allow LEFT and RIGHT as function names (#32066)
  DOCS: put LIMIT 10 to the SQL query (#32065)
  [test] turn on host io cache for opensuse (#32053)
  Tweaked Elasticsearch Service links for SEO
2018-07-17 07:36:34 +02:00
Zachary Tong 791b9b147c
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint.

- Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern.
- One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
2018-07-16 17:20:50 -04:00
Martijn van Groningen f5e2168260
Merge remote-tracking branch 'es/master' into ccr
* es/master: (21 commits)
  Tweaked Elasticsearch Service links for SEO
  Watcher: Store username on watch execution (#31873)
  Use correct formatting for links (#29460)
  Painless: Separate PainlessLookup into PainlessLookup and PainlessLookupBuilder (#32054)
  Scripting: Remove dead code from painless module (#32064)
  [Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
  [TEST] Consistent algorithm usage (#32077)
  [Rollup] Fix duplicate field names in test (#32075)
  Ensure only parent breaker trips in unit test
  Unmute field collapsing rest tests
  Fix BWC check after backport
  [Tests] Fix failure due to changes exception message (#32036)
  Remove unused params from SSource and Walker (#31935)
  [Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
  Turn off real-mem breaker in REST tests
  Turn off real-mem breaker in single node tests
  Fix broken OpenLDAP Vagrant QA test
  Cleanup Duplication in `PainlessScriptEngine` (#31991)
  SCRIPTING: Remove unused MultiSearchTemplateRequestBuilder (#32049)
  Fix compile issues introduced by merge (#32058)
  ...
2018-07-16 21:49:57 +02:00
Michael Basnight 637cac9061
Watcher: Store username on watch execution (#31873)
There is currently no way to see what user executed a watch. This commit
adds the decrypted username to each execution in the watch history, in a
new field "user".

Closes #31772
2018-07-16 14:20:16 -05:00
Jim Ferenczi a3b608d616
[Rollup] Fix duplicate field names in test (#32075)
This commit ensures that random field names do not clash with the explicit field names
set by the tests.

Closes #32067
2018-07-16 15:25:45 +02:00
Colin Goodheart-Smithe 5d3a53843a
Merge branch 'master' into index-lifecycle 2018-07-16 09:30:07 +01:00
Nhat Nguyen e26f3e0c26 Merge branch 'master' into ccr 2018-07-13 16:55:57 -04:00
Zachary Tong 867c49600c
[Rollup] Histo group config should support scaled_floats (#32048)
Metric config already whitelist scaled_floats, but it wasn't added to
the histo group config.  This centralizes the mapping types map
so that both metrics and histo (and any future configs) use the same
map.

Fixes #32035
2018-07-13 16:10:39 -04:00
Ryan Ernst 2c3ea43f45
HLRC: Add xpack usage api (#31975)
This commit adds the _xpack/usage api to the high level rest client.
Currently in the transport api, the usage data is exposed in a limited
fashion, at most giving one level of helper methods for the inner keys
of data, but then exposing thos subobjects as maps of objects. Rather
than making parsers for every set of usage data from each feature, this
PR exposes the entire set of usage data as a map of maps.
2018-07-13 09:33:27 -07:00
Michael Basnight c1a81e552f
Watcher: cleanup ensureWatchExists use (#31926)
Previously, the ensureWatchExists was overridable. This commit makes
it final so that it cannot be overridden, and cleans up some redundant
code in the process.
2018-07-13 11:12:03 -05:00
Zachary Tong b7f07f03ed
[Rollup] Use composite's missing_bucket (#31402)
We can leverage the composite agg's new `missing_bucket` feature on
terms groupings.  This means the aggregation criteria used in the indexer
will now return null buckets for missing keys.  

Because all buckets are now returned (even if a key is null),
we can guarantee correct doc counts with
"combined" jobs (where a job rolls up multiple schemas).  This was
previously impossible since composite would ignore documents that
didn't have _all_ the keys, meaning non-overlapping schemas would
cause composite to return no buckets.

Note: date_histo does not use `missing_bucket`, since a timestamp is
always required.

The docs have been adjusted to recommend a single, combined job.  It
also makes reference to the previous issue to help users that are upgrading
(rather than just deleting the sections).
2018-07-13 10:07:42 -04:00
Tim Vernum c662565f29
Access SSL contexts using names instead of Settings (#30953)
Historically we have loaded SSL objects (such as SSLContext,
SSLIOSessionStrategy) by passing in the SSL settings, constructing a
new SSL configuration from those settings and then looking for a
cached object that matches those settings.

The primary issue with this approach is that it requires a fully
configured Settings object to be available any time the SSL context
needs to be loaded. If the Settings include SecureSettings (such as
passwords for keys or keystores) then this is not true, and the cached
SSL object cannot be loaded at runtime.

This commit introduces an alternative approach of naming every cached
ssl configuration, so that it is possible to load the SSL context for
a named configuration (such as "xpack.http.ssl"). This means that the
calling code does not need to have ongoing access to the secure
settings that were used to load the configuration.

This change also allows monitoring exporters to use SSL passwords
from secure settings, however an exporter that uses a secure SSL setting
(e.g. truststore.secure_password) may not have its SSL settings updated
dynamically (this is prevented by a settings validator).
Exporters without secure settings can continue to be defined and updated
dynamically.
2018-07-13 16:40:09 +10:00
Nhat Nguyen b1f5d361b3 Merge branch 'master' into ccr
* master:
  [TEST] Mute SlackMessageTests.testTemplateRender
  Docs: Explain closing the high level client
  [ML] Re-enable memory limit integration tests (#31328)
  [test] disable packaging tests for suse boxes
  Add nio transport to security plugin (#31942)
  XContentTests : Insert random fields at random positions (#30867)
  Force execution of fetch tasks (#31974)
  Fix unreachable error condition in AmazonS3Fixture (#32005)
  Tests: Fix SearchFieldsIT.testDocValueFields (#31995)
  Add Expected Reciprocal Rank metric (#31891)
  [ML] Get ForecastRequestStats doc in RestoreModelSnapshotIT (#31973)
  SQL: Add support for single parameter text manipulating functions (#31874)
  [ML] Ensure immutability of MlMetadata (#31957)
  Tests: Mute SearchFieldsIT.testDocValueFields()
  muted tests due to #31940
  Work around reported problem in eclipse (#31960)
  Move build integration tests out of :buildSrc project (#31961)
  Tests: Remove use of joda time in some tests (#31922)
  [Test] Reactive 3rd party tests on CI (#31919)
  SQL: Support for escape sequences (#31884)
  SQL: HAVING clause should accept only aggregates (#31872)
  Docs: fix typo in datehistogram (#31972)
  Switch url repository rest tests to new style requests (#31944)
  Switch reindex tests to new style requests (#31941)
  Docs: Added note about cloud service to installation and getting started
  [DOCS] Removes alternative docker pull example (#31934)
  Add Snapshots Status API to High Level Rest Client (#31515)
  ingest: date_index_name processor template resolution (#31841)
  Test: fix null failure in watcher test (#31968)
  Switch test framework to new style requests (#31939)
  Switch low level rest tests to new style Requests (#31938)
  Switch high level rest tests to new style requests (#31937)
  [ML] Mute test failing due to Java 11 date time format parsing bug (#31899)
  [TEST] Mute SlackMessageTests.testTemplateRender
  Fix assertIngestDocument wrongfully passing (#31913)
  Remove unused reference to filePermissionsCache (#31923)
  rolling upgrade should use a replica to prevent relocations while running a scroll
  HLREST: Bundle the x-pack protocol project (#31904)
  Increase logging level for testStressMaybeFlush
  Added lenient flag for synonym token filter (#31484)
  [X-Pack] Beats centralized management: security role + licensing (#30520)
  HLRest: Move xPackInfo() to xPack().info() (#31905)
  Docs: add security delete role to api call table (#31907)
  [test] port archive distribution packaging tests (#31314)
  Watcher: Slack message empty text (#31596)
  [ML] Mute failing DetectionRulesIT.testCondition() test
  Fix broken NaN check in MovingFunctions#stdDev() (#31888)
  Date: Add DateFormatters class that uses java.time (#31856)
  [ML] Switch native QA tests to a 3 node cluster (#31757)
  Change trappy float comparison (#31889)
  Fix building AD URL from domain name (#31849)
  Add opaque_id to audit logging (#31878)
  re-enable backcompat tests
  add support for is_write_index in put-alias body parsing (#31674)
  Improve release notes script (#31833)
  [DOCS] Fix broken link in painless example
  Handle missing values in painless (#30975)
  Remove the ability to index or query context suggestions without context (#31007)
  Ingest: Enable Templated Fieldnames in Rename (#31690)
  [Docs] Fix typo in the Rollup API Quick Reference (#31855)
  Ingest: Add ignore_missing option to RemoveProc (#31693)
  Add template config for Beat state to X-Pack Monitoring (#31809)
  Watcher: Add ssl.trust email account setting (#31684)
  Remove link to oss-MSI (#31844)
  Painless: Restructure Definition/Whitelist (#31879)
  HLREST: Add x-pack-info API (#31870)
2018-07-12 17:33:57 -04:00
Tim Brooks c375d5ab23
Add nio transport to security plugin (#31942)
This is related to #27260. It adds the SecurityNioTransport to the
security plugin. Additionally, it adds support for ip filtering. And it
randomly uses the nio transport in security integration tests.
2018-07-12 11:55:38 -06:00
Dimitris Athanasiou 2cfe703299
[ML] Ensure immutability of MlMetadata (#31957)
The test failure in #31916 revealed that updating
rules on a job was modifying the detectors list
in-place. That meant the old cluster state and the
updated cluster state had no difference and thus the
change was not propagated to non-master nodes.

This commit fixes that and also reviews all of ML
metadata in order to ensure immutability.

Closes #31916
2018-07-12 11:49:06 +01:00
Colin Goodheart-Smithe 89e8f9e240
Adds concept of a safe action (#31971)
* Adds concept of a safe action

A safe action is one that does not have unwanted side effects if the
configuration of the action is change in the policy while and index is
executing the action.

This commit formalises this concept with the only current unsafe action
being ShrinkAction. It also adds testing around this and add a method
to LifecyclePolicy which returns whether the action for the provided
StepKey is safe.

* Makes IndexLifecycleRunners checks use the safe indications instead of
hardcoding shrink action

* Fixes test failure
2018-07-12 11:40:38 +01:00
Colin Goodheart-Smithe 4192f387b4
Merge branch 'master' into index-lifecycle 2018-07-12 08:57:39 +01:00
Colin Goodheart-Smithe cd89ca2685
Merge branch 'master' into index-lifecycle
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
/Users/colings86/dev/work/git/elasticsearch/.git/worktrees/elasticsearch
-ilm/MERGE_HEAD

buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlu
gin.groovy
client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLe
velClient.java
client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClien
t.java
client/rest-high-level/src/test/java/org/elasticsearch/client/PingAndInf
oIT.java
client/rest-high-level/src/test/java/org/elasticsearch/client/documentat
ion/MiscellaneousDocumentationIT.java
docs/reference/analysis/tokenfilters/synonym-graph-tokenfilter.asciidoc
docs/reference/analysis/tokenfilters/synonym-tokenfilter.asciidoc
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Grok
ProcessorTests.java
modules/ingest-common/src/test/java/org/elasticsearch/ingest/common/Json
ProcessorTests.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/test/ArchiveTestCas
e.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Archives.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Cleanup.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/FileUtils.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Installation.j
ava
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Platforms.java
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/ServerUtils.ja
va
qa/vagrant/src/main/java/org/elasticsearch/packaging/util/Shell.java
qa/vagrant/src/test/resources/packaging/tests/20_tar_package.bats
server/src/main/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
r.java
server/src/main/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rser.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymGraphTokenF
ilterFactory.java
server/src/main/java/org/elasticsearch/index/analysis/SynonymTokenFilter
Factory.java
server/src/test/java/org/elasticsearch/index/analysis/ESSolrSynonymParse
rTests.java
server/src/test/java/org/elasticsearch/index/analysis/ESWordnetSynonymPa
rserTests.java
server/src/test/java/org/elasticsearch/index/shard/IndexShardIT.java
test/framework/src/main/java/org/elasticsearch/ingest/IngestDocumentMatc
her.java
test/framework/src/test/java/org/elasticsearch/ingest/IngestDocumentMatc
herTests.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/License.java
x-pack/plugin/core/src/main/java/org/elasticsearch/license/XPackLicenseS
tate.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClien
tPlugin.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackField
.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackSetti
ngs.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/beats/Beat
sFeatureSetUsage.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/permission/Role.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStore.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/action/Tra
nsportXPackInfoActionTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/security/a
uthz/store/ReservedRolesStoreTests.java
x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseChecker.java
x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/datafeed/MlRem
oteLicenseCheckerTests.java
x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/au
thz/store/CompositeRolesStore.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/XPackInfo
Response.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/License
Status.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/L
icenseStatus.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/license/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/license/p
ackage-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/security/packag
e-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/security/
package-info.java
x-pack/protocol/src/main/java/org/elasticsearch/protocol/watcher/package
-info.java ->
x-pack/protocol/src/main/java/org/elasticsearch/protocol/xpack/watcher/p
ackage-info.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/XPackInfo
ResponseTests.java
x-pack/protocol/src/test/java/org/elasticsearch/protocol/license/License
StatusTests.java ->
x-pack/protocol/src/test/java/org/elasticsearch/protocol/xpack/license/L
icenseStatusTests.java
x-pack/qa/rolling-upgrade/src/test/resources/rest-api-spec/test/mixed_cl
uster/10_basic.yml
2018-07-11 10:26:44 +01:00
David Kyle d268b494d7
[ML] Mute test failing due to Java 11 date time format parsing bug (#31899) 2018-07-11 10:17:44 +01:00
Christoph Büscher 4b8b831517
Remove unused reference to filePermissionsCache (#31923)
Currently Role.Builder keeps a reference to the FieldPermissionsCache that is
passed into its constructors. This seems to be unused except for passing it on
to convertFromIndicesPrivileges() in the second ctor itself, but we don't need
to keep the internal reference in that case, so it can be removed.

Relates to #31876
2018-07-11 09:56:21 +02:00
Shaunak Kashyap 3189ef49a5
[X-Pack] Beats centralized management: security role + licensing (#30520)
* Adding Beats x-pack plugin + index templates

* Adding built-in roles for Beats central management

* Fixing typo

* Refactoring: extract common code into method

* More refactoring for more code reuse

* Use a single index for Beats management

* Rename "fragment" to "block"

* Adding configuration block type

* Expand kibana_system role to include Beats management index privileges

* Fixing syntax

* Adding test

* Adding asserting for reserved role

* Fixing privileges

* Updating template

* Removing beats plugin

* Fixing tests

* Fixing role variable name

* Fixing assertions

* Switching to preferred syntax for boolean false checks

* Making class final

* Making variables final

* Updating Basic license message to be more accurate
2018-07-10 10:06:41 -07:00
Nik Everett dcbb1154bf
HLRest: Move xPackInfo() to xPack().info() (#31905)
Originally I put the X-Pack info object into the top level rest client
object. I did that because we thought we'd like to squash `xpack` from
the name of the X-Pack APIs now that it is part of the default
distribution. We still kind of want to do that, but at least for now we
feel like it is better to keep the high level rest client aligned with
the other language clients like C# and Python. This shifts the X-Pack
info API to align with its json spec file.

Relates to #31870
2018-07-10 13:01:28 -04:00
Tal Levy e9cd0556c1 Merge branch 'master' into index-lifecycle 2018-07-09 19:44:12 -04:00
Nicolas Ruflin c5c860dadb
Add template config for Beat state to X-Pack Monitoring (#31809)
With https://github.com/elastic/beats/pull/7075 Beats introduces state reporting for X-Pack Monitoring. The data sent up to Elasticsearch ends up stored in the following format.

```
"beats_state": {
  "timestamp": "2018-07-05T07:21:03.581Z",
  "state": {
    "module": {
      "count": 1,
      "names": [
        "http"
      ]
    }
  },
  "beat": {
    "uuid": "594039b5-6353-4d78-9bad-778ecc0fe83f",
    "type": "metricbeat",
    "version": "7.0.0-alpha1",
    "name": "ruflin",
    "host": "ruflin"
  }
}
```

This PR adds the new fields to the template.
2018-07-09 10:04:23 +02:00
Nik Everett fb27f3e7f0
HLREST: Add x-pack-info API (#31870)
This is the first x-pack API we're adding to the high level REST client
so there is a lot to talk about here!

= Open source

The *client* for these APIs is open source. We're taking the previously
Elastic licensed files used for the `Request` and `Response` objects and
relicensing them under the Apache 2 license.

The implementation of these features is staying under the Elastic
license. This lines up with how the rest of the Elasticsearch language
clients work.

= Location of the new files

We're moving all of the `Request` and `Response` objects that we're
relicensing to the `x-pack/protocol` directory. We're adding a copy of
the Apache 2 license to the root fo the `x-pack/protocol` directory to
line up with the language in the root `LICENSE.txt` file. All files in
this directory will have the Apache 2 license header as well. We don't
want there to be any confusion. Even though the files are under the
`x-pack` directory, they are Apache 2 licensed.

We chose this particular directory layout because it keeps the X-Pack
stuff together and easier to think about.

= Location of the API in the REST client

We've been following the layout of the rest-api-spec files for other
APIs and we plan to do this for the X-Pack APIs with one exception:
we're dropping the `xpack` from the name of most of the APIs. So
`xpack.graph.explore` will become `graph().explore()` and
`xpack.license.get` will become `license().get()`.

`xpack.info` and `xpack.usage` are special here though because they
don't belong to any proper category. For now I'm just calling
`xpack.info` `xPackInfo()` and intend to call usage `xPackUsage` though
I'm not convinced that this is the final name for them. But it does get
us started.

= Jars, jars everywhere!

This change makes the `xpack:protocol` project a `compile` scoped
dependency of the `x-pack:plugin:core` and `client:rest-high-level`
projects. I intend to keep it a compile scoped dependency of
`x-pack:plugin:core` but I intend to bundle the contents of the protocol
jar into the `client:rest-high-level` jar in a follow up. This change
has grown large enough at this point.

In that followup I'll address javadoc issues as well.

= Breaking-Java

This breaks that transport client by a few classes around. We've
traditionally been ok with doing this to the transport client.
2018-07-08 11:03:56 -04:00
Nhat Nguyen 6dd3434519 Merge branch 'master' into ccr
* master:
  REST high-level client: add get index API (#31703)
  SQL: Allow long literals (#31777)
  SQL: Fix incorrect message for aliases (#31792)
  Test: Do not remove xpack templates when cleaning (#31642)
  Reduce more raw types warnings (#31780)
  Add unreleased version 6.3.2
  Scripting: Remove support for deprecated StoredScript contexts (#31394)
  [ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
  [ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800)
  [ML] Fix calendar and filter updates from non-master nodes (#31804)
  Fix license header generation on Windows (#31790)
  mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix
  mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix
  Correct exclusion of test on JDK 11
  Fix doclint jdk 11
  Add JDK11 support and enable in CI (#31644)
  Watcher: Fix check for currently executed watches (#31137)
  Watcher: Ensure correct method is used to read secure settings (#31753)
  SQL: Update CLI logo
2018-07-05 14:07:06 -04:00
Dimitris Athanasiou 894fb97ad7
[ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
It seems that java 11 tightened some validations with regard to
time formats. The random instance creator was setting an odd
time format to the data description which is invalid when run
with java 11. This commit changes it to a valid format.
2018-07-05 13:56:54 +01:00
Alpar Torok cf2295b408
Add JDK11 support and enable in CI (#31644)
* Upgrade bouncycastle

Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11

* Downgrade bouncycastle to avoid invalid manifest

* Add checksum for new jars

* Update tika permissions for jdk 11

* Mute test failing on jdk 11

* Add JDK11 to CI

* Thread#stop(Throwable) was removed

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

* Disable failing tests #31456

* Temprorarily disable doc tests

To see if there are other failures on JDK11

* Only blacklist specific doc tests

* Disable only failing tests in ingest attachment plugin

* Mute failing HDFS tests #31498

* Mute failing lang-painless tests #31500

* Fix backwards compatability builds

Fix JAVA version to 10 for ES 6.3

* Add 6.x to bwx -> java10

* Prefix out and err from buildBwcVersion for readability

```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
  [bwc] :buildSrc:compileJava
  [bwc] WARNING: An illegal reflective access operation has occurred
  [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
  [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
  [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  [bwc] WARNING: All illegal access operations will be denied in a future release
  [bwc] :buildSrc:compileGroovy
  [bwc] :buildSrc:writeVersionProperties
  [bwc] :buildSrc:processResources
  [bwc] :buildSrc:classes
  [bwc] :buildSrc:jar

```

* Also set RUNTIME_JAVA_HOME for bwcBuild

So that we can make sure it's not too new for the build to understand.

* Align bouncycastle dependency

* fix painles array tets

closes #31500

* Update jar checksums

* Keep 8/10 runtime/compile untill consensus builds on 11

* Only skip failing tests if running on Java 11

* Failures are dependent of compile java version not runtime

* Condition doc test exceptions on compiler java version as well

* Disable hdfs tests based on runtime java

* Set runtime java to minimum supported for bwc

* PR review

* Add comment with ticket for forbidden apis
2018-07-05 03:24:01 +00:00
Nhat Nguyen f736205fd5 Merge branch 'master' into ccr
* master:
  [ML] Rate limit established model memory updates (#31768)
  [Docs] Correct default window_size (#31582)
  S3 fixture should report 404 on unknown bucket (#31782)
  Detach Transport from TransportService (#31727)
  [ML] Limit ML filter items to 10K (#31731)
  [ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
  Fixture for Minio testing (#31688)
  [DOCS] Add missing get mappings docs to HLRC (#31765)
  [DOCS] Starting Elasticsearch (#31701)
  Painless: Complete Removal of Painless Type (#31699)
  Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
  Consolidate watcher setting update registration (#31762)
  Build: re-enabled bwc (#31769)
  ingest: Introduction of a bytes processor (#31733)
  Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
  Add analyze API to high-level rest client (#31577)
  [DOCS] Typos
  DOC: Add examples to the SQL docs (#31633)
  Add support for AWS session tokens (#30414)
  Watcher: Reenable start/stop yaml tests (#31754)
  Implemented XContent serialisation for GetIndexResponse (#31675)
  JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
  resolveHasher defaults to NOOP (#31723)
  Account for XContent overhead in in-flight breaker
  Split CircuitBreaker-related tests (#31659)
  Add write*Blob option to replace existing blob (#31729)
  Painless: Add Context Docs (#31190)
  Watcher: Fix chain input toXcontent serialization (#31721)
  Docs: Match the examples in the description (#31710)
  rest-high-level: added get cluster settings (#31706)
  [Docs] Correct typos (#31720)
  Clean up double semicolon code typos (#31687)
  [DOCS] Check for Windows and *nix file paths (#31648)
  [ML] Validate ML filter_id (#31535)
  Revert long lines
  Fix TransportChangePasswordActionTests
2018-07-04 09:18:27 -04:00
Simon Willnauer 3f2a241b7f
Detach Transport from TransportService (#31727)
Today TransportService is tightly coupled with Transport since it
requires an instance of TransportService in order to receive responses
and send requests. This is mainly due to the Request and Response handlers
being maintained in TransportService but also because of the lack of a proper 
callback interface.

This change moves request handler registry and response handler registration into
Transport and adds all necessary methods to `TransportConnectionListener` in order
to remove the `TransportService` dependency from `Transport`
Transport now accepts one or more `TransportConnectionListener` instances that are
executed sequentially in a blocking fashion.
2018-07-04 11:32:35 +02:00
Dimitris Athanasiou 896317fe36
[ML] Limit ML filter items to 10K (#31731)
Add hard limit to the number of items
a filter may have. This serves to protect
from excessive overhead due to the filters
taking too much memory or lookups becoming
too expensive.
2018-07-04 10:25:00 +01:00
Hendrik Muhs e9f8442bee
[ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
This change adds stats about forecasts, to the jobstats api as well as xpack/_usage. The following 
information is collected:

_xpack/ml/anomaly_detectors/{jobid|_all}/_stats:

 -  total number of forecasts
 -  memory statistics (mean/min/max)
 -  runtime statistics
 -  record statistics
 -  counts by status

_xpack/usage

 -  collected by job status as well as overall (_all):
     -  total number of forecasts
     -  number of jobs that have at least 1 forecast
     -  memory, runtime, record statistics
     -  counts by status

Fixes #31395
2018-07-04 08:15:45 +02:00
Ioannis Kakavas 49b977ba7c
resolveHasher defaults to NOOP (#31723)
* Default resolveFromHash to Hasher.NOOP

This changes the default behavior when resolving the hashing
algorithm from unrecognised hash strings, which was introduced in
 #31234

A hash string that doesn't start with an algorithm identifier can
either be a malformed/corrupted hash or a plaintext password when
Hasher.NOOP is used(against warnings).
Do not make assumptions about which of the two is true for such
strings and default to Hasher.NOOP. Hash verification will subsequently
fail for malformed hashes.
Finally, do not log the potentially malformed hash as this can very
well be a plaintext password.

Resolves #31697
Reverts 58cf95a06f
2018-07-03 11:31:48 +03:00
Christoph Büscher 31aabe4bf9
Clean up double semicolon code typos (#31687) 2018-07-02 15:14:44 +02:00
Dimitris Athanasiou 8e838ea12e
[ML] Validate ML filter_id (#31535)
Like job and datafeed ids, the filter id
should be validated with the same rules
to avoid document ids that can be problematic.
2018-07-02 12:48:04 +01:00
Nhat Nguyen 52d9012d31 Merge branch 'master' into ccr
* master:
  Do not check for object existence when deleting repository index files (#31680)
  Remove extra check for object existence in repository-gcs read object (#31661)
  Support multiple system store types (#31650)
  [Test] Clean up some repository-s3 tests (#31601)
  [Docs] Use capital letters in section headings (#31678)
  [DOCS] Add PQL language Plugin (#31237)
  Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
  TEST: Fix test task invocation (#31657)
  Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
  Fix RealmInteg test failures
  Extend allowed characters for grok field names (#21745) (#31653)
  [DOCS] Fix licensing API details (#31667)
  [TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
  Fix CreateSnapshotRequestTests Failure (#31630)
  Configurable password hashing algorithm/cost (#31234)
  [TEST] Mute failing NamingConventionsTaskIT tests
  [DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
  Core: Require all actions have a Task (#31627)
2018-06-29 10:12:03 -04:00
Ioannis Kakavas 117e9066db
Support multiple system store types (#31650)
Support multiple system store types

When falling back to using the system keystore and - most usually -
truststore, do not assume that it will be a JKS store, but deduct
its type from {@code KeyStore#getDefaultKeyStoreType}. This allows
the use of any store type the Security Provider supports by setting
the keystore.type java security property.
2018-06-29 14:41:13 +03:00
Ioannis Kakavas db6b33978e
Configurable password hashing algorithm/cost (#31234)
Make password hashing algorithm/cost configurable for the 
stored passwords of users for the realms that this applies
(native, reserved). Replaces predefined choice of bcrypt with
cost factor 10.
This also introduces PBKDF2 with configurable cost
(number of iterations) as an algorithm option for password hashing
both for storing passwords and for the user cache.
Password hash validation algorithm selection takes into
consideration the stored hash prefix and only a specific number
of algorithnm and cost factor options for brypt and pbkdf2 are 
whitelisted and can be selected in the relevant setting.
2018-06-28 19:39:39 +03:00
Nhat Nguyen 2c10cd3e1d Merge branch 'master' into ccr
* master:
  Docs: Remove duplicate test setup
  Print output when the name checker IT fails (#31660)
  Fix syntax errors in get-snapshots docs (#31656)
  Docs: Fix description of percentile ranks example example (#31652)
  Add MultiSearchTemplate support to High Level Rest client (#30836)
  Add test for low-level client round-robin behaviour (#31616)
  SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
  Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
  Correct integTest enable logic (#31646)
  Fix missing get-snapshots docs reference #31645
  Do not check for Azure container existence (#31617)
  Merge AwsS3Service and InternalAwsS3Service in a S3Service class (#31580)
  Upgrade gradle wrapper to 4.8 (#31525)
  Only set vm.max_map_count if greater than default (#31512)
  Add Get Snapshots High Level REST API (#31537)
  QA: Merge query-builder-bwc to restart test (#30979)
  Update reindex.asciidoc (#31626)
  Docs: Skip xpack snippet tests if no xpack (#31619)
  mute CreateSnapshotRequestTests
  HLRest: Fix test for explain API
  [TEST] Fix RemoteClusterConnectionTests
  Add Create Snapshot to High-Level Rest Client (#31215)
  Remove legacy MetaDataStateFormat (#31603)
  Add explain API to high-level REST client (#31387)
  Preserve thread context when connecting to remote cluster (#31574)
  Unify headers for full text queries
  Remove redundant 'minimum_should_match'
  JDBC driver prepared statement set* methods  (#31494)
  [TEST] call yaml client close method from test suite (#31591)
2018-06-28 11:13:01 -04:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 (#31525)
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes #31324. The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes #31324.

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Nhat Nguyen a55f614b85 Merge branch 'master' into ccr
* master:
  ingest: Add ignore_missing property to foreach filter (#22147) (#31578)
  Fix a formatting issue in the docvalue_fields documentation. (#31563)
  reduce log level at gradle configuration time
  [TEST] Close additional clients created while running yaml tests (#31575)
  Docs: Clarify sensitive fields watcher encryption (#31551)
  Watcher: Remove never executed code (#31135)
  Add support for switching distribution for all integration tests (#30874)
  Improve robustness of geo shape parser for malformed shapes (#31449)
  QA: Create xpack yaml features (#31403)
  Improve test times for tests using `RandomObjects::addFields` (#31556)
  [Test] Add full cluster restart test for Rollup (#31533)
  Enhance thread context uniqueness assertion
  [DOCS] Fix heading format errors (#31483)
  fix writeIndex evaluation for aliases (#31562)
  Add x-opaque-id to search slow logs (#31539)
  Watcher: Fix put watch action (#31524)
  Add package pre-install check for java binary (#31343)
  Reduce number of raw types warnings (#31523)
  Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
  turn GetFieldMappingsResponse to ToXContentObject (#31544)
  Close xcontent parsers (partial) (#31513)
  Ingest Attachment: Upgrade Tika to 1.18 (#31252)
  TEST: Correct the assertion arguments order (#31540)
2018-06-26 16:25:32 -04:00
Nik Everett 232c71b6bf
QA: Create xpack yaml features (#31403)
This creates a YAML test "features" that indices if the cluster being
tested has xpack installed (`xpack`) or if it does *not* have xpack
installed (`no_xpack`). It uses those features to centralize skipping
a few tests that fail if xpack is installed.

The plan is to use this in a followup to skip docs tests that require
xpack when xpack is not installed. We *plan* to use the declaration
of required license level on the docs page to generate the required
`skip`.

Closes #30933.
2018-06-26 09:26:48 -04:00
Christoph Büscher 86ab3a2d1a
Reduce number of raw types warnings (#31523)
A first attempt to reduce the number of raw type warnings, 
most of the time by using the unbounded wildcard.
2018-06-25 15:59:03 +02:00
Ryan Ernst 49387bf2d7
Core: Combine doExecute methods in TransportAction (#31517)
TransportAction currently contains 2 doExecute methods, one which takes
a the task, and one that does not. The latter is what some subclasses
implement, while the first one just calls the latter, dropping the given
task. This commit combines these methods, in favor of just always
assuming a task is present.
2018-06-25 09:41:22 +01:00
Nhat Nguyen 08ee9b67c5 Merge branch 'master' into ccr
* master:
  Add get field mappings to High Level REST API Client (#31423)
  [DOCS] Updates Watcher examples for code testing (#31152)
  TEST: Add bwc recovery tests with synced-flush index
  [DOCS] Move sql to docs (#31474)
  [DOCS] Move monitoring to docs folder (#31477)
  Core: Combine doExecute methods in TransportAction (#31517)
  IndexShard should not return null stats (#31528)
  fix repository update with the same settings but different type (#31458)
  Fix Mockito trying to mock IOException that isn't thrown by method (#31433) (#31527)
  Node selector per client rather than per request (#31471)
  Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
  Upgrade to Lucene 7.4.0. (#31529)
  [ML] Add ML filter update API (#31437)
  Allow multiple unicast host providers (#31509)
  Avoid deprecation warning when running the ML datafeed extractor. (#31463)
  REST high-level client: add simulate pipeline API (#31158)
  Get Mapping API to honour allow_no_indices and ignore_unavailable (#31507)
  [PkiRealm] Invalidate cache on role mappings change (#31510)
  [Security] Check auth scheme case insensitively (#31490)
  In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. (#31514)
  [DOCS] Fix REST tests in SQL docs
  [DOCS] Add code snippet testing in more ML APIs (#31339)
  Core: Remove ThreadPool from base TransportAction (#31492)
  [DOCS] Remove fixed file from build.gradle
  Rename createNewTranslog to fileBasedRecovery (#31508)
  Test: Skip assertion on windows
  [DOCS] Creates field and document level security overview (#30937)
  [DOCS] Significantly improve SQL docs
  [DOCS] Move migration APIs to docs (#31473)
  Core: Convert TransportAction.execute uses to client calls (#31487)
  Return transport addresses from UnicastHostsProvider (#31426)
  Ensure local addresses aren't null (#31440)
  Remove unused generic type for client execute method (#31444)
  Introduce http and tcp server channels (#31446)
2018-06-23 17:27:17 -04:00
Ryan Ernst 7a150ec06d
Core: Combine doExecute methods in TransportAction (#31517)
TransportAction currently contains 2 doExecute methods, one which takes
a the task, and one that does not. The latter is what some subclasses
implement, while the first one just calls the latter, dropping the given
task. This commit combines these methods, in favor of just always
assuming a task is present.
2018-06-22 15:03:01 -07:00
Tal Levy 1d11fdaad8 Merge branch 'master' into index-lifecycle 2018-06-22 10:11:15 -07:00
Dimitris Athanasiou c6cbc99f9c
[ML] Add ML filter update API (#31437)
This adds an api to allow updating a filter:

POST _xpack/ml/filters/{filter_id}/_update

The request body may have:

- description: setting a new description
- add_items: a list of the items to add
- remove_items: a list of the items to remove

This commit also changes the PUT filter api to
error when the filter_id is already used. As
now there is an api for updating filters, the
put api should only be used to create new ones.

Also, updating a filter results into a notification
message auditing the change for every job that is
using that filter.
2018-06-22 15:13:31 +01:00
Colin Goodheart-Smithe d062556754
Merge branch 'master' into index-lifecycle 2018-06-22 10:00:35 +01:00
Yogesh Gaikwad 724438a0b0
[Security] Check auth scheme case insensitively (#31490)
According to RFC 7617, the Basic authentication scheme name
should not be case sensitive.
Case insensitive comparisons are also applicable for the bearer
tokens where Bearer authentication scheme is used as per
RFC 6750 and RFC 7235

Some Http clients may send authentication scheme names in
different case types for eg. Basic, basic, BASIC, BEARER etc.,
so the lack of case-insensitive check is an issue when these
clients try to authenticate with elasticsearch.

This commit adds case-insensitive checks for Basic and Bearer
authentication schemes.

Closes #31486
2018-06-22 10:15:38 +10:00
Ryan Ernst 4f9332ee16
Core: Remove ThreadPool from base TransportAction (#31492)
Most transport actions don't need the node ThreadPool. This commit
removes the ThreadPool as a super constructor parameter for
TransportAction. The actions that do need the thread pool then have a
member added to keep it from their own constructor.
2018-06-21 11:25:26 -07:00
Tal Levy c9eec7eadc Merge branch 'master' into index-lifecycle 2018-06-21 09:56:45 -07:00
Tal Levy 901aafb946
add APIs to Maintenance Mode in ILM (#31410)
- POST _xpack/index_lifecycle/_stop
   - issues a request to be placed into STOPPED mode (maintenance mode).
     This is not immediate, since we must first verify that
     it is safe to go from STOPPING -> STOPPED.
- POST _xpack/index_lifecycle/_start
   - issues a request to be placed back into RUNNING mode (immediately)
- GET _xpack/index_lifecycle/_status
   - get back the current mode our lifecycle management is in

- update task was hardened to support uninstalled metadata
- if no metadata is installed, the start/stop actions will install metadata
  and proceed to try and change it (default start mode is RUNNING)
- rename MAINTENANCE -> STOPPED, MAINTENANCE_REQUESTED -> STOPPING, NORMAL -> RUNNING

follow-up to #31164.
2018-06-21 09:56:02 -07:00
Tal Levy fc2f982542
inherit [index.lifecycle.date] from rolled-over time (#30853)
Indices that are rolled over preserve their origin index.lifecycle.date
as the time the index was created. Although this is also fine, it would
make more sense to start the timer for moving to the warm phase from the
time it was rolled over so that we capture the time elapsed from the
latest data, as opposed to when the index was created.

This commit adds an extra step within the RolloverAction that
extracts the index.creation_date of the newly created index from the RolloverStep
and sets that as the index.lifecycle.date of the rolled-over index that is
waiting for its next phase.
2018-06-21 07:18:14 -07:00
Colin Goodheart-Smithe dec6d49bd2
Adds an API to remove ILM from an index completely (#31358)
* Adds an API to remove ILM from an index completely

The removal will only happen if the index is not in the shrink action

* Fixes compile issues
2018-06-21 09:18:06 +01:00
Ryan Ernst 00283a61e1
Remove unused generic type for client execute method (#31444)
This commit removes the request builder generic type for AbstractClient
as it was unused.
2018-06-20 16:26:26 -07:00
Tim Brooks 9ab1325953
Introduce http and tcp server channels (#31446)
Historically in TcpTransport server channels were represented by the
same channel interface as socket channels. This was necessary as
TcpTransport was parameterized by the channel type. This commit
introduces TcpServerChannel and HttpServerChannel classes. Additionally,
it adds the implementations for the various transports. This allows
server channels to have unique functionality and not implement the
methods they do not support (such as send and getRemoteAddress).

Additionally, with the introduction of HttpServerChannel this commit
extracts some of the storing and closing channel work to the abstract
http server transport.
2018-06-20 16:34:56 -06:00
Nhat Nguyen efcb9a3603 Merge branch 'master' into ccr
* master:
  [DOCS] Omit shard failures assertion for incompatible responses  (#31430)
  [DOCS] Move licensing APIs to docs (#31445)
  Add Delete Snapshot High Level REST API
  Remove QueryCachingPolicy#ALWAYS_CACHE (#31451)
  [Docs] Extend Homebrew installation instructions (#28902)
  Choose JVM options ergonomically
  [Docs] Mention ip_range datatypes on ip type page (#31416)
  Multiplexing token filter (#31208)
  Fix use of time zone in date_histogram rewrite (#31407)
  Core: Remove index name resolver from base TransportAction (#31002)
  [DOCS] Fixes code snippet testing for machine learning (#31189)
  [DOCS] Removed  and  params from MLT. Closes #28128 (#31370)
  Security: fix joining cluster with production license (#31341)
  Unify http channels and exception handling (#31379)
  [DOCS] Moves the info API to docs (#31121)
  Preserve response headers on cluster update task (#31421)
  [DOCS] Add code snippet testing for more ML APIs (#31404)
  Do not preallocate bytes for channel buffer (#31400)
  Docs: Advice for reindexing many indices (#31279)
  Mute HttpExporterTests#testHttpExporterShutdown test Tracked by #31433
  Docs: Add note about removing prepareExecute from the java client (#31401)
  Make release notes ignore the `>test-failure` label. (#31309)
2018-06-20 12:43:40 -04:00
Colin Goodheart-Smithe c5b69c87c6
Merge branch 'master' into index-lifecycle 2018-06-20 10:31:45 +01:00
Ryan Ernst 401800d958
Core: Remove index name resolver from base TransportAction (#31002)
Most transport actions don't need to resolve index names. This commit
removes the index name resolver as a super constructor parameter for
TransportAction. The actions that do need the resolver then have a
member added to keep the resolver from their own constructor.
2018-06-19 17:06:09 -07:00
Tal Levy 01d456b55c fix actions to reflect changes in master 2018-06-19 15:01:15 -07:00
Tal Levy eee1a451fc Merge branch 'master' into index-lifecycle 2018-06-19 11:37:47 -07:00
Jay Modi dc57eece75
Security: fix joining cluster with production license (#31341)
The changes made to disable security for trial licenses unless security
is explicitly enabled caused issues when a 6.3 node attempts to join a
cluster that already has a production license installed. The new node
starts off with a trial license and `xpack.security.enabled` is not
set for the node, which causes the security code to skip attaching the
user to the request. The existing cluster has security enabled and the
lack of a user attached to the requests causes the request to be
rejected.

This commit changes the security code to check if the state has been
recovered yet when making the decision on whether or not to attach a
user. If the state has not yet been recovered, the code will attach
the user to the request in case security is enabled on the cluster
being joined.

Closes #31332
2018-06-19 11:58:34 -06:00
Tim Brooks 529e704b11
Unify http channels and exception handling (#31379)
This is a general cleanup of channels and exception handling in http.
This commit introduces a CloseableChannel that is a superclass of
TcpChannel and HttpChannel. This allows us to unify the closing logic
between tcp and http transports. Additionally, the normal http channels
are extracted to the abstract server transport.

Finally, this commit (mostly) unifies the exception handling between nio
and netty4 http server transports.
2018-06-19 11:50:03 -06:00
Nhat Nguyen ec04366d29 Merge branch 'master' into ccr
* master:
  Add get stored script and delete stored script to high level REST API - post backport fix
  Add get stored script and delete stored script to high level REST API (#31355)
  Core: Combine Action and GenericAction (#31405)
  Fix reference to XContentBuilder.string() (#31337)
  Avoid sending duplicate remote failed shard requests (#31313)
  Fix defaults in GeoShapeFieldMapper output (#31302)
  RestAPI: Reject forcemerge requests with a body (#30792)
  Packaging: Remove windows bin files from the tar distribution (#30596)
  Docs: Use the default distribution to test docs (#31251)
  [DOCS] Adds testing for security APIs (#31345)
  Clarify that IP range data can be specified in CIDR notation. (#31374)
  Use system context for cluster state update tasks (#31241)
  Percentile/Ranks should return null instead of NaN when empty (#30460)
  REST high-level client: add validate query API (#31077)
  Move language analyzers from server to analysis-common module. (#31300)
  [Test] Fix :example-plugins:rest-handler on Windows
  Expose lucene's RemoveDuplicatesTokenFilter (#31275)
  Reload secure settings for plugins (#31383)
  Remove some cases in FieldTypeLookupTests that are no longer relevant. (#31381)
  Ensure we don't use a remote profile if cluster name matches (#31331)
  [TEST] Double write alias fault (#30942)
  [DOCS] Fix version in SQL JDBC Maven template
  [DOCS] Improve install and setup section for SQL JDBC
  SQL: Fix rest endpoint names in node stats (#31371)
  Support for remote path in reindex api - post backport fix Closes #22913
  [ML] Put ML filter API response should contain the filter (#31362)
  Support for remote path in reindex api (#31290)
  Add byte array pooling to nio http transport (#31349)
  Remove trial status info from start trial doc (#31365)
  [DOCS] Adds links to release notes and highlights
  add is-write-index flag to aliases (#30942)
  Add rollover-creation-date setting to rolled over index (#31144)
  [ML] Hold ML filter items in sorted set (#31338)
  [Tests] Fix edge case in ScriptedMetricAggregatorTests (#31357)
2018-06-19 09:46:39 -04:00
Ryan Ernst e67aa96c81
Core: Combine Action and GenericAction (#31405)
Since #30966, Action no longer has anything but a call to the
GenericAction super constructor. This commit renames GenericAction
into Action, thus eliminating the Action class. Additionally, this
commit removes the Request generic parameter of the class, since
it was unused.
2018-06-18 23:53:04 +02:00
Tal Levy 5d52f297cc Merge branch 'master' into index-lifecycle 2018-06-18 10:06:47 -07:00
Yannick Welsch 02a4ef38a7
Use system context for cluster state update tasks (#31241)
This commit makes it so that cluster state update tasks always run under the system context, only
restoring the original context when the listener that was provided with the task is called. A notable
exception is the clusterStatePublished(...) callback which will still run under system context,
because it's defined on the executor-level, and not the task level, and only called once for the
combined batch of tasks and can therefore not be uniquely identified with a task / thread context.

Relates #30603
2018-06-18 16:46:04 +02:00
Dimitris Athanasiou c6a5a6d924
[ML] Put ML filter API response should contain the filter (#31362) 2018-06-15 21:15:35 +01:00
Tim Brooks a705e1a9e3
Add byte array pooling to nio http transport (#31349)
This is related to #28898. This PR implements pooling of bytes arrays
when reading from the wire in the http server transport. In order to do
this, we must integrate with netty reference counting. That manner in
which this PR implements this is making Pages in InboundChannelBuffer
reference counted. When we accessing the underlying page to pass to
netty, we retain the page. When netty releases its bytebuf, it releases
the underlying pages we have passed to it.
2018-06-15 14:01:03 -06:00