Commit Graph

945 Commits

Author SHA1 Message Date
Benjamin Trent f90f06ede1
HLRC: adding machine learning open job (#32860)
* HLRC: adding machine learning open job

HLRC: adding ML open job API call

* Addressing @droberts195 comments and fixing minor style issue
2018-08-15 13:19:41 -05:00
David Roberts 5ba04e23fc
[ML] Add log structure finder functionality (#32788)
This change adds a library to ML that can be used to deduce a log
file's structure given only a sample of the log file.

Eventually this will be used to add an endpoint to ML to make the
functionality available to end users, but this will follow in a
separate change.

The functionality is split into a library so that it can also be
used by a command line tool without requiring the command line
tool to include all server code.
2018-08-15 18:04:21 +01:00
Armin Braun 986c55b830
INGEST: Add Configuration Except. Data to Metdata (#32322)
* closes #27728
2018-08-15 19:02:19 +02:00
Jason Tedor aa147cca44
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master:
  Revert "cluster formation DSL - Gradle integration -  part 2 (#32028)" (#32876)
  cluster formation DSL - Gradle integration -  part 2 (#32028)
  Introduce global checkpoint listeners (#32696)
  Move connection profile into connection manager (#32858)
  [ML] Temporarily disabling rolling-upgrade tests
  Use generic AcknowledgedResponse instead of extended classes (#32859)
  [ML] Removing old per-partition normalization code (#32816)
  Use JDK 10 for 6.4 BWC builds (#32866)
  Removed flaky test. Looks like randomisation makes these assertions unreliable.
  [test] mute IndexShardTests.testDocStats
  Introduce the dissect library (#32297)
  Security: remove password hash bootstrap check (#32440)
  Move validation to server for put user requests (#32471)
  [ML] Add high level REST client docs for ML put job endpoint (#32843)
  Test: Fix forbidden uses in test framework (#32824)
  Painless: Change fqn_only to no_import (#32817)
  [test] mute testSearchWithSignificantTermsAgg
  Watcher: Remove unused hipchat render method (#32211)
  Watcher: Remove extraneous auth classes (#32300)
  Watcher: migrate PagerDuty v1 events API to v2 API (#32285)
2018-08-15 12:30:35 -04:00
Ed Savage 51cece1900 [ML] Temporarily disabling rolling-upgrade tests
BWC tests disabled while backporting https://github.com/elastic/elasticsearch/pull/32816
2018-08-15 15:48:50 +01:00
Lee Hinman 48281ac5bc
Use generic AcknowledgedResponse instead of extended classes (#32859)
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.

While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
2018-08-15 08:06:14 -06:00
Ed Savage 8ce1ab3ed9
[ML] Removing old per-partition normalization code (#32816)
[ML] Removing old per-partition normalization code

Per-partition normalization is an old, undocumented feature that was
never used by clients. It has been superseded by per-partition maximum
scoring.

To maintain communication compatibility with nodes prior to 6.5 it is
necessary to maintain/cope with the old wire format
2018-08-15 13:13:32 +01:00
Jay Modi ac5ef8c389
Security: remove password hash bootstrap check (#32440)
This change removes the PasswordHashingBootstrapCheck and replaces it
with validation on the setting itself. This ensures we always get a
valid value from the setting when it is used.
2018-08-14 13:34:22 -06:00
Jay Modi 7d8a64d703
Move validation to server for put user requests (#32471)
This change moves the validation for values of usernames and passwords
from the request to the transport action. This is done to prevent
the need to move more classes into protocol once we add this API to the
high level rest client. Additionally, this resolves an issue where
validation depends on settings and we always pass empty settings
instead of the actual settings.

Relates #32332
2018-08-14 13:18:07 -06:00
David Roberts c985f500f4
[ML] Add high level REST client docs for ML put job endpoint (#32843)
Relates #29827
Relates #32726
2018-08-14 19:52:54 +01:00
Nhat Nguyen 6556186d9a Merge branch 'master' into ccr 2018-08-14 12:11:35 -04:00
Michael Basnight 6fde9e5c2f
Watcher: Remove unused hipchat render method (#32211)
The HipChatMessage#render is no longer used, and instead the
HipChatAccount#render is used in the ExecutableHipChatAction. Only a
test that validated the HttpProxy used this render method still. This
commit cleans it up.
2018-08-14 11:07:58 -05:00
Michael Basnight e1206a6272
Watcher: Remove extraneous auth classes (#32300)
The auth.basic package was an example of a single implementation
interface that leaked into many different classes. In order to clean
this up, the HttpAuth interface, factories, and Registries all were
removed and the single implementation, BasicAuth, was substituted in all
cases. This removes some dependenies between Auth and the Templates,
which can now use static methods on BasicAuth. BasicAuth was also moved
into the http package and all of the other classes were removed.
2018-08-14 11:06:46 -05:00
Michael Basnight 4c90a61a35
Watcher: migrate PagerDuty v1 events API to v2 API (#32285)
The PagerDuty v1 API is EOL and will stop accepting new accounts
shortly. This commit swaps out the watcher use of the v1 API with the
new v2 API. It does not change anything about the existing watcher
API.

Closes #32243
2018-08-14 11:06:18 -05:00
Yannick Welsch 00b006f033
HLRC: Add Delete License API (#32586)
Relates to #29827
2018-08-14 11:55:54 +02:00
Nhat Nguyen cba7fceb53 AwaitsFix case-functions.sql-spec
Relates #32589
2018-08-13 15:06:33 -04:00
Ioannis Kakavas 06758f579c
Mute security-cli tests in FIPS JVM (#32812)
All Unit tests in this module are muted in FIPS 140 JVMs and
 as such the CI run fails. This commit disables test task for the
module in a  FIPS JVM and reverts adding a dummy test in 
4cbcc1.
2018-08-13 21:27:06 +03:00
Zachary Tong 4da0b52a19
[TEST] Force a stop to save rollup state before continuing (#32787)
We only upgrade the ID when the state is saved in one of four scenarios:

- when we reach a checkpoint (every 50 pages)
- when we run out of data
- when explicitly stopped
- on failure

The test was relying on the pre-upgrade to finish, save state and then
the post-upgrade to start, hit the end of data and upgrade ID.  THEN
get the new doc and apply the new ID.

But I think this is vulnerable to timing issues. If the pre-upgrade
portion shutdown before it saved the state, when restarting we would run
through all the data from the beginning with the old ID, meaning both
docs would still have the old scheme.

This change makes the pre-upgrade wait for the job to go back to STARTED
so that we know it persisted the end point.  Post-upgrade, it stops and
restarts the job to ensure the state was persisted and the ID upgraded.

That _should_ rule out the above timing issue.

Closes #32773
2018-08-13 12:53:22 -04:00
Dimitris Athanasiou 4d2144a2a0
[ML][DOCS] Refer to rules feature as custom rules (#32785) 2018-08-13 14:51:13 +01:00
Ed Savage d147cd72cc
[ML] Partition-wise maximum scores (#32748)
Added infrastructure to push through the 'person name field value' to
the normalizer process. This is required by the normalizer to retrieve
the maximum scores for individual partitions.
2018-08-13 10:31:17 +01:00
Luca Cavanna 3e692c3f3d
HLRC: migration get assistance API (#32744)
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.

Relates to #29827
2018-08-13 11:05:27 +02:00
Yogesh Gaikwad 8114646e12
[Kerberos] Add debug log statement for exceptions (#32663)
This commit adds missing debug log statements for exceptions
that occur during ticket validation. I thought these
get logged somewhere else in authentication chain
but even after enabling trace logs I could not see them
logged. As the Kerberos exception messages are cryptic
adding full stack trace would help debugging faster.
2018-08-12 00:49:08 +10:00
Tim Brooks 38ec0ff6ca
Make x-pack core pull transport-nio (#32757)
The security nio transports depend on transport-nio. This commit
modifies x-pack core to include the transport-nio jar into the x-pack
core module.
2018-08-10 17:36:29 -06:00
Benjamin Trent b08416b899
Clear Job#finished_time when it is opened (#32605) (#32755)
* Clear Job#finished_time when it is opened (#32605)

* not returning failure when Job#finished_time is not reset

* Changing error log string and source string
2018-08-10 13:52:00 -05:00
Nik Everett f5ba801c6b Test: Only sniff host metadata for node_selectors (#32750)
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
2018-08-10 13:35:47 -04:00
Albert Zaharovits 1dcf80795a
[TEST] Certificate NONE not allowed in FIPS JVM (#32753)
Certificate NONE not allowed when running in a FIPS JVM
2018-08-10 19:37:05 +03:00
Dimitris Athanasiou c7b1ba33aa
[ML] Refactor ProcessCtrl into Autodetect and Normalizer builders (#32720)
This moves the helper functionality for creating the autodetect
and mormalizer processes into corresponding builders.
2018-08-10 17:28:20 +01:00
Alexander Reelsen e49717393a
Tests: Disable rolling upgrade tests with system key on fips JVM (#32775)
This disables the x-pack rolling upgrade tests using a fips JVM, as
there are problems creating the keystore.

Relates #32737
2018-08-10 16:50:11 +02:00
Benjamin Trent 94a9b253db
Adding ML HLRC wrapper and put_job API call (#32726)
* Adding ML HLRC wrapper and put_job API call

* Changing integration test job to have consistent stucture
2018-08-10 07:16:55 -05:00
David Roberts ae0c303dad
Move icu4j and super-csv version numbers to versions file (#32769)
The upcoming ML log structure finder functionality will use these
libraries, and it makes sense to use the same versions that are
being used elsewhere in Elasticsearch.  This is especially true
with icu4j, which is pretty big.
2018-08-10 12:19:06 +01:00
Alexander Reelsen 3fa3af1d11 Tests: Mute test in KerberosRealmTests
Relates #32701
2018-08-10 10:40:37 +02:00
Tim Vernum 93d4f84b21
[TEST] Add "ne" as an unsupported SimpleKdc locale (#32700)
SimpleKdcServer cannot handle some locales, so we explicitly avoid
using them in tests.
Adding "ne" (Nepali) to that list.
2018-08-10 13:36:36 +10:00
Yogesh Gaikwad 0161bbc77f
[Test] Handle file permissions for Windows (#32681)
This commit modifies the test to handle file permission
tests in windows/dos environments. The test requires access
to UserPrincipal and so have modified the plugin-security policy
to access user information.

Closes#32637
2018-08-10 09:55:43 +10:00
Alpar Torok 5bbed5ed9a Merge branch 'master' of github.com:elastic/elasticsearch 2018-08-09 21:58:03 +03:00
Alpar Torok c5e66cb345 mute test #32737 2018-08-09 21:54:58 +03:00
Andrei Stefan 7b618f3db4
SQL: Bug fix for the optional "start" parameter usage inside LOCATE function (#32576)
The incorrect NodeInfo is created when the optional parameter is not used, leading to the incorrect constructor being used. Simplified LocateFunctionProcessorDefinition by using one constructor instead of two.
Fixes https://github.com/elastic/elasticsearch/issues/32554
2018-08-09 19:30:02 +03:00
Andrei Stefan 6750e15773
SQL: Ignore H2 comparative tests for uppercasing/lowercasing string functions (#32604)
Skip the comparative tests using lowercasing/uppercasing against H2 (which considers the Locale).
ES-SQL is, so far, ignoring the Locale.
Still, the same queries are executed against ES-SQL alone and results asserted to be correct.
2018-08-09 19:28:48 +03:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Lee Hinman 7af28c48c3
Switch WritePipelineResponse to AcknowledgedResponse (#32722)
We previously discussed moving the classes extending `AcknowledgedResponse` to
simply use `AcknowledgedResponse`, making the class non-abstract.

This moves the first class to do this, removing `WritePipelineResponse` in the
process.

If we like the way this looks, I will switch the remaining classes over to using
`AcknowledgedResponse`.
2018-08-08 16:21:58 -06:00
Benjamin Trent d586e4cfd3
Adding `Job` and `AnalysisConfig` for HLRC (#32687)
* Adding `Job` and `AnalysisConfig` for HLRC

* Removing println used for local debugging

* Adding null checks and removing unnecessary field
2018-08-08 11:54:11 -05:00
Jim Ferenczi f1869cca35
Fix role query that can match nested documents (#32705)
This change makes sure that the role query excludes nested documents
when it is ran to select the parent documents allowed by the role.
2018-08-08 16:36:51 +02:00
Armin Braun 580d59e2d7
CORE: Upgrade to Jackson 2.8.11 (#32670)
* closes #30352
2018-08-08 12:04:25 +02:00
Tim Vernum 4cbcc1d659 [TEST] Add empty test case for FIPS
All other tests in security:cli are skipped on FIPS JVMs
2018-08-08 13:47:18 +10:00
Tanguy Leroux 2e65bac5dd
[Rollup] Remove builders from RollupJobConfig (#32669) 2018-08-07 18:54:42 +02:00
Benjamin Trent 6d50d8b5a9
Adding job process pojos to protocol pkg (#32657)
* Adding job process pojos to protocol pkg

* Removing unused `RESULTS_FIELD`

* Addressing PR comments, removing unnecessary methods
2018-08-07 10:51:52 -05:00
Albert Zaharovits 1f50950099 Add @AwaitsFix for #32673 2018-08-07 13:22:12 +03:00
Tanguy Leroux 1122314b3b
[Rollup] Remove builders from GroupConfig (#32614) 2018-08-07 09:39:24 +02:00
Benjamin Trent b2a0f38a0c
Adding xpack.core.ml.datafeed to protocol.xpack.ml.datafeed (#32625)
* Adding org.elasticsearch.xpack.core.ml.datafeed to org.elasticsearch.protocol.xpack.ml.datafeed

* removing unused ParseField and import

* Addressing PR feed back and fixing tests

* Simplifying Datafeed(Config|Update) ctor parser
2018-08-06 15:33:18 -05:00
Nhat Nguyen 919888eba7 TEST: Enable debug log testValidateFollowingIndexSettings 2018-08-06 14:55:56 -04:00
Nhat Nguyen c394eb9ae9 CCR: Expose the operation primary term
Relates #32442
2018-08-06 10:55:37 -04:00
Nhat Nguyen 5881322b3f Merge branch 'master' into ccr
* master:
  Cross-cluster search: preserve cluster alias in shard failures (#32608)
  Handle AlreadyClosedException when bumping primary term
  [TEST] Allow to run in FIPS JVM (#32607)
  [Test] Add ckb to the list of unsupported languages (#32611)
  SCRIPTING: Move Aggregation Scripts to their own context (#32068)
  Painless: Use LocalMethod Map For Lookup at Runtime (#32599)
  [TEST] Enhance failure message when bulk updates have failures
  [ML] Add ML result classes to protocol library (#32587)
  Suppress LicensingDocumentationIT.testPutLicense in release builds (#32613)
  [Rollup] Update wire version check after backport
  Suppress Wildfly test in FIPS JVMs (#32543)
  [Rollup] Improve ID scheme for rollup documents (#32558)
  ingest: doc: move Dot Expander Processor doc to correct position (#31743)
  [ML] Add some ML config classes to protocol library (#32502)
  [TEST]Split transport verification mode none tests (#32488)
  Core: Move helper date formatters over to java time (#32504)
  [Rollup] Remove builders from DateHistogramGroupConfig (#32555)
  [TEST} unmutes SearchAsyncActionTests and adds debugging info
  [ML] Add Detector config classes to protocol library (#32495)
  [Rollup] Remove builders from MetricConfig (#32536)
  Tests: Add rolling upgrade tests for watcher (#32428)
  Fix race between replica reset and primary promotion (#32442)
2018-08-06 10:27:18 -04:00
Igor Motov e641fccfe3
Rest HL client: Add get license action (#32438)
Rest HL client: Add get license action

Continues to use String instead of a more complex License class to
hold the license text similarly to put license.

Relates #29827
2018-08-06 07:15:40 -07:00
Yogesh Gaikwad 615aa85f4e
[Kerberos] Use canonical host name (#32588)
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
2018-08-06 23:51:43 +10:00
Armin Braun 0a67cb4133
LOGGING: Upgrade to Log4J 2.11.1 (#32616)
* LOGGING: Upgrade to Log4J 2.11.1
* Upgrade to `2.11.1` to fix memory leaks in slow logger when logging large requests
   * This was caused by a bug in Log4J https://issues.apache.org/jira/browse/LOG4J2-2269 and is fixed in `2.11.1` via https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9496c0c
* Fixes #32537
* Fixes #27300
2018-08-06 14:56:21 +02:00
Jason Tedor 3b739b9fd5
Avoid NPE on shard changes action (#32630)
If a leader index is deleted while there is an active follower, the
follower will send shard changes requests bound for the leader
index. Today this will result in a null pointer exception because there
will not be an index routing table for the index. A null pointer
exception looks like a bug to a user so this commit addresses this by
throwing an index not found exception instead.
2018-08-06 08:01:47 -04:00
Ioannis Kakavas 66edba2012 [TEST] Allow to run in FIPS JVM (#32607)
* Change SecurityNioHttpServerTransportTests to use PEM key and
certificate files instead of a JKS keystore so that this tests
can also run in a FIPS 140 JVM
* Do not attempt to run cases with ssl.verification_mode NONE in
SessionFactoryTests so that the tests can run in a FIPS 140 JVM
2018-08-06 07:42:26 +03:00
Ioannis Kakavas ceb1ae4d7b [Test] Add ckb to the list of unsupported languages (#32611) 2018-08-06 10:00:45 +10:00
Jason Tedor 1a39f1d6c5
Fix CCR stats assertions
This commit addresses a race that can happen in the basic CCR stats REST
tests. Namely, peek reads can fire before the REST test client fires the
stats request. This means that we have to weaken our assertions about
the expected stats response.
2018-08-05 08:53:08 -04:00
David Roberts b99aa81fe4
[ML] Add ML result classes to protocol library (#32587)
This commit adds the ML results classes to the X-Pack protocol
library used by the high level REST client.

(Other commits will add the config classes and stats classes.)

These classes:

- Are publically immutable
- Are privately mutable - this is perhaps not as nice as the
  config classes, but to do otherwise would require adding
  builders and the corresponding server-side classes that the
  old transport client used don't have builders
- Have little/no validation of field values beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names and getter names as the corresponding
  classes in X-Pack core to ease migration for transport client
  users
- Don't reproduce all the methods that do calculations or
  transformations that the the corresponding classes in X-Pack core
  have
2018-08-03 20:48:38 +01:00
Zachary Tong 992ec4be5d [Rollup] Update wire version check after backport
Bumping down the version to 6.4 since the backport is complete.  Also
adds some missing version checks to the bwc tests to make sure it
only runs on the correct versions
2018-08-03 14:09:01 -04:00
Zachary Tong fc9fb64ad5
[Rollup] Improve ID scheme for rollup documents (#32558)
Previously, we were using a simple CRC32 for the IDs of rollup documents.
This is a very poor choice however, since 32bit IDs leads to collisions
between documents very quickly.

This commit moves Rollups over to a 128bit ID.  The ID is a concatenation
of all the keys in the document (similar to the rolling CRC before),
hashed with 128bit Murmur3, then base64 encoded.  Finally, the job
ID and a delimiter (`$`) are prepended to the ID.

This gurantees that there are 128bits per-job.  128bits should
essentially remove all chances of collisions, and the prepended
job ID means that _if_ there is a collision, it stays "within"
the job.

BWC notes:

We can only upgrade the ID scheme after we know there has been a good
checkpoint during indexing.  We don't rely on a STARTED/STOPPED
status since we can't guarantee that resulted from a real checkpoint,
or other state.  So we only upgrade the ID after we have reached
a checkpoint state during an active index run, and only after the
checkpoint has been confirmed.

Once a job has been upgraded and checkpointed, the version increments
and the new ID is used in the future.  All new jobs use the
new ID from the start
2018-08-03 11:13:25 -04:00
Jason Tedor 32c2759bb9
Remove extra blank line in CcrStatsAction.java
This commit removes an extra blank line that was accidentally committed
to CcrStatsAction.java.
2018-08-03 09:55:04 -04:00
Jason Tedor d640c9ddf9
Introduce CCR stats endpoint (#32350)
This commit introduces the CCR stats endpoint which provides shard-level
stats on the status of CCR follower tasks.
2018-08-03 09:09:45 -04:00
David Roberts bc274b2ff2
[ML] Add some ML config classes to protocol library (#32502)
This commit adds four ML config classes to the X-Pack protocol
library used by the high level REST client.

(Other commits will add the remaining config classes, plus results
and stats classes.)

These classes:

- Are immutable
- Have little/no validation of field values beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names, member names and getter/setter names
  as the corresponding classes in X-Pack core to ease migration
  for transport client users
- Don't reproduce all the methods that do calculations or
  transformations that the the corresponding classes in X-Pack core
  have
2018-08-03 13:21:08 +01:00
Ioannis Kakavas 1ee6393117
[TEST]Split transport verification mode none tests (#32488)
This commit splits SecurityNetty4TransportTests in two methods
one handling verification mode certificate and full and one
handling verification mode none. This is done so that the second
method can be muted in a FIPS 140 JVM where verification mode none
cannot be used.
2018-08-03 14:44:40 +03:00
Tanguy Leroux 21f660d801
[Rollup] Remove builders from DateHistogramGroupConfig (#32555)
Same motivation as #32507 but for the DateHistogramGroupConfig
configuration object. This pull request also changes the format of the
time zone from a Joda's DateTimeZone to a simple String.

It should help to port the API to the high level rest client and allows
clients to not be forced to use the Joda Time library. Serialization is
impacted but does not need a backward compatibility layer as
DateTimeZone are serialized as String anyway. XContent also expects
a String for timezone, so I found it easier to move everything to String.

Related to #29827
2018-08-03 13:11:00 +02:00
David Roberts eb17128b9c
[ML] Add Detector config classes to protocol library (#32495)
This commit adds the Detector class and its dependencies to the
X-Pack protocol library used by the high level REST client.

(Future commits will add the remaining config classes, plus results
and stats classes.)

These classes:

- Are immutable, with builders, but the builders do no validation
  beyond null checks
- Are convertible to and from X-Content, but NOT wire transportable
- Have lenient parsers to maximize compatibility across versions
- Have the same class names, member names and getter/setter names
  as the corresponding classes in X-Pack core to ease migration
  for transport client users
- Don't reproduce all the methods that do calculations or
  transformations that the the corresponding classes in X-Pack core
  have
2018-08-03 10:39:29 +01:00
Tanguy Leroux 937dcfd716
[Rollup] Remove builders from MetricConfig (#32536)
Related to #29827
2018-08-03 10:01:20 +02:00
Alexander Reelsen f809d6fff4
Tests: Add rolling upgrade tests for watcher (#32428)
These tests ensure, that the basic watch APIs are tested in the rolling
upgrade tests. After initially adding a watch, the tests try to get,
execute, deactivate and activate a watch. Watcher stats are tested as
well, and an own java based test has been added for restarting, as that
requires waiting for a state change. Watcher history is also checked.

Closes #31216
2018-08-03 09:41:29 +02:00
Nhat Nguyen 6eeb628d6d Merge branch 'master' into ccr
* master:
  HLRC: Move commercial clients from XPackClient (#32596)
  Add cluster UUID to Cluster Stats API response (#32206)
  Security: move User to protocol project (#32367)
  [TEST] Test for shard failures, add debug to testProfileMatchesRegular
  Minor fix for javadoc (applicable for java 11). (#32573)
  Painless: Move Some Lookup Logic to PainlessLookup (#32565)
  TEST: Avoid merges in testSeqNoAndCheckpoints
  [Rollup] Remove builders from HistoGroupConfig (#32533)
  Mutes failing SQL string function tests due to #32589
  fixed elements in array of produced terms (#32519)
  INGEST: Enable default pipelines (#32286)
  Remove cluster state initial customs (#32501)
  Mutes LicensingDocumentationIT due to #32580
  [ML] Remove multiple_bucket_spans (#32496)
  [ML] Rename JobProvider to JobResultsProvider (#32551)
  Correct minor typo in explain.asciidoc for HLRC
  Build: Add elastic maven to repos used by BuildPlugin (#32549)
  Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
  Revert "[test] turn on host io cache for opensuse (#32053)"
  Enable packaging tests on suse boxes
  [ML] Improve error when no available field exists for rule scope (#32550)
  [ML] Improve error for functions with limited rule condition support (#32548)
  Painless: Clean Up PainlessField (#32525)
  Add @AwaitsFix for #32554
  Remove broken @link in Javadoc
  Scripting: Conditionally use java time api in scripting (#31441)
  [ML] Fix thread leak when waiting for job flush (#32196) (#32541)
  Add AwaitsFix to failing test - see #32546
  Core: Minor size reduction for AbstractComponent (#32509)
  SQL: Added support for string manipulating functions with more than one parameter (#32356)
  [DOCS] Reloadable Secure Settings (#31713)
  Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
  [Rollup] Remove builders from TermsGroupConfig (#32507)
  Use hostname instead of IP with SPNEGO test (#32514)
  Switch x-pack rolling restart to new style Requests (#32339)
  NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
  [DOCS] Small fixes in rule configuration page (#32516)
  Painless: Clean up PainlessMethod (#32476)
  Build: Remove shadowing from benchmarks (#32475)
  Docs: Add all JDKs to CONTRIBUTING.md
  Add licensing enforcement for FIPS mode (#32437)
  SQL: Add test for handling of partial results (#32474)
  Mute testFilterCacheStats
  [ML][DOCS] Fix typo applied_to => applies_to
  Scripting: Fix painless compiler loader to know about context classes (#32385)
2018-08-02 23:14:37 -04:00
Jason Tedor 2387616c80
Remove _xpack from CCR APIs (#32563)
For a new feature like CCR we will go without this extra layer of
indirection. This commit replaces all /_xpack/ccr/_(\S+) endpoints by
/_ccr/$1 endpoints.
2018-08-02 20:21:43 -04:00
Shaunak Kashyap 0a83968650
Add cluster UUID to Cluster Stats API response (#32206)
* Make cluster stats response contain cluster UUID

* Updating constructor usage in Monitoring tests

* Adding cluster_uuid field to Cluster Stats API reference doc

* Adding rest api spec test for expecting cluster_uuid in cluster stats response

* Adding missing newline

* Indenting do section properly

* Missed a spot!

* Fixing the test cluster ID
2018-08-02 17:14:19 -07:00
Jay Modi eb3accb721 Security: move User to protocol project (#32367)
The User class has been moved to the protocol project for upcoming work
to add more security APIs to the high level rest client. As part of
this change, the toString method no longer uses a custom output method
from MetadataUtils and instead just relies on Java's toString
implementation.
2018-08-03 00:21:34 +03:00
Andrei Stefan 9e1e38ff51
Minor fix for javadoc (applicable for java 11). (#32573) 2018-08-02 23:46:08 +03:00
Tanguy Leroux 08e4f4be42
[Rollup] Remove builders from HistoGroupConfig (#32533)
Related to #29827
2018-08-02 17:55:00 +02:00
Colin Goodheart-Smithe 4cdbb42542
Mutes failing SQL string function tests due to #32589 2018-08-02 16:15:59 +01:00
Yannick Welsch db6e8c736d
Remove cluster state initial customs (#32501)
This infrastructure was introduced in #26144 and made obsolete in #30743
2018-08-02 15:49:59 +02:00
Dimitris Athanasiou f30bb0ebf8
[ML] Remove multiple_bucket_spans (#32496)
This commit removes the never released multiple_bucket_spans
configuration parameter. This is now replaced with the new
multibucket feature that requires no configuration.
2018-08-02 11:25:56 +01:00
David Kyle 15679315e3
[ML] Rename JobProvider to JobResultsProvider (#32551) 2018-08-02 09:53:47 +01:00
Dimitris Athanasiou 8bf83647f5
[ML] Improve error when no available field exists for rule scope (#32550)
Closes #32542
2018-08-01 18:58:20 +01:00
Dimitris Athanasiou c5140170f7
[ML] Improve error for functions with limited rule condition support (#32548)
Closes #32545
2018-08-01 18:15:46 +01:00
David Turner 2db420a81d Add @AwaitsFix for #32554 2018-08-01 17:26:45 +01:00
David Turner 621350c1b9 Remove broken @link in Javadoc 2018-08-01 17:12:55 +01:00
Benjamin Trent 9fb790dcc3
[ML] Fix thread leak when waiting for job flush (#32196) (#32541) 2018-08-01 10:38:04 -05:00
Andrei Stefan 4c388539a0
SQL: Added support for string manipulating functions with more than one parameter (#32356)
Added support for string manipulating functions with more than one parameter:
CONCAT, LEFT, RIGHT, REPEAT, POSITION, LOCATE, REPLACE, SUBSTRING, INSERT
2018-08-01 12:29:06 +03:00
Alexander Reelsen 7d7cddc6ed
Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
The error message mentioned in #30094 does not link to to a cause by the
test itself, as there are still inflight requests according to the
circuit breaker.

I ran this test class 100k times on bare metal and could not reproduce
it. I will reenable the test for now.

Closes #30094
2018-08-01 10:46:02 +02:00
Tanguy Leroux 82fe67b225
[Rollup] Remove builders from TermsGroupConfig (#32507)
While working on adding the Create Rollup Job API to the 
high level REST client (#29827), I noticed that the configuration 
objects like TermsGroupConfig rely on the Builder pattern in 
order to create or parse instances. These builders are doing 
some validation but the same validation could be done within 
the constructor itself or on the server side when appropriate.

This commit removes the builder for TermsGroupConfig, 
removes some other methods that I consider not really usefull 
once the TermsGroupConfig object will be exposed in the 
high level REST client. It also simplifies the parsing logic.

Related to #29827
2018-08-01 09:43:32 +02:00
Jay Modi f2f33f3149 Use hostname instead of IP with SPNEGO test (#32514)
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes #32498
2018-08-01 12:57:33 +10:00
Nik Everett 99d9a0a40c
Switch x-pack rolling restart to new style Requests (#32339)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:rolling-upgrade*` projects to use
the new versions.
2018-07-31 21:04:33 -04:00
Armin Braun 4b199dde8d
NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
* Upgrade to `4.1.28` since the problem reported in #32487 is a bug in Netty itself (see https://github.com/netty/netty/issues/7337)
* Fixed other leaks in test code that now showed up due to fixes improvements in leak reporting in the newer version
* Needed to extend permissions for netty common package because it now sets a classloader at runtime after changes in 63bae0956a
* Adjusted forbidden APIs check accordingly
* Closes #32487
2018-08-01 02:34:58 +02:00
Lisa Cawley cc6d6cae7c
[DOCS] Small fixes in rule configuration page (#32516) 2018-07-31 16:28:09 -07:00
Jay Modi 0788188574
Add licensing enforcement for FIPS mode (#32437)
This commit adds licensing enforcement for FIPS mode through the use of
a bootstrap check, a node join validator, and a check in the license
service. The work done here is based on the current implementation of
the TLS enforcement with a production license.

The bootstrap check is always enforced since we need to enforce the
licensing and this is the best option to do so at the present time.
2018-07-31 12:16:22 -06:00
Igor Motov 5fd7202808
SQL: Add test for handling of partial results (#32474)
Verifies that partial results are rejected by SQL requests.

Closes #32284
2018-07-31 11:06:00 -07:00
Dimitris Athanasiou b88b3d81c8 [ML][DOCS] Fix typo applied_to => applies_to 2018-07-31 17:00:55 +01:00
Nhat Nguyen 036cb3f864 Merge branch 'master' into ccr
* master:
  Logging: Make node name consistent in logger (#31588)
  Mute SSLTrustRestrictionsTests on JDK 11
  Increase max chunk size to 256Mb for repo-azure (#32101)
  Docs: Fix README upgrade mention (#32313)
  Changed ReindexRequest to use Writeable.Reader (#32401)
  Mute KerberosAuthenticationIT
  Fix AutoIntervalDateHistogram.testReduce random failures (#32301)
  fix no=>not typo (#32463)
  Mute QueryProfilerIT#testProfileMatchesRegular()
  HLRC: Add delete watch action (#32337)
  High-level client: fix clusterAlias parsing in SearchHit (#32465)
  Fix calculation of orientation of polygons (#27967)
  [Kerberos] Add missing javadocs (#32469)
  [Kerberos] Remove Kerberos bootstrap checks (#32451)
  Make get all app privs requires "*" permission (#32460)
  Switch security to new style Requests (#32290)
  Switch security spi example to new style Requests (#32341)
  Painless: Add PainlessConstructor (#32447)
  update rollover to leverage write-alias semantics (#32216)
  Update Fuzzy Query docs to clarify default behavior re max_expansions (#30819)
  INGEST: Clean up Java8 Stream Usage (#32059)
  Ensure KeyStoreWrapper decryption exceptions are handled (#32464)
2018-07-31 10:56:10 -04:00
Nik Everett 22459576d7
Logging: Make node name consistent in logger (#31588)
First, some background: we have 15 different methods to get a logger in
Elasticsearch but they can be broken down into three broad categories
based on what information is provided when building the logger.

Just a class like:
```
private static final Logger logger = ESLoggerFactory.getLogger(ActionModule.class);
```
or:
```
protected final Logger logger = Loggers.getLogger(getClass());
```

The class and settings:
```
this.logger = Loggers.getLogger(getClass(), settings);
```

Or more information like:
```
Loggers.getLogger("index.store.deletes", settings, shardId)
```

The goal of the "class and settings" variant is to attach the node name
to the logger. Because we don't always have the settings available, we
often use the "just a class" variant and get loggers without node names
attached. There isn't any real consistency here. Some loggers get the
node name because it is convenient and some do not.

This change makes the node name available to all loggers all the time.
Almost. There are some caveats are testing that I'll get to. But in
*production* code the node name is node available to all loggers. This
means we can stop using the "class and settings" variants to fetch
loggers which was the real goal here, but a pleasant side effect is that
the ndoe name is now consitent on every log line and optional by editing
the logging pattern. This is all powered by setting the node name
statically on a logging formatter very early in initialization.

Now to tests: tests can't set the node name statically because
subclasses of `ESIntegTestCase` run many nodes in the same jvm, even in
the same class loader. Also, lots of tests don't run with a real node so
they don't *have* a node name at all. To support multiple nodes in the
same JVM tests suss out the node name from the thread name which works
surprisingly well and easy to test in a nice way. For those threads
that are not part of an `ESIntegTestCase` node we stick whatever useful
information we can get form the thread name in the place of the node
name. This allows us to keep the logger format consistent.
2018-07-31 10:54:24 -04:00
Daniel Mitterdorfer 5f302580f9 Mute SSLTrustRestrictionsTests on JDK 11
This commit adds an assumption to two test methods in
SSLTrustRestrictionsTests that we are not on JDK 11 as the tests
currently fail there.

Relates #29989
2018-07-31 16:43:05 +02:00
Daniel Mitterdorfer adb93da974 Mute KerberosAuthenticationIT
Relates #32498
2018-07-31 15:55:52 +02:00
Yannick Welsch 11f6983d69
HLRC: Add delete watch action (#32337)
Adds the "delete watch" API to the High-Level Rest Client.

Relates #29827
2018-07-31 10:29:22 +02:00
Yogesh Gaikwad d4ea440e37
[Kerberos] Add missing javadocs (#32469)
This commit adds missing javadocs and fixes few where
the build failed when using JDK 11 for compilation.

Closes#32461
2018-07-31 11:18:08 +10:00
Yogesh Gaikwad f0b36679ec
[Kerberos] Remove Kerberos bootstrap checks (#32451)
This commit removes Kerberos bootstrap checks as they were more
validation checks and better done in Kerberos realm constructor
than as bootstrap checks. This also moves the check
for one Kerberos realm per node to where we initialize realms.
This commit adds few validations which were missing earlier
like missing read permissions on keytab file or if it is directory
to throw exception with error message.
2018-07-31 10:59:36 +10:00
Nhat Nguyen 8cfbb64d6e
ShardFollowNodeTask should fetch operation once (#32455)
Today ShardFollowNodeTask might fetch some operations more than once.
This happens because we ask the leading for up to max_batch_count
operations (instead of the left-over size) for the left-over request.
The leading then can freely respond up to the max_batch_count, and at
the same time, if one of the previous requests completed, we might issue
another read request whose range overlaps with the response of the
left-over request.

Closes #32453
2018-07-30 20:53:09 -04:00
Tim Vernum d75efbcf68
Make get all app privs requires "*" permission (#32460)
The default behaviour for "GetPrivileges" is to get all application
privileges. This should only be allowed if the user has access to
the "*" application.
2018-07-31 09:07:47 +10:00
Nik Everett 4101fc4e3d
Switch security to new style Requests (#32290)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin/security` project to use the new
versions.
2018-07-30 18:16:26 -04:00
Nik Everett 670630948b
Switch security spi example to new style Requests (#32341)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/qa/security-example-spi-extension`
project to use the new versions.
2018-07-30 18:07:49 -04:00
Nhat Nguyen 2245812ef7 Merge branch 'master' into ccr
* master:
  Tests: Fix convert error tests to use fixed value (#32415)
  IndicesClusterStateService should replace an init. replica with an init. primary with the same aId (#32374)
  REST high-level client: parse back _ignored meta field (#32362)
  [CI] Mute DocumentSubsetReaderTests testSearch
2018-07-30 14:02:58 -04:00
David Roberts 14a4a740ac [CI] Mute DocumentSubsetReaderTests testSearch
Relates #32457
2018-07-30 09:04:17 +01:00
Nhat Nguyen aa3b6e098c
Reject follow request if following setting not enabled on follower (#32448)
Today we do not check if the `following_index` setting of the follower
is enabled or not when processing a follow-request. If that setting is
disabled, the follower will use the default engine, not the following
engine. This change checks and rejects such invalid follow requests.

Relates #30086
2018-07-29 21:57:45 -04:00
Nhat Nguyen 2f756b00f6 Merge branch 'master' into ccr
* master:
  Remove reference to non-existent store type (#32418)
  [TEST] Mute failing FlushIT test
  Fix ordering of bootstrap checks in docs (#32417)
  [TEST] Mute failing InternalEngineTests#testSeqNoAndCheckpoints
  [TEST] Mute failing testConvertLongHexError
  bump lucene version after backport
  Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
  [Kerberos] Avoid vagrant update on precommit (#32416)
  TESTS: Move netty leak detection to paranoid level (#32354)
  [DOCS] Fixes formatting of scope object in job resource
  Copy missing segment attributes in getSegmentInfo (#32396)
  AbstractQueryTestCase should run without type less often (#28936)
  INGEST: Fix Deprecation Warning in Script Proc. (#32407)
  Switch x-pack/plugin to new style Requests (#32327)
  Docs: Correcting a typo in tophits (#32359)
  Build: Stop double generating buildSrc pom (#32408)
  TEST: Avoid triggering merges in FlushIT
  Fix missing JavaDoc for @throws in several places in KerberosTicketValidator.
  Switch x-pack full restart to new style Requests (#32294)
  Release requests in cors handler (#32364)
  Painless: Clean Up PainlessClass Variables (#32380)
  Docs: Fix callouts in put license HL REST docs (#32363)
  [ML] Consistent pattern for strict/lenient parser names (#32399)
  Update update-settings.asciidoc (#31378)
  Remove some dead code (#31993)
  Introduce index store plugins (#32375)
  Rank-Eval: Reduce scope of an unchecked supression
  Make sure _forcemerge respects `max_num_segments`. (#32291)
  TESTS: Fix Buf Leaks in HttpReadWriteHandlerTests (#32377)
  Only enforce password hashing check if FIPS enabled (#32383)
2018-07-27 16:24:03 -04:00
Nhat Nguyen 8474f8a01c
Validate source of an index in LuceneChangesSnapshot (#32288)
Today it's possible to encounter an Index operation in Lucene whose
_source is disabled, and _recovery_source was pruned by the MergePolicy.
If it's the case, we create a Translog#Index without source and let the
caller validate it later. However, this approach is challenging for the
caller.

Deletes and No-Ops don't allow invoking "source()" method. The caller
has to make sure to call "source()" only on index operations. The
current implementation in CCR does not follow this and fail to replica
deletes or no-ops. Moreover, it's easier to reason if a Translog#Index
always has the source.
2018-07-27 08:16:52 -04:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Yogesh Gaikwad 9198bcf78e
[Kerberos] Avoid vagrant update on precommit (#32416)
This commit avoids dependency during compile on copy keytab to
be present in the generated sources so pre-commit does not
stall for updating vagrant box.

Closes#32387
2018-07-27 12:01:52 +10:00
lcawl 67a884ec88 [DOCS] Fixes formatting of scope object in job resource 2018-07-26 12:28:26 -07:00
Nik Everett 643235d46a
Switch x-pack/plugin to new style Requests (#32327)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin` project to use the new versions.
2018-07-26 13:31:16 -04:00
Jack Conradson eb73dde7c8 Fix missing JavaDoc for @throws in several places in
KerberosTicketValidator.
2018-07-26 09:14:43 -07:00
Nik Everett 11a2189c6f
Switch x-pack full restart to new style Requests (#32294)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:qa:full-cluster-restart` project to use
the new versions.
2018-07-26 12:11:31 -04:00
David Roberts 0afa265ac9
[ML] Consistent pattern for strict/lenient parser names (#32399)
Previously we had two patterns for naming of strict
and lenient parsers.

Some classes had CONFIG_PARSER and METADATA_PARSER,
and used an enum to pass the parser type to nested
parsers.

Other classes had STRICT_PARSER and LENIENT_PARSER
and used ternary operators to pass the parser type
to nested parsers.

This change makes all ML classes use the second of
the patterns described above.
2018-07-26 16:55:40 +01:00
Christoph Büscher 35ae87125d
Remove some dead code (#31993)
Removing some dead code or supressing warnings where apropriate. Most of the
time the variable tested for null is dereferenced earlier or never used before.
2018-07-26 17:12:51 +02:00
Jason Tedor eb675a1c4d
Introduce index store plugins (#32375)
Today we allow plugins to add index store implementations yet we are not
doing this in our new way of managing plugins as pull versus push. That
is, today we still allow plugins to push index store providers via an on
index module call where they can turn around and add an index
store. Aside from being inconsistent with how we manage plugins today
where we would look to pull such implementations from plugins at node
creation time, it also means that we do not know at a top-level (for
example, in the indices service) which index stores are available. This
commit addresses this by adding a dedicated plugin type for index store
plugins, removing the index module hook for adding index stores, and by
aggregating these into the top-level of the indices service.
2018-07-26 08:05:49 -04:00
Jason Tedor 467a60ba0c
Only enforce password hashing check if FIPS enabled (#32383)
This commit modifies the FIPS password hashing algorithm check to only
be executed if FIPS mode is enabled.
2018-07-25 20:57:03 -04:00
Nhat Nguyen caefed4d33 Merge branch 'master' into ccr
* master:
  [DOCS] Fix formatting error in Slack action
  Painless: Fix documentation links to use existing refs (#32335)
  Painless: Decouple PainlessLookupBuilder and Whitelists (#32346)
  [DOCS] Adds recommendation for xpack.security.enabled (#32345)
  [TEST] Mute ConvertProcessortTests.testConvertIntHexError
  [TEST] Fix failure due to exception message in java11 (#32321)
  [DOCS] Fixes typo in ML aggregations page
  [DOCS] Adds link from bucket_span property to common time units
  [ML][DOCS] Add documentation for detector rules and filters (#32013)
  Add opaque_id to index audit logging (#32260)
  Add 6.5.0 version to master
  fixes broken build for third-party-tests (#32353)
2018-07-25 20:12:13 -04:00
lcawl a577fb3381 [DOCS] Fix formatting error in Slack action 2018-07-25 14:19:19 -07:00
Jason Tedor b8548973e3
Maintain ordering in X-Pack project substitutions
This commit fixes the ordering of the project substitutions for
X-Pack. This is not a bug, simply keeping them in order helps maintain
them.
2018-07-25 13:13:53 -04:00
Zachary Tong 6cf7588c3d
[TEST] Fix failure due to exception message in java11 (#32321)
Java 11 uses more verbose exceptions messages, causing this assertion
to fail.  Changed the test to be less restrictive and only look
for the classes we care about.
2018-07-25 11:34:26 -04:00
lcawl 5a12c63b4e [DOCS] Fixes typo in ML aggregations page 2018-07-25 08:32:01 -07:00
lcawl 6832aa6797 [DOCS] Adds link from bucket_span property to common time units 2018-07-25 08:31:52 -07:00
Dimitris Athanasiou 9a7a649755
[ML][DOCS] Add documentation for detector rules and filters (#32013) 2018-07-25 16:10:32 +01:00
Igor Motov e12e2e0cdd
Add opaque_id to index audit logging (#32260)
Logs opaque_id if it is available with all audit log messages using
index-based audit log.

Closes #31521
2018-07-25 08:07:09 -07:00
Nik Everett d2d78415a9 Build: Fix jarHell error I caused by last backport
I fixed this locally but didn't commit it.... Ooops. Shouldn't merge
late a night....
2018-07-25 00:16:34 -04:00
Nhat Nguyen cd8b80da58 Use shadow plugin in ccr/qa 2018-07-25 00:16:33 -04:00
Tim Vernum 33e3822c5d Fix JarHell on X-Pack protocol
Ports 466bc8c from 6.x
2018-07-25 14:14:19 +10:00
Nhat Nguyen a5d8f0b55a CCR: use shadow plugin
Relates #32240
2018-07-24 22:48:11 -04:00
Nhat Nguyen ab4deefbe0 Merge branch 'master' into ccr
* master:
  Security: revert to old way of merging automata (#32254)
  Networking: Fix test leaking buffer (#32296)
  Undo a debugging change that snuck in during the field aliases merge.
  Painless: Update More Methods to New Naming Scheme (#32305)
  [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests
  Ingest: Support integer and long hex values in convert (#32213)
  Introduce fips_mode setting and associated checks (#32326)
  Add V_6_3_3 version constant
  [DOCS] Removed extraneous callout number.
  Rest HL client: Add put license action (#32214)
  Add ERR to ranking evaluation documentation (#32314)
  Introduce Application Privileges with support for Kibana RBAC (#32309)
  Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
  [Kerberos] Add Kerberos authentication support (#32263)
  [ML] Extract persistent task methods from MlMetadata (#32319)
  Add Restore Snapshot High Level REST API
  Register ERR metric with NamedXContentRegistry (#32320)
  fixes broken build for third-party-tests (#32315)
  Allow Integ Tests to run in a FIPS-140 JVM (#31989)
  [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280)
  awaitsfix testRandomClusterStateUpdates
  [TEST] add version skip to weighted_avg tests
  Consistent encoder names (#29492)
  Add WeightedAvg metric aggregation (#31037)
  Switch monitoring to new style Requests (#32255)
  Rename ranking evaluation `quality_level` to `metric_score` (#32168)
  Fix a test bug around nested aggregations and field aliases. (#32287)
  Add new permission for JDK11 to load JAAS libraries (#32132)
  Silence SSL reload test that fails on JDK 11
  [test] package pre-install java check (#32259)
  specify subdirs of lib, bin, modules in package (#32253)
  Switch x-pack:core to new style Requests (#32252)
  awaitsfix SSLConfigurationReloaderTests
  Painless: Clean up add methods in PainlessLookup (#32258)
  Fail shard if IndexShard#storeStats runs into an IOException (#32241)
  AwaitsFix RecoveryIT#testHistoryUUIDIsGenerated
  Remove unnecessary warning supressions (#32250)
  CCE when re-throwing "shard not available" exception in TransportShardMultiGetAction (#32185)
  Add new fields to monitoring template for Beats state (#32085)
2018-07-24 21:33:50 -04:00
Jay Modi e43375bf9a
Security: revert to old way of merging automata (#32254)
This commit reverts to the pre-6.3 way of merging automata as the
change in 6.3 significantly impacts the performance for roles with a
large number of concrete indices. In addition, the maximum number of
states for security automata has been increased to 100,000 in order
to allow users to use roles that caused problems pre-6.3 and 6.3 fixed.

As an escape hatch, the maximum number of states is configurable with
a setting so that users with complex patterns in roles can increase
the states with the knowledge that there is more memory usage.
2018-07-24 16:26:50 -06:00
Lee Hinman 73be9840dd [TEST] Fix assumeFalse -> assumeTrue in SSLReloadIntegTests 2018-07-24 13:10:55 -06:00
Ioannis Kakavas be40a6982a
Introduce fips_mode setting and associated checks (#32326)
* Introduce fips_mode setting and associated checks

Introduce xpack.security.fips_mode.enabled setting ( default false)
When it is set to true, a number of Bootstrap checks are performed:
- Check that Secure Settings are of the latest version (3)
- Check that no JKS keystores are configured
- Check that compliant algorithms ( PBKDF2 family ) are used for
  password hashing
2018-07-24 22:05:00 +03:00
Igor Motov 29c802f88e
Rest HL client: Add put license action (#32214)
In the HL REST client we replace the License object with a string, because of 
complexity of this class. It is also not really needed on the client side since 
end-users are not interacting with the license besides passing it as a string 
to the server.

Relates #29827
2018-07-24 11:19:07 -07:00
Tim Vernum 387c3c7f1d Introduce Application Privileges with support for Kibana RBAC (#32309)
This commit introduces "Application Privileges" to the X-Pack security
model.

Application Privileges are managed within Elasticsearch, and can be
tested with the _has_privileges API, but do not grant access to any
actions or resources within Elasticsearch. Their purpose is to allow
applications outside of Elasticsearch to represent and store their own
privileges model within Elasticsearch roles.

Access to manage application privileges is handled in a new way that
grants permission to specific application names only. This lays the
foundation for more OLS on cluster privileges, which is implemented by
allowing a cluster permission to inspect not just the action being
executed, but also the request to which the action is applied.
To support this, a "conditional cluster privilege" is introduced, which
is like the existing cluster privilege, except that it has a Predicate
over the request as well as over the action name.

Specifically, this adds
- GET/PUT/DELETE actions for defining application level privileges
- application privileges in role definitions
- application privileges in the has_privileges API
- changes to the cluster permission class to support checking of request
  objects
- a new "global" element on role definition to provide cluster object
  level security (only for manage application privileges)
- changes to `kibana_user`, `kibana_dashboard_only_user` and
  `kibana_system` roles to use and manage application privileges

Closes #29820
Closes #31559
2018-07-24 10:34:46 -06:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Yogesh Gaikwad a525c36c60 [Kerberos] Add Kerberos authentication support (#32263)
This commit adds support for Kerberos authentication with a platinum
license. Kerberos authentication support relies on SPNEGO, which is
triggered by challenging clients with a 401 response with the
`WWW-Authenticate: Negotiate` header. A SPNEGO client will then provide
a Kerberos ticket in the `Authorization` header. The tickets are
validated using Java's built-in GSS support. The JVM uses a vm wide
configuration for Kerberos, so there can be only one Kerberos realm.
This is enforced by a bootstrap check that also enforces the existence
of the keytab file.

In many cases a fallback authentication mechanism is needed when SPNEGO
authentication is not available. In order to support this, the
DefaultAuthenticationFailureHandler now takes a list of failure response
headers. For example, one realm can provide a
`WWW-Authenticate: Negotiate` header as its default and another could
provide `WWW-Authenticate: Basic` to indicate to the client that basic
authentication can be used in place of SPNEGO.

In order to test Kerberos, unit tests are run against an in-memory KDC
that is backed by an in-memory ldap server. A QA project has also been
added to test against an actual KDC, which is provided by the krb5kdc
fixture.

Closes #30243
2018-07-24 08:44:26 -06:00
David Kyle 99426eb4f8
[ML] Extract persistent task methods from MlMetadata (#32319)
Move ML persistent task helper functions to the new class MlTasks
and remove MLMetadataField after moving the string constant to 
MlMetadata.
2018-07-24 15:22:57 +01:00
Ioannis Kakavas a2dbd83db1
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
* Complete changes for running IT in a fips JVM

- Mute :x-pack:qa:sql:security:ssl:integTest as it
  cannot run in FIPS 140 JVM until the SQL CLI supports key/cert.
- Set default JVM keystore/truststore password in top level build
  script for all integTest tasks in a FIPS 140 JVM
- Changed top level x-pack build script to use keys and certificates
  for trust/key material when spinning up clusters for IT
2018-07-24 12:48:14 +03:00
David Kyle 177750719d [DOCS] Rollup Caps API incorrectly mentions GET Jobs API (#32280) 2018-07-24 10:23:13 +01:00
Zachary Tong 6ba144ae31
Add WeightedAvg metric aggregation (#31037)
Adds a new single-value metrics aggregation that computes the weighted 
average of numeric values that are extracted from the aggregated 
documents. These values can be extracted from specific numeric
fields in the documents.

When calculating a regular average, each datapoint has an equal "weight"; it
contributes equally to the final value.  In contrast, weighted averages
scale each datapoint differently.  The amount that each datapoint contributes 
to the final value is extracted from the document, or provided by a script.

As a formula, a weighted average is the `∑(value * weight) / ∑(weight)`

A regular average can be thought of as a weighted average where every value has
an implicit weight of `1`.

Closes #15731
2018-07-23 18:33:15 -04:00
Nik Everett 55a2d3e0dd
Switch monitoring to new style Requests (#32255)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack/plugin/monitoring` project to use the new
versions.
2018-07-23 16:50:15 -04:00
Christoph Büscher fe6bb75eb4
Rename ranking evaluation `quality_level` to `metric_score` (#32168)
The notion of "quality" is an overloaded term in the search ranking evaluation 
context. Its usually used to decribe certain levels of "good" vs. "bad" of a 
seach result with respect to the users information need. We currently report the 
result of the ranking evaluation as `quality_level` which is a bit missleading.
This changes the response parameter name to `metric_score` which fits better.
2018-07-23 22:25:02 +02:00
Jason Tedor 0b72132553 Silence SSL reload test that fails on JDK 11
This commit adds an assumption for an SSL reload test that we are not on
JDK 11 as the test currently fails there.

Relates #32293
2018-07-23 14:52:56 -04:00
Nik Everett 4804da06bf
Switch x-pack:core to new style Requests (#32252)
In #29623 we added `Request` object flavored requests to the low level
REST client and in #30315 we deprecated the old `performRequest`s. This
changes all calls in the `x-pack:core` project to use the new versions.
2018-07-23 14:07:18 -04:00
Andy Bristol d43c3a18b9 awaitsfix SSLConfigurationReloaderTests 2018-07-23 10:57:40 -07:00
Christoph Büscher ff87b7aba4
Remove unnecessary warning supressions (#32250) 2018-07-23 11:31:04 +02:00
Nicolas Ruflin 8f1d15ce61
Add new fields to monitoring template for Beats state (#32085)
New data is reported from Beats to the monitoring endpoint. This PR adds the template change necessary for it. See https://github.com/elastic/beats/issues/7521 for more details.

Queue data is skipped for now as implementation is not finished yet.
2018-07-23 07:38:19 +02:00
Nhat Nguyen 88190299df
CCR: Fix incorrect read request completion condition (#32266)
Today we consider a read request is exhausted if from_seqno is equal to
or greater than the max_required_seqno. However, if we stop when
from_seqno equals to the max_required_seqno, we will miss an operation
whose seqno is max_required_seqno because we have not seen that 
operation yet.
2018-07-22 22:14:27 -04:00
Martijn van Groningen b6b596e471
[CCR] Add random shard follow task test (#32188)
Added shard follow task unit tests that tests whether the shard follow task is able to process randomly generated shard changes api responses.
2018-07-21 12:38:05 +02:00
Martijn van Groningen 8ed3624f47
Merge remote-tracking branch 'es/master' into ccr
* es/master: (23 commits)
  Switch full-cluster-restart to new style Requests (#32140)
  [DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016)
  Remove BouncyCastle dependency from runtime (#32193)
  INGEST: Extend KV Processor (#31789) (#32232)
  INGEST: Make a few Processors callable by Painless (#32170)
  Add region ISO code to GeoIP Ingest plugin (#31669)
  [Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236)
  Make sure that field aliases count towards the total fields limit. (#32222)
  Switch rolling restart to new style Requests (#32147)
  muting failing test for internal auto date histogram to avoid failure before fix is merged
  MINOR: Remove unused `IndexDynamicSettings` (#32237)
  Fix multi level nested sort (#32204)
  Enhance Parent circuit breaker error message (#32056)
  [ML] Use default request durability for .ml-state index (#32233)
  Remove indices stats timeout from monitoring docs
  Rename ranking evaluation response section (#32166)
  Dependencies: Upgrade to joda time 2.10 (#32160)
  Remove aliases resolution limitations when security is enabled (#31952)
  Ensure that field aliases cannot be used in multi-fields. (#32219)
  TESTS: Check for Netty resource leaks (#31861)
  ...
2018-07-21 09:06:13 +02:00
Nhat Nguyen 8e15504443 TEST: Fix range issue in ShardChangesActionTests
We modified the way we calculate to_seqno in #32121 but did not adjust
this test accordingly. If min_seqno equals to max_seqno, the size should be
one instead of zero.

Relates #32121
2018-07-20 17:20:41 -04:00
Ioannis Kakavas aaa8f842d6
Remove BouncyCastle dependency from runtime (#32193)
* Remove BouncyCastle dependency from runtime

This commit introduces a new gradle  project that contains
 the classes that have a dependency on BouncyCastle. For 
the default distribution, It builds  a jar from those and
 in puts it in a subdirectory of lib
 (/tools/security-cli) along with the BouncyCastle jars. 
This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools 
that use these classes.

BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.
2018-07-21 00:03:58 +03:00
David Kyle ac960bfa6b
[ML] Use default request durability for .ml-state index (#32233)
The initial decision to use async durability was made a long time ago
for performance reasons. That argument no longer applies and we
prefer the safety of request durability.
2018-07-20 15:49:37 +01:00
Nhat Nguyen fe574f89f8 CCR: Translog op on primary should have versionType
Normally translog operations will not be replayed on the primary.
Following engine is an exception where we replay translog on both
primary and replica as a non-primary strategy.  Even though we won't use
the version_type in the following engine, we still need to pass a valid
value for the primary operation in order not to trip assertions in an
engine.

This commit passes version_type EXTERNAL for translog operation if its
origin is primary.

Relates #31945
2018-07-20 08:39:38 -04:00
Martijn van Groningen a6b7497fdc
[CCR] Add more unit tests for shard follow task (#32121)
The added tests are based on specific scenarios as described in the test plan.
Before this change the ShardFollowNodeTaskTests contained more random like tests,
but these have been removed and in a followup pr better random tests will
be added in a new test class as is described in the test plan.
2018-07-20 14:12:05 +02:00
Alexander Reelsen c5cde96691
Dependencies: Upgrade to joda time 2.10 (#32160)
Changelog: http://www.joda.org/joda-time/changes-report.html
2018-07-20 10:18:38 +02:00
Luca Cavanna 00a6ad0e9e
Remove aliases resolution limitations when security is enabled (#31952)
Resolving wildcards in aliases expression is challenging as we may end
up with no aliases to replace the original expression with, but if we
replace with an empty array that means _all which is quite the opposite.
Now that we support and serialize the original requested aliases,
whenever aliases are replaced we will be able to know what was
initially requested. `MetaData#findAliases` can then be updated to not
return anything in case it gets empty aliases, but the original aliases
were not empty. That means that empty aliases are interpreted as _all
only if they were originally requested that way.

Relates to #31516
2018-07-20 09:23:32 +02:00
Nhat Nguyen d0f3ed5abd Merge branch 'master' into ccr
* master:
  Painless: Simplify Naming in Lookup Package (#32177)
  Handle missing values in painless (#32207)
  add support for write index resolution when creating/updating documents (#31520)
  ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
  Remove indication of future multi-homing support (#32187)
  Rest test - allow for snapshots to take 0 milliseconds
  Make x-pack-core generate a pom file
  Rest HL client: Add put watch action (#32026)
  Build: Remove pom generation for plugin zip files (#32180)
  Fix comments causing errors with Java 11
  Fix rollup on date fields that don't support epoch_millis (#31890)
  Detect and prevent configuration that triggers a Gradle bug (#31912)
  [test] port linux package packaging tests (#31943)
  Revert "Introduce a Hashing Processor (#31087)" (#32178)
  Remove empty @return from JavaDoc
  Adjust SSLDriver behavior for JDK11 changes (#32145)
  [test] use randomized runner in packaging tests (#32109)
  Add support for field aliases. (#32172)
  Painless: Fix caching bug and clean up addPainlessClass. (#32142)
  Call setReferences() on custom referring tokenfilters in _analyze (#32157)
  Fix BwC Tests looking for UUID Pre 6.4 (#32158)
  Improve docs for search preferences (#32159)
  use before instead of onOrBefore
  Add more contexts to painless execute api (#30511)
  Add EC2 credential test for repository-s3 (#31918)
  A replica can be promoted and started in one cluster state update (#32042)
  Fix Java 11 javadoc compile problem
  Fix CP for namingConventions when gradle home has spaces (#31914)
  Fix `range` queries on `_type` field for singe type indices (#31756)
  [DOCS] Update TLS on Docker for 6.3 (#32114)
  ESIndexLevelReplicationTestCase doesn't support replicated failures but it's good to know what they are
  Remove versionType from translog (#31945)
  Switch distribution to new style Requests (#30595)
  Build: Skip jar tests if jar disabled
  Painless: Add PainlessClassBuilder (#32141)
  Build: Make additional test deps of check (#32015)
  Disable C2 from using AVX-512 on JDK 10 (#32138)
  Build: Move shadow customizations into common code (#32014)
  Painless: Fix Bug with Duplicate PainlessClasses (#32110)
  Remove empty @param from Javadoc
  Re-disable packaging tests on suse boxes
  Docs: Fix missing example script quote (#32010)
  [ML] Wait for aliases in multi-node tests (#32086)
  [ML] Move analyzer dependencies out of categorization config (#32123)
  Ensure to release translog snapshot in primary-replica resync (#32045)
  Handle TokenizerFactory  TODOs (#32063)
  Relax TermVectors API to work with textual fields other than TextFieldType (#31915)
  Updates the build to gradle 4.9 (#32087)
  Mute :qa:mixed-cluster indices.stats/10_index/Index - all’
  Check that client methods match API defined in the REST spec (#31825)
  Enable testing in FIPS140 JVM (#31666)
  Fix put mappings java API documentation (#31955)
  Add exclusion option to `keep_types` token filter (#32012)
  [Test] Modify assert statement for ssl handshake (#32072)
2018-07-19 23:03:01 -04:00
Tim Vernum 6f2b7dc9fe Fix AD / vagrant based tests for #30953
These tests were creating a SSL service that was not aware of the
realm that they were trying to test.
This no longer works.
2018-07-20 12:51:52 +10:00
Tim Vernum c32981db6b
Detect old trial licenses and mimic behaviour (#32209)
Prior to 6.3 a trial license default to security enabled. Since 6.3
they default to security disabled. If a cluster is upgraded from <6.3
to >6.3, then we detect this and mimic the old behaviour with respect
to security.
2018-07-20 10:09:28 +10:00
Tal Levy 9ae6905657
add support for write index resolution when creating/updating documents (#31520)
Now write operations like Index, Delete, Update rely on the write-index associated with 
an alias to operate against. This means writes will be accepted even when an alias points to multiple indices, so long as one is the write index. Routing values will be used from the AliasMetaData for the alias in the write-index. All read operations are left untouched.
2018-07-19 09:17:49 -07:00
Ryan Ernst 9aa562d3dd Make x-pack-core generate a pom file
This was a forgotten part of #32180
2018-07-19 01:53:08 -07:00
Alexander Reelsen 202894b832
Rest HL client: Add put watch action (#32026)
Relates #29827

This implementation behaves like the current transport client, that you basically cannot configure a Watch POJO representation as an argument to the put watch API, but only a bytes reference. You can use the the `WatchSourceBuilder` from the `org.elasticsearch.plugin:x-pack-core` dependency to build watches.

This commit also changes the license type to trial, so that watcher is available in high level rest client tests.

/cc @hub-cap
2018-07-19 10:40:54 +02:00
Jim Ferenczi 644a92f158
Fix rollup on date fields that don't support epoch_millis (#31890)
The rollup indexer uses a range query to select the next page
of results based on the last time bucket of the previous round
and the `delay` configured on the rollup job. This query uses
the `epoch_millis` format implicitly but doesn't set the `format`.
This result in errors during the rollup job if the field
definition doesn't allow this format. It can also miss documents
if the format is not accepted but another format in the field
definition is able to parse the query (e.g.: `epoch_second`).
This change ensures that we use `epoch_millis` as the only format
to parse the rollup range query.
2018-07-19 09:34:23 +02:00
Alpar Torok 38e2e1d553
Detect and prevent configuration that triggers a Gradle bug (#31912)
* Detect and prevent configuration that triggers a Gradle bug

As we found in #31862, this can lead to a lot of wasted time as it's not
immediatly obvius what's going on.
Givent how many projects we have it's getting increasingly easier to run
into gradle/gradle#847.
2018-07-19 06:46:58 +00:00
Tal Levy 27d663b64b
Revert "Introduce a Hashing Processor (#31087)" (#32178)
This reverts commit 8c78fe7114.
2018-07-18 16:44:05 -07:00
Michael Basnight 63749498ea Remove empty @return from JavaDoc 2018-07-18 13:28:25 -05:00
Tim Brooks 90fcb38448
Adjust SSLDriver behavior for JDK11 changes (#32145)
This is related to #32122. A number of things changed related to adding
TLS 1.3 support in JDK11. Some exception messages and other SSLEngine
behavior changed. This commit fixes assertions on exception messages.
Additionally it identifies two bugs related to how the SSLDriver behaves
in regards to JDK11 changes. Finally, it mutes a tests until correct
behavior can be identified. There is another open issue for that muted
test (#32144).
2018-07-18 11:49:42 -06:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
Toby McLaughlin 487cfc3b3f
[DOCS] Update TLS on Docker for 6.3 (#32114)
Remove references to the `platinum` image and add a self-generated trial
licence to the example for TLS on Docker.

Fixes elastic/elasticsearch-docker#176
2018-07-18 17:07:31 +10:00
Nik Everett 1b97652a4c
Build: Move shadow customizations into common code (#32014)
Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:
* The non-shadow jar will not be built
* The shadow jar will not have a "classifier"
* Tests will run against the shadow jar
* Javadoc will include all of the shadowed classes
* Service files in `META-INF/services` will be merged
2018-07-17 14:20:41 -04:00
Ioannis Kakavas 9cdbd1bd85 Remove empty @param from Javadoc 2018-07-17 20:25:38 +03:00
Martijn van Groningen d88c76e02b
[CCR] Initial replication group based tests (#32024)
Tests shard follow task in the context of a leader and follower ReplicationGroup,
in order to test how the shard follow logic reacts to certain shard related
failure scenarios.

More tests will need to be added, but this indicates what changes need to be made
to have these tests.

Relates to #30102
2018-07-17 17:39:49 +02:00
David Kyle 2bd40a7927
[ML] Wait for aliases in multi-node tests (#32086) 2018-07-17 16:21:53 +01:00
David Roberts 99c2a82c04
[ML] Move analyzer dependencies out of categorization config (#32123)
The ML config classes will shortly be moved to the X-Pack protocol
library to allow the ML APIs to be moved to the high level REST
client.  Dependencies on server functionality should be removed
from the config classes before this is done.

This change is entirely about moving code between packages.  It
does not add or remove any functionality or tests.
2018-07-17 15:01:12 +01:00
Armin Braun ed3b44fb4c
Handle TokenizerFactory TODOs (#32063)
* Don't replace Replace TokenizerFactory with Supplier, this approach was rejected in #32063 
* Remove unused parameter from constructor
2018-07-17 14:14:02 +02:00
Martijn van Groningen 006c79a80d
[CCR] Improve retry mechanism when making remote calls from shard follow task (#31930)
Closes #31816
2018-07-17 10:25:51 +02:00
Ioannis Kakavas 9e529d9d58
Enable testing in FIPS140 JVM (#31666)
Ensure our tests can run in a FIPS JVM

JKS keystores cannot be used in a FIPS JVM as attempting to use one
in order to init a KeyManagerFactory or a TrustManagerFactory is not
allowed.( JKS keystore algorithms for private key encryption are not
FIPS 140 approved)
This commit replaces JKS keystores in our tests with the
corresponding PEM encoded key and certificates both for key and trust
configurations.
Whenever it's not possible to refactor the test, i.e. when we are
testing that we can load a JKS keystore, etc. we attempt to
mute the test when we are running in FIPS 140 JVM. Testing for the
JVM is naive and is based on the name of the security provider as
we would control the testing infrastrtucture and so this would be
reliable enough.
Other cases of tests being muted are the ones that involve custom
TrustStoreManagers or KeyStoreManagers, null TLS Ciphers and the
SAMLAuthneticator class as we cannot sign XML documents in the
way we were doing. SAMLAuthenticator tests in a FIPS JVM can be
reenabled with precomputed and signed SAML messages at a later stage.

IT will be covered in a subsequent PR
2018-07-17 10:54:10 +03:00
Yogesh Gaikwad 6717df3c2d
[Test] Modify assert statement for ssl handshake (#32072)
There have been changes in error messages for `SSLHandshakeException`.
This has caused a couple of failures in our tests.
This commit modifies test verification to assert on exception type of
class `SSLHandshakeException`.
There was another issue in Java11 which caused NPE. The bug has now
been fixed on Java11 - early access build 22.
Bug Ref: https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8206355
Enable the skipped tests due to this bug.

Closes #31940
2018-07-17 15:43:03 +10:00
Martijn van Groningen d0c9cf26a9
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Add Index UUID to `/_stats` Response (#31871)
  Painless: Move and Rename Several Methods in the lookup package (#32105)
  Bypass highlight query terms extraction on empty fields (#32090)
  Switch non-x-pack to new style requests (#32106)
  [Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
  Revert "[test] disable packaging tests for suse boxes"
  SQL: allow LEFT and RIGHT as function names (#32066)
  DOCS: put LIMIT 10 to the SQL query (#32065)
  [test] turn on host io cache for opensuse (#32053)
  Tweaked Elasticsearch Service links for SEO
2018-07-17 07:36:34 +02:00
Armin Braun 4b5071f2d0
Add Index UUID to `/_stats` Response (#31871)
* Add "uuid" field to each index's section in the `/_stats` response
* closes #31791
2018-07-17 06:50:21 +02:00
Zachary Tong 791b9b147c
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint.

- Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern.
- One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
2018-07-16 17:20:50 -04:00
Costin Leau 20ea72e2f2
SQL: allow LEFT and RIGHT as function names (#32066)
Due to the way ANTLR works, any declared tokens need to be accounted for
manually inside function names (otherwise a different rule gets applied).

Fix #32046
2018-07-16 23:42:44 +03:00
Martijn van Groningen f5e2168260
Merge remote-tracking branch 'es/master' into ccr
* es/master: (21 commits)
  Tweaked Elasticsearch Service links for SEO
  Watcher: Store username on watch execution (#31873)
  Use correct formatting for links (#29460)
  Painless: Separate PainlessLookup into PainlessLookup and PainlessLookupBuilder (#32054)
  Scripting: Remove dead code from painless module (#32064)
  [Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
  [TEST] Consistent algorithm usage (#32077)
  [Rollup] Fix duplicate field names in test (#32075)
  Ensure only parent breaker trips in unit test
  Unmute field collapsing rest tests
  Fix BWC check after backport
  [Tests] Fix failure due to changes exception message (#32036)
  Remove unused params from SSource and Walker (#31935)
  [Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
  Turn off real-mem breaker in REST tests
  Turn off real-mem breaker in single node tests
  Fix broken OpenLDAP Vagrant QA test
  Cleanup Duplication in `PainlessScriptEngine` (#31991)
  SCRIPTING: Remove unused MultiSearchTemplateRequestBuilder (#32049)
  Fix compile issues introduced by merge (#32058)
  ...
2018-07-16 21:49:57 +02:00
Michael Basnight 637cac9061
Watcher: Store username on watch execution (#31873)
There is currently no way to see what user executed a watch. This commit
adds the decrypted username to each execution in the watch history, in a
new field "user".

Closes #31772
2018-07-16 14:20:16 -05:00
Zachary Tong 59191b4998
[Rollup] Replace RollupIT with a ESRestTestCase version (#31977)
The old RollupIT was a node IT, an flaky for a number of reasons.
This new version is an ESRestTestCase and should be a little more robust.

This was added to the multi-node QA tests as that seemed like the most
appropriate location.  It didn't seem necessary to create a whole new
QA module.

Note: The only test that was ported was the "Big" test for validating
a larger dataset.  The rest of the tests are represented in existing
yaml tests.

Closes #31258
Closes #30232
Related to #30290
2018-07-16 10:47:46 -04:00
Ioannis Kakavas ef7ccd1c07
[TEST] Consistent algorithm usage (#32077)
Ensure that the same algorithm is used for settings and
change password requests for consistency, even if we
do not expext to reach the code where the algorithm is
checked for now.
Completes a7eaa409e8
2018-07-16 16:41:56 +03:00
Jim Ferenczi a3b608d616
[Rollup] Fix duplicate field names in test (#32075)
This commit ensures that random field names do not clash with the explicit field names
set by the tests.

Closes #32067
2018-07-16 15:25:45 +02:00
Jim Ferenczi a14db2f9d6 [Test] Mute MlJobIT#testDeleteJobAfterMissingAliases
Relates #32034
2018-07-16 10:53:51 +02:00
Tim Vernum edbea73f24 Fix broken OpenLDAP Vagrant QA test
This was broken due to c662565 but the problem didn't get detected as
CI builds typically don't run vagrant tests
2018-07-16 15:43:41 +10:00
Tim Brooks a612404b1f
Fix compile issues introduced by merge (#32058)
The build was broken due to some issues with the merging of #32018. A
method that was public went private before the PR was merged. That did
not cause a merge conflict (so the PR was merged successfully). But it
did cause the build to fail.
2018-07-13 23:37:15 -06:00
Tim Brooks 305bfea9c3
Add nio http transport to security plugin (#32018)
This is related to #27260. It adds the SecurityNioHttpServerTransport
to the security plugin. It randomly uses the nio http transport in
security integration tests.
2018-07-13 16:41:02 -06:00
Nhat Nguyen e26f3e0c26 Merge branch 'master' into ccr 2018-07-13 16:55:57 -04:00
Zachary Tong 867c49600c
[Rollup] Histo group config should support scaled_floats (#32048)
Metric config already whitelist scaled_floats, but it wasn't added to
the histo group config.  This centralizes the mapping types map
so that both metrics and histo (and any future configs) use the same
map.

Fixes #32035
2018-07-13 16:10:39 -04:00
Michael Basnight 1f72afa773
Watcher: Make settings reloadable (#31746)
This commit allows for rebuilding watcher secure secrets via the
reload_secure_settings API call. The commit also renames a method in the
Notification Service to make it a bit more readable.
2018-07-13 13:49:47 -05:00
Ryan Ernst 2c3ea43f45
HLRC: Add xpack usage api (#31975)
This commit adds the _xpack/usage api to the high level rest client.
Currently in the transport api, the usage data is exposed in a limited
fashion, at most giving one level of helper methods for the inner keys
of data, but then exposing thos subobjects as maps of objects. Rather
than making parsers for every set of usage data from each feature, this
PR exposes the entire set of usage data as a map of maps.
2018-07-13 09:33:27 -07:00