Commit Graph

624 Commits

Author SHA1 Message Date
Adrien Grand eaaf37a1f9 AwaitFix FullClusterRestartIT#testRollupIDSchemeAfterRestart. 2018-08-16 16:35:10 +02:00
Benjamin Trent aedc2c1c49
HLRC: adding machine learning delete job (#32820)
* HLRC: adding machine learning delete job

* Fixing whitespace

* Moving docs and tests around

* Unifying ml asciidoc file naming convention
2018-08-16 07:18:43 -05:00
David Kyle 7f6802cb51 [ML] Choose seconds to fix intermittent DatafeeedConfigTest failure 2018-08-16 10:48:56 +01:00
Jim Ferenczi 996ed73d73 Test: Fix unpredictive merges in DocumentSubsetReaderTests
The merge policy that was used could lead to unpredictive merges due to the
randomization of `setDeletesPctAllowed`.

Closes #32457
2018-08-16 11:44:57 +02:00
debadair 93ada98eb5
[DOCS] Fixing cross doc link to Stack Overview security topic. 2018-08-15 15:09:41 -07:00
Jay Modi 1a45b27d8b
Move CharArrays to core lib (#32851)
This change cleans up some methods in the CharArrays class from x-pack, which
includes the unification of char[] to utf8 and utf8 to char[] conversions that
intentionally do not use strings. There was previously an implementation in
x-pack and in the reloading of secure settings. The method from the reloading
of secure settings was adopted as it handled more scenarios related to the
backing byte and char buffers that were used to perform the conversions. The
cleaned up class is moved into libs/core to allow it to be used by requests
that will be migrated to the high level rest client.

Relates #32332
2018-08-15 15:26:00 -06:00
Benjamin Trent f90f06ede1
HLRC: adding machine learning open job (#32860)
* HLRC: adding machine learning open job

HLRC: adding ML open job API call

* Addressing @droberts195 comments and fixing minor style issue
2018-08-15 13:19:41 -05:00
David Roberts 5ba04e23fc
[ML] Add log structure finder functionality (#32788)
This change adds a library to ML that can be used to deduce a log
file's structure given only a sample of the log file.

Eventually this will be used to add an endpoint to ML to make the
functionality available to end users, but this will follow in a
separate change.

The functionality is split into a library so that it can also be
used by a command line tool without requiring the command line
tool to include all server code.
2018-08-15 18:04:21 +01:00
Armin Braun 986c55b830
INGEST: Add Configuration Except. Data to Metdata (#32322)
* closes #27728
2018-08-15 19:02:19 +02:00
Ed Savage 51cece1900 [ML] Temporarily disabling rolling-upgrade tests
BWC tests disabled while backporting https://github.com/elastic/elasticsearch/pull/32816
2018-08-15 15:48:50 +01:00
Lee Hinman 48281ac5bc
Use generic AcknowledgedResponse instead of extended classes (#32859)
This removes custom Response classes that extend `AcknowledgedResponse` and do nothing, these classes are not needed and we can directly use the non-abstract super-class instead.

While this appears to be a large PR, no code has actually changed, only class names have been changed and entire classes removed.
2018-08-15 08:06:14 -06:00
Ed Savage 8ce1ab3ed9
[ML] Removing old per-partition normalization code (#32816)
[ML] Removing old per-partition normalization code

Per-partition normalization is an old, undocumented feature that was
never used by clients. It has been superseded by per-partition maximum
scoring.

To maintain communication compatibility with nodes prior to 6.5 it is
necessary to maintain/cope with the old wire format
2018-08-15 13:13:32 +01:00
Jay Modi ac5ef8c389
Security: remove password hash bootstrap check (#32440)
This change removes the PasswordHashingBootstrapCheck and replaces it
with validation on the setting itself. This ensures we always get a
valid value from the setting when it is used.
2018-08-14 13:34:22 -06:00
Jay Modi 7d8a64d703
Move validation to server for put user requests (#32471)
This change moves the validation for values of usernames and passwords
from the request to the transport action. This is done to prevent
the need to move more classes into protocol once we add this API to the
high level rest client. Additionally, this resolves an issue where
validation depends on settings and we always pass empty settings
instead of the actual settings.

Relates #32332
2018-08-14 13:18:07 -06:00
David Roberts c985f500f4
[ML] Add high level REST client docs for ML put job endpoint (#32843)
Relates #29827
Relates #32726
2018-08-14 19:52:54 +01:00
Michael Basnight 6fde9e5c2f
Watcher: Remove unused hipchat render method (#32211)
The HipChatMessage#render is no longer used, and instead the
HipChatAccount#render is used in the ExecutableHipChatAction. Only a
test that validated the HttpProxy used this render method still. This
commit cleans it up.
2018-08-14 11:07:58 -05:00
Michael Basnight e1206a6272
Watcher: Remove extraneous auth classes (#32300)
The auth.basic package was an example of a single implementation
interface that leaked into many different classes. In order to clean
this up, the HttpAuth interface, factories, and Registries all were
removed and the single implementation, BasicAuth, was substituted in all
cases. This removes some dependenies between Auth and the Templates,
which can now use static methods on BasicAuth. BasicAuth was also moved
into the http package and all of the other classes were removed.
2018-08-14 11:06:46 -05:00
Michael Basnight 4c90a61a35
Watcher: migrate PagerDuty v1 events API to v2 API (#32285)
The PagerDuty v1 API is EOL and will stop accepting new accounts
shortly. This commit swaps out the watcher use of the v1 API with the
new v2 API. It does not change anything about the existing watcher
API.

Closes #32243
2018-08-14 11:06:18 -05:00
Yannick Welsch 00b006f033
HLRC: Add Delete License API (#32586)
Relates to #29827
2018-08-14 11:55:54 +02:00
Nhat Nguyen cba7fceb53 AwaitsFix case-functions.sql-spec
Relates #32589
2018-08-13 15:06:33 -04:00
Ioannis Kakavas 06758f579c
Mute security-cli tests in FIPS JVM (#32812)
All Unit tests in this module are muted in FIPS 140 JVMs and
 as such the CI run fails. This commit disables test task for the
module in a  FIPS JVM and reverts adding a dummy test in 
4cbcc1.
2018-08-13 21:27:06 +03:00
Zachary Tong 4da0b52a19
[TEST] Force a stop to save rollup state before continuing (#32787)
We only upgrade the ID when the state is saved in one of four scenarios:

- when we reach a checkpoint (every 50 pages)
- when we run out of data
- when explicitly stopped
- on failure

The test was relying on the pre-upgrade to finish, save state and then
the post-upgrade to start, hit the end of data and upgrade ID.  THEN
get the new doc and apply the new ID.

But I think this is vulnerable to timing issues. If the pre-upgrade
portion shutdown before it saved the state, when restarting we would run
through all the data from the beginning with the old ID, meaning both
docs would still have the old scheme.

This change makes the pre-upgrade wait for the job to go back to STARTED
so that we know it persisted the end point.  Post-upgrade, it stops and
restarts the job to ensure the state was persisted and the ID upgraded.

That _should_ rule out the above timing issue.

Closes #32773
2018-08-13 12:53:22 -04:00
Dimitris Athanasiou 4d2144a2a0
[ML][DOCS] Refer to rules feature as custom rules (#32785) 2018-08-13 14:51:13 +01:00
Ed Savage d147cd72cc
[ML] Partition-wise maximum scores (#32748)
Added infrastructure to push through the 'person name field value' to
the normalizer process. This is required by the normalizer to retrieve
the maximum scores for individual partitions.
2018-08-13 10:31:17 +01:00
Luca Cavanna 3e692c3f3d
HLRC: migration get assistance API (#32744)
The request and response classes have been extracted from `IndexUpgradeInfoAction` into top-level classes, and moved to the protocol jar. The `UpgradeActionRequired` enum is also moved.

Relates to #29827
2018-08-13 11:05:27 +02:00
Yogesh Gaikwad 8114646e12
[Kerberos] Add debug log statement for exceptions (#32663)
This commit adds missing debug log statements for exceptions
that occur during ticket validation. I thought these
get logged somewhere else in authentication chain
but even after enabling trace logs I could not see them
logged. As the Kerberos exception messages are cryptic
adding full stack trace would help debugging faster.
2018-08-12 00:49:08 +10:00
Tim Brooks 38ec0ff6ca
Make x-pack core pull transport-nio (#32757)
The security nio transports depend on transport-nio. This commit
modifies x-pack core to include the transport-nio jar into the x-pack
core module.
2018-08-10 17:36:29 -06:00
Benjamin Trent b08416b899
Clear Job#finished_time when it is opened (#32605) (#32755)
* Clear Job#finished_time when it is opened (#32605)

* not returning failure when Job#finished_time is not reset

* Changing error log string and source string
2018-08-10 13:52:00 -05:00
Nik Everett f5ba801c6b Test: Only sniff host metadata for node_selectors (#32750)
Our rest testing framework has support for sniffing the host metadata on
startup and, before this change, it'd sniff that metadata before running
the first test. This prevents running these tests against
elasticsearch installations that won't support sniffing like Elastic
Cloud. This change allows tests to only sniff for metadata when they
encounter a test with a `node_selector`. These selectors are the things
that need the metadata anyway and they are super rare. Tests that use
these won't be able to run against installations that don't support
sniffing but we can just skip them. In the case of Elastic Cloud, these
tests were never going to work against Elastic Cloud anyway.
2018-08-10 13:35:47 -04:00
Albert Zaharovits 1dcf80795a
[TEST] Certificate NONE not allowed in FIPS JVM (#32753)
Certificate NONE not allowed when running in a FIPS JVM
2018-08-10 19:37:05 +03:00
Dimitris Athanasiou c7b1ba33aa
[ML] Refactor ProcessCtrl into Autodetect and Normalizer builders (#32720)
This moves the helper functionality for creating the autodetect
and mormalizer processes into corresponding builders.
2018-08-10 17:28:20 +01:00
Alexander Reelsen e49717393a
Tests: Disable rolling upgrade tests with system key on fips JVM (#32775)
This disables the x-pack rolling upgrade tests using a fips JVM, as
there are problems creating the keystore.

Relates #32737
2018-08-10 16:50:11 +02:00
Benjamin Trent 94a9b253db
Adding ML HLRC wrapper and put_job API call (#32726)
* Adding ML HLRC wrapper and put_job API call

* Changing integration test job to have consistent stucture
2018-08-10 07:16:55 -05:00
David Roberts ae0c303dad
Move icu4j and super-csv version numbers to versions file (#32769)
The upcoming ML log structure finder functionality will use these
libraries, and it makes sense to use the same versions that are
being used elsewhere in Elasticsearch.  This is especially true
with icu4j, which is pretty big.
2018-08-10 12:19:06 +01:00
Alexander Reelsen 3fa3af1d11 Tests: Mute test in KerberosRealmTests
Relates #32701
2018-08-10 10:40:37 +02:00
Tim Vernum 93d4f84b21
[TEST] Add "ne" as an unsupported SimpleKdc locale (#32700)
SimpleKdcServer cannot handle some locales, so we explicitly avoid
using them in tests.
Adding "ne" (Nepali) to that list.
2018-08-10 13:36:36 +10:00
Yogesh Gaikwad 0161bbc77f
[Test] Handle file permissions for Windows (#32681)
This commit modifies the test to handle file permission
tests in windows/dos environments. The test requires access
to UserPrincipal and so have modified the plugin-security policy
to access user information.

Closes#32637
2018-08-10 09:55:43 +10:00
Alpar Torok 5bbed5ed9a Merge branch 'master' of github.com:elastic/elasticsearch 2018-08-09 21:58:03 +03:00
Alpar Torok c5e66cb345 mute test #32737 2018-08-09 21:54:58 +03:00
Andrei Stefan 7b618f3db4
SQL: Bug fix for the optional "start" parameter usage inside LOCATE function (#32576)
The incorrect NodeInfo is created when the optional parameter is not used, leading to the incorrect constructor being used. Simplified LocateFunctionProcessorDefinition by using one constructor instead of two.
Fixes https://github.com/elastic/elasticsearch/issues/32554
2018-08-09 19:30:02 +03:00
Andrei Stefan 6750e15773
SQL: Ignore H2 comparative tests for uppercasing/lowercasing string functions (#32604)
Skip the comparative tests using lowercasing/uppercasing against H2 (which considers the Locale).
ES-SQL is, so far, ignoring the Locale.
Still, the same queries are executed against ES-SQL alone and results asserted to be correct.
2018-08-09 19:28:48 +03:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Lee Hinman 7af28c48c3
Switch WritePipelineResponse to AcknowledgedResponse (#32722)
We previously discussed moving the classes extending `AcknowledgedResponse` to
simply use `AcknowledgedResponse`, making the class non-abstract.

This moves the first class to do this, removing `WritePipelineResponse` in the
process.

If we like the way this looks, I will switch the remaining classes over to using
`AcknowledgedResponse`.
2018-08-08 16:21:58 -06:00
Benjamin Trent d586e4cfd3
Adding `Job` and `AnalysisConfig` for HLRC (#32687)
* Adding `Job` and `AnalysisConfig` for HLRC

* Removing println used for local debugging

* Adding null checks and removing unnecessary field
2018-08-08 11:54:11 -05:00
Jim Ferenczi f1869cca35
Fix role query that can match nested documents (#32705)
This change makes sure that the role query excludes nested documents
when it is ran to select the parent documents allowed by the role.
2018-08-08 16:36:51 +02:00
Armin Braun 580d59e2d7
CORE: Upgrade to Jackson 2.8.11 (#32670)
* closes #30352
2018-08-08 12:04:25 +02:00
Tim Vernum 4cbcc1d659 [TEST] Add empty test case for FIPS
All other tests in security:cli are skipped on FIPS JVMs
2018-08-08 13:47:18 +10:00
Tanguy Leroux 2e65bac5dd
[Rollup] Remove builders from RollupJobConfig (#32669) 2018-08-07 18:54:42 +02:00
Benjamin Trent 6d50d8b5a9
Adding job process pojos to protocol pkg (#32657)
* Adding job process pojos to protocol pkg

* Removing unused `RESULTS_FIELD`

* Addressing PR comments, removing unnecessary methods
2018-08-07 10:51:52 -05:00
Albert Zaharovits 1f50950099 Add @AwaitsFix for #32673 2018-08-07 13:22:12 +03:00