Commit Graph

4969 Commits

Author SHA1 Message Date
Martijn van Groningen 29451bb7e3 [TEST] select timestamp differently for test documents
Original commit: elastic/x-pack-elasticsearch@679273012c
2017-01-24 11:07:24 +01:00
David Roberts 215410e93f Rename list to filter (elastic/elasticsearch#774)
Part of the endpoint rename Sophie and Steve agreed with Clint

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@6ded117849
2017-01-24 10:01:24 +00:00
Martijn van Groningen a7d1918461 [TEST] added more logging
Original commit: elastic/x-pack-elasticsearch@062a0b41b8
2017-01-23 17:54:14 +01:00
David Roberts d693f8317c Run build as well as check from CI to bundle Kibana plugin (elastic/elasticsearch#773)
The check task bundles the Elasticsearch part of the plugin but not
Kibana

Fixes elastic/elasticsearch#772

Original commit: elastic/x-pack-elasticsearch@ead2ff3686
2017-01-23 15:48:42 +00:00
Simon Willnauer 1998b7ef46 Remove RequestContext from Security (elastic/elasticsearch#4710)
RequestContext is a leftover from when we had no thread context. This
commit removes the last place where it was used and uses the thread context
instead.

Original commit: elastic/x-pack-elasticsearch@50a2bff400
2017-01-23 15:44:40 +01:00
Colin Goodheart-Smithe db0f4c0977 Fixes projects refs in build (elastic/elasticsearch#771)
The build would fail certain task (e.g. `gradle tasks` and `gradle :prelert-legacy:test`) if run from the elasticsearch root directory because the project references in the build were not abolute project paths from the root, they were relative to the `prelert-legacy` project. This change fixes the references so they are all absolute project paths.

Original commit: elastic/x-pack-elasticsearch@e549533834
2017-01-23 14:09:22 +00:00
Simon Willnauer 8651c0ad9f Log exceptions in onFailure instead of bubbling up (elastic/elasticsearch#4709)
Today we wrap the checked IOException in an unchecked exception when
sending back a failure the security transport interceptor. Yet, if that failure handling
in-turn fails due to a broken response channel we should rather log the exception instead
of bubbling it up since it can have unforeseeable side-effects.

Relates to elastic/elasticsearch#4706

* fix line len

Original commit: elastic/x-pack-elasticsearch@03c3826112
2017-01-23 15:08:48 +01:00
David Roberts cd2332730b Move the named pipe no bootstrap test to a separate qa module (elastic/elasticsearch#769)
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch.  This should make merging to
x-pack easier.

Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.

Original commit: elastic/x-pack-elasticsearch@5547f457b6
2017-01-23 12:08:35 +00:00
Dimitris Athanasiou b3b8a7edc9 Restructure packages (elastic/elasticsearch#767)
Restructure packages according to plan described in elastic/elasticsearch#730.
Copying here for completeness.

* Move config classes
* DataCounts/Quantiles/state are process.autodetect.state
* AutodetectProcessManager move to process.autodetect
* lists -> config
* PageParams, QueryPage -> action.util
* StatusReporter -> process.DataCountsReporter
* CountingInputStream -> process
* JobManager -> job
* CppLog* -> process.logging
* DataProcessor -> collapse into implementation
* job.audit -> ml.notifications

Closes elastic/elasticsearch#730

Original commit: elastic/x-pack-elasticsearch@769ea1ed69
2017-01-23 10:12:21 +00:00
Simon Willnauer b6703c1515 Preserve `forceExecution` flag when forking off handler threads after authentication (elastic/elasticsearch#4706)
Today we might get a rejection on a critical operation where `forceExecution=true` but
due to the fact that the forceExecution flag is not passed to the transport interceptor
interface we were not able to preserve this flag when forking off the request after authentication.
This causes serious issues if for instance a replication handler is rejected.

Closes elastic/elasticsearch#4704

Original commit: elastic/x-pack-elasticsearch@f0aad7dede
2017-01-23 11:07:36 +01:00
Martijn van Groningen 0c40317ed2 Remove unneeded inject annotations, because they are no longer needed since rest controllers have been de-guiced.
Original commit: elastic/x-pack-elasticsearch@d272ae1a1c
2017-01-23 10:05:09 +01:00
polyfractal ae7446f78e De-guice getRestHandlers() to follow upstream changes
Original commit: elastic/x-pack-elasticsearch@04bd9e688f
2017-01-20 17:13:35 -05:00
Jay Modi 9005e9fdb9 security: filter content of known requests with passwords (elastic/elasticsearch#4700)
This commit adds a mechanism for defining known sensitive values in rest bodies so that
these can be filtered when auditing the request body.

Original commit: elastic/x-pack-elasticsearch@d138a6bff7
2017-01-20 14:05:23 -05:00
David Kyle ecd462bf89 Fix bug updating normalised results (elastic/elasticsearch#765)
The bulk request needed resetting after it was executed otherwise stale documents are persisted repeatedly after they have been updated causing a versioning error

Original commit: elastic/x-pack-elasticsearch@263fa9d25d
2017-01-20 17:33:37 +00:00
Nik Everett d690c5f789 Deguice rest handlers (elastic/elasticsearch#4598)
Since some of the rest handlers depend on components constructed
in `createComponents` we use `SetOnce` to save a reference to
the components at the class level and reuse the reference in
`initRestHandlers`. This does require that `initRestHandlers`
is called after `createComponents` but I think that is fairly
reasonable.

Original commit: elastic/x-pack-elasticsearch@4fd87ad911
2017-01-20 11:49:20 -05:00
Colin Goodheart-Smithe 4c6989212a Gets build to use elasticsearch-extras (elastic/elasticsearch#758)
* Gets build to use elasticsearch-extras

Also adds ci script for building repo on CI servers

To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory

* Adds USE_SSH option to ci script

* iter

Original commit: elastic/x-pack-elasticsearch@ea127dfef0
2017-01-20 15:11:21 +00:00
Yannick Welsch 8f70653233 Close XContentParser when executing searches from Watcher (elastic/elasticsearch#4696)
Also removes unused method convertToObject from XContentUtils that does not close XContentParser either.

Original commit: elastic/x-pack-elasticsearch@99ce977c55
2017-01-20 12:40:48 +01:00
Alexander Reelsen 84936d57ad Watcher: Fix XContentFilterKeysUtils when using nested lists (elastic/elasticsearch#4635)
The ability to filter JSON inside of arrays was stopping after the first element was found.

Closes elastic/elasticsearch#4614

Original commit: elastic/x-pack-elasticsearch@452cf1c49d
2017-01-20 12:22:20 +01:00
Alexander Reelsen 8011912ce7 Tests: Added some tests for proper 404 responses on watch actions (elastic/elasticsearch#4620)
Relates elastic/elasticsearch#4616 elastic/elasticsearch#4617

Original commit: elastic/x-pack-elasticsearch@7f73792507
2017-01-20 12:01:44 +01:00
David Roberts bddfac59ed Minor adjustments to datafeed endpoints (elastic/elasticsearch#761)
1. get_datafeeds_stats -> get_datafeed_stats

2. get_datafeeds now accepts implicit _all

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@1fc0f69ee2
2017-01-20 10:57:06 +00:00
David Kyle 2eb0499454 Remove JobDataDeleterFactory and OldDataDeleter (elastic/elasticsearch#759)
Original commit: elastic/x-pack-elasticsearch@ac5b75eb58
2017-01-20 09:49:24 +00:00
Boaz Leskes 630b5fd836 Move ContextPreservingActionListener to core (elastic/elasticsearch#4692)
Original commit: elastic/x-pack-elasticsearch@c3e5762ffc
2017-01-20 10:14:34 +01:00
Martijn van Groningen 9665368755 Changed job lifecycle to be task oriented.
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.

Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.

Original commit: elastic/x-pack-elasticsearch@194a058dd2
2017-01-19 23:15:00 +01:00
Martijn van Groningen f20f56e2e1 fixed compile errors due upstream change
Original commit: elastic/x-pack-elasticsearch@0a9d73d2be
2017-01-19 20:42:44 +01:00
Colin Goodheart-Smithe 3fa87c0994 Removes upload pack task from build (elastic/elasticsearch#757)
* removes upload pack task from build

This is preventing us from being an elasticsearch-extra project and we cannot have this task when we move to x-pack. Once we are in X-Pack the unified build will be uploading the final artifact so for now we will change the CI build to add a build step to upload the pack artifact.

* Removes OS specific stuff from the build

the CPP_LOCAL_DIST will now look for any `ml-cpp` artifacts for the same version in the specified directory.

* review corrections

Original commit: elastic/x-pack-elasticsearch@be15e55ddb
2017-01-19 16:14:08 +00:00
Colin Goodheart-Smithe 55dd438557 Fixes typo on cpp dependencies
Original commit: elastic/x-pack-elasticsearch@1bf51ac6f5
2017-01-19 15:36:45 +00:00
Colin Goodheart-Smithe 62cb7a17c5 Changes build to get c++ lib as a standard dependency (elastic/elasticsearch#756)
Original commit: elastic/x-pack-elasticsearch@d46990da49
2017-01-19 15:22:55 +00:00
Colin Goodheart-Smithe d0b36fd52a Change pack artifact to be more maven-like
Original commit: elastic/x-pack-elasticsearch@e8eed37553
2017-01-19 14:03:40 +00:00
Colin Goodheart-Smithe 33800bae5e Changes build to make cpp artifact download correct with ml-cpp changes (elastic/elasticsearch#754)
https://github.com/elastic/machine-learning-cpp/pull/3 changes the artifact names and paths for the ml-cpp build. This change makes it so the machine learning build references the artifacts in their new location.

Original commit: elastic/x-pack-elasticsearch@d3916b6a7f
2017-01-19 13:51:01 +00:00
Dimitris Athanasiou 0b084ea0e6 Treat timestamps without timezone as UTC (elastic/elasticsearch#753)
Original commit: elastic/x-pack-elasticsearch@33ab2fb781
2017-01-19 13:49:14 +00:00
David Roberts 36bdcaff5d Rename scheduler/scheduled to datafeed (elastic/elasticsearch#755)
Relates elastic/elasticsearch#630

The more subtle changes to the datafeed endpoints required by elastic/elasticsearch#630
are NOT in this commit, as they would be drowned out by the rename

Original commit: elastic/x-pack-elasticsearch@3318971da9
2017-01-19 13:44:19 +00:00
David Roberts 10441a3e38 More endpoint adjustments (elastic/elasticsearch#750)
This commit contains some more of the endpoint changes Sophie and Steve
agreed with Clint:

1. get_jobs_stats renamed to get_job_stats

2. Revert snapshot must now be done using an ID - other options removed

3. Renamed "categorydefinitions" to "categories" in endpoints

4. get_jobs now has an implicit _all if no job ID/wildcard is specified

5. There is an option to retrieve a specific model snapshot by ID in
   get_model_snapshots

Relates elastic/elasticsearch#630

Original commit: elastic/x-pack-elasticsearch@9dd71c64a8
2017-01-19 11:41:35 +00:00
David Kyle e826a56212 Make document Ids unique if in a shared index (elastic/elasticsearch#749)
Original commit: elastic/x-pack-elasticsearch@ecc7e876ce
2017-01-19 09:31:03 +00:00
Martijn van Groningen d3c589c33d Moved waiting for scheduler started logic into StartSchedulerAction.TransportAction and moved the logic that was original there to a new action named InternalStartSchedulerAction.
This change prepares for elastic/elasticsearch/elastic/elasticsearch#22575, where we don't have ClusterService available in rest actions.

Original commit: elastic/x-pack-elasticsearch@87658c7fe8
2017-01-19 09:38:10 +01:00
Daniel Mitterdorfer 6ed83cc8ea Merge pull request elastic/elasticsearch#4380 from danielmitterdorfer/strict-booleans
This PR removes all leniency in the conversion of Strings to booleans: "true" is converted to the boolean value true, "false" is converted to the boolean value false. Everything else raises an error.

Relates elastic/elasticsearchelastic/elasticsearch#22200

Original commit: elastic/x-pack-elasticsearch@a505df1f5d
2017-01-19 08:00:06 +01:00
Ali Beyad 5aae30f722 [TEST] fixes known handler test
Original commit: elastic/x-pack-elasticsearch@98a8f23d23
2017-01-18 16:42:08 -05:00
Dimitris Athanasiou c33f26976d Improve field extraction in scheduler (elastic/elasticsearch#748)
This commit performs the following improvements:

- the time field is always requested as doc_value. This makes
specifying a time format for scheduled jobs unnecessary.
- adds DataDescription as a param to the PostDataAction. When set,
it overrides the job's DataDescription. This allows the scheduler to
override the job's DataDescription since it knows the data format (JSON)
and the time format (epoch_ms). This is not exposed in the REST API to
discourage users from using it.
- by default, data extractor search now requests doc_values for analysis fields. This is
expected to result in increased performance.
- a `_source` field is added to the scheduler config. This needs to be
set to true when one or more of the analysis fields do not have
doc_values.
- the ELASTICSEARCH data format is removed as is now redundant.
- fixes the usage of `script_fields`. Previously, setting
`script_fields` would result to none of the source to be returned. Thus,
is the analysis fields were a mixture of script and non-script fields it
would not work.
- ensures nested fields are handled properly

Closes elastic/elasticsearch#679, Closes elastic/elasticsearch#267 

Original commit: elastic/x-pack-elasticsearch@fed35ed354
2017-01-18 18:46:43 +00:00
Ali Beyad db129051c8 [TEST] fix test to use the stored field instead of id for
stored_template_role

Original commit: elastic/x-pack-elasticsearch@18c9a78587
2017-01-18 12:34:12 -05:00
Ali Beyad 159bf63cd4 [TEST] fix security's request handler tests to check in the
knownHandlers set, not the knownActions

Original commit: elastic/x-pack-elasticsearch@63f3744853
2017-01-18 12:19:42 -05:00
Simon Willnauer 5b5c77f573 Follow up for elastic/elasticsearchelastic/elasticsearch#22677 (elastic/elasticsearch#4670)
This commit picks up some improvments from elastic/elasticsearch#22677 that streamlines correct restore of stored contexts.

Original commit: elastic/x-pack-elasticsearch@0259de50c8
2017-01-18 16:18:10 +01:00
David Kyle 4c0d2a492d Refactor get methods (elastic/elasticsearch#747)
Original commit: elastic/x-pack-elasticsearch@d300be2dde
2017-01-18 13:35:25 +00:00
Daniel Mitterdorfer 226d6872ff Merge remote-tracking branch 'origin/master' into strict-booleans
Original commit: elastic/x-pack-elasticsearch@a81d65f77f
2017-01-18 13:39:52 +01:00
James Gowdy b2917376f0 Merge branch 'master' of github.com:elastic/prelert-legacy
Original commit: elastic/x-pack-elasticsearch@c198cef9d3
2017-01-18 09:53:15 +00:00
Simon Willnauer 533b525b13 Add cross cluster search proxy actions
Original commit: elastic/x-pack-elasticsearch@13f78dffe4
2017-01-18 10:25:40 +01:00
Martijn van Groningen 40332c7e1c use client instead of transport action directly in rest actions
Original commit: elastic/x-pack-elasticsearch@4c3380ceb9
2017-01-17 20:38:53 +01:00
Martijn van Groningen d9a75424d0 fixed wrong mockito import in test
Original commit: elastic/x-pack-elasticsearch@c6a7232a87
2017-01-17 20:22:30 +01:00
David Roberts 449a74b2fd C++ log message handler now remembers C++ process copyright message (elastic/elasticsearch#743)
Once we're in x-pack this (or the portion of it containing the version)
can be returned in the ml feature info of the x-pack info endpoint

Relates elastic/elasticsearch#566

Original commit: elastic/x-pack-elasticsearch@b2ea740a6d
2017-01-17 17:19:48 +00:00
David Kyle cfb94b6627 Rename result iterators (elastic/elasticsearch#740)
Original commit: elastic/x-pack-elasticsearch@c462e9595a
2017-01-17 16:02:02 +00:00
Tim Brooks 17492777a9 Wrap ServerSocket usage in doPrivileged blocks (elastic/elasticsearch#4631)
This is related to elastic/elasticsearch#22116. In the tests there are multiple usages of
server implementations that `accept()` socket connections. To avoid
throwing `SecurityException` when using the `SecurityManager` these
operations must be privileged. Additionally, x-pack:elasticsearch
requires `accept` permission. This was added in the plugin-security
policy file.

Original commit: elastic/x-pack-elasticsearch@057e2abb1f
2017-01-17 09:33:41 -06:00
David Roberts 92c808fd97 Remove C++ code
It now lives in the machine-learning-cpp repository

Original commit: elastic/x-pack-elasticsearch@11ac8212d7
2017-01-17 14:46:11 +00:00