Commit Graph

5020 Commits

Author SHA1 Message Date
Dimitris Athanasiou 3e43b591df [ML] Correctly return job stats for _all (elastic/x-pack-elasticsearch#540)
Original commit: elastic/x-pack-elasticsearch@a286ea36e9
2017-02-13 09:10:35 +00:00
Jason Tedor 1b4fec642c Enforce Java version
The Elastic Secrets vault is served via HTTPS with a Let's Encrypt
certificate. The root certificate that cross-signed the Let's Encrypt
certificates were not trusted by the JDK until 8u101. This commit adds a
version check at the start of the build to make it clear the cause of
the underlying issue, and what the fix is.

Relates elastic/x-pack-elasticsearch#541

Original commit: elastic/x-pack-elasticsearch@6bf8076cb6
2017-02-11 12:12:26 -05:00
Ryan Ernst 89f996eed2 Adding new build files after split
Original commit: elastic/x-pack-elasticsearch@ddd5a5bfa6
2017-02-10 11:02:42 -08:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Ryan Ernst ae04627152 Remove references to logstash and kibana builds
Original commit: elastic/x-pack-elasticsearch@da6c2c92dd
2017-02-10 11:02:42 -08:00
Ryan Ernst 1fb742a0ad Rename core plugin and transport dirs
Original commit: elastic/x-pack-elasticsearch@1844685f77
2017-02-10 11:02:42 -08:00
Ryan Ernst 46f7eeaf54 Add migration script to split xpack (elastic/elasticsearch#4949)
* Add migration script

Original commit: elastic/x-pack-elasticsearch@f2ca8ccf2c
2017-02-10 08:00:51 -08:00
jaymode 9f16d5e355 ServerTransportFilterIntegrationTests cannot use mock zen pings
The ServerTransportFilterIntegrationTests cannot use the mock zen pings as the tests rely on the actual sending and
handling of zen pings to test connectivity to the client profile. The server profile test also needs to use the same
type of pings as the test cluster.

Original commit: elastic/x-pack-elasticsearch@fc389921ca
2017-02-10 10:43:09 -05:00
Dimitris Athanasiou 2a904ae581 Merge pull request elastic/elasticsearch#4955 from dimitris-athanasiou/add-datafeed-update-api
Add update datafeed endpoint

Original commit: elastic/x-pack-elasticsearch@57983bbd21
2017-02-10 15:41:42 +00:00
Dimitrios Athanasiou 9c60ee076b Add update datafeed endpoint
- Adds /_xpack/ml/datafeeds/{datafeed_id}/_update

Fixes elastic/elasticsearch#4954

Original commit: elastic/x-pack-elasticsearch@7a1887089d
2017-02-10 15:40:39 +00:00
Jay Modi 51cff2f3e4 SecurityIndexSearcherWrapper should build a filter instead of a query (elastic/elasticsearch#4953)
The SecurityIndexSearcherWrapper was calling toQuery instead of toFilter, which in certain cases can trip the max
clause count check for a boolean query. The same query works fine as a filter and that is what users would expect when
using the query for document level security.

Original commit: elastic/x-pack-elasticsearch@40330636ec
2017-02-10 10:37:35 -05:00
David Roberts 406741c2f8 Load the mlmodel config file from the correct location (elastic/elasticsearch#4958)
There were two problems:

1. The location for the autodetect process was ES_HOME/config/ml
2. The location for the normalize process was its current working
   directory

In both cases the correct location is ES_HOME/config/x-pack

Original commit: elastic/x-pack-elasticsearch@7d8a384fda
2017-02-10 15:19:47 +00:00
Martijn van Groningen 299f9ab74b [ml] Datafeed task improvements
When retrying datafeed task only start datafeed when job is in started state.
Run datafeed task on the node where the job task is running.
Let start datafeed api waits until datafeed state has been set to started.

Original commit: elastic/x-pack-elasticsearch@ebf1d3b9aa
2017-02-10 15:29:55 +01:00
Martijn van Groningen dd33fae50f [ml] [test] changed error handling
Original commit: elastic/x-pack-elasticsearch@75cacd82a4
2017-02-10 15:27:51 +01:00
Adrien Grand 5386beec29 Upgrade to Lucene 6.5.0. (elastic/elasticsearch#4950)
This is a sibling of elastic/elasticsearchelastic/elasticsearch#23087.

Original commit: elastic/x-pack-elasticsearch@ab99fbc874
2017-02-10 15:09:04 +01:00
Jay Modi 16ef39073a Separate the NativeRealmMigrator and the NativeUsersStore (elastic/elasticsearch#4932)
This commit removes the NativeRealmMigrator's dependency on the NativeUserStore and instead directly uses the
InternalClient for the migration operations. There are pros and cons to doing it both ways, but I feel this
method makes it more explicit that this is what the migrator is going to do. The downside here is that there
are two places in the code that need to know the inner details of how we store users.

Additionally, by doing this we avoid a race condition between the NativeUsersStore starting and the
NativeRealmMigrator attempting to get all of the reserved users. This race causes the
OldSecurityIndexBackwardsCompatibility tests to fail intermittently.

Original commit: elastic/x-pack-elasticsearch@6c388db535
2017-02-10 08:17:39 -05:00
Martijn van Groningen b6146e906f Update README.asciidoc
Original commit: elastic/x-pack-elasticsearch@679c5ff71b
2017-02-10 13:49:50 +01:00
Alexander Reelsen 232f3e76a4 Watcher: Refresh watches index after write operation (elastic/elasticsearch#4865)
When a watch gets written by an external process, we should refresh
immediately, so that watches and their state are immediately available,
also via search which is needed for the Watcher UI.

Closes elastic/elasticsearch#4695

Original commit: elastic/x-pack-elasticsearch@8bed60c928
2017-02-10 10:50:32 +01:00
Alexander Reelsen 497147c30d Watcher: Include watch status in history (elastic/elasticsearch#4875)
In order to help watcher UI easily gather the status of a watch
and its actions, we should write the watch status in the history
as well.

This commit is doing exactly that. Tests ensure, that the status is not searchable,
as in the watch mapping itself.

This also requires the history template to be changed, thus the counter of the template
had to be increased as well.

Some minor refactorings by making ctors private that dont need to be public
have been happening as well.

Closes elastic/elasticsearch#4735

Original commit: elastic/x-pack-elasticsearch@df8352ceb5
2017-02-10 08:44:08 +01:00
Jason Tedor be3b5f49d0 Introduce issue migration script
This commit introduces the issue migration script for splitting x-pack
into x-pack-elasticsearch, x-pack-kibana, and x-pack-logstash.

Relates elastic/elasticsearch#4935

Original commit: elastic/x-pack-elasticsearch@33a00e5d06
2017-02-09 17:09:30 -05:00
Ryan Ernst e51b850d75 Build: Make ml credentials load only if necessary (elastic/elasticsearch#4938)
Also, this change allows the credentials to be passed in through
gradle properties, which will allow the unified release to use its own
aws credentials, supplied securely to the build.


Original commit: elastic/x-pack-elasticsearch@62f7a30e59
2017-02-09 14:08:36 -08:00
Zachary Tong dc07b593b7 [ML] Support job deletion from multiple indices (elastic/elasticsearch#4918)
This extends the DBQ to delete from a pattern, rather than a specific index.  Once shared/rollover
indices are implemented, this will be capable of purging results from the matching set.

Original commit: elastic/x-pack-elasticsearch@4ec0944173
2017-02-09 15:37:37 -05:00
Pier-Hugues Pellerin 04a4c816bd Merge pull request elastic/elasticsearch#4936 from ph/fix/append-snapshot-to-artifact
Make sure we add the right suffix to the generated artifact

Original commit: elastic/x-pack-elasticsearch@9984c69c99
2017-02-09 15:20:53 -05:00
Pier-Hugues Pellerin 4bd3eb6bf1 Merge pull request elastic/elasticsearch#4930 from ph/fix/logstash-remove-snapshot
Remove the `SNAPSHOT` suffix in the VERSION file

Original commit: elastic/x-pack-elasticsearch@6c3b609525
2017-02-09 15:14:46 -05:00
Pier-Hugues Pellerin e7b786734d Merge pull request elastic/elasticsearch#4928 from ph/fix/impose-strict-versioning-on-logstash-core
Logstash's x-pack should have a strict dependency

Original commit: elastic/x-pack-elasticsearch@51f62a6f0d
2017-02-09 15:09:32 -05:00
Colin Goodheart-Smithe 56e983b5f3 After auth with Vault retry until can see bucket (elastic/elasticsearch#4912)
After authenticating with vault it can take time for the credentials to be propagated by the AWS API. previously we would just blindly wait for 10 seconds and then try to continue. This change introduces a retry loop where we will do a `headBucket` request every 0.5 seconds until the bucket is accessible or until we have tried 15 times. This means the build is only held up for at most 0.5 seconds after the bucket is accessible.

This is a step towards the final solution since  the authentication with vault still happens on every build in the configuraiton phase. A subsequent change will be made to move this out of the configuration phase so that it only runs when the dependencies are required.

Original commit: elastic/x-pack-elasticsearch@ab3abba1ea
2017-02-09 15:28:46 +00:00
Martijn van Groningen c6763489d5 Moved job lifecycle over to persistent tasks
Also replaced the DELETING status from JobState with a boolean flag on Job. The state of a job is now stored inside a persistent task in cluster state. Jobs that aren't running don't have a persistent task, so I moved that notion of being deleted to the job config itself.

Original commit: elastic/x-pack-elasticsearch@b0ed82124d
2017-02-09 16:20:34 +01:00
Colin Goodheart-Smithe af2486b834 Adds named pipes access for windows to security policy (elastic/elasticsearch#4927)
Windows requires named pipes to be created at a specific location so we need to allow access to this special location in the security policy for the machine learning plugin to run on X-Pack

Original commit: elastic/x-pack-elasticsearch@3079e88713
2017-02-09 15:09:31 +00:00
David Roberts de5e65b687 Allow github.token to work on Windows (elastic/elasticsearch#4925)
Original commit: elastic/x-pack-elasticsearch@95c8b79af1
2017-02-09 14:09:51 +00:00
Jay Modi 8c2ce2c504 [Monitoring] _xpack/monitoring/_bulk action support content stream (elastic/elasticsearch#4916)
This commit marks the monitoring rest bulk action as supporting a content stream. This endpoint takes the same format as the normal bulk endpoint so we need to also accept the newline delimited JSON content type header here.

Closes elastic/elasticsearch#4913

Original commit: elastic/x-pack-elasticsearch@a312cd1256
2017-02-09 06:27:52 -05:00
Martijn van Groningen 5185e06631 [Watcher] Removed not needed usages of `com.google.common.*` classes.
Original commit: elastic/x-pack-elasticsearch@a8dea17a90
2017-02-09 10:40:30 +01:00
Alexander Reelsen 2838946d8b Watcher: Add REST test for execute watch API
to ensure that a 404 is returned, when the watch does not exist.

Relates elastic/elasticsearch#4919

Original commit: elastic/x-pack-elasticsearch@494cd50804
2017-02-09 09:20:54 +01:00
Colin Goodheart-Smithe fe933b027e temporarily increase vault wait
This is temporary fix to allow more time for the AWS credentials to be progagated by the AWS API. Following this we will implement a retry mechanism to keep trying to find the dependency for an amount of time

Original commit: elastic/x-pack-elasticsearch@a300b8698b
2017-02-08 18:15:14 +00:00
Colin Goodheart-Smithe 2f5865756b Migrates Machine Learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@611f3590bb
2017-02-08 17:28:40 +00:00
Colin Goodheart-Smithe b26594cf00 Adds post-migration Elasticsearch changes
Original commit: elastic/x-pack-elasticsearch@f842107076
2017-02-08 16:58:57 +00:00
Colin Goodheart-Smithe 1da752152a Migrates machine learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@9ad22980b8
2017-02-08 16:58:56 +00:00
Colin Goodheart-Smithe e761b76765 Migrates QA files
Original commit: elastic/x-pack-elasticsearch@ac651f51d4
2017-02-08 16:58:55 +00:00
Colin Goodheart-Smithe 1b2381d355 Migrates Elasticsearch files
Original commit: elastic/x-pack-elasticsearch@370af88d14
2017-02-08 16:58:55 +00:00
Colin Goodheart-Smithe fa7a82a945 Removes files no longer needed
Original commit: elastic/x-pack-elasticsearch@8f197075a3
2017-02-08 16:58:55 +00:00
Martijn van Groningen 14a677396e [TEST] Reduce size of large documents to be more heap memory friendly in xpack build
Original commit: elastic/x-pack-elasticsearch@d3864a5021
2017-02-08 17:45:50 +01:00
Alexander Reelsen 1ba5f8fb30 Watcher: Enable with STANDARD license (elastic/elasticsearch#4838)
In order to have monitoring use alerts on cloud, we need to
enable watcher when the STANDARD license is in place. For
more information, please refer to the referenced issue.

Closes elastic/elasticsearch#4766

Original commit: elastic/x-pack-elasticsearch@9d5547274c
2017-02-08 17:38:00 +01:00
Zachary Tong 1591003c7d Fix rest of mocking issues, remove awaitsFix
Original commit: elastic/x-pack-elasticsearch@d5e876e867
2017-02-08 10:57:17 -05:00
David Kyle 6e929fb290 Fix test
Original commit: elastic/x-pack-elasticsearch@0e656d8906
2017-02-08 15:52:23 +00:00
Zachary Tong 99ce9be6ca Fix (some) mocking issues due to upstream changes.
Two suites marked as awaitsFix while being worked on.

Original commit: elastic/x-pack-elasticsearch@06eb352b1e
2017-02-08 10:36:15 -05:00
Colin Goodheart-Smithe 30b6425b3a Convert ml-cpp repo to be part of elasticsearch-extra (elastic/elasticsearch#890)
This means we can reference the local build from within the prelert-legacy build script and build it directly

Original commit: elastic/x-pack-elasticsearch@14024841ab
2017-02-08 14:39:22 +00:00
Zachary Tong 91883ad57b Upstream fixes: use getter methods instead of (now) private members
Original commit: elastic/x-pack-elasticsearch@80786e4f84
2017-02-08 09:37:10 -05:00
David Kyle 9dc4a2f31c Online updates to the running autodetect process (elastic/elasticsearch#886)
* Methods to update the running process with new settings

* Task to update the running autodetect process

* Don’t start process update task if not config specified

Original commit: elastic/x-pack-elasticsearch@4364b141b5
2017-02-08 14:19:24 +00:00
Simon Willnauer 0e779b41de Followup from elastic/elasticsearchelastic/elasticsearch#23042 (elastic/elasticsearch#4895)
This change accomodates for the renamings done in elastic/elasticsearchelastic/elasticsearch#23042

Original commit: elastic/x-pack-elasticsearch@c290c8ecc4
2017-02-08 14:40:17 +01:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Boaz Leskes aadbe81767 TransportService.connectToNode should validate remote node ID (elastic/elasticsearch#4866)
companion PR for elastic/elasticsearchelastic/elasticsearch#22828

Original commit: elastic/x-pack-elasticsearch@2465a03ebc
2017-02-07 22:11:54 +02:00