Commit Graph

5099 Commits

Author SHA1 Message Date
Martijn van Groningen 3572cff0a8 [TEST] slightly changed workaround cleanup for ml integ tests.
Original commit: elastic/x-pack-elasticsearch@3eba1e76ab
2017-02-16 14:28:06 +01:00
Martijn van Groningen 55b2111bdf [TEST] wait number of nodes instead of green
Original commit: elastic/x-pack-elasticsearch@d5e47d4e28
2017-02-16 13:54:30 +01:00
Tim Vernum de10d26bfc [Security] Treat LDAP exceptions as auth failures (elastic/x-pack-elasticsearch#565)
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.

Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.

Closes: elastic/x-pack-elasticsearch#564

Original commit: elastic/x-pack-elasticsearch@5282290b00
2017-02-16 22:57:12 +11:00
David Kyle e77a71827d Fix integ test after field rename
Original commit: elastic/x-pack-elasticsearch@4ebb93fb44
2017-02-16 11:19:15 +00:00
David Kyle 26df31a3d0 Rename Job.index_name -> Job.results_index_name (elastic/x-pack-elasticsearch#570)
Original commit: elastic/x-pack-elasticsearch@50fc6515e7
2017-02-16 10:26:38 +00:00
Martijn van Groningen d03c6befd5 [ML] Don't set node attributes when ml is disabled.
Original commit: elastic/x-pack-elasticsearch@d450750f41
2017-02-16 10:52:30 +01:00
Colin Goodheart-Smithe 19fc532961 Restricts certain ml endpoints if license forbids ml (elastic/x-pack-elasticsearch#568)
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:

* Create job
* Create data feed
* Open job
* Start data feed

This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.

The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.

Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
2017-02-16 09:40:53 +00:00
Martijn van Groningen 54d57f6398 [ml] When trying to find a node to open a job take `max_running_jobs` into account instead of just trying and then failing.
Also before even creating a persistent task for the job, check if there ml nodes available to run the job, otherwise fail quickly.

relates elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@63fd72e56a
2017-02-16 09:58:44 +01:00
Zach e441e0dc4e [ML] Fix order that DBQ is constructed
Also use Job.ID constant instead of string literal.

Kudos to @dimitris-athanasiou for this one :)

Original commit: elastic/x-pack-elasticsearch@906b080110
2017-02-15 15:53:45 -05:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
jaymode c8b5be186d remove signing of authentication information
Now that TLS is required for node to node communication, we no longer need to sign the
authentication information to prevent tampering.

Original commit: elastic/x-pack-elasticsearch@1f86cf2395
2017-02-15 12:55:24 -05:00
Ali Beyad b3d72af644 [TEST] bumps rolling upgrade bwc version to 5.4.0-snapshot
Original commit: elastic/x-pack-elasticsearch@2f321682d2
2017-02-15 11:00:11 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Jim Ferenczi 87563e96f2 Fix default path for bwc index
Original commit: elastic/x-pack-elasticsearch@72bcf992f4
2017-02-15 12:44:15 +01:00
Brandon Kobel e39ab93a3d Fixing the x-pack path (elastic/x-pack-elasticsearch#562)
Original commit: elastic/x-pack-elasticsearch@dc311d4a78
2017-02-14 17:16:18 -05:00
Martijn van Groningen 76ea6444cf [test] if enabled always specify setting other randomly specify setting to false.
Original commit: elastic/x-pack-elasticsearch@eacdb979e2
2017-02-14 22:28:43 +01:00
Zachary Tong cb29cbd8a9 [ML] Allow result indices to be shared. (elastic/x-pack-elasticsearch#555)
Essentially an update to https://github.com/elastic/prelert-legacy/pull/736 .  Still does not
default to using shared indices, but adds the capability for two jobs to share the same one
without conflict

Still does not default to using shared indices, just adds the capability for two jobs to share the same one without conflict.

Original commit: elastic/x-pack-elasticsearch@60d93a06ea
2017-02-14 14:59:28 -05:00
Zachary Tong 72ef1acdc0 Move domainSplit injection to ScrollDataExtractor (elastic/x-pack-elasticsearch#554)
By injecting immediately before use, we don't have to store it in the config and return
huge blobs to the user when they request the config

Original commit: elastic/x-pack-elasticsearch@2e53c930ea
2017-02-14 12:26:21 -05:00
Adrien Grand ac8f3ffec4 Fix security source filtering. (elastic/x-pack-elasticsearch#561)
Source filtering is currently written as if the fact that an object is accepted
means that all sub fields and objects are implicitly accepted, which is not
true.

Original commit: elastic/x-pack-elasticsearch@afc4440e7a
2017-02-14 17:44:20 +01:00
Martijn van Groningen 6e411c6118 [ml] Added allocation enabled node attribute setting
This node attribute setting defaults to true when ml is enabled and jobs are only opened on nodes where the allocation enabled node attribute is set to true.

Original commit: elastic/x-pack-elasticsearch@fd3f1b3058
2017-02-14 16:58:08 +01:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Martijn van Groningen 67fe584e0f [ml] Also allow closing failed jobs
Relates to elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@1c814afbf0
2017-02-14 15:38:46 +01:00
Dimitris Athanasiou 4976feb53b [ML] Extract MlInitializationService steps into respective methods (elastic/x-pack-elasticsearch#560)
Original commit: elastic/x-pack-elasticsearch@c43d02e752
2017-02-14 14:05:39 +00:00
David Kyle 27a90f4f0d Remove the job ignore_downtime option (elastic/x-pack-elasticsearch#550)
Original commit: elastic/x-pack-elasticsearch@5a7c4fdfea
2017-02-14 09:43:25 +00:00
Jason Tedor c4be485514 Add support for dispatching bad requests
This commit is a response to an upstream change in Elasticsearch which
added a bad dispatch request handler.

Relates elastic/x-pack-elasticsearch#553

Original commit: elastic/x-pack-elasticsearch@c28baee799
2017-02-13 17:39:44 -05:00
jaymode 82a80dbf42 Update seriailization after backport of content type changes
Original commit: elastic/x-pack-elasticsearch@0ef4dda41d
2017-02-13 13:18:56 -05:00
Martijn van Groningen 5eb8414215 [ml] Specify the minimum supported version,
so that we don't send ml metadata to older nodes.

Original commit: elastic/x-pack-elasticsearch@f196b5c461
2017-02-13 18:15:48 +01:00
Tanguy Leroux a6e67fe51e Monitoring: remove collectors setting elastic/elasticsearch#4957
This commit removes the undocumented, test used only, setting xpack.monitoring.collection.collectors. It should not be used by users at all, and has been initially created for collector tests.

From  elastic/elasticsearch#4957

Original commit: elastic/x-pack-elasticsearch@0fa39a94b5
2017-02-13 16:36:59 +01:00
Martijn van Groningen fbcac2710a [ml] changed start datafeed validation
Original commit: elastic/x-pack-elasticsearch@266d360cae
2017-02-13 14:04:32 +01:00
Dimitris Athanasiou 3e43b591df [ML] Correctly return job stats for _all (elastic/x-pack-elasticsearch#540)
Original commit: elastic/x-pack-elasticsearch@a286ea36e9
2017-02-13 09:10:35 +00:00
Jason Tedor 1b4fec642c Enforce Java version
The Elastic Secrets vault is served via HTTPS with a Let's Encrypt
certificate. The root certificate that cross-signed the Let's Encrypt
certificates were not trusted by the JDK until 8u101. This commit adds a
version check at the start of the build to make it clear the cause of
the underlying issue, and what the fix is.

Relates elastic/x-pack-elasticsearch#541

Original commit: elastic/x-pack-elasticsearch@6bf8076cb6
2017-02-11 12:12:26 -05:00
Ryan Ernst 89f996eed2 Adding new build files after split
Original commit: elastic/x-pack-elasticsearch@ddd5a5bfa6
2017-02-10 11:02:42 -08:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Ryan Ernst ae04627152 Remove references to logstash and kibana builds
Original commit: elastic/x-pack-elasticsearch@da6c2c92dd
2017-02-10 11:02:42 -08:00
Ryan Ernst 1fb742a0ad Rename core plugin and transport dirs
Original commit: elastic/x-pack-elasticsearch@1844685f77
2017-02-10 11:02:42 -08:00
Ryan Ernst 46f7eeaf54 Add migration script to split xpack (elastic/elasticsearch#4949)
* Add migration script

Original commit: elastic/x-pack-elasticsearch@f2ca8ccf2c
2017-02-10 08:00:51 -08:00
jaymode 9f16d5e355 ServerTransportFilterIntegrationTests cannot use mock zen pings
The ServerTransportFilterIntegrationTests cannot use the mock zen pings as the tests rely on the actual sending and
handling of zen pings to test connectivity to the client profile. The server profile test also needs to use the same
type of pings as the test cluster.

Original commit: elastic/x-pack-elasticsearch@fc389921ca
2017-02-10 10:43:09 -05:00
Dimitris Athanasiou 2a904ae581 Merge pull request elastic/elasticsearch#4955 from dimitris-athanasiou/add-datafeed-update-api
Add update datafeed endpoint

Original commit: elastic/x-pack-elasticsearch@57983bbd21
2017-02-10 15:41:42 +00:00
Dimitrios Athanasiou 9c60ee076b Add update datafeed endpoint
- Adds /_xpack/ml/datafeeds/{datafeed_id}/_update

Fixes elastic/elasticsearch#4954

Original commit: elastic/x-pack-elasticsearch@7a1887089d
2017-02-10 15:40:39 +00:00
Jay Modi 51cff2f3e4 SecurityIndexSearcherWrapper should build a filter instead of a query (elastic/elasticsearch#4953)
The SecurityIndexSearcherWrapper was calling toQuery instead of toFilter, which in certain cases can trip the max
clause count check for a boolean query. The same query works fine as a filter and that is what users would expect when
using the query for document level security.

Original commit: elastic/x-pack-elasticsearch@40330636ec
2017-02-10 10:37:35 -05:00
David Roberts 406741c2f8 Load the mlmodel config file from the correct location (elastic/elasticsearch#4958)
There were two problems:

1. The location for the autodetect process was ES_HOME/config/ml
2. The location for the normalize process was its current working
   directory

In both cases the correct location is ES_HOME/config/x-pack

Original commit: elastic/x-pack-elasticsearch@7d8a384fda
2017-02-10 15:19:47 +00:00
Martijn van Groningen 299f9ab74b [ml] Datafeed task improvements
When retrying datafeed task only start datafeed when job is in started state.
Run datafeed task on the node where the job task is running.
Let start datafeed api waits until datafeed state has been set to started.

Original commit: elastic/x-pack-elasticsearch@ebf1d3b9aa
2017-02-10 15:29:55 +01:00
Martijn van Groningen dd33fae50f [ml] [test] changed error handling
Original commit: elastic/x-pack-elasticsearch@75cacd82a4
2017-02-10 15:27:51 +01:00
Adrien Grand 5386beec29 Upgrade to Lucene 6.5.0. (elastic/elasticsearch#4950)
This is a sibling of elastic/elasticsearchelastic/elasticsearch#23087.

Original commit: elastic/x-pack-elasticsearch@ab99fbc874
2017-02-10 15:09:04 +01:00
Jay Modi 16ef39073a Separate the NativeRealmMigrator and the NativeUsersStore (elastic/elasticsearch#4932)
This commit removes the NativeRealmMigrator's dependency on the NativeUserStore and instead directly uses the
InternalClient for the migration operations. There are pros and cons to doing it both ways, but I feel this
method makes it more explicit that this is what the migrator is going to do. The downside here is that there
are two places in the code that need to know the inner details of how we store users.

Additionally, by doing this we avoid a race condition between the NativeUsersStore starting and the
NativeRealmMigrator attempting to get all of the reserved users. This race causes the
OldSecurityIndexBackwardsCompatibility tests to fail intermittently.

Original commit: elastic/x-pack-elasticsearch@6c388db535
2017-02-10 08:17:39 -05:00
Martijn van Groningen b6146e906f Update README.asciidoc
Original commit: elastic/x-pack-elasticsearch@679c5ff71b
2017-02-10 13:49:50 +01:00
Alexander Reelsen 232f3e76a4 Watcher: Refresh watches index after write operation (elastic/elasticsearch#4865)
When a watch gets written by an external process, we should refresh
immediately, so that watches and their state are immediately available,
also via search which is needed for the Watcher UI.

Closes elastic/elasticsearch#4695

Original commit: elastic/x-pack-elasticsearch@8bed60c928
2017-02-10 10:50:32 +01:00
Alexander Reelsen 497147c30d Watcher: Include watch status in history (elastic/elasticsearch#4875)
In order to help watcher UI easily gather the status of a watch
and its actions, we should write the watch status in the history
as well.

This commit is doing exactly that. Tests ensure, that the status is not searchable,
as in the watch mapping itself.

This also requires the history template to be changed, thus the counter of the template
had to be increased as well.

Some minor refactorings by making ctors private that dont need to be public
have been happening as well.

Closes elastic/elasticsearch#4735

Original commit: elastic/x-pack-elasticsearch@df8352ceb5
2017-02-10 08:44:08 +01:00
Jason Tedor be3b5f49d0 Introduce issue migration script
This commit introduces the issue migration script for splitting x-pack
into x-pack-elasticsearch, x-pack-kibana, and x-pack-logstash.

Relates elastic/elasticsearch#4935

Original commit: elastic/x-pack-elasticsearch@33a00e5d06
2017-02-09 17:09:30 -05:00
Ryan Ernst e51b850d75 Build: Make ml credentials load only if necessary (elastic/elasticsearch#4938)
Also, this change allows the credentials to be passed in through
gradle properties, which will allow the unified release to use its own
aws credentials, supplied securely to the build.


Original commit: elastic/x-pack-elasticsearch@62f7a30e59
2017-02-09 14:08:36 -08:00