Commit Graph

162 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe 1da752152a Migrates machine learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@9ad22980b8
2017-02-08 16:58:56 +00:00
Colin Goodheart-Smithe e761b76765 Migrates QA files
Original commit: elastic/x-pack-elasticsearch@ac651f51d4
2017-02-08 16:58:55 +00:00
Simon Willnauer 0e779b41de Followup from elastic/elasticsearchelastic/elasticsearch#23042 (elastic/elasticsearch#4895)
This change accomodates for the renamings done in elastic/elasticsearchelastic/elasticsearch#23042

Original commit: elastic/x-pack-elasticsearch@c290c8ecc4
2017-02-08 14:40:17 +01:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Jay Modi bd04b30acd security: transport always uses TLS (elastic/elasticsearch#4738)
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.

For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.

Closes elastic/elasticsearch#4332

Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
2017-02-07 11:39:31 -05:00
David Kyle 5a1cd69a6a More checkstyle fixes
Original commit: elastic/x-pack-elasticsearch@4c454c5061
2017-02-03 16:58:19 +00:00
Jason Tedor 02bcd9510f Enable disabled rolling upgrade tests
These tests were disabled due to a bug introduced when
sequence-number-based recovery was introduced. This commit enables these
tests again.

Relates elastic/elasticsearch#4801

Original commit: elastic/x-pack-elasticsearch@e555bfec1a
2017-02-03 08:12:52 -05:00
Jay Modi 1f32ef21a2 Update to pass content type from the rest APIs to transport requests (elastic/elasticsearch#4689)
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.

Original commit: elastic/x-pack-elasticsearch@57475fd403
2017-02-02 14:08:34 -05:00
polyfractal 3504608a1e [TEST] more robust regex for tests missing subdomains
Original commit: elastic/x-pack-elasticsearch@28e5d14c22
2017-02-01 11:04:55 -05:00
Zachary Tong a11ddd1e04 Integrate domainSplit function into datafeeds (elastic/elasticsearch#841)
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.

The majority of this PR is actually test-related.  Adds a unit test to check for the injected
script/params.  Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly.  The QA test can
be simplified greatly once the Preview API is added.

Original commit: elastic/x-pack-elasticsearch@c7c35a982c
2017-02-01 10:20:00 -05:00
Ali Beyad 8cab4fec4b [TEST] adds AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@9e8f6ce047
2017-01-31 22:50:17 -05:00
Martijn van Groningen ce6dc4a506 Make job stats api task aware.
This will allow the job stats api to redirect the request to node where job is running.

Original commit: elastic/x-pack-elasticsearch@9f1d12dfcb
2017-01-31 22:35:54 +01:00
Jack Conradson 7f0ecc4b30 Change Namespace for Stored Script to Only Use Id (elastic/elasticsearch#4387)
Changes the behavior in x-pack necessary to support the elasticsearch change elastic/elasticsearch#22206.

Original commit: elastic/x-pack-elasticsearch@916e72e263
2017-01-31 13:28:03 -08:00
Colin Goodheart-Smithe f804bb1917 Removes ensureGreen from PlainlessDomainSplitIT (elastic/elasticsearch#839)
This shouldn’t be needed as the cluster no longer goes red when an index is created.

Original commit: elastic/x-pack-elasticsearch@b554ea9caf
2017-01-31 16:09:25 +00:00
Nik Everett cdfa20b83f Handle moving some classes from reindex to core
Original commit: elastic/x-pack-elasticsearch@cdf98a70f7
2017-01-27 16:08:32 -05:00
Zachary Tong 9395ef81b1 Painless DomainSplit tests in new Single-Node QA Module (elastic/elasticsearch#787)
This contains the Painless-based DomainSplit function, generated static maps and basic tests.  Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response


Original commit: elastic/x-pack-elasticsearch@c6c2942e01
2017-01-27 08:52:48 -05:00
Colin Goodheart-Smithe 603fa47580 Adds an option to disable the ML plugin (elastic/elasticsearch#785)
Adds an `xpack.ml.enabled` node level setting that can be used to enable and disable the plugin. This will be important when we migrate to X-Pack

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@e5c4969a96
2017-01-24 16:14:56 +00:00
Martijn van Groningen 5684caece0 [TEST] print hot threads when op job fails
Original commit: elastic/x-pack-elasticsearch@8820758495
2017-01-24 15:51:37 +01:00
David Roberts cd2332730b Move the named pipe no bootstrap test to a separate qa module (elastic/elasticsearch#769)
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch.  This should make merging to
x-pack easier.

Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.

Original commit: elastic/x-pack-elasticsearch@5547f457b6
2017-01-23 12:08:35 +00:00
Colin Goodheart-Smithe 4c6989212a Gets build to use elasticsearch-extras (elastic/elasticsearch#758)
* Gets build to use elasticsearch-extras

Also adds ci script for building repo on CI servers

To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory

* Adds USE_SSH option to ci script

* iter

Original commit: elastic/x-pack-elasticsearch@ea127dfef0
2017-01-20 15:11:21 +00:00
Martijn van Groningen 9665368755 Changed job lifecycle to be task oriented.
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.

Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.

Original commit: elastic/x-pack-elasticsearch@194a058dd2
2017-01-19 23:15:00 +01:00
Ali Beyad db129051c8 [TEST] fix test to use the stored field instead of id for
stored_template_role

Original commit: elastic/x-pack-elasticsearch@18c9a78587
2017-01-18 12:34:12 -05:00
Lee Hinman a75fecb472 flush after deleting docs
Original commit: elastic/x-pack-elasticsearch@1cd245c398
2017-01-10 14:26:21 -07:00
Lee Hinman 760c960c98 Revert "Revert "Add a qa/rolling-upgrade test that does single-document index and deletes""
This reverts commit elastic/x-pack@94b65dcf52.

Original commit: elastic/x-pack-elasticsearch@abd90463f8
2017-01-10 12:12:09 -07:00
Lee Hinman 05f72c0b91 Revert "Add a qa/rolling-upgrade test that does single-document index and deletes"
This reverts commit elastic/x-pack@ff05f28b7e.

Original commit: elastic/x-pack-elasticsearch@a05fc1f9e6
2017-01-10 11:19:15 -07:00
Lee Hinman 320ec6716d Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@23761f3e16
2017-01-09 16:27:42 -07:00
Lee Hinman 45e84cde6e Add PreventFailingBuildIT in qa/smoke-test-watcher
The watcher tests were recently marked with `@Network`, which prevents them from
normally being run. Unfortunately, this means no tests run by default and the
entire suite fails.

Original commit: elastic/x-pack-elasticsearch@40cfc75b26
2017-01-09 15:59:32 -07:00
Lee Hinman 8326b6d83b Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@98f4e74d2e
2017-01-09 14:22:18 -07:00
Jay Modi e0f0b4b7b8 rename the kibana role to kibana_system
This commit renames the kibana role to kibana_system and provides a backwards compatibility
layer so that kibana access still works properly during a rolling upgrade.

Closes elastic/elasticsearch#4525

Original commit: elastic/x-pack-elasticsearch@5c5796e53a
2017-01-09 16:06:50 -05:00
Lee Hinman 93720505b8 Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@089fa9977d
2017-01-09 11:39:37 -07:00
Lee Hinman 7387d04139 Add a qa/rolling-upgrade test that does single-document index and deletes
Original commit: elastic/x-pack-elasticsearch@5850439b22
2017-01-09 11:16:33 -07:00
Tanguy Leroux a86b112f1e [Watcher] Delete JIRA issues after integration tests execution (elastic/elasticsearch#4552)
This commit delete the JIRA issues after the integration test execution. All issues from the testing project XWT are deleted, even if they have not been created during this specific test execution.

closes elastic/elasticsearch#4535

Original commit: elastic/x-pack-elasticsearch@0362463633
2017-01-09 10:41:48 +01:00
Tanguy Leroux 19cbab4ac3 [Watcher] Update triggering interval in Jira integration tests
In Jira integration tests, some watches are triggered every second whereas they are executed using the watch execute API. This commit increases the triggering interval to 1d so that the watches are not executed on slow machines.

Original commit: elastic/x-pack-elasticsearch@4d0462bc00
2017-01-05 21:00:53 +01:00
Tanguy Leroux 3d2d1d49b6 [Watcher] Move Jira integration tests to smoke-test-watcher (elastic/elasticsearch#4534)
This commit moves the Jira rest integration tests from the smoke-test-watcher-with-mustache project to the smoke-test-watcher project.

Original commit: elastic/x-pack-elasticsearch@c6b03d557f
2017-01-05 15:22:59 +01:00
Alexander Reelsen 63f4bbba98 Watcher: Use Apache HttpClient for internal Watcher HttpClient (elastic/elasticsearch#4434)
Watcher: Use Apache HttpClient for internal Watcher HttpClient

The current implementation based on URLConnection has several drawbacks.

* If server returned HTTP header but then got stuck, no timeout would help, the connection remained stuck
* GET requests with a body were not supported, the method was silently changed to POST
* More complex handling of input/error stream handling, the body could not be read from a single input stream

NOTE: This is a BWC breaker. From now on every part of the URL needs to be encoded properly before it is configured in the requeust builder. This requires an upgrade of all watches.

Closes elastic/elasticsearch#1141

Original commit: elastic/x-pack-elasticsearch@bbc8f85dd8
2017-01-05 14:25:58 +01:00
Tanguy Leroux 125a2c9c03 [Watcher] Enable Jira integration tests (elastic/elasticsearch#4331)
This commit enables the Jira integration tests with the Jira project and account provided by Edward Sy.

closes elastic/infraelastic/elasticsearch#1498

Original commit: elastic/x-pack-elasticsearch@78d1005064
2017-01-05 13:44:04 +01:00
Nik Everett 93b16f0615 Fix another qa project
Original commit: elastic/x-pack-elasticsearch@9609036dde
2017-01-05 10:56:09 +01:00
Nik Everett f2ae490b32 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integTest only projects that is what we want.

Original commit: elastic/x-pack-elasticsearch@f576dfdfbb
2017-01-05 10:56:09 +01:00
Nik Everett 5b6bfffa9a Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.

Also fix some license headers.

Original commit: elastic/x-pack-elasticsearch@3d5549d170
2017-01-05 10:56:09 +01:00
Jay Modi 18a2cf23d4 Build a single role that represents a user's permissions (elastic/elasticsearch#4449)
This PR changes how we use roles and how we look at the roles of a user. Previously we looked up each role individually, parsed each into their own `Role` object, and had a wrapper that essentially served as an iterator over the roles. The same pattern was also followed for the permissions that composed a role (ClusterPermission, IndicesPermission, and RunAsPermission). This resulted in a lot of code that was hard to follow and could be inefficient. 

Now, we look up the roles for a user in bulk and only get the RoleDescriptor for each role. Once all role descriptors have been retrieved, we build a single Role that represents the user's permissions and we also cache this combination for better performance as authorization can happen many times for a single top level request as we authorize the top level request and any sub requests, which could be a large number in the case of shard requests.

This change also enabled a large cleanup of our permission and privilege classes, which should reduce the footprint of what needs to be followed. Some of the notable changes are:

* Consolidation of GeneralPrivilege and AbstractAutomatonPrivilege into the Privilege class
* The DefaultRole class has been removed and the permissions it provided were moved into the AuthorizationService
* The GlobalPermission class was removed as there is a single role that represents a user's permissions
* The Global inner classes for the various permissions were removed
* The Core inner class was removed and ClusterPermission, IndexPermission, RunAsPermission became final classes instead of interfaces
* The Permission interface has been removed. The isEmpty() method defined by this interface is not needed as we can simply evaluate the permission to get the same effect
* The ClusterPermission#check method only takes the action name again
* The AutomatonPredicate class was removed and replaced by Automatons#predicate
* IndicesAccessControl objects no longer need to be merged when evaluating permissions
* MergedFieldPermissions has been removed
* The Name class that was used to hold an array of strings has been removed and replaced with the use of a Set
* Privilege resolution is more efficient by only combining automata once

Other items:
* NativeRolesStore no longer does caching, so the RoleAndVersion class could be removed
* FileRolesStore doesn't need to be an AbstractLifecycleComponent

Relates elastic/elasticsearch#4327

Original commit: elastic/x-pack-elasticsearch@c1901bc82e
2016-12-30 09:27:49 -05:00
Nik Everett 4ff6279865 Remove exception from client suite
Original commit: elastic/x-pack-elasticsearch@51e94561d9
2016-12-22 09:10:42 -05:00
Areek Zillur 84db1b8731 x-pack changes for elasticsearchelastic/elasticsearch#21964
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).

This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.

Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
2016-12-22 02:35:35 -05:00
jaymode 880808c428 test: disable security manager for custom realm integration tests
Original commit: elastic/x-pack-elasticsearch@4bcd7675b2
2016-12-21 16:05:43 -05:00
jaymode 9288401767 test: do not use security manager for client tests
Original commit: elastic/x-pack-elasticsearch@1200a1b757
2016-12-21 14:46:18 -05:00
Ryan Ernst 789df7d2fa Internal: Update signatures for EnvironmentAwareCommand (elastic/elasticsearch#4367)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#22175

Original commit: elastic/x-pack-elasticsearch@4359cb1947
2016-12-19 15:23:53 -08:00
Ryan Ernst 6d4d599f91 Build: Convert xplugins to use new extra projects setup (elastic/elasticsearch#4175)
* Build: Convert xplugins to use new extra projects setup

This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.

The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.

Original commit: elastic/x-pack-elasticsearch@09398aea5a
2016-12-14 15:02:28 -08:00
Alexander Reelsen b57c4f6ebe Watcher: Remove in memory watch store (elastic/elasticsearch#4201)
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.

Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when

* Put
* Get
* Ack
* Activate/Deactivate
* Execute

Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569

Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.

Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.

Original commit: elastic/x-pack-elasticsearch@c47465b47c
2016-12-13 08:54:03 +01:00
Jay Modi ac34c3c37f Remove deprecated methods from the Realm class
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.

Original commit: elastic/x-pack-elasticsearch@cff21e21ee
2016-12-09 07:28:11 -05:00
jaymode 2b7c03848f test: disable xpack plugins on tribe client nodes
The license test for the tribe node disables xpack features but these settings
are not passed to the tribe client nodes, so they need to manually be added.

Original commit: elastic/x-pack-elasticsearch@7273817c47
2016-12-03 14:02:55 -05:00
Simon Willnauer 7f17896927 Revert "Followup for elastic/elasticsearchelastic/elasticsearch#21915 - removal of legacy BWC test infrastructure (elastic/elasticsearch#4247)"
This reverts commit elastic/x-pack@c6c0ffd5d9.

Original commit: elastic/x-pack-elasticsearch@8b7386fb99
2016-12-02 10:55:46 +01:00