"cloud_internal" license type enables dynamically updating license operation mode via a config file.
When the installed license is "cloud_internal", the node level operation mode can be updated by writing
a `license_mode` file in the x-pack config directory (config/x-pack/license_mode). The file is expected
to have a string representing the desired license mode (e.g. "gold", "basic"). In case of a failure to
read a valid license mode from the `license_mode` file, the operation mode for "cloud_internal" license
defaults to PLATINUM.
This change also ensures that the correct operation mode is reported via the _xpack endpoint.
closeselastic/elasticsearch#2042
Original commit: elastic/x-pack-elasticsearch@6a2d788e45
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#19440. It adds a
getter on XPackExtension for extensions that add custom rest headers, in
addition to the headers registered for xpack itself.
Original commit: elastic/x-pack-elasticsearch@bd142b88c6
This commit moves an awaits fix from
MonitoringIndexNameResolverTestCase#testResolver to
MonitoringIndexNameResolverTestCase#testSource as the previous commit
elastic/x-pack@98e76642ea marked the incorrect test as
awaits fix.
Original commit: elastic/x-pack-elasticsearch@62e9aaa628
The InternalClient is used by xpack code making requests to other nodes,
to add the xpack user to the thread context. To do this, it uses has a
dependency on the AuthenticationService, which in turn transitively
depends on InternalClient (eg IndexAuditTrail). But to add the xpack
user, the full authentication service is not necessary. Only having the
crypto service is needed in order to encrypt the header.
This change simplifes construction of InternalClient both making it a
real class instead of an interface, and removing the dependency on the
AuthenticationService. It also removes the use of
Provider<InternalClient> in users of the client.
Original commit: elastic/x-pack-elasticsearch@10f633bdf3
Both logfile and index audit trails currently depend on injection of
Transport in order to find the bound address of the local node. However,
the ClusterService provides access to information about the local node,
including the bound addresses. This change makes the audit trails use
the cluster service, and also makes the logging audit trail not use a
lifecycle.
Original commit: elastic/x-pack-elasticsearch@d747d64ee1
This provides the same -- more useful -- index memory chart to the index page, instead of just "Lucene Memory", it now breaks down what it shows just like the Node page as as more generalized "Index Memory" chart. It also rolls Fielddata into the Index Memory. With Fielddata now in the Index Memory chart, I decided to add the Segment Count chart to the Index page.
Original commit: elastic/x-pack-elasticsearch@4e3490ce9c
elastic/elasticsearchelastic/elasticsearch#19428 removes `node.mode` and `node.local` this PR
fixes xplugins to configure networking explicitly.
Original commit: elastic/x-pack-elasticsearch@ee8daa36dd
This change removes some of the complexity around licensing classes in
xpack. It removes the interfaces for registration and management so the
remaining LicensesService class is the thing that components wanting to
interact with the license should use. It also removes complexity around
the Licensee interface, removing generics and the registration at
construction time, as well as making the licensees no longer have a
lifecycle. There is still more to be done with simplification of license
classes construction, but this is a step towards a simpler world.
Original commit: elastic/x-pack-elasticsearch@5307d67b5b
This makes use of the registerAsDeprecatedHandler method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a Warning header for anyone using HTTP clients (though they have to be looking for it...).
- This also adds deprecated `/_licenses` variants of the endpoint. Users are consistently making that mistake, and it's easy enough to support this way. Can remove it if people disagree though.
License portion only
Original commit: elastic/x-pack-elasticsearch@431c871fcf
This makes use of the registerAsDeprecatedHandler method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a Warning header for anyone using HTTP clients (though they have to be looking for it...).
- This also changes from PUT _start, _restart, _stop (Watcher endpoints) to POST _start, _restart, _stop
- The deprecated variant still honors PUT
- Nothing about the hijack endpoints was deprecated because they did not change from 2.x
Watcher portion only
Original commit: elastic/x-pack-elasticsearch@36f87a6526
When the monitoring tests are run in isolation, they succeed. However, when the whole suite of
REST tests is being run at the same time, the "Bulk indexing of monitoring data" intermittently
fails with a timeout. Therefore, a timeout of 60 seconds has been added.
Closeselastic/elasticsearch#2809
Original commit: elastic/x-pack-elasticsearch@d11dc7a2be
This particular test requires Elasticsearch to run on port 9400, which is not
guaranteed if the clients run their own tests, as it is a matter of configuration
in gradle.
Therefore these tests need to run in their project.
Original commit: elastic/x-pack-elasticsearch@da38407766
The callback replaces the ability to fully replace the http client instance. By doing that, one used to lose any default that the RestClient had set for the underlying http client. Given that you'd usually override one or two things only, like a couple of timeout values, the ssl factory or the default credentials providers, it is not uder friendly if by doing that users end up replacing the whole http client instance and lose any default set by us.
Original commit: elastic/x-pack-elasticsearch@03adca6f62
Currently each node monitors the cluster state for a license, and if it
does not find one, it sends a request to the master to generate a trial
license. However, the master node has this same logic. Since the master
node is the only thing that can change the cluster state, we know that
once some node becomes master, it will notice the lack of license,
generate a trial license, and send a cluster state update. The trigger
from every node telling the master to generate the trial license is not
needed.
This change removes the register_trial_license action that the non
master nodes used. It removes the need for the TransportService in the
LicensesService, which will help with deguicing.
Original commit: elastic/x-pack-elasticsearch@a71656847e
This is the first deguicing for xplugins to use the new
createComponents(). The removal was very straightforward. One thing to
note is HttpAuthFactory only has one implementation (basic auth), but I
kept the registry and such for now. Also, HttpRequest.Parser is only
used in 2 tests, not at all in main code, it should probably be removed.
Original commit: elastic/x-pack-elasticsearch@839e7e4900
Since the "Should throw 404 after license deletion" test deletes the license,
a subsequent test run fails for many tests in other components.
This patch adds a `teardown` section to install back a trial development license.
Closeselastic/elasticsearch#2782
Original commit: elastic/x-pack-elasticsearch@45ec5d4244
This change creates the CryptoService during xpack plugin construction,
and also collapses the only implementation of CryptoService into a
class instead of an interface.
Original commit: elastic/x-pack-elasticsearch@7f00a5d6ef
The delete test has been relying on the license already being installed
in the cluster from the other test. However, many runners run tests
in arbitrary order, so it has been failing.
This patch adds a "Trial Development" license as the first action
in the test.
Related: elastic/elasticsearch#2777Closeselastic/elasticsearch#2778
Original commit: elastic/x-pack-elasticsearch@cd198cf67a
The actions in the `do` sections of tests definitions have not been indented, which
confused the Ruby YAML parser.
Also, the calls to `license.*` have not been prefixed with the `xpack` API.
Related: elastic/elasticsearch#2777
Original commit: elastic/x-pack-elasticsearch@b2da7d2703
The check is failing, because even when the document is deleted before
the test, the version of the document in Elasticsearch sticks around
for a while.
I have changed the test to use the `is_true` operator to check for
the presence of the field only.
Original commit: elastic/x-pack-elasticsearch@308ffe34f0
The shorthand format has been failing in the Ruby runner. Not sure if
this is particular to the Ruby YAML library, but I think there's no
harm in making the syntax standard.
Original commit: elastic/x-pack-elasticsearch@d753cad64a
This changes the xpack.monitoring.agent.* settings so that they fall under xpack.monitoring.collection.*.
- This also changes Kibana's equivalent data_collection to collection.
Original commit: elastic/x-pack-elasticsearch@a69bc18055
Added missing IOException and simplified filtering out mock transports
for security integration tests.
Relates to elastic/elasticseachelastic/elasticsearch#19332
Original commit: elastic/x-pack-elasticsearch@8a5c015c3d
The SecretService implements a subset of the api provided by the
CryptoService. Currently the CryptoService is simply bound to null when
in the transport client or when security is disabled. This change
converts all users of the SecretService to the CryptoService, and any
tests using SecretService.Insecure to simply using null.
Original commit: elastic/x-pack-elasticsearch@8251c65ffe
The CryptoService currently has a lot of variations of methods that are
unused. It really only uses sign/unsign, encrypt/decrypt. This change
trims the api down to those needed methods.
Original commit: elastic/x-pack-elasticsearch@92e83efeb7
The Clock interface, which basically allows testing in watcher to "time
warp" is currently constructed using guice. This change constructs it
using a protected method on XPackPlugin which can be overriden in tests.
This allows removing the ClockModule. For now, the Clock still needs to
be bound in guice, but this at least removes one guice construction, and
shows how other things can be overriden for tests.
Original commit: elastic/x-pack-elasticsearch@7addaea086
This makes use of the registerAsDeprecatedHandler method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a Warning header for anyone using HTTP clients (though they have to be looking for it...).
Security portion only
Original commit: elastic/x-pack-elasticsearch@ab1a50fe06
This adds the following stats to the usage stats
* Across all active watches
* Number of triggers per type
* Number of inputs per type
* Number of transforms per type
* Number of conditions per type
* Number of actions per type
* Total number of watches
* Total number of active watches
* Per action
* Execution time total
* Execution time mean
* Invocation count
* Total use of watch metadata
Internally this uses a `Counters` helper class, so that creating counters does not require
you to decide about the map structure.
Relates elastic/elasticsearch#2210
Original commit: elastic/x-pack-elasticsearch@e6f95ba290
This changes the way that Kibana (and future applications) send their monitoring stats to Elasticsearch.
Instead of sending their payloads with the System ID (e.g., "kibana") and System Version (e.g., "5.0.0-alpha4"), it now expects the System ID and System _API_ Version (e.g., "2"). This means a few things:
- Future releases are automatically compatible with previous releases as long as the API version doesn't change.
- Users don't have to update Kibana at the exact same time as their cluster (which technically means rolling updates were temporarily blockers of Kibana monitoring before).
- We can accept old API versions (if we need to make a breaking change) and automatically up-convert them to the latest API version. (We are in full control of how far back we choose to accept)
In general, this change implies that users should be updating their Monitoring cluster before their _monitored_ cluster(s) to get the best opportunity of monitoring backwards compatibility. That way if any API change does occur, then it can up-convert as needed. Then, any ES node should be updated, and only then should Kibana be updated. This is not required in any way, but it will give the smoothest experience.
Original commit: elastic/x-pack-elasticsearch@d3c24936e1
While testing another PR, I noticed that when Security is disabled, the _xpack/usage API throws a NullPointerException.
This checks for null before using it, and adds tests to verify behavior.
Original commit: elastic/x-pack-elasticsearch@6474af6569
When a painless exception is raised in the script condition, it was not bubbled up due to
catching exceptions on during execution. This removes the different catching of exceptions
and allows the watch record construct to contain an exception that is also serialized correctly
so that it can be stored in the watch history but also returned in the execute watch API.
This also updates the watch history template, so that exceptions are not indexed, but logged.
Relates elastic/elasticsearch#2587
Original commit: elastic/x-pack-elasticsearch@4dffb672bf
The top-level class Throwable represents all errors and exceptions in
Java. This hierarchy is divided into Error and Exception, the former
being serious problems that applications should not try to catch and the
latter representing exceptional conditions that an application might
want to catch and handle. This commit renames
org.elasticsearch.cli.UserError to org.elasticsearch.UserException to
make its name consistent with where it falls in this hierarchy.
Relates elastic/elasticsearch#2701
Original commit: elastic/x-pack-elasticsearch@589e159ec0
Today throughout the codebase, catch throwable is used with reckless
abandon. This is dangerous because the throwable could be a fatal
virtual machine error resulting from an internal error in the JVM, or an
out of memory error or a stack overflow error that leaves the virtual
machine in an unstable and unpredictable state. This commit removes
catch throwable from the codebase and removes the temptation to use it
by modifying listener APIs to receive instances of Exception instead of
the top-level Throwable.
Relates elastic/elasticsearch#2694
Original commit: elastic/x-pack-elasticsearch@7ecdd7d978
This looks like it predates settings validation in core, and only had a
single use inside the watcher ExecutionService. This change moves the
settings inside ExecutionService to be validated settings, and removes
the watcher specific validation.
Original commit: elastic/x-pack-elasticsearch@82843ce56c
When a license exception is raised, we returned 401 as HTTP error code
in there. However this seems to have triggered some browsers to actually
ask for login credentials, which wont have any impact here.
Closeselastic/elasticsearch#1863
Original commit: elastic/x-pack-elasticsearch@cc63abdac8
We are going to parse the body anyways whenever it's in json format as it is going to be stashed. It is not useful to lazily parse it anymore. Also this allows us to not rely on automatic detection of the xcontent type based on the content of the response, but rather read the content type from the response headers.
Original commit: elastic/x-pack-elasticsearch@11be4684ae
The internal representation of the object that JsonPath gives access to is a map. That is independent of the initial input format, which is json but could also be yaml etc.
This commit renames JsonPath to ObjectPath and adds a static method to create an ObjectPath from an XContent
Original commit: elastic/x-pack-elasticsearch@bc84c68161
Currently, the REST tests for security (and possibly others) don't clean up the environment
after they have run, eg. they don't delete the users and roles they create. This leads to
test failures, because in a subsequent run, a user or role already exists, so eg. a test
like `match: { role: { created: true } }` fails.
This patch adds a `teardown` section to the test, with `do` actions which are to be
executed _after_ the test runs.
This patch assumes that REST tests runners for all languages support the `teardown` directive
in a xUnimt nomenclature -- similarly to the `setup` directive, which they already support.
Original commit: elastic/x-pack-elasticsearch@70d0ff4ee9
This is the xplugins side of moving the client dependency for rest
handlers to the handleRequest method
Original commit: elastic/x-pack-elasticsearch@ce66e35e7b
This commit modifies the construction of the poll interval setting in
the native realm tests in response to upstream change
elastic/elasticsearchelastic/elasticsearch#2f638b5a23597967a98b1ced1deac91d64af5a44.
Original commit: elastic/x-pack-elasticsearch@c6f60f51f4
This adds the `bin/shield/migrate` tool that allows migrating users and
roles from the files to the native (API-based) store.
It looks like this:
```
λ bin/shield/migrate native -U http://localhost:9200 -u test_user -p changeme -n lee,foo -r role1,role2,role3,role4,foo
starting migration of users and roles...
importing users from [/home/hinmanm/scratch/elasticsearch-2.4.0-SNAPSHOT/config/shield/users]...
found existing users: [test_user, joe3, joe2]
migrating user [lee]
{"user":{"created":true}}
no user [foo] found, skipping
importing roles from [/home/hinmanm/scratch/elasticsearch-2.4.0-SNAPSHOT/config/shield/roles.yml]...
found existing roles: [marvel_user, role_query_fields, admin_role, role3, admin, remote_marvel_agent, power_user, role_new_format_name_array, role_run_as, logstash, role_fields, role_run_as1, role_new_format, kibana4_server, user, transport_client, role1.ab, role_query]
migrating role [role1]
{"role":{"created":true}}
migrating role [role2]
{"role":{"created":true}}
role [role3] already exists, skipping
migrating role [role4]
failed to migrate role [role4] with body: {"indices":[{"names":["idx2"]},{"names":["idx2"]},{"names":["idx1"]}]}
java.io.IOException: {"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse indices privileges for role [role4]. missing required [privileges] field"}],"type":"parse_exception","reason":"failed to parse indices privileges for role [role4]. missing required [privileges] field"},"status":400}
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.postURL(ESNativeRealmMigrateTool.java:206)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.importRoles(ESNativeRealmMigrateTool.java:389)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.execute(ESNativeRealmMigrateTool.java:171)
at org.elasticsearch.common.cli.CliTool.execute(CliTool.java:153)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool.main(ESNativeRealmMigrateTool.java:91)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost:9200/_shield/role/role4
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.postURL(ESNativeRealmMigrateTool.java:192)
... 4 more
no role [foo] found, skipping
users and roles imported.
```
Original commit: elastic/x-pack-elasticsearch@3ce47c0ffd
If an attachment is configured of disposition type INLINE, and is referred to
in HTML body parts, then some email clients can display images inside of an HTML
email and refer to those attachments.
Watcher already had support for inlined attachments, however this could not be configured
from a watch, but just via the Java API. Also it was not tested.
This commit changes the attachment to decide on creation if it should be inline or a regular
attachment and adds a test.
Relates elastic/elasticsearch#2381
Relates elastic/elasticsearch#2464Closeselastic/elasticsearch#2557
Original commit: elastic/x-pack-elasticsearch@84935ffb18
When parsing the privileges, we now no longer throw an exception if
there haven't been any names parsed out. This is not an issue though,
because we validate that the `names` array is not empty when we parse
it, and that it's not `null` before returning from the function.
Adds a rest test that sends things out of order to test this still
works.
Resolveselastic/elasticsearch#2606
Original commit: elastic/x-pack-elasticsearch@62a38bea8f
Add new REST API endpoint to acknoweldging actions. The old endpoints
have not been removed as part of this PR, but can be in the next major
version.
Update the documentation to remove the parameter based example, and
mention that the old endpoints are going to be removed in the future.
Closeselastic/elasticsearch#2517
Original commit: elastic/x-pack-elasticsearch@e2558e9e1f
The search type is overridden from default to query_and_fetch by the search action if the number of shards to request is equal to 1
Original commit: elastic/x-pack-elasticsearch@a9552c697c
Template support is going to be removed from the Search API to its own Search Template API in the lang-mustache module (see elastic/elasticsearch#17906, elastic/elasticsearch#18765). This commit changes Watcher's SearchInput and SearchTransform classes so that it now uses a WatcherSearchTemplateRequest that contains both the search request and the template. Search request and template are rendered using WatcherSearchTemplateRequestService before being executed.
Original commit: elastic/x-pack-elasticsearch@bfa16ab80f
We were ignoring the response code which is always 401 because the license is not good to start watcher. Plus all tests run fine without these methods.
Original commit: elastic/x-pack-elasticsearch@f93e1c2777
The watch index action was using the _timestamp field by default.
This functionality now needs to be configured explicitely for a special
field that is part of that document which is going to be indexed.
Relates elastic/elasticsearchelastic/elasticsearch#18980
Original commit: elastic/x-pack-elasticsearch@dfa4cf2296
This class should have never existed, I fixed all places where we messed aroudn with this
and resolved dependencies or let guice deal with it.
Original commit: elastic/x-pack-elasticsearch@6a42c4153d
This commit handles the use of `*` as a field in a role as effectively disabling field level
security. We do this to take advantage of caches that we disable when field level security
is active.
See elastic/elasticsearch#2407
Original commit: elastic/x-pack-elasticsearch@d96e18d57c
This change makes the default role check the authenticating realm when authorizing
a request for the current user (or run as user) where the user is trying to change their
own password. We need to do this, otherwise we open up the potential of a user in one
realm changing the password of a user in another realm.
As part of this work, the authentication service has been refactored and simplified. A
new object, Authentication, is now returned when authenticating. Currently, this object
contains the user, authenticating realm information, and if it is a run as request the
information of the realm that looked up the user.
Closeselastic/elasticsearch#2089
Original commit: elastic/x-pack-elasticsearch@3fd9c37a16
The WWW-Authenticate header can optionally specify the charset that the server uses after
decoding credentials. If this is not specified, most clients will limit the available characters to
ISO-8859-1, which causes issues for certain characters.
See RFC 7617
Closeselastic/elasticsearch#2290
Original commit: elastic/x-pack-elasticsearch@44411eebe7
The active state was not serialized in the PutWatchRequest leading to
to always setting it to active, when a different node than the master
node was hit with a put watch request.
Closeselastic/elasticsearch#2490
Original commit: elastic/x-pack-elasticsearch@060c0fa35f
s/request.setRefresh/request.setRefreshPolicy/
setRefresh is still supported on the builder for backwards
compatibility but not on the request itself.
Original commit: elastic/x-pack-elasticsearch@8763e2e65f
A RestClient instance is now created whenever EsIntegTestCase#getRestClient is invoked for the first time. It is then kept until the cluster is cleared (depending on the cluster scope of the test).
Renamed other two restClient methods to createRestClient, as that instance needs to be closed and managed in the tests.
Original commit: elastic/x-pack-elasticsearch@3a9d6f6e90
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
Original commit: elastic/x-pack-elasticsearch@304487ecc5
Removes handlers internal:cluster/node/index/deleted and internal:cluster/node/index_store/deleted that have been removed in core as part of elastic/elasticsearchelastic/elasticsearch#18602
Original commit: elastic/x-pack-elasticsearch@e040871e5a
This commit fixes the version compatability test by updating the version
to reflect the current version in core.
Original commit: elastic/x-pack-elasticsearch@0bb6dbc1c3
If a chained input was used, and inside of this a search input was
used, that hat dots in its field names somewhere (like when sorting
or using a compare condition), then storing this in the history failed.
The reason for this was the broken watch history template, that did not take
nested requests bodies into account and thus tried to create an index mapping
for requests that were inside of a chained input.
This commit fixes the watch history index template.
Closeselastic/elasticsearch#2338
Original commit: elastic/x-pack-elasticsearch@d9f48234d3
Areek Zillur
Ryan Ernst
Jason Tedor
Chris Earle
Lee Hinman
Simon Willnauer
Karel Minarik
Alexander Reelsen
Simon Willnauer
Nik Everett
javanna
jaymode
Robert Muir
Robert Muir
Tanguy Leroux
Adrien Grand
Boaz Leskes
Colin Goodheart-Smithe
Jay Modi
Martijn van Groningen
Yannick Welsch
Luca Cavanna
Daniel Mitterdorfer
Tim Sullivan
Timothy Sullivan
Jonathan Budzenski
Jim Ferenczi
markharwood
Christoph Büscher