Fixes to x-plugins code now that DateMathParser accepts a LongSupplier rather than a Callable to get the value of now
Relates to elastic/elasticsearchelastic/elasticsearch#20796
Original commit: elastic/x-pack-elasticsearch@99fc47a8a7
This change moves to using SSLParameters as the configuration source for SSLEngine and SSLSocket
objects that are configured by the SSLService. Previously we used a mix of specific methods and
SSLParameters, which resulted in issues where ordering of calls is important. For example, if configuring
client authentication directly on the engine prior to setting the SSLParameters resulted in the client
authentication configuration being reset to the default.
Additionally, this change also sets use cipher suite order to true to ensure preferred ciphers will be used.
Original commit: elastic/x-pack-elasticsearch@8ddecdc20c
This change ensures we wait for a response before the async http client is closed. Otherwise we can
close the client during the connection to the remote endpoint or never even connect to the remote
endpoint.
Closeselastic/elasticsearch#3640
Original commit: elastic/x-pack-elasticsearch@54900b1b4a
This changes does two things in the tribe tests. The first is that when we split data up between
multiple clusters, we always force create the security index so that randomization does not cause
edge cases like the index not existing in the preferred cluster. The second is we look at the cluster
state of the nodes and ensure the tribe node sees the indices and has all primaries active.
Separate tests were also added to cover the scenario where the security index only exists in the non
preferred node.
Original commit: elastic/x-pack-elasticsearch@17b78ec837
This is the last action that needs additional support for proxies.
You can set a proxy in the JSON like this:
```
"actions" : {
"notify-pagerduty" : {
"pagerduty" : {
"description" : "Main system down, please check!",
"proxy" : { "host" : "localhost", "port" : 8080 }
}
}
}
```
Closeselastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@b99969fd6b
You can set it like this in the JSON
"actions" : {
"notify-slack" : {
"slack" : {
"account" : "integration-account",
"proxy" : {
"host" : "localhost",
"port" : 8080
},
"message" : {
...
}
}
}
}
Relates elastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@de86233d4f
Watcher uses a custom thread pool. This is because executing watches can
be long-running tasks that often block on I/O and it is best to not
consume the core thread pools with these tasks. Today this thread pool
is fixed, and sized at five times the bounded number of cores (so 160 on
a 32-core box). It makes sense for there to possibly be so many threads,
again because these tasks can block on I/O and having excess capacity
lets unblocked watches execute. It's the fixed size that can cause
problem, all these threads are always consuming resources even when
there are no or not that many watches running. This commit changes this
thread pool to be a scaling thread pool.
Relates elastic/elasticsearch#3660
Original commit: elastic/x-pack-elasticsearch@3cafab6e83
We need to special case IndicesAliasesRequest as it doesn't implement CompositeIndicesRequest anymore. Note that the similar loop for CompositeIndicesRequests's subrequests will soon go away
Relates to elastic/elasticsearch#3638
Original commit: elastic/x-pack-elasticsearch@50d119ff61
This change allows native users and roles to be used on tribe nodes. The tribe node will actually
use the security index of one of the tribes, which must be specified with the `tribe.on_conflict`
setting. User and role modifications are not permitted when running on a tribe node.
Closeselastic/elasticsearch#3451
Original commit: elastic/x-pack-elasticsearch@2b762ca648
When deleting a watch the version was used as part of
the delete request. However a watch deletion means the
user really wants to get rid of it and not accidentally
run into a version exception because the watch was running in
between.
Original commit: elastic/x-pack-elasticsearch@e585f717f1
In preparation for elastic/elasticsearch#2957, I found we have things both in the root level
dev-tools, as well as elasticsearch/x-dev-tools. Most of this stuff can
be removed as it had to do with the old manual release process. There
was also a signed license file checked in. I removed it here, we really
should not have licenses checked in IMO, and it is unclear what the
purpose of this license was for. The two remaining scripts were moved to
the root dev-tools.
Original commit: elastic/x-pack-elasticsearch@3e24ea2d56
When running as a node, we check the `xpack.security.transport.filter.enabled` setting to see
if we should create the IPFilter but this check is not really correct. The HTTP filter could be
enabled or a profile filter could be enabled so there are times when we may not be filtering connections
when we should. Additionally, since we do not bind the IPFilter to a null provider, Guice will try to create
one during startup to inject into the security transport. This results in an exception and startup fails.
This change always creates the IPFilter when running as a node. This IPFilter has its own settings and
logic to determine whether it should be filtering on a given network transport.
Closeselastic/elasticsearch#3592
Original commit: elastic/x-pack-elasticsearch@95c25651c4
Some methods have been renamed in elastic/elasticsearchelastic/elasticsearch#20560. This commit change a .bytes() call to a .getBytes() call.
Original commit: elastic/x-pack-elasticsearch@4a0ff77361
This commit fixes the cat.templates REST tests so that it works when other templates exist (like monitoring)
Original commit: elastic/x-pack-elasticsearch@2e27ad88b4
This change switches the build to use the licensing prod key when
building the xpack jar for release.
Original commit: elastic/x-pack-elasticsearch@54a21dae5b
This adds an "interval" placeholder parameter that is required to the Monitoring Bulk API, and adds it to the Kibana side of the plumbing.
Having this will allow us to add it to all incoming documents and start to report against it with the Insights, as well as to detect the _lack_ of incoming documents.
By adding it now, we can avoid having a non-BWC API change for Kibana in 5.1. We'll just pickup new data in our documents.
Original commit: elastic/x-pack-elasticsearch@5ba8aafe03
This is a followup for elastic/elasticsearchelastic/elasticsearch#20526 removes the pluggability of
transport / http server transport via guice.
Original commit: elastic/x-pack-elasticsearch@5fb84949aa
The x-pack client jar and api jar are exactly the same: the entirety of
x-pack. Since we added the x-pack-transport jar, we no longer really
want the client jar as it is confusing. Additionally, it causes jar hell
when a test for an extension uses the transport client. This change
removes the client jar, and makes the x-pack transport client use the
api jar instead. This sounds odd at first, but since transport client is
going away eventually, it is a stopgap, and works.
closeselastic/elasticsearch#3309
Original commit: elastic/x-pack-elasticsearch@ee7a2c12c0
Due to untested code there was an NPE happening in production,
when a chained input execution failed, but the chained input
tried to access the resulting payload (which is never set on
failures). This payload now defaults to being empty.
This commit also drive-by fixes a broken logging statement, that
on the one side returned not the watch id, but a useless watch
toString() representation, and on the other hand only logs an error
message, but not a stack trace into the log, as this is what the
history is for.
Original commit: elastic/x-pack-elasticsearch@7dbe1afd90
Whenever a watch is updated (put, delete, set state), until now we
happened to reject those operations when a watch was executed at the
same time. However with long running reporting this might mean, that a
watch can never be changed, because it always gets executed.
* Removes the ability of write requests to obtain a lock at all (executing watches is still protected by a lock)
* Replaced the FairKeyedLock in watcher with the KeyedLock in Elasticsearch, which also takes a fair option, removed the FairKeyedLock
* Removed all the timeout parameters that are no longer needed, because there is no lock anymore
* Removed also the force parameter for watch deletion. Just do it[tm]
* Added a test that deleting a watch while it is being executed does not leave any leftovers
In case of a deletion of a watch during an execution, so that updating the status of the watch fails,
a warning is logged.
Closeselastic/elasticsearch#3417
Original commit: elastic/x-pack-elasticsearch@22fad1b797
As part of the review of elastic/elasticsearch#3287, the stopping of the IndexAuditTrail was moved to the tearDown
method. This works sometimes but other times it fails because tearDown is run after
ESIntegTestCase#after, so the IndexAuditTrail is still running during the after checks which will
cause the test to fail since the shard lock cannot be obtained.
Closeselastic/elasticsearch#3520
Original commit: elastic/x-pack-elasticsearch@4cb52b15a2
Today the operation mode can be set to default for a short amout of
time until it's reset to the actual mode this can cause weird sideeffects
for users if it's read concurrently. Also the test relies on a certain
happens before relationship that is not guaranteed since the operation
mode is set before the listerner is run. This change also rewrites the test
to not use busy waiting but wait for the actual listern to be executed.
Original commit: elastic/x-pack-elasticsearch@a2a42b89e5
1. We only support indexes created by Marvel 2.3+. All other indexes
are just ignored.
2. The tests don't assert a ton of interesting stuff because there
isn't a java API for Monitoring that we can just use. Instead we assert
that a few objects are there and look sane.
3. We don't migrate the contents of the data index. Instead we just
rely on Monitoring recreating it.
Original commit: elastic/x-pack-elasticsearch@86216c2d61
The IndexAuditTrail had both a stop and close method that needed to be called in order
to stop the service. There was a race where we called either flush or close in a non
blocking fashion and then immediately closed the underlying client. This change makes
the stop method wait for up to 10 seconds when closing the bulk processor.
Closeselastic/elasticsearch#3279
Original commit: elastic/x-pack-elasticsearch@0d776bc91a
TransportService is not pluggable anymore in core. Instead we now have a interceptor
infrastructure that allows to intercept send and receive calls on the transport layer.
Relates to elastic/elasticsearchelastic/elasticsearch#20505
Original commit: elastic/x-pack-elasticsearch@04194ecb09
This adds proxy support to the hipchat action. Right now
neither hipchat nor slack nor pagerduty allow for this,
but if you dont need a proxy for internal http connections,
but you do for external, then this configuration cannot be done
without setting a proxy for those actions.
You can set it like this in the JSON
```
"actions" : {
"notify-hipchat" : {
"hipchat" : {
"account" : "integration-account",
"proxy" : {
"host" : "localhost",
"port" : 8080
},
"message" : {
...
}
}
}
}
```
Relates elastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@4e8447ce37
This cleans up some of the reported stats to be a little clearer, including making the JVM Heap chart behave like Kibana's memory chart. This solves two problems: you can now determine the max heap size and you know what "x%" actually means relative to it.
Original commit: elastic/x-pack-elasticsearch@450f6fd546
This commit fixes the logging audit tests which were broken due to an
upstream change in core Elasticsearch relating to the fact that prefixes
are no longer considered part of the log message, but are instead
implemented via markers.
Original commit: elastic/x-pack-elasticsearch@abd7ec23d8
To deny access to a fields users can name exceptions to field permissions with the following syntax:
"fields": {
"grant": [list of field names patterns],
"except": [list of patterns that are forbidden]
}
See doc for the rules for this.
This commit also reverts elastic/elasticsearch#2720closeselastic/elasticsearch#2681
Original commit: elastic/x-pack-elasticsearch@d6537028ec
The enabled and username fields are both now allowed in the request body for the put user
request. This makes it easier to perform a get and update a user without needing to edit more
of the request body than necessary.
Closeselastic/elasticsearch#3391
Original commit: elastic/x-pack-elasticsearch@ab763e843b
Start dates are a required feature for cloud. This functionality adds support
for specifying and enforcing a start date on licenses.
Behaviour: If the start date is > than now, the license will be rejected.
Due to another field in the license class, the version of the License class as well
as its serialization methods are adapted to this.
Closeselastic/elasticsearch#3370
Original commit: elastic/x-pack-elasticsearch@eb2a6f5be3
We were starting nodes at weird times and then shutting them down again,
slowing down the tests and causing the watcher tests to fail because
watcher wasn't being shut down with its traditional kid gloves.
Original commit: elastic/x-pack-elasticsearch@2fd81b3eaf
When the HTTP attachment was not able to successfully retrieve the
data from and endpoint, there was no indication in the watch history
of what went wrong. Instead a logger was used, which is not useful
for the person running the watches.
This commit removes the logger statement and throws an exception,
so that the exception message can be stored in the watch history.
Source of this issue was a forum post:
https://discuss.elastic.co/t/sending-e-mail-with-generated-report-fails/60263/6
Original commit: elastic/x-pack-elasticsearch@acdaf7abef
In our tests you have to explicitly shut down watcher rather than shut
down the node it is running on because of thread leak detection. Just
shutting down the node that it is running on will cause it to start up
on another node if there is another one running and then not properly
shut down. This is probably something that should be fixed in watcher
somehow but for now lets just be more careful with the tests.
Closeselastic/elasticsearch#2365Closeselastic/elasticsearch#2588
Original commit: elastic/x-pack-elasticsearch@fb8a172972
Previously core Elasticsearch had methods in a test class for removing
and adding appenders. However, these methods were moved to production
code. This commit adjusts x-plugins for this change.
Original commit: elastic/x-pack-elasticsearch@83e37ef65a
This particular change focuses on upgrading the source of a watch when it comes to scripts that have no language specified explicitly.
The default language in version 5 changed to painless from whatever is specified in `script.default_lang` setting (this defaulted to groovy). In order to make sure that scripts in watcher remain to work we should rewrite the search source upon startup and set the legacy default language explicitly. The legacy script language is now controlled by `script.legacy.default_lang` setting and that defaults to groovy.
Changing the source upon startup should do the trick and only change the source of watches with scripts that don't have an explicit language set. For new watches the default language used in scripts is painless and because we now always serialize the language explicitly in scripts these watches won't be changed on startup.
The upgrade logic added here tries to upgrade scripts in the following places in a watch:
* script condition
* script transform
* any script defined inside of a search input
Original commit: elastic/x-pack-elasticsearch@4d578819eb
This commit cleans most of the methods of XContentBuilder so that:
- Jackson's convenience methods are used instead of our custom ones (ie field(String,long) now uses Jackson's writeNumberField(String, long) instead of calling writeField(String) then writeNumber(long))
- null checks are added for all field names and values
- methods are grouped by type in the class source
- methods have the same parameters names
- duplicated methods like field(String, String...) and array(String, String...) are removed
- varargs methods now have the "array" name to reflect that it builds arrays
- unused methods like field(String,BigDecimal) are removed
- all methods now follow the execution path: field(String,?) -> field(String) then value(?), and value(?) -> writeSomething() method. Methods to build arrays also follow the same execution path.
Original commit: elastic/x-pack-elasticsearch@d83f3aa6e2
This fixes a bug I found with a customer when he updated from 1.x to 2.x.
Due to an BWC incompatible change in the watch history mapping and a thread
pool rejection during execution a watch was not removed from the triggered
watches and tried to be executed again.
While trying to fix it it turned out that the execution of the failure
test case was still done in the transport thread and thus required some
offloading to another thread pool.
Original commit: elastic/x-pack-elasticsearch@df04ce31f2
If someone deletes the watch index (i.e. by deleting all indices), the watcher
in memory store still contains all the watches and tries to execute watches -
which results in exceptions as the watch itself cannot be updated anymore.
In order to minimize this problem (it cant be get rid of completely), we should
act accordingly if the watch index goes missing (either deleted or closed) and
clear out the memory representation of watches in the watchstore as well as trying
to finish all the current executions.
Closeselastic/elasticsearch#2794
Original commit: elastic/x-pack-elasticsearch@12d98cd566
This change moves the logfile audit output from determining what to log based on the
logger level to a enum based configuration that is used by the index output.
A few notable changes were made:
* We alway log all the information we have except for the request body
* The request body is no longer logged by default for REST events; the user needs to
explicitly opt in as there could be sensitive data in the body
* Added a `realm_authentication_failed` event that separates overall authentication
failure from that of an individual realm
Original commit: elastic/x-pack-elasticsearch@343a2bcdd9
This change adds support for disabling users. Users can be disabled by setting the enabled
property to false and the AuthenticationService will check to make sure that the user is enabled.
If the user is not enabled, this will be audited as an authentication failure.
Also as part of this work, the AnonymousUser was cleaned up to remove having a static instance
that caused issues with tests.
Finally, the poller of users was removed to simplify the code in the NativeUsersStore. In our other
realms we rely on the clear cache APIs and the timeout of the user cache. We should have the
same semantics for the native realm.
Closeselastic/elasticsearch#2172
Original commit: elastic/x-pack-elasticsearch@0820e40183
This rewrites the HTTP Exporter to use the REST client underneath. Functionality is improved in resource blocking (templates and pipelines existing) and the majority of the code fundamentall simplified by removing direct HTTP calls.
This is blocked by the SSLService pull request. After that is merged, the I will update this PR to reflect those changes and it could possibly allow us to remove the security privileges required for monitoring.
Original commit: elastic/x-pack-elasticsearch@1ad25f17f8
Basic backwards compatibility support for watcher.
Closeselastic/elasticsearch#3230
Relates to elastic/elasticsearch#3231 - this actually should fix all the failures caused
by fractional time values but it does so by being able to parse them.
Being able to parse them is important for 2.x compatibility but 5.0
watches shouldn't produce fractional time values. This fixes the
particular way of making fractional time values mentioned in elastic/elasticsearch#3231
but I expect there are a half dozen more places to fix. The actual
watcher tests are fairly basic.
Original commit: elastic/x-pack-elasticsearch@328717455c
This publishes X-Pack usage data to the cluster info from the elected master node. This allows phone home to retrieve this data from the index, rather than fetching it live from the connected cluster (thereby not getting it from any n - 1 clusers that are not connceted).
Original commit: elastic/x-pack-elasticsearch@79bfaaaf0b
This removes the "agent" package from org.elasticsearch.xpack.monitoring.agent.*, so that now everything is simply org.elasticsearch.xpack.monitoring.*.
Follow-on work will be refactoring some of the other code, but this is a first step now that it's always the agent (in effect).
Original commit: elastic/x-pack-elasticsearch@14025cb17c
This change migrates xpack (security, watcher, and monitoring) to use the common ssl
configuration for the elastic stack. As part of this work, several aspects of how we deal
with SSL has been modified.
From a functionality perspective, an xpack wide configuration for SSL was added and
all of the code that needs SSL uses the SSLService now. The following is a list of all
of the aspects of xpack that can have their own SSL configuration, which are separate
from the xpack wide configuration:
* Transport
* Transport profiles
* HTTP Transport
* Realms
* Monitoring Exporters
* HTTP Client
In terms of the code, some cleanups were made with these changes. SSLConfiguration is
now a concrete class and SSLConfiguration.Custom and SSLConfiguration.Global have been
removed. The validate method on key and trust configurations has been removed and these
classes will now throw exceptions when they are constructed with bad values. The
OptionalSettings helper class has been removed as it was just a file with one line functions
that made the code harder to understand. The SSL configuration and service classes have
been moved from the security source directories to the main xpack source set. The SSLService
now handles more of the configuration of the SSLEngine it returns to prevent callers from
having to handle those aspects. The settings that get registered for SSL have been moved to
XPackSettings.
Also included in this PR is a update to the docs around SSL. This includes a large simplification to
the documentation in that the certificate authority configuration section has been removed and the
process that is documented for generating certificates only includes the CLI tool that we bundle.
Closeselastic/elasticsearch#3104Closeselastic/elasticsearch#2971Closeselastic/elasticsearch#3164
Original commit: elastic/x-pack-elasticsearch@5bd9e5ef38
* master:
Changes tests to conform with new cluster health API, calling setWaitForNoRelocatingShards(true) instead of setWaitForRelocatingShards(0)
Original commit: elastic/x-pack-elasticsearch@bde6ad8c8a
* master:
Use releasable locks in NativeRolesStore
security: limit the size of the role store cache
security: remove explicit handshake wait in netty4 transport
test: smoke-test-plugins-ssl no longer relies on logging to start
kibana monitoring/uuid config key reference update
Docs: Updated release date for 2.4 in RNs.
Update README.md
Build: Add apijar task to assemble so it gets built with other artifacts
monitoring ui/license: cluster listing status cell treatment for basic/unsupported cluster
monitoring ui:fix cluster overview when cluster has no indices/shards
monitoring ui/license: logic cleanup per feedback
monitoring ui/license: primary cluster asterisk styling
monitoring ui/license: allow clicking into primary cluster if all are basic
monitoring ui: add isPrimary property to cluster listing response
Security: throw exception if we cannot extract indices from an indices request
Security: add tests for delete and update by query
Original commit: elastic/x-pack-elasticsearch@3cb41739ee
Previously the roles store cache was unbounded as it was a just using a ConcurrentHashMap,
which could lead to excessive memory usage in cases where there are a large number of roles
as we tried to eagerly load the roles into the cache if they were not present. The roles store now
loads roles on demand and caches them for a finite period of time.
Additionally, the background polling of roles has been removed to reduce complexity. A best effort
attempt is made to clear the roles cache upon modification and if necessary the cache can be
cleared manually.
See elastic/elasticsearch#1837
Original commit: elastic/x-pack-elasticsearch@450dd779c8
Netty 4's SslHandler does not require the application to wait for the handshake to
be completed before data is written. This change removes the explicit wait on each
handshake future.
Original commit: elastic/x-pack-elasticsearch@c19bcebb83
This commit removes some unnecessary calls to Path#toString when logging
a path in OperationFileModeWatcher#onChage. The calls to Path#toString
are not necessary since the logging infrastructure will do this anyway.
Original commit: elastic/x-pack-elasticsearch@da1bc30d60
This commit removes an unnecessary logging of an exception message from
HttpEmailAttachementParser#toAttachment since the full exception is
logged anyway.
Original commit: elastic/x-pack-elasticsearch@55d4bada00
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
Original commit: elastic/x-pack-elasticsearch@955ab89b8b
This used to be an assertion but we move it to an exception to be able to catch this at all times without requiring assertion enabled
Original commit: elastic/x-pack-elasticsearch@fcb5fbe852
When migrating the package and class names from shield to xpack.security, the logging.yml
file was updated incorrectly which resulted in audit entries going to the wrong file. This change
specifies the correct package name so audit logs go to the correct file.
Closeselastic/elasticsearch#3063
Original commit: elastic/x-pack-elasticsearch@5c07108392
This change allows extensions to define their own filtered settings. This is used when there
may be sensitive credentials stored in a custom realm that should be filtered out of some
API calls.
Closeselastic/elasticsearch#2847
Original commit: elastic/x-pack-elasticsearch@952474daba
With elastic/elasticsearchelastic/elasticsearch#19865 the XContentBuilder has changed to support both inclusive and exclusive filters and now uses Set<String> instead of arrays of Strings. This change updates the various places in x-plugins where string arrays were used.
Original commit: elastic/x-pack-elasticsearch@1f8d4485f4
The actual backwards compatibility support is handled by core's ability to downgrade `keyword` and `text` into `string` for indexes created in 2.x.
Original commit: elastic/x-pack-elasticsearch@6b615d9a45
Although most of the security functionality was disabled when a basic license was applied,
some of the functionality still could be executed such as using the authenticate API or using
a transport client. The issue here is the UI calls the authenticate API and this gives the impression
that security is really in use when it is not.
Original commit: elastic/x-pack-elasticsearch@881453fc4c
The realtime request interceptor was added to stop realtime requests from bypassing DLS
or FLS as the request could read a document from the translog. After
elastic/elasticsearchelastic/elasticsearch#20102 we no longer read documents from the translog so we can
allow realtime requests even when DLS or FLS is enabled.
Original commit: elastic/x-pack-elasticsearch@069b501500
This commit simplifies the code used for resolving the files used for security to always
resolve against the `config` file. Elasticsearch no longer offers a way to disable the security
manager, so the files read by x-pack should not really be configurable and only exist in their
default locations since that is what can be read by the process.
As part of this, the documentation was updated to indicate that these files should always be in
the default location and the settings to change the locations have been removed. Also, a bug
was fixed in a few places where settings were still using `shield.` instead of `xpack.security.`.
Finally, some outdated and unused files were deleted from the repository.
Closeselastic/elasticsearch#305
Original commit: elastic/x-pack-elasticsearch@3884f080a0
This change cleans up some of the log messages and levels that could now be considered misleading.
While performing these cleanups, the following was done:
* remove creation of dummy user for gradle run as we have the `elastic` user
* Request interceptors are not bound if field and document level security is disabled
* FLS/DLS interceptors skip execution if document and field level security is disabled by the license state
* The roles store that loaded the role is logged at the TRACE level
* The TransportXPackUsageAction was using the incorrect action name when registering a handler
Closeselastic/elasticsearch#2096Closeselastic/elasticsearch#1861Closeselastic/elasticsearch#2229
See elastic/elasticsearch#1879
Original commit: elastic/x-pack-elasticsearch@ac16b21c0c
This change allows users with the superuser role to access the security index. We previously allowed
the XPackUser to access this with the intent that the XPackUser was also the `elastic` user. When the
`elastic` user was split out into the ElasticUser, we did not update the check to allow this user access
to the security index.
Original commit: elastic/x-pack-elasticsearch@fa556d9845
This commit changes how we get the file and directory name for certificates in the tool. The
tool now prompts the user for the filename. If the provided instance name will result in a
valid filename, this is provided as a default. Otherwise the user must provide a valid
filename.
Closeselastic/elasticsearch#2854
Original commit: elastic/x-pack-elasticsearch@3c923d736b
This adds back (again) building a transport client plugin jar for
x-pack, and also adds producing an "api" jar which extension authors can
build against. For now, both these jars are exactly the same, but
eventually they could differ, and be reduced to less than the real
x-pack jar.
see elastic/stackelastic/elasticsearch#7
Original commit: elastic/x-pack-elasticsearch@0a989de18b
This reverts building a separate client jar for xpack. It is not
necessary because we already build x-pack as a jar and publish to
elastic maven, since extension authors need that.
Original commit: elastic/x-pack-elasticsearch@2fab06b42c
SecurityTransportModule handled binding two things in guice. First, IPFilter,
for which createComponents already had the necessary dependencies. The
second was ClientTransportFilter. For transport clients, this was a
noop and could be removed. For nodes, this just attaches the system
user, which could be done directly from SecurityServerTransportService.
Original commit: elastic/x-pack-elasticsearch@da327de476
In some cases, the random ascii value could match a forbidden name, which will cause this
test to fail as it expects valid names. This changes ensures the names are valid.
Original commit: elastic/x-pack-elasticsearch@c79f8fc4cc
This commit marks WatcherUtilsTests#testDeserializeSearchRequest which
is failing with seed 2203D3AD59DB5223 as awaits fix.
Original commit: elastic/x-pack-elasticsearch@4b6cbe544b
This commit marks SearchTransformTests#testParser which is failing with
seed 97BC9E2543410D55 as awaits fix.
Original commit: elastic/x-pack-elasticsearch@3d69e9648c
This commit marks WatcherUtilsTests#testSerializeSearchRequest which is
failing with seed 97BC9E2543410D55 as awaits fix.
Original commit: elastic/x-pack-elasticsearch@45a174c1f2
With the latest clean ups and changes in Watcher, the default search template lang has been switched to WatcherScript.DEFAULT_LANG which points to "groovy" but it should be "mustache" instead.
Original commit: elastic/x-pack-elasticsearch@1d9ef1963e
This adds a "condition" to every action (via the ActionWrapper) that prevents execution of the action if the condition fails. An action-level condition is only useful when there is more than one action, but nothing checks to ensure that it's only used in that scenario.
Original commit: elastic/x-pack-elasticsearch@704cfb1a86
Instead watcher search template should be agnostic of SearchSourceBuilder and just work with BytesReference, so that serializing to a string before templating isn't needed.
Original commit: elastic/x-pack-elasticsearch@36d21ec819
This roughly matches what was done in 2.x for the license tools (except
without any dependency exclusions).
Original commit: elastic/x-pack-elasticsearch@4e1f07b5fa
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#20000, using the new
SearchRequestParsers.
Original commit: elastic/x-pack-elasticsearch@d7cee93a8f
This adds an empty _ingest/pipeline named after the _current_ version of the Monitoring API (currently 2) from both exporters.
This allows us to eventually change the API version (to 3, 4, etc.) and go _back_ and modify the pipeline that exists. The modified pipeline would then "fix" the documents as best as possible and rename the index. As a result, different versions (starting with 5.0) will be able to report to the same monitoring cluster regardless of the running API version.
Note: This has no impact on stale data (e.g., the day before the upgrade) _and_ it implies that the monitoring cluster should always be updated first. A simple reindexing script can be supplied for old data, which can be done at the discretion of the user.
Original commit: elastic/x-pack-elasticsearch@45df5ee87b
The SSLConfigurationReloaderTests rarely failed during some local runs. This turned out to be due to
signaling that the reload happened before we actually reloaded. This led to a race condition where we
attempted to validate the config was reloaded properly and actually reloading. This change fixes the
ordering of operations and uses a CountDownLatch instead of a AtomicInteger and awaitBusy.
Original commit: elastic/x-pack-elasticsearch@9615f225d6
The XPackFeatureSet.Usage writeTo method failed to write the name to the
StreamOutput whereas the deserialization code expected it to be the first value
present in the StreamInput. This causes xpack usage requests made to nodes
other than the master to fail.
This change fixes the serialization and adds tests to all of the feature sets to ensure
that the usage is serialized properly.
Closeselastic/elasticsearch#3072
Original commit: elastic/x-pack-elasticsearch@eccab616ca
This change refactors the ssl services into a single class and reduces the number of
public methods that are exposed by the class. As part of this change, ssl configurations
must be loaded during construction of the service otherwise an exception will be thrown
when trying to retrieve the configuration. Additionally, the reloading of key material and
trust material is now handled by the SSLService instead of the SSLConfiguration. The
SSLConfiguration class has been made package private as its usage has been contained
to the ssl package entirely.
See elastic/elasticsearch#2971
Original commit: elastic/x-pack-elasticsearch@46e4535bb1
Currently there are a number of helper methods, all used at various
stages of initializing xpack, for reading "enabled" settings. Almost all
of these read the setting directly rather than use the new settings
infra. This change adds an XPackSettings class to contain these settings
constants, and adds all the enabled settings to it.
Note: While there is an existing MonitoringSettings, I think we do not
have enough settings (especially for all features) to warrant a separate
class per "feature" (eg graph would have nothing). We should decide
if/when this is necessary on a case by case basis, but I think the
enabled settings should all go here in XPackSettings.
Original commit: elastic/x-pack-elasticsearch@086152d5a5
When security is disabled, we currently set the transport and HTTP type
to Netty 4. However, this is unnecessary as these are the default
settings so this commit removes these explicit settings.
Relates elastic/elasticsearch#3065
Original commit: elastic/x-pack-elasticsearch@afa360ceec
This commit reenable some Watcher tests that were muted in elastic/elasticsearch#724. It removes the Groovy dependency for tests that don't really use Groovy features and replace scripts with mocked scripts. It converts a GroovyScriptConditionIT test into a REST test and moves it in the smoke-test-watcher-with-groovy project.
closeselastic/elasticsearch#724
Original commit: elastic/x-pack-elasticsearch@f4c8161946
These mock Netty plugins were needed to assist in setting
sun.nio.ch.bugLevel. Yet, settings this property should no longer be
needed on JDK 8 and the missing privileged block when writing this
property can be fixed upstream.
Relates elastic/elasticsearch#3047
Original commit: elastic/x-pack-elasticsearch@e2aa4f6fb5
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects.
Original commit: elastic/x-pack-elasticsearch@fb23208e7d
This change removes guice from creating notification componenents. It
also removes the Notification helper class, as it just makes looking at
what stuff xpack adds more obfuscated.
Original commit: elastic/x-pack-elasticsearch@69b8ea4735
When a watch is executed, but the condition is not met, it is not automatically marked as dirty.
This results in a divergence of what is displayed when someone is just getting a watch via the get
watch API (due to using the in memory store) compare to when someone is using the GET API to get the
watch via the .watches index call.
This commit sets the watch status always to dirty after a condition check, thus forcing an update.
Closeselastic/elasticsearch#2699
Original commit: elastic/x-pack-elasticsearch@e6a00260f7
Users are allowed to create query objects with an empty string value as we do
not currently validate the input against a query parser. In this case, we can ignore
the empty value when parsing the role. If we pass an empty BytesReference in
the role then trying to determine the XContentType will fail in the
SecurityIndexSearcherWrapper.
Closeselastic/elasticsearch#2997
Original commit: elastic/x-pack-elasticsearch@fc593943c4
This commit collapes some duplicated setup code in every test in the
Netty 4 security transport tests to a common method.
Original commit: elastic/x-pack-elasticsearch@0a088db78d
* master: (25 commits)
docs: remove unused file and add link to invalid role examples
Remove interfaces for notification services
Redirect to URL specified by next parameter, if it is set
Fix fixture and tests so they pass again
Update error message to be more actionable
Switch to NamedWriteable pull based extension in xpack
Fixing issue with infinite redirect loop
Toggle display of security nav controls more dynamically
Pass in xpackMainPlugin instead of xpackMainPlugin.info
Wrap the return in a Promise
Only unset the cookie if it is currently set
Clarifying intent of code
Updating tests fixtures + adding assertion for client cookie deletion
If security is disabled, do not attempt to call the authenticate ES API
Disambiguate between resolve function names
Revert to not using xpackMainPlugin.info until the xpackMainPlugin is ready
Redirect /login => / if security is disabled in ES
Register/deregister security management items depending on whether there's an auth'd user
Show/hide the username + logout button depending on whether there is an auth'd user
If security is disabled, continue without auth + delete client cookie
...
Original commit: elastic/x-pack-elasticsearch@16b92a1a59
We have 4 types of notification services, and all of them have an
interface with a single implementation class. They also all
unnecessarily are lifecycle componenets, but the only thing start does
is read the settings.
This change converts all 4 notification services to classes, and makes
them regular components instead of lifecycle services.
Original commit: elastic/x-pack-elasticsearch@897115ae65
After this commit, we have "security3" and "security4" as possible
transport and HTTP settings, we default to "security4" if it is not set,
and we randomize the selection in some of the integration tests.
Original commit: elastic/x-pack-elasticsearch@e56718911a
This commit cleans up the Netty handshake waiting handlers. We rename
the Netty 3 implementation to include "Netty3" in the name, the Netty 4
implementation is not needed, and we improve the handling of waiting for
the handshakes to complete when connecting.
Original commit: elastic/x-pack-elasticsearch@f736fdc8f0
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#19764. It converts
the one use of registering custom NamedWriteable readers in xpack for
xpack feature sets to register them up front with the new pull based
registration.
Original commit: elastic/x-pack-elasticsearch@48e2020816
Since elastic/elasticsearch#19621 MockScriptEngine is stricter and expects scripts to be defined before being used in tests. Because watcher makes heavy use of scripts without really need of custom logic, this commit changed the MockMustacheScriptEngine implementation so that it always returns the script's source as a result.
Original commit: elastic/x-pack-elasticsearch@09cffa9517
This commit moves the reloading and monitoring of files from the trust/key configuration
classes into a separate class that will reload for the whole SSLConfiguration object.
SSLContexts are loaded lazily by most of security, so a listener interface was added to
notify the reloader that there may be other paths to monitor.
Original commit: elastic/x-pack-elasticsearch@1633cc14a7
The tests have been moved back to xpack, turned into a rest test or removed.
For example testing specific inline, file or stored mustache template functionality is already covered in the `lang-mustache` module. The smoke-test-watcher-with-mustache should tests watcher mustach specific things like the if the watcher variables are available.
Original commit: elastic/x-pack-elasticsearch@e434bcd3fa
This makes use of the `registerAsDeprecatedHandler` method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a `Warning` header for anyone using HTTP clients (though they have to be looking for it...).
Graph portion only
Original commit: elastic/x-pack-elasticsearch@d6452a75c1
The value of the setting name had a typo.
Also added a message to show the value of globally configured proxy on startup
to help admins spot if this is configured.
Original commit: elastic/x-pack-elasticsearch@bdc41ff733
This changes the usage stats for the realms to not use nested objects but to instead group
values by the realm type. Additionally, the realms now return the actual size of the users that
are contained or have recently logged in (in case of LDAP/AD).
Additionally, the audit section will always contain an enabled flag and a new field has been
added to capture whether anonymous access is enabled or not.
See elastic/elasticsearch#2210
Original commit: elastic/x-pack-elasticsearch@9cc2852585
There are still tests left to be cleanup, but these tests need to be turned into a rest test and moved the smoke test xpack with mustache module.
Original commit: elastic/x-pack-elasticsearch@3b88b15b97
We currently have the license:base project, as well as license:licensor.
Xpack depends on license base. This change moves all the base license
code into xpack, and makes the license-tools (previously called
licensor) depenend on x-pack.
Original commit: elastic/x-pack-elasticsearch@7fd462ad6a
In 2.x, 1.x license format in cluster state was upgraded
to the 2.x format. This commit removes the code to read
1.x license format from cluster state in 5.x, as 2.x clusters
will already upgrade the license format to 2.x format.
Original commit: elastic/x-pack-elasticsearch@77f18ffc76
This change moves all license related code in xpack under
org.elasticsearch.license. Some classes become package private (more can
become package private later with further deguicing). It also adds
package level javadocs. Note, I did not move anything from the actual
licensing jar. This should be cleaned up in a followup.
Original commit: elastic/x-pack-elasticsearch@2413c4f288
This change removes guice from construction of monitoring classes.
Additionally, it removes disk threshold watermark and enabled setting
from node stats collector. These were not node stats, just cluster
settings. If we want to add back actual percentage of disk threshold
used, it should be in node stats directly.
Original commit: elastic/x-pack-elasticsearch@4cd49557cf
If the result of a search actions fails (i.e. because the index you queried does not exist yet),
the watch record failed to store into the Watch History because of a mapping issue, as the
template path match regular expression did not match properly.
Closeselastic/elasticsearch#2913
Original commit: elastic/x-pack-elasticsearch@3c2d4b3ca9
This is dependent on elastic/elasticsearchelastic/elasticsearch#19589. It also fixes the
existing files that either were missing the license header or had the
open source license header.
Original commit: elastic/x-pack-elasticsearch@2642c20381
The response status code was stored in the result of an http input,
but inaccessible in the payload itself and could not be used in
scripts.
This puts the status code in the payload under the name '_status_code',
similar to the '_headers' variable, which already stores the headers.
Original commit: elastic/x-pack-elasticsearch@dff2a39535
Background: When a watch has been acked, but the condition evaluates to false again,
the watch must be marked as dirty - which means it needs to be persisted to the watches
index - so in case of a master node switch this information is not lost.
This commit fixes the setting of the `dirty` field in the watch status, in case
the condition is not met, but some actions have been acked.
Original commit: elastic/x-pack-elasticsearch@1a55a45b14
The license code currently has a Licensee implementation per feature,
which is updated by the license service. This meant maintaining a
listener type feature for the license service, and having an abstract
listener and such. The licensee class also mixed in stuff only needed by
the license service (acknowledgement messages).
This change collapses all the methods from licensees into
XPackLicenseState. The naming was inconsistent across licensee
implementations, so here it is standardized on `is*Allowed()`. There are
also a number of tests which should be consolidated for testing the
license service but that is left for a future change. I also removed
collector tests that were testing license: that is better left for the
direct tests of the license state in XPackLicenseStateTests.
Original commit: elastic/x-pack-elasticsearch@734871e870
The LicenseState class exists to distinguish when a license is enabled,
vs being inside its grace period. However, the consumers of this state
do not care whether the license is in the grace period, they view that
and an active license as the same thing. The only part that cares about
the grace period is in the license service which logs a warning when a
license begins its grace period.
This change removes the LicenseState enum in favor of a simple boolean
indicating whether the license is active.
Original commit: elastic/x-pack-elasticsearch@5a90a0e3d4
As the specified integration tests were better suited as REST tests, I changed them into
those and removed the (partially already unused) java classes.
Original commit: elastic/x-pack-elasticsearch@f26d8d94e8
We only have one license in 5.x. This change renames the service that
mantains the license state on each node to reflect that fact.
Original commit: elastic/x-pack-elasticsearch@bb241b30cb
This change removes guice from most of the rest of security. It also
converts the last use of onModule in xpack extensions to a pull based
extension.
Original commit: elastic/x-pack-elasticsearch@9de072550e
Both AuthenticationService and AuthorizationService are currently
interfaces with single implementations. This is unnecessary, and makes
it harder to deguice. This change removes the abstractions and leaves
just AuthenticationService and AuthorizationService.
Original commit: elastic/x-pack-elasticsearch@d04c897ae4
AuditTrailService
We currently have a number of actions and components which try to write
to the audit trail. But they do so by expecting a bound AuditTrail
object. In reality, this should always be AuditTrailService, except when
security is disabled. This change removes the use of the AuditTrail
interface for that purpose, and instead makes the AuditTrailService
allow an empty list of trails, so that it is always bound when running
on a node.
Original commit: elastic/x-pack-elasticsearch@9559dbd6c1
This commit removes the SelfReschedulingRunnable and changes the native stores
to use the threadpool for scheduling again since we have now fixed the issue in core.
Original commit: elastic/x-pack-elasticsearch@50030e31ff
This is useful if an index is shared with many small customers, which are to small to have their own index or shard,
and in order to share an index safely they will need to use document level security. This processor can then automatically
add the username or roles of the current authenticated user to the documents being indexed, so that the DLS query can be simple. (`username: abc` only return data inserted by user abc)
Closeselastic/elasticsearch#2738
Original commit: elastic/x-pack-elasticsearch@f4df2f6d6f
Currently the security context is an object passed around to code
needing to check the user for the current request. Like recent
InternalClient changes, it current depends on the AuthenticationService,
but can be simplified by only knowing about the thread context and
crypto service. This change makes SecurityContext a class, instead of an
interface, and removes the dependency on AuthenticationService.
Original commit: elastic/x-pack-elasticsearch@b8af75e8cb
Followup of elastic/elasticsearchelastic/elasticsearch#19435
Relates to elastic/elasticsearchelastic/elasticsearch#19412
Original commit: elastic/x-pack-elasticsearch@60f7047ea9
This commit is a combination of enhancements and fixes to the active directory
and ldap realms. The active directory realm has been enhanced to add support
for authentication against multiple domains in a forest. The ldap realm has
been updated so that:
* attributes required for group resolution are loaded eagerly if possible
* user search can now be executed using unpooled connections
* the default search filter for groups now includes posixGroup and memberUid
to avoid users needed to understand ldap filters
Finally, the UnboundID LDAP SDK was upgraded to the latest version and some
long standing AwaitsFix were addressed.
Closeselastic/elasticsearch#20Closeselastic/elasticsearch#26Closeselastic/elasticsearch#1950Closeselastic/elasticsearch#2145Closeselastic/elasticsearch#2363
Original commit: elastic/x-pack-elasticsearch@63c9be2337
This commit adds the ability to define metadata for roles. This metadata is currently
only used for the API and to indicate that a role is reserved. We can continue passing
on the metadata as needed, when necessary.
Closeselastic/elasticsearch#2036
Original commit: elastic/x-pack-elasticsearch@8b5f606138
This commit adds a CLI tool that can be used to generate a CA and signed certificates in PEM
format. The tool only requires a name of an instance to be provided by the user; ip and dns values
are supported but optional. By default, the tool is interactive and will prompt the user for input but
an option exists to provide a yaml file that contains the necessary information to generate certificates
or signing requests.
The output is in the form of a zip file with subfolders for each instance. Neither the zip file or the PEM
files are encrypted as some parts of our stack do not support encrypted PEM files.
Original commit: elastic/x-pack-elasticsearch@3dc0f8d495
When a HTTP input has a configured response content, then this should
always be treated as preferred over the content type that is returned
by the server in order to give the user the power to decide.
This also refactors the code a bit to make it more readable.
Closeselastic/elasticsearch#2211
Original commit: elastic/x-pack-elasticsearch@ecdb4f931c
ScriptServiceProxy is a thin wrapper around the ScriptService which does
a runAs the xpack user when compiling. But script services know nothing
about xpack users, so this has no real effect. I believe this is a
remnant of when we had indexed scripts, where the compilation may have
done a get on the scripts index.
This change removes the ScriptServiceProxy. It also renames Script in
watcher to WatcherScript, to remove confusion between elasticsearch's
Script and watchers Script.
Original commit: elastic/x-pack-elasticsearch@4e2fdbc518
This change makes the internal realms factories, as well as those added
by extensions, constructed directly instead of via guice. Adding realms
in extensions is now pull based. Finally, all of the generics for realms
and realm factories have been removed.
Original commit: elastic/x-pack-elasticsearch@f0de9d2340
"cloud_internal" license type enables dynamically updating license operation mode via a config file.
When the installed license is "cloud_internal", the node level operation mode can be updated by writing
a `license_mode` file in the x-pack config directory (config/x-pack/license_mode). The file is expected
to have a string representing the desired license mode (e.g. "gold", "basic"). In case of a failure to
read a valid license mode from the `license_mode` file, the operation mode for "cloud_internal" license
defaults to PLATINUM.
This change also ensures that the correct operation mode is reported via the _xpack endpoint.
closeselastic/elasticsearch#2042
Original commit: elastic/x-pack-elasticsearch@6a2d788e45
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#19440. It adds a
getter on XPackExtension for extensions that add custom rest headers, in
addition to the headers registered for xpack itself.
Original commit: elastic/x-pack-elasticsearch@bd142b88c6
This commit moves an awaits fix from
MonitoringIndexNameResolverTestCase#testResolver to
MonitoringIndexNameResolverTestCase#testSource as the previous commit
elastic/x-pack@98e76642ea marked the incorrect test as
awaits fix.
Original commit: elastic/x-pack-elasticsearch@62e9aaa628
The InternalClient is used by xpack code making requests to other nodes,
to add the xpack user to the thread context. To do this, it uses has a
dependency on the AuthenticationService, which in turn transitively
depends on InternalClient (eg IndexAuditTrail). But to add the xpack
user, the full authentication service is not necessary. Only having the
crypto service is needed in order to encrypt the header.
This change simplifes construction of InternalClient both making it a
real class instead of an interface, and removing the dependency on the
AuthenticationService. It also removes the use of
Provider<InternalClient> in users of the client.
Original commit: elastic/x-pack-elasticsearch@10f633bdf3
Both logfile and index audit trails currently depend on injection of
Transport in order to find the bound address of the local node. However,
the ClusterService provides access to information about the local node,
including the bound addresses. This change makes the audit trails use
the cluster service, and also makes the logging audit trail not use a
lifecycle.
Original commit: elastic/x-pack-elasticsearch@d747d64ee1
This provides the same -- more useful -- index memory chart to the index page, instead of just "Lucene Memory", it now breaks down what it shows just like the Node page as as more generalized "Index Memory" chart. It also rolls Fielddata into the Index Memory. With Fielddata now in the Index Memory chart, I decided to add the Segment Count chart to the Index page.
Original commit: elastic/x-pack-elasticsearch@4e3490ce9c
elastic/elasticsearchelastic/elasticsearch#19428 removes `node.mode` and `node.local` this PR
fixes xplugins to configure networking explicitly.
Original commit: elastic/x-pack-elasticsearch@ee8daa36dd
This change removes some of the complexity around licensing classes in
xpack. It removes the interfaces for registration and management so the
remaining LicensesService class is the thing that components wanting to
interact with the license should use. It also removes complexity around
the Licensee interface, removing generics and the registration at
construction time, as well as making the licensees no longer have a
lifecycle. There is still more to be done with simplification of license
classes construction, but this is a step towards a simpler world.
Original commit: elastic/x-pack-elasticsearch@5307d67b5b
This makes use of the registerAsDeprecatedHandler method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a Warning header for anyone using HTTP clients (though they have to be looking for it...).
- This also adds deprecated `/_licenses` variants of the endpoint. Users are consistently making that mistake, and it's easy enough to support this way. Can remove it if people disagree though.
License portion only
Original commit: elastic/x-pack-elasticsearch@431c871fcf
This makes use of the registerAsDeprecatedHandler method to automatically warn users when they're using deprecated functionality.
This will also automatically provide a Warning header for anyone using HTTP clients (though they have to be looking for it...).
- This also changes from PUT _start, _restart, _stop (Watcher endpoints) to POST _start, _restart, _stop
- The deprecated variant still honors PUT
- Nothing about the hijack endpoints was deprecated because they did not change from 2.x
Watcher portion only
Original commit: elastic/x-pack-elasticsearch@36f87a6526
When the monitoring tests are run in isolation, they succeed. However, when the whole suite of
REST tests is being run at the same time, the "Bulk indexing of monitoring data" intermittently
fails with a timeout. Therefore, a timeout of 60 seconds has been added.
Closeselastic/elasticsearch#2809
Original commit: elastic/x-pack-elasticsearch@d11dc7a2be
This particular test requires Elasticsearch to run on port 9400, which is not
guaranteed if the clients run their own tests, as it is a matter of configuration
in gradle.
Therefore these tests need to run in their project.
Original commit: elastic/x-pack-elasticsearch@da38407766