Removed SchedulerStats as scheduler status is all we need and start and end times are only needed in start scheduler api.
Original commit: elastic/x-pack-elasticsearch@80c563cb69
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.
Original commit: elastic/x-pack-elasticsearch@cff21e21ee
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.
As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Original commit: elastic/x-pack-elasticsearch@cced57b884
The latest release of the mock web server requires more security permissions, and
we dont need all the functionality anyway.
This introduces a small MockWebServer using the JDK internal HttpServer, yet fullfilling
all our needs and supporting SSL as well for testing.
The MockWebServer allows to enqueue responses and also requires you to enqueue as many responses
as you requests will be executed - there is no fallback at the moment.
SSL is also supported by passing an SSL context - for which the TestsSSLService is needed, which
makes the required methods public.
Original commit: elastic/x-pack-elasticsearch@55f4a172a2
* Use well-defined IDs for records and influencers
Removes the reliance on ES autogenerated UUIDs for all types that will
be renormalized
* Address some review comments
Original commit: elastic/x-pack-elasticsearch@85fde8b957
* Make ModelDebugOutput a result type
* Delete unused ElasticsearchBatchedModelDebugOutputIterator
* Add result_type field to ModelDebugOutput
* Address review comments
Original commit: elastic/x-pack-elasticsearch@a48e4cd946
The start scheduler api call will run until the scheduler has completed. Either when lookback only scheduler completes or the scheduler has been stopped.
The start scheduler api will first update the scheduler status from STOPPED to STARTED on master node and then start running the scheduler.
Once a scheduled job completes it updates the scheduler status from STARTED to STOPPED and then the start schedule api returns.
The STARTING and STOPPING statuses are no longer used, so have been removed.
The stop scheduler api is a sugar api that uses the task list and cancel apis stop the scheduler.
Renamed ScheduledJobService to ScheduledJobRunner
Original commit: elastic/x-pack-elasticsearch@ab504fe3d9
Now when a cluster gets updated with a new license,
the license uid and type will be logged as info. Making
it easier to verify the current license mode of the cluster.
Original commit: elastic/x-pack-elasticsearch@c205cb42ae
Make jobId and from/size mutually exclusive options.
This approach has the main properties of not allowing an invalid Request to be built, and alerting the user if they set an incorrect configuration. It has the downside that PageParams can be null so the consumer will have to check for it. Since jobId could be null before, this seemed acceptable.
Original commit: elastic/x-pack-elasticsearch@106dcdf61a
Adds a delete_list endpoint. If a list is currently in use by a job, it is not allowed to be deleted
Original commit: elastic/x-pack-elasticsearch@7d9a984b3a
There were some test failures as LicenseServiceClusterTests was not properly setting minimum_master_nodes while having autoMinMasterNodes disabled.
Original commit: elastic/x-pack-elasticsearch@2030db2424
This avoids the confusing situation that a there is no allocation when a job hasn't been opened yet. Now it complains about the fact that the job status is closed.
Original commit: elastic/x-pack-elasticsearch@3159dc6954
* Collapse ElasticsearchBulkDeleter into JobDataDeleter
* Add blocking delete to JobDataDeleter
* Delete interim results only after all the results are parsed.
* Remove unused deleteModelSizeStats and deleteModelDebugOutput methods.
Document missing javadoc tags
Original commit: elastic/x-pack-elasticsearch@1997541673
* A job now has the following statuses: OPENING, OPENED, CLOSING, CLOSED and FAILED.
* The open job and close job APIs wait until the job gets into a OPENED or CLOSED state.
* The post data api no longer lazily opens a job and fails if the job has not been opened.
* When a job gets into a failed state also the reason is recorded in the allocation.
* Removed pause and resume APIs.
* Made `max_running_jobs` setting dynamically updatedable.
Original commit: elastic/x-pack-elasticsearch@3485ec5317
A companion PR to https://github.com/elastic/elasticsearch/pull/21846 where the above methods were removed. See ES PR for details.
With the concurrent starting the issues with licenses and time freeze became more apparent and I had to apply my suggestion to only freeze time once the license has been applied (as opposed to freeze on node start up). Since this also means that a node that starts up after the cluster time has been frozen need to also immediately freeze, it felt natural to use a `ServiceDisruptionScheme`. Although the name doesn't really make sense here, it all has all the logic we need.
Original commit: elastic/x-pack-elasticsearch@5641742f60
This commit migrates the actual authentication calls that are performed by internal realms
to be asynchronous so that we do not execute blocking calls on network threads. The realms
that used LDAP were especially bad as they can issue many different requests and/or open
multiple connections per authentication attempt.
The LdapRealm now uses the ThreadPool to fork a thread for authentication. This is necessary
because a LDAP bind operation is not allowed to be performed asynchronously and must be done
in a blocking manner. After the bind request has completed, all other requests can be done
asynchrnously. The LdapUtils class now provides asynchronous methods for searching and also
includes the handling of referral following as automated following is not supported by
UnboundID when using the asynchronous calls.
Original commit: elastic/x-pack-elasticsearch@dfb259e498
* [Watcher] HistoryStore should hide secrets
Some watcher actions like Jira stores the whole HTTP request in case of a failure. This is very helpful when something goes wrong, but it has to hide the password used for Basic authentication otherwise it will appear in the watch record in the .watcher-history index. In general, I would consider better to always hide secrets when storing something in the history index.
This commit removes the XContentParams "hide secrets" set at the JiraIssue so that it is now set at the HistoryStore level.
It also fixes test failures that can be reproduced with:
gradle :x-plugins:elasticsearch:test -Dtests.seed=BA730F93E726AECC -Dtests.class=org.elasticsearch.xpack.notification.jira.JiraIssueTests -Dtests.method="testToXContent" -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=Asia/Choibalsan
Original commit: elastic/x-pack-elasticsearch@d686b3b3ad
The local exporter previously fired off asynchronous requests every time a cluster state was
observed that did not contain all of the required items for monitoring. This change adds a
flag so that monitoring can wait for the pending requests to complete before retrying. This
will reduce the number of duplicated log messages as well.
Additionally, the log message for adding modern aliases now contains the name of the indices.
Closeselastic/elasticsearch#3756
Original commit: elastic/x-pack-elasticsearch@727a0adfbe
* Add test to read model size stats
* Most recent model_size_stats document should have the name ‘model_size_stats’
Original commit: elastic/x-pack-elasticsearch@e192d4c34d
* Allow POST with body to get records
* Allow records endpoint to accept POST requests with body
* CategoryDefinition can accept POST requests with body parameters
Original commit: elastic/x-pack-elasticsearch@2edb7a9c47
* Convert security rest filter to rest handler wrapper
This is the xpack side of elastic/elasticsearchelastic/elasticsearch#21905
Original commit: elastic/x-pack-elasticsearch@38bfa771b6
* Make persist DataCounts a non-blocking operation
* Add trace and debug level logging to the persist data counts action listener.
Remove dead code from test
Original commit: elastic/x-pack-elasticsearch@84bbfa880a
This commit removes some unnecessary code: collectors do not need to extend AbstractLifecycleComponent and do not need to be started/stopped. The extra
interface Collector is removed and AbstractCollector renamed to Collector.
Original commit: elastic/x-pack-elasticsearch@75893d66e3
Transform and condition scripts were only compiled on its initial creation, so
when a new watch is created or when the master node loads all the watches. However
changing a script (like a stored one) did not lead to any changes in the in memory
watch store and thus the old script was executed again.
We do however have a mechanism in Elasticsearch's ScriptService that already does some
caching, and should reuse that one.
Closeselastic/elasticsearch#4237
Original commit: elastic/x-pack-elasticsearch@477548e237
This commit improves the output of the migrate tool in cases when there are errors parsing entries
in the roles or users files. This is done through the use of a logger that delegates its output to
the terminal.
Additionally, the `-c` option has been removed. This option was used to set the configuration directory
but this should be handled one way only and that is through the use of the `-Epath.conf` setting.
Closeselastic/elasticsearch#3757Closeselastic/elasticsearch#3758
Original commit: elastic/x-pack-elasticsearch@811e367766
Currently we have these logs for integration tests only.
This adds the following log at the start:
```
logger.info("[{}]: before test", getTestName());
```
and this is logged at the end, but before any clean up done in sub classes
```
logger.info("[{}]: after test", getTestName());
```
Original commit: elastic/x-pack-elasticsearch@2ca7296665
It may take a few clicks before the templates are added.
Waiting for started doesn't guarantee that the index templates have been added.
Original commit: elastic/x-pack-elasticsearch@ee94b740a8
The changes the way the Elasticsearch plugin module gets the cpp code when running the bundlePlugin task. Instead of manually going into the cpp builds output and grabbing the folders it depends on the buildZip task of the :cpp project and grabs the resource that task builds. This means that it does not matter if the cpp buildZip task is changed as long as it results in a zip being generated.
Original commit: elastic/x-pack-elasticsearch@257e5df2e7
This commit adds tests for DNS only hostname verification. This is a followup of elastic/elasticsearchelastic/elasticsearch#21828, which fixes issues with this type of hostname verification
due to some addresses losing the host information.
Original commit: elastic/x-pack-elasticsearch@8a63bb113d
Make sure CountDownLatch gets removed when it is no longer needed
Also add CountDownLatch is it is missing when we ack a flush id,
we may ack before we wait for it
Original commit: elastic/x-pack-elasticsearch@83a993b9ad
When we encounter a composite request, we authorize at first without looking at the indices, to see whether the action can be executed at all. We then rely on the action to delegate to an inner action per sub-request, which will be authorized based on the indices it refers to. The first step works great for the simulate mode of search template, as it doesn't involve any index. The second step will make sure that when search template involves a search, it will be authorized as a normal search request would, based on the indices it reads from.
Note that the wildcard expansion happens now on the search side, it doesn't have to happen when executing the first authorization step, hence SearchTemplateRequest doesn't have to implement IndicesRequest, only SearchRequest has to (which it does already).
Closeselastic/elasticsearch#4171
Original commit: elastic/x-pack-elasticsearch@d586bd90cb
* Persist DataCounts every 10 seconds from a thread pool scheduled task.
Also rework the isReportingBoundary function changing the function as the boundary changes
* Remove overloaded constructors from StatusReporter and DummyStatusReporter
* Persist dataCounts in a background thread from the status reporter
* Use generic threadpool
Original commit: elastic/x-pack-elasticsearch@f00c1067aa
* Renamed REST path parameters to snake case.
And added missing parameter descriptions
* Document endpoint which accept body parameters
* Endpoints that support body parameters must also accept POST
Original commit: elastic/x-pack-elasticsearch@2cad2e8af6
This commit applies the same fix merged in elastic/elasticsearch#4179 for XDocsClientYamlTestSuiteIT. It adds a waitForSecurityTemplate() method in order to wait for the security-index-template to be created by the SecurityTemplateService.
Original commit: elastic/x-pack-elasticsearch@1476f30e2d
The problem is that the Java security manager can open the pipe and then
quickly close it, leading to a need to reconnect from the C++ side.
Original commit: elastic/x-pack-elasticsearch@772b57f443
it can happen we encounter more then one cluster states with no usage index and causing this class to send two or more create index requests, but only one request will succeed.
Original commit: elastic/x-pack-elasticsearch@524a1dda61
This commit moves the iteration of realms for authentication and user lookup to
be done in an asynchronous fashion. The existing blocking methods have been deprecated
to allow custom realm implementors time to switch. All internal realms implement the
asynchronous methods.
This PR is another step toward the full migration to async authentication, but does not
complete the work. Additional work is needed for the LDAP realms, which make blocking
network calls. These blocking calls will be handled in a follow-up PR.
See elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@a65a9b2bb4
Also moved all creation logic that is required to run at cluster startup into PrelertInitializationService.
Original commit: elastic/x-pack-elasticsearch@453ba3efa3
* Add result_type field to bucket
* Query and delete buckets/records by result_type
* Add a filter to the ElasticsearchBatchedDocumentsIterator subclasses for result_type:bucket
* De-type Influencers, BucketInfluencers and Category Definitions
* Revert de-typing CategoryDefinition
* Resolve merge errors after rebase
Original commit: elastic/x-pack-elasticsearch@65605432e8
If the HTTP response is an invalid one, it is still logged as success.
This commit changes the behaviour, that if the response status code is
set to -1 (which means it could not be interpreted), than an IOException
is thrown and thus the execution will be marked as a failure.
Closeselastic/elasticsearch#4152
Original commit: elastic/x-pack-elasticsearch@5736fbe3c0
In c++ the results are built all together under a bucket hierarchy.
This buckets was written out and java would read it and split the
bucket into its parts: the bucket itself, its records and its
influencers.
During the migration, the bucket started being persisted as a whole,
including its records and influencers.
This commit is changing this by modifying the way results are written
in c++. This way, the java and c++ results writing/reading are in sync.
To achieve this, the change involved writing records and influencers as
top level results from c++. In addition, they are written as an array
object in order to allow the java side to persist them in a bulk
request.
* Fix bucket counting in results processor
Original commit: elastic/x-pack-elasticsearch@feadf3f887
Elasticsearch will now attempt to spawn a controller daemon called simply
`controller` when it starts up, but ours was called `prelert_controller`.
For consistency it makes sense to remove the prefix from all our programs.
Original commit: elastic/x-pack-elasticsearch@d06714b231
* Make close job a master node action
The close job action now:
- Is a master node action
- Sets the job status to CLOSING
- Waits for the job status to change to CLOSED before it responds
JobLifeCycleService picks up on a job status change to CLOSING and
closes the job. At the end, it sets the job status to CLOSED.
* Assert job status is closed after close in integration test
This also correctly passes an ActionListener to JobManager.setJobStatus
in order to ensure the job status request has completed and to properly
propagate failures.
Original commit: elastic/x-pack-elasticsearch@1546c77fca
The PID used needs to be the PID of autodetect's parent proecss. The parent
is of course the controller daemon rather than the JVM.
Original commit: elastic/x-pack-elasticsearch@607481e1e2
* Do not try bind to port range but use free ephemeral port
* Start a new email server in all tests, do not use static one
* Remove selection of username/password, as it was static anyway
* Remove Listener.Handle class, as it is not needed, when not running in static context
Original commit: elastic/x-pack-elasticsearch@8816cc25f6
* Added a setting that determines the maximum number of jobs that can run on a single node.
* Fail to start autodetect process if a user attempts to start more jobs than is allowed on a single node.
* Prevent concurrent data write, flush and close operation to the autodetect process.
Original commit: elastic/x-pack-elasticsearch@aca15fd51c
QueryPage now requires a ParseField in the constructor, which is used for the name of the array of results. I considered putting an enum of different field names in QueryPage itself, but it seemed better to keep the field name local to each respective object.
hitCount was also renamed to count, and getters updated appropriately.
Finally, added a static helper to throw the ResourceNotFoundException, just to make life easier/more consistent.
Closeselastic/elasticsearch#359
Original commit: elastic/x-pack-elasticsearch@9ba42ad4a1
Putting the various endpoints back to their plural form (and adjusting Java class names as necessary).
Original commit: elastic/x-pack-elasticsearch@5c8f3c7341
OldMonitoringIndicesBackwardsCompatibilityTests fails because it waits for more shards stats to be collected but that can only work if new indices are created in the meanwhile.
Original commit: elastic/x-pack-elasticsearch@003c28cf93
Also uploads the pack zip to the nas instead of the elasticsearch plugin.
The cpp build can be disabled with `-Pxpack.cpp.build=false`
Original commit: elastic/x-pack-elasticsearch@1efb1b2e7e
This commit updates the watch_history.json file so that it includes mappings for the new Jira action. It also update the JiraIssue format so that it now includes the name of the account used to create the Jira issue. It also update the REST tests to check that Jira action result are searchable and hide the user's password.
Original commit: elastic/x-pack-elasticsearch@75888f7748
* Add scheduler status filter to jobs endpoint
* For scheduled jobs set the initial scheduler state
* Add status filter to job endpoint
Original commit: elastic/x-pack-elasticsearch@c7ed1627e2
This commit adds tests to ensure that user lookup in caching realms works as expected. An unclear
contract in the Cache#computeIfAbsent method allowed for null values to be returned from this method
even if there should have been exception reported to the loader. This has been fixed in the cache
implementation and we add tests to verify that the caching of user lookups is done properly under
concurrent operations.
Closeselastic/elasticsearch#4054
Original commit: elastic/x-pack-elasticsearch@41567c6ed9
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
Currently, a tribe node ignored underlying cluster licenses
due to inablity to select an appropriate license from multiple
licenses. Now that tribe node supports merging custom metadata
(elasticsearch#elastic/elasticsearch#21552), we can enable license support in tribe
node.
Now, tribe node chooses license with the highest operation
mode from underlying cluster licenses. This commit also
adds integration tests for licensing to verify that:
- autogenerated trial license propagates to tribe node
- tribe node chooses the highest operation mode license
- removing a license from underlying cluster license is
removed from tribe
closeselastic/elasticsearch#3212
Original commit: elastic/x-pack-elasticsearch@b5c003decd
This commit changes the toXContent() method so that it generates content that can be parsed again by the parse() method. Before this commit, the authentication of the HTTP request is rendered as:
{ "auth": {"username": "foo", "password": "bar" } }
but the parsing method expects the authentication type to be a root object:
{ "auth": { "basic" : {"username": "foo", "password": "bar" } } }
The toXContent method has been adapted to include the type of authentication in the generated content.
Original commit: elastic/x-pack-elasticsearch@b740466109
This removes the mock webserver trying to bind to a port range, just
try to bind to any port.
Also removed unneeded wrapper class in the process.
Original commit: elastic/x-pack-elasticsearch@19b6ac393a
When a fatal error is thrown on the network layer, such an error never
makes its way to the uncaught exception handler. This prevents the node
from being torn down if an out of memory error or other fatal error is
thrown while handling HTTP or transport traffic. This commit adds logic
to ensure that such errors bubble their way up to the uncaught exception
handler, even though Netty tries really hard to swallow everything.
Original commit: elastic/x-pack-elasticsearch@f76757674f
In the same vein of GetBucket/GetBuckets and GetCategory/GetCategories.
This one was a bit more involved, since previously GetJobs didn't support
configuring the various metrics. So now all metrics can be configured when
requesting both a single job, or a range of jobs. It also unifies the request
body handling between the two and adds POST handlers.
And similar to the other refactorings, the SingleDoc response is gone in
favor of always returning an array of hits.
Original commit: elastic/x-pack-elasticsearch@ac47bb9bf6
These classes are no longer needed. They used to be
necessary in order to handle compressed data and sending
data to multiple jobs. Compressed data is now handled
by elasticsearch and multi-job upload is no longer supported.
Original commit: elastic/x-pack-elasticsearch@08a9f29855
This commit aims to fix the regularly failing test OldMonitoringIndicesBackwardsCompatibilityTests. It enables/disables monitoring data collection using time interval "-1" as well as stopping AgentService instances directly and checks multiple times if indices are still present at the end of the test.
closeselastic/elasticsearch#3999
Original commit: elastic/x-pack-elasticsearch@8ac785061a
* Remove DataCounts and ModelSizeStats from Job
* Delete unused ElasticsearchJobDetailsMapper
Conflicts:
elasticsearch/src/main/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapper.java
elasticsearch/src/test/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapperTests.java
* Add missing mappings for DataCounts
Fixes yaml test failures
* Add test to assert revert a model snapshot sets DataCounts.latest_record_timestamp
* Resolve merge errors
* Add NORELEASE for persisting dataCounts in cluster update
Original commit: elastic/x-pack-elasticsearch@46099d4db6
* Record audit trail even if indicies-resolver throws exception
If the IndicesAndAliasesResolver throws an exception, treat is as "accessDenied" for the purpose of the audit-trail.
This can occur when an index request has a wildcard that doesn't match (and "allowNoIndices" is false)
Closeselastic/elasticsearch#3719
Original commit: elastic/x-pack-elasticsearch@ca6567e5ed
This change moves the Java code from `/java/apps/engine-node` to `/elasticsearch` so it matches the structure required by X-Pack. This will make it easier to get the gradle build working correct and building the full Pack zip.
Original commit: elastic/x-pack-elasticsearch@2fd6539e85
RemoteIndexAuditTrailStartingTests has been failing quite often due to several
race conditions. This change simplifies the test (one node per cluster, local and
remote) and fixes some issues in IndexAuditTrail. The interrupt based shutdown has
been removed from the `QueueConsumer` and an additional health call has been added
to trigger another start attempt if the first one failed. This stabilize the test
that now run in 1 second rather than 1 minute.
Original commit: elastic/x-pack-elasticsearch@a1c8131cd2
This commit responds to a chance in core which modified the interface
for ThreadPool#executor and ThreadPool#generic to return an executor
service rather than an executor.
Relates elastic/elasticsearch#4107
Original commit: elastic/x-pack-elasticsearch@7c7c6a3b90
This is a followup from elastic/elasticsearchelastic/elasticsearch#21590 and needs to be
committed first or at the same time since netty_3 is removed
Original commit: elastic/x-pack-elasticsearch@131d74dd6b
If these tests were in `:core` then then `IT` would be an appropriate
suffix, but in xplugins they should have `Tests`. This moves them
from the `integTest` task to the `test` task which is a good idea
because it lets them run in parallel with the other tests. Naming
them `IT` means that they are run not in parallel and they are run
with a running instance of Elasticsearch which they don't need.
This cuts the build from 15.5 minutes to 13.5 minutes for me.
Original commit: elastic/x-pack-elasticsearch@7b4bcf8bc5
This commit fixes a compilation issue introduced by the merge of
sequence numbers branch into master. This merge introduced a new
constructor parameter for ShardStats.
Original commit: elastic/x-pack-elasticsearch@b234b8ee50
This commit migrates the authentication service to an asynchronous model where we use listeners
instead of blocking and waiting for the authentication to return. This is the first part of making
authentication asynchronous as we still have blocking I/O inside of realms.
See elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@9339af4af8
In the authentication service, we currently only copy the username and roles of the
user that was authenticated but we should instead preserve all of their information
in the newly created user object. This change does that through the user of a new
constructor in the user class that takes in both users.
Closeselastic/elasticsearch#3877
Original commit: elastic/x-pack-elasticsearch@7455078841
If a authenticated user with run as permission attempts to run as an unknown user, the unknown
user will be assigned the default role and anonymous role if enabled. This change prevents this
from happening as we require the run as user to have been looked up by a realm.
Closeselastic/elasticsearch#3878
Original commit: elastic/x-pack-elasticsearch@034f44757d
The internal test cluster now auto manages min_master_nodes ( see https://github.com/elastic/elasticsearch/pull/21458 ). This requires some code changes but also changes the timings of forming a cluster. This has had a funny side effect where the master is no longer always the first node to be started in the cluster. This caused issues with watcher tests which freeze time.
Original commit: elastic/x-pack-elasticsearch@1e5ea8ae94
This change mocks the ClusterService and sets it up so that when remote indexing, we see the
current node as the master.
Original commit: elastic/x-pack-elasticsearch@1f26b3df1f
Template test is easy to fix. I have no idea what is up with the
dynamic index name test. It is finding results on the wrong time.
Original commit: elastic/x-pack-elasticsearch@535676164c
Today when a message is not fully read on a response, we log (among
other details) the handler name. Unfortunately, if the handler is a
wrapper, all that we see is
o.e.x.s.t.TransportService$ContextRestoreResponseHandler@7446ba18
completely losing the offending handler. This commit adds an override
for TransportService$ContextRestoreResponseHandler#toString so that the
underlying offender can be discovered.
Relates elastic/elasticsearch#4043
Original commit: elastic/x-pack-elasticsearch@3eb1003d5b
This removes all guice module classes from the watcher codebase, so that guice is
only used for the transport and rest actions, but nowhere else in the codebase.
Also it ensures, that only ticker/schedule are valid trigger engine options.
Original commit: elastic/x-pack-elasticsearch@400ba24c33
There was an unused and unconverted xpack.watcher.plugin.version
in the watch history index template, which resulted in this template output:
```
"watch_history_1" : {
"order" : 2147483647,
"template" : ".watcher-history-1*",
"settings" : {
... ,
"xpack" : {
"watcher" : {
"plugin" : {
"version" : "${xpack.watcher.plugin.version}"
},
"template" : {
"version" : "1"
}
}
}
}
```
As everything is one plugin now, this can be safely removed.
Original commit: elastic/x-pack-elasticsearch@c5c0bcaaaa
This change adds the restoration of the original context inside the listeners and handlers where
we restore another context. This prevents us from polluting the context of the thread that called
the listener and leaving around a different user in the thread context.
Original commit: elastic/x-pack-elasticsearch@0f30363ef7
As we are not dependent on any versions any more and we never exposed
an version type functionality, nor documented it nor tested it, we
should remove this for the next major release.
Note, this just removes the ability to set those options, which we ignored
anyway in the transport action.
Original commit: elastic/x-pack-elasticsearch@3830203f50
This commit stops the index audit trail from executing blocking calls on the cluster
state update thread. Blocking calls were executed when indexing to a remote cluster
to get that cluster's state and also possibly put a template and mappings.
Closeselastic/elasticsearch#3989
Original commit: elastic/x-pack-elasticsearch@a8c0269fad
Ensure that tests never sleep, but run using timewarping,
so that sleeping can be removed from the test code.
Original commit: elastic/x-pack-elasticsearch@40fc3814f9
This prevents a possible NPE when sending emails, as some host have
a perfectly fine internet connection, but cannot resolve their localhost.
In addition I also removed a EmailService.send() method that was only used
in tests and thus not needed.
Closeselastic/elasticsearch#3227
Original commit: elastic/x-pack-elasticsearch@d2e29b4c92
The clock interface is merely used to create mock clocks and inject them into tests.
We can do this with the java8 based java.time.Clock class as well, so there is no need
to keep this interface.
Original commit: elastic/x-pack-elasticsearch@ae30dc29ca
The base test case class has been adapted to assert that no warn nor
error messages are logged to the Log4j status logger. An issue that
exists in x-pack uncovered by this change is that the mock appender in
the capturing logger was never started. This commit addresses this
issue.
Relates elastic/elasticsearch#3988
Original commit: elastic/x-pack-elasticsearch@8fc0b99bae
Make them do things slightly less frequently and wait longer
to make sure everything is shut down. Sometimes it takes a while
but it does get there.
Original commit: elastic/x-pack-elasticsearch@76fc4acf47
This explicitly shuts down monitoring at the end of every index
which should help prevent the shutdown problems we were seeing.
Original commit: elastic/x-pack-elasticsearch@2be0ee0df8
We only need it to be so fast when starting up. We can slow it
down once it has started so shutdown isn't problematic.
Original commit: elastic/x-pack-elasticsearch@052d64402f
* [Monitoring] Use Suffix with Marvel Index alias
Using a suffix allows a non-aliased Monitoring index get created for _today_. Otherwise the time-based index for today cannot be created.
Original commit: elastic/x-pack-elasticsearch@92e7695a2c
These tests were failing for me locally with this reproduction
line about 80% of the time:
```
gradle :x-plugins:elasticsearch:integTest -Dtests.seed=660D249EDCC648E5 -Dtests.class=org.elasticsearch.xpack.monitoring.OldMonitoringIndicesBackwardsCompatibilityIT -Dtests.method="testOldIndexes" -Dtests.security.manager=true -Dtests.jvms=12 -Dtests.locale=vi-VN -Dtests.timezone=Arctic/Longyearbyen
```
This was due to the ten second collection interval lining up with the
10 second sleep in `assertBusy`. Eventually the collection interval
lines up in such a way that it take more than ten seconds total to
create the alias for the monitoring index. 😢
This fixes that by dropping the interval to 100ms, making the test
no longer fail with that seed and succeed much more quickly. 😄
Relates to elastic/elasticsearch#3951
Original commit: elastic/x-pack-elasticsearch@b5dfa6ef7d
Those tests ran 35 seconds on my local notebook. By not using a sleep based
scripting engine and some other tweaks the time is down to 13 seconds.
Also renamed the class to remove the `Slow` prefix.
Original commit: elastic/x-pack-elasticsearch@5289fe8dab
Moves the tests for aliases starting with `-` into the backwards
compatibility tests because we can no longer create such aliases.
Original commit: elastic/x-pack-elasticsearch@3639fe4d46
* [Monitoring UI] Show Replica Count not Replication Factor in Overview
This changes it to only show the replica count as `total - primaries` rather than showing the replication factor, which is particularly unhelpful when different indices have different replica counts.
Original commit: elastic/x-pack-elasticsearch@552f94bf8f
`action.destructive_requires_name` setting was ignored by the security plugin as wildcards got expanded and resolved in the plugin before es core could actually check if the operation was supposed to be allowed or not. We are discussing how we could perform the check earlier in es core, but anyways it is good to perform the same check in the security plugin just to make sure.
Closeselastic/elasticsearch#3689
Original commit: elastic/x-pack-elasticsearch@3414cb3471
The old handling was not thread safe, as it used to replace volatile
objects in the code. This implementation uses a concurrent hashmap
to easily allow adding/removing schedules without having to replace
whole objects
Original commit: elastic/x-pack-elasticsearch@0aa618b372
When parsing chain inputs there were possibilities to write invalid
JSON that resulting in losing the order of the inputs without any
exception being thrown.
This commit makes the parsing more strict.
Closeselastic/elasticsearch#3736
Original commit: elastic/x-pack-elasticsearch@963641ee2b
Currently security always parses the permissions filters with a shard id equal
to `0` even if the query is executed on a different shard. Also it does not
protect against queries that may rely on the current timestamp even though we
don`t currently have ways to make sure that all shards use a consistent
timestamp.
Sibling of elastic/elasticsearchelastic/elasticsearch#21196.
Original commit: elastic/x-pack-elasticsearch@cab47f2ed2
The execution time of the trigger tests was extremely slow, because it
really waited until executions happened. This uses the mock clock to
advance in time manually.
This also allows to remove the bad apples annotation and make sure that
the schedule engine tests for both implementations are run all the time.
Relates elastic/elasticsearch#1007
Original commit: elastic/x-pack-elasticsearch@f9436f506f
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
Original commit: elastic/x-pack-elasticsearch@d7c7bd7362
My current assumption is, that creating the templates is not
yet finished (as this is async), so that we need to add
another check that the templates have been added before
continuing.
Relates elastic/elasticsearch#3892
Original commit: elastic/x-pack-elasticsearch@3880d200a1
Today we have the same madness in two places and no dedicated test. This
change moves the real madness into a single place and adds a test for it
to make sure it actually works and isn't just crazy.
Original commit: elastic/x-pack-elasticsearch@dabf5fdd63
We used to be very lenient with all kinds of exceptions related to the
`.security` index. Yet, sometimes in tests the index is not yet there but
transport clients already pinging the node this causes issues and transport
clients disconnect. Now if the index is not present we simply return no role.
Original commit: elastic/x-pack-elasticsearch@60948d0c2a
TransportGetRolesAction optimizes for single role case while this
optimization can be simply inside the NativeRoleStore and being
way more contained.
Original commit: elastic/x-pack-elasticsearch@c43d8ba341
`TransportGetUsersAction` does some funky blocking calls even though
it's specifying `SAME` as the thread-pool indicating that it's fast or
forking off quickly. Both might not be true today. This change adds
async support to the methods it calls without breaking the existing
Realm interface. Yet, we might need to do this down the road.
Original commit: elastic/x-pack-elasticsearch@d0959f87f3
This changes adds a special value for negative role lookups so that we can avoid scenarios
where we overload the cluster due to continually trying to load non-existing roles as is often
the case when `unmapped_groups_as_roles` is used with the active directory realm.
Relates elastic/elasticsearch#3530
Original commit: elastic/x-pack-elasticsearch@62567b4c22
* security: restore the correct user when switching to the system user
For internal actions where we need to switch to the SystemUser, we should always restore the proper
context after execution. We were restoring an empty context for actions executed by the SystemUser
in the SecurityServerTransportInterceptor.
In order to accomplish this, a few changes have been made. Both the SecurityServerTransportInterceptor
and the SecurityActionFilter delegate to `SecurityContext#executeAsUser` when a user switch is necessary.
Tests were added for this method to ensure that the consumer is executed as the correct user and the proper
user is restored.
While working on this, a few other cleanups were made:
* SecurityContext can never have a null CryptoService, so a null check was removed
* We no longer replace the user with the system user when the system user is already associated with the request
* The security transport interceptor checks the license state and if auth is not allowed, delegate and return
* The security transport interceptor sendWithUser method now requires authentication to be present or a hard
exception is thrown.
* The TransportFilters integration test has been deleted. This was integration test that relied on the ability to
get instances from a node and trace the execution. This has been replaced by additional unit tests in
ServerTransportFilterTests
Closeselastic/elasticsearch#3845
Original commit: elastic/x-pack-elasticsearch@d8bcb59cb7
This change removes the blocking notion from fetching the roles
from a remote index. This also removes the blocking client calls
that can potentially deadlock a request if executed on the transport
thread.
Relates to elastic/elasticsearch#3790
Original commit: elastic/x-pack-elasticsearch@c2eda39043
This change now installs a signed license that has been generated at runtime so the
BWC tests can run without hitting licensing issues. The x-pack BWC tests pull in the
full cluster state, which contains the trial license from when the indices and state
was generated. After the trial license period and grace period issues arise with the
tests.
Closeselastic/elasticsearch#3858
Original commit: elastic/x-pack-elasticsearch@1c79e874e5
Today when a request is executed with InternalClient the thread context might
be lost if another component like security exchanges it by executing an async call
or an internal action. This can be a serious security problem since if the async
call executes as the system user all subsequent calls made by the response
thread will also execute as the system user instead.
Original commit: elastic/x-pack-elasticsearch@80682f338d
This commit changes the permissions of the files generated by the certgen tool to 600 (like syskeygen does)
Original commit: elastic/x-pack-elasticsearch@bca74e9c92
The calls made by the native users and roles store use the internal xpack user to make the request
and this user has a built-in role that has a single instance. A bug was introduced when fixing the logic
for applying the anonymous role to all users in elastic/elasticsearch#3716. The anonymous role was now being added to
the xpack user, even though the additional role would have no effect as this user is a superuser.
When the anonymous role is applied to the xpack user and exists as a native role or doesn't exist
at all, we run into a deadlock since we wind up querying for the role as a user that also has the
anonymous role.
This change special cases the XPackUser when getting the collection of roles so that the only role
applied to this user is the superuser role.
Closeselastic/elasticsearch#3822
Original commit: elastic/x-pack-elasticsearch@e3093904f1
This commit removes the dependency on the briks automatons library and instead uses the lucene
version. Shield was originally implemented using the lucene version, but issues arose with supporting
multiple versions of elasticsearch and API changes, so we moved to using the briks library.
x-pack and elasticsearch are always the same version so we can use the lucene version of the
automatons and remove the briks library. This also brings with it protection from huge automatons
that we did not have before.
Original commit: elastic/x-pack-elasticsearch@e3f34b6b55
This changes updates the unboundid ldapsdk to the latest version to stay up to date
with their releases.
Original commit: elastic/x-pack-elasticsearch@b9e4f7f062
This change simplifies the creation of Actions and Transformations.
It moves all instantiation away from guice into straight forward
constructor based initialization.
Original commit: elastic/x-pack-elasticsearch@3c0bca2bea
A commit in core removed the UUID parameter from the
ClusterStatsResponse constructor. This commit adjusts x-plugins to this.
Original commit: elastic/x-pack-elasticsearch@6f2f26168e
This change is a first step towards a real abstraction on top of all the
notification services. There are a bunch of followup changes coming for this
that will remove most of the classes in here but this is a first small step
to actually have a notification service interface.
Original commit: elastic/x-pack-elasticsearch@e14abf8a8b
This commit changes the logging to only log if we actually loaded the system key, otherwise
the message is misleading as the key file may not even exist but we output that it was
loaded.
Original commit: elastic/x-pack-elasticsearch@0af7953c64
Instead of using the long running and long blocking single polling HTTP attachment for our reporting,
we should use the async API provided by kibana. The new workflow (all blocking and in a single watch)
looks like this:
1. An initial request is sent to trigger the report generation, which returns a path
2. This path is used to continuously check if the report is done (then it is sent back) or kibana sends another HTTP error code, which will result in watcher to sleep for another interval until the report is finally returned.
Features include configurable interval time and retry count, so that the total amount of waiting can be tweaked into two directions.
This is what the reporting type looks like right now
```
{
"my-attachment":{
"reporting":{
"url":"http://www.example.org/my-dashboard",
"retries":6, // optional, default 40
"interval":"1s", // optional, default 15s
"auth":{
"basic":{
"username":"foo",
"password":"secret"
}
}
}
}
}
```
The interval/retries can also be configured via settings.
Note, that this is just a temporal workaround until the watcher execution can execute in an asynchronous fashion.
Closeselastic/elasticsearch#3524
Original commit: elastic/x-pack-elasticsearch@d1eaa856b9
The `.triggered-watches`, `.watches` and `.security` indices should load
as early as possible, and not wait for other indices (especially not
for time-based indices, that are old).
This commit adds an index.priority to the template for those indices.
The values 1000, 900 and 800 were chosen rather arbitrary, mainly we
did not want to go with 10, because it was used in the sample documentation.
Security should always be loaded first, because we might need this index for
other operations.
Any administrator can still change all the values in the indices, but this
cares for better defaults.
Original commit: elastic/x-pack-elasticsearch@6ed0fb7975
As discussed in #elastic/elasticsearch-migration/79 supporting aliases for watcher allows
the migration plugin to work.
This adds the relevent checks in the WatchStore and the TriggeredWatchStore that aliases are
supported, as the current assumption was always to just load an index.
Also, this rarely sets those indices as aliases in all the integration tests, so that this
case gets tested.
Note: The new WatchStoreUtils.getConcreteIndex() method will be put into core, as this is a
useful helper for others.
Original commit: elastic/x-pack-elasticsearch@4a98af691d
This change reduces the Condition infrastructure to a single interface called
`Condition` this interface is used to produce and parse requests but also
encapsulates the executable condition. The per class Result, Factory and Executable
are removed and replaced by a single class containing all logic.
Original commit: elastic/x-pack-elasticsearch@2870dff7ad
Watcher does not require any unique build info anymore, as all is put into
the MANIFEST.MF file during the build.
Also the xpack-properties is unused now and can be deleted.
Original commit: elastic/x-pack-elasticsearch@62f121c979
ExecutableActions is really an unnecessary abstraction on top of
List and Map. This commit remove the class and all its usage.
Original commit: elastic/x-pack-elasticsearch@b938499fcf
The system user gets used to put mappings for an index during recovery from local shards, which
is how the shrink index process works. The system user previously had this privilege in 2.x as
we did not have the ThreadContext and dynamic mapping updates would be done by the system user;
with the ThreadContext, these mapping updates are done by the actual user so this privilege
was removed from the SystemUser.
Closeselastic/elasticsearch#3766
Original commit: elastic/x-pack-elasticsearch@cd5d7bea53
The security indices resolver checks through an assertion that shard level requests always have their wildcard expressions resolved. Index names that start with `-` or `+` though shouldn't be considered wild card expressions. Up to 6.x there can be indices with names starting with `-` or `+` and we have to take that into account.
Also moved from assertion to explicit exception so we can also test it better.
Original commit: elastic/x-pack-elasticsearch@a520bbf247
If we create index test1 and alias test1-alias, and tests configure access for test* for some users, this is going to cause problems when verifying exclusions like -test2, as the index itself gets excluded but the alias that points to it doesn't. That is expected behaviour, with this commit we modify the way aliases are named to use a prefix rather than a suffix (e.g. from test1-alias to alias-test1).
Changed also the way aliases creation is randomized.
Original commit: elastic/x-pack-elasticsearch@7f9877e858
missing `.get()` :) the create index request was never sent. The indices were being automatically created when indexing a document into them.
Original commit: elastic/x-pack-elasticsearch@129d69c88e
The checkNodeStats method in this test checks for many fields in every documents of all bwc indices, but some fields like disk_threshold_enabled have been removed in 5.x. This commit changes the method so that it checks for the right fields in the right version.
closeselastic/elasticsearch#3672
Original commit: elastic/x-pack-elasticsearch@c95209cc3b
This commit responds to an API change in core migrating from
EsExecutors#boundedNumberOfProcessors to EsExecutors#numberOfProcessors.
Original commit: elastic/x-pack-elasticsearch@87d6fad971
extracted loading of authorized indices and aliases to separate class (AuthorizedIndices) with reduced dependencies. Allows also to lazily load authorized indices the first time they are required, and reuse them if they are needed again later. Removes AuthzService dependency in indices resolver.
Removed array of resolvers in authorization service as we support only one. Removed IndicesAndAliasesResolver interface and rename DefaultIndicesAndAliasesResolver to IndicesAndAliasesResolver.
Original commit: elastic/x-pack-elasticsearch@a267fefa07
FieldAndDocumentLevelSecurityRequestInterceptor really support intercepting only subclasses of IndicesRequests, we shouldn't have logic that is never used around intercepting CompositeIndicesRequest. Also we can guarantee at compile time, using generics, that only supported subclasses are intercepted through it, no need to verify that at runtime.
Original commit: elastic/x-pack-elasticsearch@6ab6e2d50e
Eagerly authorizing CompositeIndicesRequests allowed the security plugin to fail fast up until now, but it makes it very hard to reason about each specific item in a multi items request. Either all items fail, or none do. We would rather want to adopt a similar behaviour to es core, where individual items fail without affecting other items that are part of the same request. We can rely on the fact that es core always authorizes both main action and every subaction too, and skip authorization for the main action. By subaction we mean either all sub search requests in msearch, as well as each shard level get in mget or shard level bulk request for bulk.
BulkRequestInterceptor was converted to intercept BulkShardRequests rather than BulkRequest as that is where bulk is authorized after this change.
Split IndicesAndAliasesResolverIntegrationTests into ReadActionsTests and WriteActionsTests as they require different set of permissions, lots of tests added.
Explicitly listing the composite actions makes sure that the actions that can bypass security are known, somebody adding a similar action must to add it to the list, so we know it doesn't happen by mistake. At this point the CompositeIndicesRequest can be used as a marker interface only (it is not really needed but can be used to verify that composite actions use a request that implements such interface).
Given that we don't authorize composite actions based on their indices anymore, but only their sub-requests which implement IndicesRequest, printing out the indices names in the audit log for requests like bulk and msearch is confusing. Removed support for that.
Authorize composite indices actions based on their name only, their indices will be authorized at the sub-request/shard level
Rather than simply granting bulk, mget, msearch etc. and relying on authorization at the sub-request/shard level, we check that the current user can at least execute the action. This justifies the grant line that gets written in the audit log, the action is potentially possible without looking at the indices. Each specific item will fail or succeed later and will yield its own specific audit log entry.
Original commit: elastic/x-pack-elasticsearch@4570caf019
Like es core does in TransportIndicesExistsAction, we should only consider expandWildcardsOpen and expandWildcardsClosed out of the indices options passed in with IndicesExistsRequest. ignore_unavailable and allow_no_indices should always be considered both true, to prevent the request from throwing exception as it is supposed to return true or false, no exceptions.
Original commit: elastic/x-pack-elasticsearch@daa274b3fd
Supporting allowNoIndices means that the security plugin has a behaviour much more similar to vanilla es when dealing with wildcard expressions that match no indices, or empty clusters. The default for most request is to allow no indices, but security plugin could only disallow no indices all the time up until now.
The technical problem was that when anything gets resolved to an empty set of indices, we couldn't let that go through to es core, as that would become resolved to all indices by es core, which would be a security hole. We have now found a way though to replace an empty set of indices with something that es core will for sure resolve to no indices, so we can let the request through. We simply replace empty indices with '-*'.
Multi apis requests (e.g. _msearch) have yet to be fixed, as all their indices end up in the same bucket while they should each be authorized separately, so that every specific item can fail or be let through.
Original commit: elastic/x-pack-elasticsearch@0f67a0bfea
For all the requests that support multiple indices and wildcards, hence implementing IndicesRequest.Replaceable, we replace the wildcard expressions with the explicit names of the authorized indices they match. _all or empty indices is treated as a wildcard expression. We can also honour the ignore_unavailable option by going over all the explicit names and filter out the non authorized ones when ignore_unavailable is set to true. If ignore_unavailable is set to false, we leave everything as-is, which will cause an authorization exception to be thrown if only one of those explicit indices is not authorized for the current user.
This is the first step towards resolving elastic/elasticsearch#1250. The remaining issue is that in case we are left with no indices after stripping out the ones that the user is not authorized for, we throw an authorization exception rather than returning an empty response. That will require honouring the allow_no_indices option, which will also change the behaviour when a cluster is empty.
Relates to elastic/elasticsearch#1250
Original commit: elastic/x-pack-elasticsearch@e4ca940d05
The superuser role is the only user assignable role that grants access to the .security index, but when
resolving wildcards the index was not getting resolved. The resolution of indices and aliases explicitly
excludes the .security index for users that are not the internal user without checking if the user has the
superuser role. This commit adds a check in for the superuser role.
Original commit: elastic/x-pack-elasticsearch@02ee0a8740
The role cache was previously using the wrong time unit for its expire after write time; the
value passed to the cache was milliseconds instead of nanoseconds.
Original commit: elastic/x-pack-elasticsearch@65f7b08763
The anonymous role was being applied to other users for index access control but was not being applied
in terms of action level access control. This change makes the minimum required change to apply the
anonymous role for all users when anonymous is enabled. Additionally, some minor changes were made to the native roles store to not lookup roles before the service is started.
Closeselastic/elasticsearch#3711
Original commit: elastic/x-pack-elasticsearch@a9398e178d
When adding a watch which has a painless component, the scriptexception
was wrapped into a deprecated exception which means, that the awesome
painless descriptions were lost. This wrapping has been removed.
Closeselastic/elasticsearch#3161
Original commit: elastic/x-pack-elasticsearch@1703fe4eb6
This test has been blacklisted and deactivated months ago. This commit reenables this test and moves it at the right place. It also change the test to use the Execute Watch API instead of being sleep based.
Original commit: elastic/x-pack-elasticsearch@e7a9689375
in core we wrap request handlers with an asserting one to ensure we can serialize messages
with different versions. Yet, xpack uses the same functionality to add security aspects to
the network layer. These tests assert that the right handlers are in-place.
Original commit: elastic/x-pack-elasticsearch@e39c8995ae
Fixes to x-plugins code now that DateMathParser accepts a LongSupplier rather than a Callable to get the value of now
Relates to elastic/elasticsearchelastic/elasticsearch#20796
Original commit: elastic/x-pack-elasticsearch@99fc47a8a7
This change moves to using SSLParameters as the configuration source for SSLEngine and SSLSocket
objects that are configured by the SSLService. Previously we used a mix of specific methods and
SSLParameters, which resulted in issues where ordering of calls is important. For example, if configuring
client authentication directly on the engine prior to setting the SSLParameters resulted in the client
authentication configuration being reset to the default.
Additionally, this change also sets use cipher suite order to true to ensure preferred ciphers will be used.
Original commit: elastic/x-pack-elasticsearch@8ddecdc20c
This change ensures we wait for a response before the async http client is closed. Otherwise we can
close the client during the connection to the remote endpoint or never even connect to the remote
endpoint.
Closeselastic/elasticsearch#3640
Original commit: elastic/x-pack-elasticsearch@54900b1b4a
This changes does two things in the tribe tests. The first is that when we split data up between
multiple clusters, we always force create the security index so that randomization does not cause
edge cases like the index not existing in the preferred cluster. The second is we look at the cluster
state of the nodes and ensure the tribe node sees the indices and has all primaries active.
Separate tests were also added to cover the scenario where the security index only exists in the non
preferred node.
Original commit: elastic/x-pack-elasticsearch@17b78ec837
This is the last action that needs additional support for proxies.
You can set a proxy in the JSON like this:
```
"actions" : {
"notify-pagerduty" : {
"pagerduty" : {
"description" : "Main system down, please check!",
"proxy" : { "host" : "localhost", "port" : 8080 }
}
}
}
```
Closeselastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@b99969fd6b
You can set it like this in the JSON
"actions" : {
"notify-slack" : {
"slack" : {
"account" : "integration-account",
"proxy" : {
"host" : "localhost",
"port" : 8080
},
"message" : {
...
}
}
}
}
Relates elastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@de86233d4f
Watcher uses a custom thread pool. This is because executing watches can
be long-running tasks that often block on I/O and it is best to not
consume the core thread pools with these tasks. Today this thread pool
is fixed, and sized at five times the bounded number of cores (so 160 on
a 32-core box). It makes sense for there to possibly be so many threads,
again because these tasks can block on I/O and having excess capacity
lets unblocked watches execute. It's the fixed size that can cause
problem, all these threads are always consuming resources even when
there are no or not that many watches running. This commit changes this
thread pool to be a scaling thread pool.
Relates elastic/elasticsearch#3660
Original commit: elastic/x-pack-elasticsearch@3cafab6e83
We need to special case IndicesAliasesRequest as it doesn't implement CompositeIndicesRequest anymore. Note that the similar loop for CompositeIndicesRequests's subrequests will soon go away
Relates to elastic/elasticsearch#3638
Original commit: elastic/x-pack-elasticsearch@50d119ff61
This change allows native users and roles to be used on tribe nodes. The tribe node will actually
use the security index of one of the tribes, which must be specified with the `tribe.on_conflict`
setting. User and role modifications are not permitted when running on a tribe node.
Closeselastic/elasticsearch#3451
Original commit: elastic/x-pack-elasticsearch@2b762ca648
When deleting a watch the version was used as part of
the delete request. However a watch deletion means the
user really wants to get rid of it and not accidentally
run into a version exception because the watch was running in
between.
Original commit: elastic/x-pack-elasticsearch@e585f717f1
In preparation for elastic/elasticsearch#2957, I found we have things both in the root level
dev-tools, as well as elasticsearch/x-dev-tools. Most of this stuff can
be removed as it had to do with the old manual release process. There
was also a signed license file checked in. I removed it here, we really
should not have licenses checked in IMO, and it is unclear what the
purpose of this license was for. The two remaining scripts were moved to
the root dev-tools.
Original commit: elastic/x-pack-elasticsearch@3e24ea2d56
When running as a node, we check the `xpack.security.transport.filter.enabled` setting to see
if we should create the IPFilter but this check is not really correct. The HTTP filter could be
enabled or a profile filter could be enabled so there are times when we may not be filtering connections
when we should. Additionally, since we do not bind the IPFilter to a null provider, Guice will try to create
one during startup to inject into the security transport. This results in an exception and startup fails.
This change always creates the IPFilter when running as a node. This IPFilter has its own settings and
logic to determine whether it should be filtering on a given network transport.
Closeselastic/elasticsearch#3592
Original commit: elastic/x-pack-elasticsearch@95c25651c4
Some methods have been renamed in elastic/elasticsearchelastic/elasticsearch#20560. This commit change a .bytes() call to a .getBytes() call.
Original commit: elastic/x-pack-elasticsearch@4a0ff77361
This commit fixes the cat.templates REST tests so that it works when other templates exist (like monitoring)
Original commit: elastic/x-pack-elasticsearch@2e27ad88b4
This change switches the build to use the licensing prod key when
building the xpack jar for release.
Original commit: elastic/x-pack-elasticsearch@54a21dae5b
This adds an "interval" placeholder parameter that is required to the Monitoring Bulk API, and adds it to the Kibana side of the plumbing.
Having this will allow us to add it to all incoming documents and start to report against it with the Insights, as well as to detect the _lack_ of incoming documents.
By adding it now, we can avoid having a non-BWC API change for Kibana in 5.1. We'll just pickup new data in our documents.
Original commit: elastic/x-pack-elasticsearch@5ba8aafe03
This is a followup for elastic/elasticsearchelastic/elasticsearch#20526 removes the pluggability of
transport / http server transport via guice.
Original commit: elastic/x-pack-elasticsearch@5fb84949aa
The x-pack client jar and api jar are exactly the same: the entirety of
x-pack. Since we added the x-pack-transport jar, we no longer really
want the client jar as it is confusing. Additionally, it causes jar hell
when a test for an extension uses the transport client. This change
removes the client jar, and makes the x-pack transport client use the
api jar instead. This sounds odd at first, but since transport client is
going away eventually, it is a stopgap, and works.
closeselastic/elasticsearch#3309
Original commit: elastic/x-pack-elasticsearch@ee7a2c12c0
Due to untested code there was an NPE happening in production,
when a chained input execution failed, but the chained input
tried to access the resulting payload (which is never set on
failures). This payload now defaults to being empty.
This commit also drive-by fixes a broken logging statement, that
on the one side returned not the watch id, but a useless watch
toString() representation, and on the other hand only logs an error
message, but not a stack trace into the log, as this is what the
history is for.
Original commit: elastic/x-pack-elasticsearch@7dbe1afd90
Whenever a watch is updated (put, delete, set state), until now we
happened to reject those operations when a watch was executed at the
same time. However with long running reporting this might mean, that a
watch can never be changed, because it always gets executed.
* Removes the ability of write requests to obtain a lock at all (executing watches is still protected by a lock)
* Replaced the FairKeyedLock in watcher with the KeyedLock in Elasticsearch, which also takes a fair option, removed the FairKeyedLock
* Removed all the timeout parameters that are no longer needed, because there is no lock anymore
* Removed also the force parameter for watch deletion. Just do it[tm]
* Added a test that deleting a watch while it is being executed does not leave any leftovers
In case of a deletion of a watch during an execution, so that updating the status of the watch fails,
a warning is logged.
Closeselastic/elasticsearch#3417
Original commit: elastic/x-pack-elasticsearch@22fad1b797
As part of the review of elastic/elasticsearch#3287, the stopping of the IndexAuditTrail was moved to the tearDown
method. This works sometimes but other times it fails because tearDown is run after
ESIntegTestCase#after, so the IndexAuditTrail is still running during the after checks which will
cause the test to fail since the shard lock cannot be obtained.
Closeselastic/elasticsearch#3520
Original commit: elastic/x-pack-elasticsearch@4cb52b15a2
Today the operation mode can be set to default for a short amout of
time until it's reset to the actual mode this can cause weird sideeffects
for users if it's read concurrently. Also the test relies on a certain
happens before relationship that is not guaranteed since the operation
mode is set before the listerner is run. This change also rewrites the test
to not use busy waiting but wait for the actual listern to be executed.
Original commit: elastic/x-pack-elasticsearch@a2a42b89e5
1. We only support indexes created by Marvel 2.3+. All other indexes
are just ignored.
2. The tests don't assert a ton of interesting stuff because there
isn't a java API for Monitoring that we can just use. Instead we assert
that a few objects are there and look sane.
3. We don't migrate the contents of the data index. Instead we just
rely on Monitoring recreating it.
Original commit: elastic/x-pack-elasticsearch@86216c2d61
The IndexAuditTrail had both a stop and close method that needed to be called in order
to stop the service. There was a race where we called either flush or close in a non
blocking fashion and then immediately closed the underlying client. This change makes
the stop method wait for up to 10 seconds when closing the bulk processor.
Closeselastic/elasticsearch#3279
Original commit: elastic/x-pack-elasticsearch@0d776bc91a
TransportService is not pluggable anymore in core. Instead we now have a interceptor
infrastructure that allows to intercept send and receive calls on the transport layer.
Relates to elastic/elasticsearchelastic/elasticsearch#20505
Original commit: elastic/x-pack-elasticsearch@04194ecb09
This adds proxy support to the hipchat action. Right now
neither hipchat nor slack nor pagerduty allow for this,
but if you dont need a proxy for internal http connections,
but you do for external, then this configuration cannot be done
without setting a proxy for those actions.
You can set it like this in the JSON
```
"actions" : {
"notify-hipchat" : {
"hipchat" : {
"account" : "integration-account",
"proxy" : {
"host" : "localhost",
"port" : 8080
},
"message" : {
...
}
}
}
}
```
Relates elastic/elasticsearch#3372
Original commit: elastic/x-pack-elasticsearch@4e8447ce37
This cleans up some of the reported stats to be a little clearer, including making the JVM Heap chart behave like Kibana's memory chart. This solves two problems: you can now determine the max heap size and you know what "x%" actually means relative to it.
Original commit: elastic/x-pack-elasticsearch@450f6fd546
This commit fixes the logging audit tests which were broken due to an
upstream change in core Elasticsearch relating to the fact that prefixes
are no longer considered part of the log message, but are instead
implemented via markers.
Original commit: elastic/x-pack-elasticsearch@abd7ec23d8
To deny access to a fields users can name exceptions to field permissions with the following syntax:
"fields": {
"grant": [list of field names patterns],
"except": [list of patterns that are forbidden]
}
See doc for the rules for this.
This commit also reverts elastic/elasticsearch#2720closeselastic/elasticsearch#2681
Original commit: elastic/x-pack-elasticsearch@d6537028ec
The enabled and username fields are both now allowed in the request body for the put user
request. This makes it easier to perform a get and update a user without needing to edit more
of the request body than necessary.
Closeselastic/elasticsearch#3391
Original commit: elastic/x-pack-elasticsearch@ab763e843b
Start dates are a required feature for cloud. This functionality adds support
for specifying and enforcing a start date on licenses.
Behaviour: If the start date is > than now, the license will be rejected.
Due to another field in the license class, the version of the License class as well
as its serialization methods are adapted to this.
Closeselastic/elasticsearch#3370
Original commit: elastic/x-pack-elasticsearch@eb2a6f5be3
We were starting nodes at weird times and then shutting them down again,
slowing down the tests and causing the watcher tests to fail because
watcher wasn't being shut down with its traditional kid gloves.
Original commit: elastic/x-pack-elasticsearch@2fd81b3eaf
When the HTTP attachment was not able to successfully retrieve the
data from and endpoint, there was no indication in the watch history
of what went wrong. Instead a logger was used, which is not useful
for the person running the watches.
This commit removes the logger statement and throws an exception,
so that the exception message can be stored in the watch history.
Source of this issue was a forum post:
https://discuss.elastic.co/t/sending-e-mail-with-generated-report-fails/60263/6
Original commit: elastic/x-pack-elasticsearch@acdaf7abef
In our tests you have to explicitly shut down watcher rather than shut
down the node it is running on because of thread leak detection. Just
shutting down the node that it is running on will cause it to start up
on another node if there is another one running and then not properly
shut down. This is probably something that should be fixed in watcher
somehow but for now lets just be more careful with the tests.
Closeselastic/elasticsearch#2365Closeselastic/elasticsearch#2588
Original commit: elastic/x-pack-elasticsearch@fb8a172972
Previously core Elasticsearch had methods in a test class for removing
and adding appenders. However, these methods were moved to production
code. This commit adjusts x-plugins for this change.
Original commit: elastic/x-pack-elasticsearch@83e37ef65a
This particular change focuses on upgrading the source of a watch when it comes to scripts that have no language specified explicitly.
The default language in version 5 changed to painless from whatever is specified in `script.default_lang` setting (this defaulted to groovy). In order to make sure that scripts in watcher remain to work we should rewrite the search source upon startup and set the legacy default language explicitly. The legacy script language is now controlled by `script.legacy.default_lang` setting and that defaults to groovy.
Changing the source upon startup should do the trick and only change the source of watches with scripts that don't have an explicit language set. For new watches the default language used in scripts is painless and because we now always serialize the language explicitly in scripts these watches won't be changed on startup.
The upgrade logic added here tries to upgrade scripts in the following places in a watch:
* script condition
* script transform
* any script defined inside of a search input
Original commit: elastic/x-pack-elasticsearch@4d578819eb
This commit cleans most of the methods of XContentBuilder so that:
- Jackson's convenience methods are used instead of our custom ones (ie field(String,long) now uses Jackson's writeNumberField(String, long) instead of calling writeField(String) then writeNumber(long))
- null checks are added for all field names and values
- methods are grouped by type in the class source
- methods have the same parameters names
- duplicated methods like field(String, String...) and array(String, String...) are removed
- varargs methods now have the "array" name to reflect that it builds arrays
- unused methods like field(String,BigDecimal) are removed
- all methods now follow the execution path: field(String,?) -> field(String) then value(?), and value(?) -> writeSomething() method. Methods to build arrays also follow the same execution path.
Original commit: elastic/x-pack-elasticsearch@d83f3aa6e2
This fixes a bug I found with a customer when he updated from 1.x to 2.x.
Due to an BWC incompatible change in the watch history mapping and a thread
pool rejection during execution a watch was not removed from the triggered
watches and tried to be executed again.
While trying to fix it it turned out that the execution of the failure
test case was still done in the transport thread and thus required some
offloading to another thread pool.
Original commit: elastic/x-pack-elasticsearch@df04ce31f2
If someone deletes the watch index (i.e. by deleting all indices), the watcher
in memory store still contains all the watches and tries to execute watches -
which results in exceptions as the watch itself cannot be updated anymore.
In order to minimize this problem (it cant be get rid of completely), we should
act accordingly if the watch index goes missing (either deleted or closed) and
clear out the memory representation of watches in the watchstore as well as trying
to finish all the current executions.
Closeselastic/elasticsearch#2794
Original commit: elastic/x-pack-elasticsearch@12d98cd566
This change moves the logfile audit output from determining what to log based on the
logger level to a enum based configuration that is used by the index output.
A few notable changes were made:
* We alway log all the information we have except for the request body
* The request body is no longer logged by default for REST events; the user needs to
explicitly opt in as there could be sensitive data in the body
* Added a `realm_authentication_failed` event that separates overall authentication
failure from that of an individual realm
Original commit: elastic/x-pack-elasticsearch@343a2bcdd9
This change adds support for disabling users. Users can be disabled by setting the enabled
property to false and the AuthenticationService will check to make sure that the user is enabled.
If the user is not enabled, this will be audited as an authentication failure.
Also as part of this work, the AnonymousUser was cleaned up to remove having a static instance
that caused issues with tests.
Finally, the poller of users was removed to simplify the code in the NativeUsersStore. In our other
realms we rely on the clear cache APIs and the timeout of the user cache. We should have the
same semantics for the native realm.
Closeselastic/elasticsearch#2172
Original commit: elastic/x-pack-elasticsearch@0820e40183
This rewrites the HTTP Exporter to use the REST client underneath. Functionality is improved in resource blocking (templates and pipelines existing) and the majority of the code fundamentall simplified by removing direct HTTP calls.
This is blocked by the SSLService pull request. After that is merged, the I will update this PR to reflect those changes and it could possibly allow us to remove the security privileges required for monitoring.
Original commit: elastic/x-pack-elasticsearch@1ad25f17f8
Basic backwards compatibility support for watcher.
Closeselastic/elasticsearch#3230
Relates to elastic/elasticsearch#3231 - this actually should fix all the failures caused
by fractional time values but it does so by being able to parse them.
Being able to parse them is important for 2.x compatibility but 5.0
watches shouldn't produce fractional time values. This fixes the
particular way of making fractional time values mentioned in elastic/elasticsearch#3231
but I expect there are a half dozen more places to fix. The actual
watcher tests are fairly basic.
Original commit: elastic/x-pack-elasticsearch@328717455c
This publishes X-Pack usage data to the cluster info from the elected master node. This allows phone home to retrieve this data from the index, rather than fetching it live from the connected cluster (thereby not getting it from any n - 1 clusers that are not connceted).
Original commit: elastic/x-pack-elasticsearch@79bfaaaf0b
This removes the "agent" package from org.elasticsearch.xpack.monitoring.agent.*, so that now everything is simply org.elasticsearch.xpack.monitoring.*.
Follow-on work will be refactoring some of the other code, but this is a first step now that it's always the agent (in effect).
Original commit: elastic/x-pack-elasticsearch@14025cb17c
This change migrates xpack (security, watcher, and monitoring) to use the common ssl
configuration for the elastic stack. As part of this work, several aspects of how we deal
with SSL has been modified.
From a functionality perspective, an xpack wide configuration for SSL was added and
all of the code that needs SSL uses the SSLService now. The following is a list of all
of the aspects of xpack that can have their own SSL configuration, which are separate
from the xpack wide configuration:
* Transport
* Transport profiles
* HTTP Transport
* Realms
* Monitoring Exporters
* HTTP Client
In terms of the code, some cleanups were made with these changes. SSLConfiguration is
now a concrete class and SSLConfiguration.Custom and SSLConfiguration.Global have been
removed. The validate method on key and trust configurations has been removed and these
classes will now throw exceptions when they are constructed with bad values. The
OptionalSettings helper class has been removed as it was just a file with one line functions
that made the code harder to understand. The SSL configuration and service classes have
been moved from the security source directories to the main xpack source set. The SSLService
now handles more of the configuration of the SSLEngine it returns to prevent callers from
having to handle those aspects. The settings that get registered for SSL have been moved to
XPackSettings.
Also included in this PR is a update to the docs around SSL. This includes a large simplification to
the documentation in that the certificate authority configuration section has been removed and the
process that is documented for generating certificates only includes the CLI tool that we bundle.
Closeselastic/elasticsearch#3104Closeselastic/elasticsearch#2971Closeselastic/elasticsearch#3164
Original commit: elastic/x-pack-elasticsearch@5bd9e5ef38
Martijn van Groningen
Jay Modi
Daniel Mitterdorfer
Alexander Reelsen
Ryan Ernst
Jason Tedor
David Roberts
Zachary Tong
David Kyle
Tanguy Leroux
Dimitris Athanasiou
Areek Zillur
Yannick Welsch
jaymode
Jim Ferenczi
Boaz Leskes
Colin Goodheart-Smithe
Simon Willnauer
Simon Willnauer
javanna
Adrien Grand
Luca Cavanna
Tim Vernum
Nik Everett
Jack Conradson
Chris Earle
Clinton Gormley
Igor Motov
jaymode
Chris Earle
Britta Weber
Britta Weber