* Upgrades to ES 6.0.0-alpha1-SNAPSHOT
* Kibana changes to run upgrade to 6.0.0-alpha1-SNAPSHOT
* Other version changes to 6.0.0-alpha1-SNAPSHOT
Original commit: elastic/x-pack-elasticsearch@574d8573ab
This commit contains around half of the endpoint changes Sophie and Steve
agreed with Clint:
1) Automatic job ID generation is removed
2) Job IDs must now be specified in the URL when putting a job; to avoid
breaking many test configs, job IDs may also be specified in the job config
body, but in this case the value specified must match the URL argument
3) The endpoint name for posting data is now post_data instead of job_data
4) The post_data endpoint ends with _data instead of data
5) modelsnapshots is renamed to model_snapshots in all related endpoints
6) PUT model_snapshots/description is changed to POST model_snapshots/_update
Relates elastic/elasticsearch#630
Original commit: elastic/x-pack-elasticsearch@c379a23f3c
The `influencer_field_name` field was declared two in the results mapping. Once directly from `ElasticsearchMappings.resultsMapping()` and again from `addInfluencerFieldsToMapping(XContentBuilder)` which the `resultsMapping()` method calls.
this change removes the duplicate.
Original commit: elastic/x-pack-elasticsearch@5707a5ee53
The CompositeRolesStoreTests#testNegativeLookupsAreCached test had a bug where it was expected to
retrieve the superuser role but the mockito verification on the call failed. This was because there
is also randomization on the number of times to call, which could be 0.
Closeselastic/elasticsearch#4562
Original commit: elastic/x-pack-elasticsearch@5c62df15b7
We frequently have support requests to diagnose LDAP realm problems.
One of the tools that would be useful in those cases is to be able to turn on trace logging and be able to see the LDAP searches and their results
Original commit: elastic/x-pack-elasticsearch@632d8e4f19
This commit marks ActionThrottleTests#testFailingActionDoesGetThrottled
as awaits fix as this test fails reliably.
Original commit: elastic/x-pack-elasticsearch@465c48b603
Changes required are for two reasons:
1) SearchResponse is a self contained valid object which doesn't need to be wrapped in a new object anymore
2) RestToXContentBuilderListener requires ToXContentObject, hence GraphExploreResponse needs to be moved over
Original commit: elastic/x-pack-elasticsearch@12277d0220
Allow deletes to proceed even if index is missing
Also adds some tests. All non-IndexNotFound exceptions will still abort the delete.
We can revisit this if we find other edge-cases.
Original commit: elastic/x-pack-elasticsearch@823d00d8a7
and FixBlockingClientOperations in two places where blocking client calls are ok,
because these methods aren't called from a network thread.
Original commit: elastic/x-pack-elasticsearch@a6dc34651c
Merged categoryDefinition(...) into categoryDefinitions(...) as the two did similar things. The get call has been replaced with a search with a query on the _uid field and routing on category id, so that the response handling code can be reused.
Original commit: elastic/x-pack-elasticsearch@4243917b00
The start scheduler waits until the scheduler state has been set to started before returning.
Before this change after the scheduler state has been set to started, the scheduler would link itself to the task the start scheduler api has created.
If the stop scheduler api was called immediately after the start scheduler api then this could lead the stop scheduler api cancelling the task without
stopping the scheduler, as the scheduler could not have been linked to the task.
Now the scheduler gets linked to the task before the scheduler state is set to started, fixing the problematic situation discribed above.
Original commit: elastic/x-pack-elasticsearch@8334ae1967
This commit renames the current AgentService into MonitoringService and changes the way it works: it was previously based on thread sleeping and it now use thread scheduling instead.
At every given time interval, a MonitoringExecution is executed. It first checks if monitoring data can be collected and if so it will collect data from Collectors and then export the data using the Exporters. There are cases where the data cannot be collected: when the service is stopping, when the interval has been set to -1 after the MonitoringExecution has been scheduled, or when the previous data collection is not yet terminated. In this last case, MonitoringExecution will still be executed at the given interval but will not collect any data.
All tasks are executed on the generic thread pool.
closeselastic/elasticsearch#2866
Original commit: elastic/x-pack-elasticsearch@d37b4d3731
Watcher: Use Apache HttpClient for internal Watcher HttpClient
The current implementation based on URLConnection has several drawbacks.
* If server returned HTTP header but then got stuck, no timeout would help, the connection remained stuck
* GET requests with a body were not supported, the method was silently changed to POST
* More complex handling of input/error stream handling, the body could not be read from a single input stream
NOTE: This is a BWC breaker. From now on every part of the URL needs to be encoded properly before it is configured in the requeust builder. This requires an upgrade of all watches.
Closeselastic/elasticsearch#1141
Original commit: elastic/x-pack-elasticsearch@bbc8f85dd8
Also merged the JobProvider#getBucket(...) method into Jobprovider#getBuckets(...) method, because
it contained a lot of similar logic, otherwise it had to be converted to use non blocking client calls too.
Part of elastic/elasticsearch#127
Original commit: elastic/x-pack-elasticsearch@b1e66b62cb
There was an N-squared algorithm in the state processing code, leading
to large state persistence eventually timing out. Large state documents
are read from the network in 8KB chunks, and the old code was checking
ALL previously read chunks for separators every time a new chunk was read.
Fixeselastic/elasticsearch#635
Original commit: elastic/x-pack-elasticsearch@c814858c2c
An integration test was failing due to the possibility of the
.watch index being an alias (which only happens rarely).
Original commit: elastic/x-pack-elasticsearch@d53a4ebdd4
Watcher: Remove async code in sync code block
When removing the watch store this code snippet sneaked in.
A call to get a watch in order to find out if it exists was async, but
the code checking the result was called immediately afterwards without
waiting for the result, thus always using the default value.
This also removes some unused code in TriggeredWatchStore.
Original commit: elastic/x-pack-elasticsearch@c47e70bf8a
Deleting a job now starts a three-step process:
1. Job status updated to DELETING
2. Physical index is deleted
3. Job removed from cluster state
When jobs are in DELETING, they cannot be modified/updated/changed at all. Only jobs that are DELETING can actually be removed from the CS.
Original commit: elastic/x-pack-elasticsearch@2cd99a240c
Add `logstash`, in addition to `kibana` type mappings if they are missing from the _existing_ .monitoring-data-2 indices. This allows existing indices to be upgraded in place to support the Logstash monitoring feature added in 5.2 without having to delete the .monitoring-data-2 index.
Original commit: elastic/x-pack-elasticsearch@864343a116
* [Monitoring] BWC Allow New Types to be added to .monitoring-data-2 index
This adds the new Logstash (and technically Kibana) types to the index mapping if they're not already there. Pre-existing indices will be blocked from creating new types in the index due to the index setting. The index setting cannot be flipped without opening/closing the index, so manually adding the new types is easier.
Original commit: elastic/x-pack-elasticsearch@e85e800335
Currently, roles making use of field or document level security are still applied when
the license level does not enable field and document level security. There is no indication
that these roles are not being applied so it is misleading to users. This change prevents
these roles for applying to authorization and also adds a transient metadata to the response
that indicates which features of a role is unlicensed.
Additionally, this PR prevents the addition or modification of roles to include field or
document level security.
Closeselastic/elasticsearch#2472
Original commit: elastic/x-pack-elasticsearch@c9455958f5
with the fix we also make sure that prelert metatadata is taken into account when verifying the cluste state consistency
Original commit: elastic/x-pack-elasticsearch@1deaec3836
The interval value set is "minimumShouldMatch" and also the documented query dsl
parameter is `minimum_should_match` so we can deprecate
`minimum_number_should_match` and its getters/setters in 5.x and remove them in 6.
Original commit: elastic/x-pack-elasticsearch@53ce64b7b3
* [Monitoring] Use Cgroup Data from _nodes/stats
This adds Cgroup data from _nodes/stats.
- This also adds 5m and 15m from system load, which are sent from ES, but
were not recorded. Kibana does record/report those values though.
Original commit: elastic/x-pack-elasticsearch@8e04452c60
This PR adds UI for visualizing Logstash internal metrics with contributions from @ph, @jsvd and @pickypg
Changes include:
* x-pack component for Logstash which installs a monitoring exporter plugin to Logstash core. This periodically ships
monitoring data to the monitoring bulk API endpoint.
* Adds xpack.monitoring.* setting to logstash.yml when x-pack is installed.
* UI changes to graph Logstash monitoring data. Logstash processes are called nodes. They send separate monitoring info
which gets aggregated and displayed at cluster level and also at individual node level.
* Adds gradle build support for Logstash x-pack which can be controled via `xpack.logstash.build=false`
Fixeselastic/elasticsearch#4169
Original commit: elastic/x-pack-elasticsearch@f58ef406c1
When migrating realms to use the new settings for SSL, the removal of the
hostname_verification setting was missed in addition to actually using the
ssl.verification_mode setting. This change deprecates the
hostname_verification and enables the use of the ssl.verification_mode setting.
Original commit: elastic/x-pack-elasticsearch@ea24e83fd3
This PR changes how we use roles and how we look at the roles of a user. Previously we looked up each role individually, parsed each into their own `Role` object, and had a wrapper that essentially served as an iterator over the roles. The same pattern was also followed for the permissions that composed a role (ClusterPermission, IndicesPermission, and RunAsPermission). This resulted in a lot of code that was hard to follow and could be inefficient.
Now, we look up the roles for a user in bulk and only get the RoleDescriptor for each role. Once all role descriptors have been retrieved, we build a single Role that represents the user's permissions and we also cache this combination for better performance as authorization can happen many times for a single top level request as we authorize the top level request and any sub requests, which could be a large number in the case of shard requests.
This change also enabled a large cleanup of our permission and privilege classes, which should reduce the footprint of what needs to be followed. Some of the notable changes are:
* Consolidation of GeneralPrivilege and AbstractAutomatonPrivilege into the Privilege class
* The DefaultRole class has been removed and the permissions it provided were moved into the AuthorizationService
* The GlobalPermission class was removed as there is a single role that represents a user's permissions
* The Global inner classes for the various permissions were removed
* The Core inner class was removed and ClusterPermission, IndexPermission, RunAsPermission became final classes instead of interfaces
* The Permission interface has been removed. The isEmpty() method defined by this interface is not needed as we can simply evaluate the permission to get the same effect
* The ClusterPermission#check method only takes the action name again
* The AutomatonPredicate class was removed and replaced by Automatons#predicate
* IndicesAccessControl objects no longer need to be merged when evaluating permissions
* MergedFieldPermissions has been removed
* The Name class that was used to hold an array of strings has been removed and replaced with the use of a Set
* Privilege resolution is more efficient by only combining automata once
Other items:
* NativeRolesStore no longer does caching, so the RoleAndVersion class could be removed
* FileRolesStore doesn't need to be an AbstractLifecycleComponent
Relates elastic/elasticsearch#4327
Original commit: elastic/x-pack-elasticsearch@c1901bc82e
When attempting to a get a watch that does not exist, the listener is called once inside
an if statement and the execution of the method continues as if the watch existed. This
causes failures to happen including a NPE. This commit wraps the execution in a if-else
to prevent this.
Original commit: elastic/x-pack-elasticsearch@27f09852e7
We are trying to get rid of guava, so it doesn't make much sense to use it as a dependency unless we really need to.
Original commit: elastic/x-pack-elasticsearch@e2e2d58fba
Switches custom cluster state components from PROTO-based de-serialization to named objects based de-serialization
Original commit: elastic/x-pack-elasticsearch@e1be8597ee
Note that the change in elasticsearch allowed us to store scheduler
config's query and scriptFields as typed objects instead of
ByteReference.
Original commit: elastic/x-pack-elasticsearch@38c5aef2ef
* Check the bulk request contains actions before executing.
This suppresses an validation exception about no requests being added.
* Persist bulk request before refreshing the indexes on a flush acknowledgment
Original commit: elastic/x-pack-elasticsearch@22543e46c8
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).
This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.
Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
This commit adapts to the change of the method name
ESTestCase#randomPostivieLong to ESTestCase#randomNonNegativeLong.
Original commit: elastic/x-pack-elasticsearch@689429cb54
* Put model state in the .mlstate index
* Revert results index rename
* Put ModelSnapshots in the results index
* Change state index in C++
* Fix logging
* Rename state index ‘.ml-state’
Original commit: elastic/x-pack-elasticsearch@dbe5f6b525
* Strict parse search parts of schedulerConfig
This commit adds methods to build the typed objects
for the search parts of a scheduler config. Those are:
query, aggregations and script_fields.
As scheduler configs are stored in the cluster state and parsing
the search parts requires a SearchRequestParsers object, we cannot
store them as typed fields. Instead, they are now stored as
BytesReferences.
This change is in preparation for switching over to using a
client based data extractor.
Point summary of changes:
- query, aggregations and scriptFields are now stored as BytesReference
- adds methods to build the corresponding typed objects
- putting a scheduler now builds the search parts to to validate that
the config is valid
Relates to elastic/elasticsearch#478
Original commit: elastic/x-pack-elasticsearch@e6d5a85871
This doesn't happen initially when buckets are output by the C++, but
buckets can contain records at the moment they're sent for persistence
during normalization or during integration tests. It's safest if the
persistence code specifically doesn't persist these records.
Original commit: elastic/x-pack-elasticsearch@a93135d8c0
This "super registry" will eventually replace things like
`IndiciesQueriesRegistry` but for now it is just another thing
to plumb across requests.
Original commit: elastic/x-pack-elasticsearch@da26a42b36
When the index action is used to do some bulk indexing, the single
items of the response were not checked to have been indexed successful.
This could lead to NPEs due to an index response being null when the index
operation had failed. The action was still logged as a success though.
This commit only returns SUCCESS for the action, if all items were indexed
successfully. If all items failed, the result will be FAILED as well. Lastly
the result status PARTIAL_FAILURE is used if there were successful and unsuccessful
index operations.
Additionally some minor cleanups happened, like changing equals/hashcode.
Closeselastic/elasticsearch#4416
Original commit: elastic/x-pack-elasticsearch@692687e1af
* Add job config option to set the index name
* Check index does not already exist if ’index_name’ is set
* Don’t create alias if ‘index_name’ is the same as ‘job_id’
* Default index_name value
Set it to job_id if null and only create the index alias if job_id != index_name
* Fix compile errors after rebasing
* Address review comments
* Test if the index exists by checking the cluster state
* Update comment
Original commit: elastic/x-pack-elasticsearch@a3e7f1a5bb
Removing the WatchLockService could result in duplication of wids, because of a wrong
call to replace underscores with dashes. As UUIDs.createBase64UUID() can contain underscores
but they are kind of reserved in the Wid class due to handling of watch ids, this just uses
the toString() representation of a random UUID.
Closeselastic/elasticsearch#4422
Original commit: elastic/x-pack-elasticsearch@dceb01ae5e
* Persist quantile documents with the jobId in the document Id
* Add job Id to snapshot Id
* Add job Id to categoriser state document Id
* Rename quantiles doc to start with job id as the other state docs do
* Fix restoring categoriser state
Original commit: elastic/x-pack-elasticsearch@3e5d3368b5
This test fails due to a changing cluster state(?). The
test checks that a local exporter is ready and then continues.
However, during the test, we see output similar to:
skipping exporter [_local] as it isn't ready yet
Which indicates that the cluster state has changed and the
exporter does not return a bulk anymore. Hence, the test is
failing although at one point in time it returned a bulk.
By enabling trace logging we should be able to find out
what's going on.
Original commit: elastic/x-pack-elasticsearch@d7e2200dd9
The threadpool that supplies the threads used for job IO cannot be
resized, so the number of jobs cannot be dynamic either
Original commit: elastic/x-pack-elasticsearch@c584bf7147
The watch lock service is not really needed, as there is already
a data structure that has information about the currently executing
watches, that can be consulted before executed.
This change will now check, if there is already a watch running with
the current id. If there is not, execution will happen as usual. If
there is however, than a watch record will be created, stating that
the watch is currently being executed - which means that it is either
being executed or in the list of planned executions.
This way users can check in the watch history, if a watch has been executed
more often than it should.
In order to easily search for this, a new execution state called
`NOT_EXECUTED_ALREADY_QUEUED` has been added.
Original commit: elastic/x-pack-elasticsearch@867acec3c3
* Redesign the get anomaly_detectors APIs
This commit redesigns the APIs to get anomaly_detectors.
The new design has 2 GET APIs:
- An API to get the configurations: /anomaly_detectors/{job_id}
- An API to get the stats: /anomaly_detectors/{job_id}/_stats
For both APIs entering "_all" as the job_id returns results for
all jobs.
Note that page params have been removed. They were useful
when the configs were stored in an index. Now that they are part
of cluster state there is no need. Additionally, future support
for wildcard job_id expressions will give users a tool to narrow
down the GET actions to a certain subset of jobs which will be
more useful than the from/size approach.
Follow up:
- Implement similar GET APIs for schedulers
- Remove scheduler_stats from the anomaly_detectors _stats API
as it will be part of the schedulers _stats API
Closeselastic/elasticsearch#548
Original commit: elastic/x-pack-elasticsearch@046a0db8f5
* Added the Elastic copyright header to C++ files
* Added the name of the copyright holder to Java files
Original commit: elastic/x-pack-elasticsearch@aea1b5a656
Aggs does not need to be a separate member field. There can simply
be an aggs parse field which also then stored onto the aggregations
parse field.
Finally, retrieve_whole_source is unnecessary as we move towards
a node client based data extraction.
Original commit: elastic/x-pack-elasticsearch@14024c2ee5
In LoggingAuditTrail and the audit-event filtering, we distinguish between single-realm authentication failures ("realm_authentication_failure") and global failures ("authentication_failure").
Update the message output of IndexAuditTrail to reflect this distinction.
Original commit: elastic/x-pack-elasticsearch@56802ae2df
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default for all XContent types (not only JSON).
We have also changed the name of the system property to disable this feature
from `es.json.strict_duplicate_detection` to the now more appropriate name
`es.xcontent.strict_duplicate_detection`.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Relates elastic/elasticsearchelastic/elasticsearch#22073
Original commit: elastic/x-pack-elasticsearch@c92e2a6c55
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
Original commit: elastic/x-pack-elasticsearch@6400f18911
* snake_case model debug output fields
* Rename CategorizerState and BucketInfluencer types to snake_case
Original commit: elastic/x-pack-elasticsearch@da94dc7ec1
* Build: Convert xplugins to use new extra projects setup
This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.
The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.
Original commit: elastic/x-pack-elasticsearch@09398aea5a
changed stop scheduler api to wait with returning a response until the scheduler status has been set to STOPPED.
Original commit: elastic/x-pack-elasticsearch@e20fcd1ae9
* Check use of mappings
* Add unit tests for JobProvider.createJobRelatedIndices
* Remove ‘index: no’ from mappings as no longer required
The entire type mapping has ‘enabled: false’
* Restore “index.analysis.analyzer.default.type” setting
* Remove include_in_all from nested mappings
* Add audit and usage mappings to the job index
* Revert ‘Restore “index.analysis.analyzer.default.type” setting’
Original commit: elastic/x-pack-elasticsearch@c7d62e0c7e
This builds on PR elastic/elasticsearch#526 to get normalization working end-to-end using the
native normalizer process.
The ShortCircuitingRenormalizer class is basically doing what the old
BlockingQueueRenormaliser class did but using the ES threadpool instead
of its own thread.
Also fixed a bug where the C++ was calling the score field of partition_score
documents normalized_probability but the Java was calling it anomaly_score.
Original commit: elastic/x-pack-elasticsearch@d4cecae150
The new MockWebServer assumed that responses are inserted at the beginning
and removed later on. This was not thread safe.
Also this fixes a bug in the HttpExporterIT where there was no wait time
for a bulk request, even though the request execution is asynchronous.
Closeselastic/elasticsearch#4335
Original commit: elastic/x-pack-elasticsearch@11f31f68bd
These test are just checking response / status code. It is cleaner to have these tests as yaml tests.
Original commit: elastic/x-pack-elasticsearch@61c323059e
* Test for job existence before updating its state
* Add unit tests covering expected missing job exceptions
Original commit: elastic/x-pack-elasticsearch@bcd270dafd
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.
Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when
* Put
* Get
* Ack
* Activate/Deactivate
* Execute
Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569
Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.
Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.
Original commit: elastic/x-pack-elasticsearch@c47465b47c
Modified the definition and loading of settings in Security to provide early detection and failure of invalid (unrecognised or syntactically invalid) settings.
Also consolidates the number of places where settings were defined.
Each realm now defines its supported settings.
This is facilitated for custom realms via a new "getRealmSettings" method on XPackExtension.
The realm group setting performs validation of the child settings with reference to the "type".
For backwards compatibility reasons, realm types that have no configuration defined, will be accepted during validation, but may fail at realm creation time.
All SSL settings have been centralised into SSLConfigurationSettings, which supports a variable "prefix" to accommodate the multiple places we define SSL config.
HTTP Proxy settings are explicitly defined rather than being a generic group.
Where possible all security settings now reference a Setting object, and there are less magic strings scattered in the code.
Closes: elastic/elasticsearch#3965
Original commit: elastic/x-pack-elasticsearch@2c76a137a9
* Added back Normalizable classes
* Added back normalization process management classes
* Added back the scores updater
Original commit: elastic/x-pack-elasticsearch@ac8edf6ed6
In the triggered watch store a list of integers was returned to keep track
which watches had been successfully stored and thus should be executed.
This means, that an arraylist, plus autoboxing/unboxing needs to be done for
all the triggered watches. This data structure can easily be replaced with a BitSet,
resulting in much less objects being created or parsed - also it's a bit faster.
Original commit: elastic/x-pack-elasticsearch@e9fba67e34
This also shuffles results under /anomaly_detectors/. Note: the cluster state still refers to
"jobs" which should probably be fixed in a separate PR
Original commit: elastic/x-pack-elasticsearch@c9e634621c
Removed SchedulerStats as scheduler status is all we need and start and end times are only needed in start scheduler api.
Original commit: elastic/x-pack-elasticsearch@80c563cb69
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.
Original commit: elastic/x-pack-elasticsearch@cff21e21ee
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.
As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.
Relates elastic/elasticsearchelastic/elasticsearch#19614
Original commit: elastic/x-pack-elasticsearch@cced57b884
The latest release of the mock web server requires more security permissions, and
we dont need all the functionality anyway.
This introduces a small MockWebServer using the JDK internal HttpServer, yet fullfilling
all our needs and supporting SSL as well for testing.
The MockWebServer allows to enqueue responses and also requires you to enqueue as many responses
as you requests will be executed - there is no fallback at the moment.
SSL is also supported by passing an SSL context - for which the TestsSSLService is needed, which
makes the required methods public.
Original commit: elastic/x-pack-elasticsearch@55f4a172a2
* Use well-defined IDs for records and influencers
Removes the reliance on ES autogenerated UUIDs for all types that will
be renormalized
* Address some review comments
Original commit: elastic/x-pack-elasticsearch@85fde8b957
* Make ModelDebugOutput a result type
* Delete unused ElasticsearchBatchedModelDebugOutputIterator
* Add result_type field to ModelDebugOutput
* Address review comments
Original commit: elastic/x-pack-elasticsearch@a48e4cd946
The start scheduler api call will run until the scheduler has completed. Either when lookback only scheduler completes or the scheduler has been stopped.
The start scheduler api will first update the scheduler status from STOPPED to STARTED on master node and then start running the scheduler.
Once a scheduled job completes it updates the scheduler status from STARTED to STOPPED and then the start schedule api returns.
The STARTING and STOPPING statuses are no longer used, so have been removed.
The stop scheduler api is a sugar api that uses the task list and cancel apis stop the scheduler.
Renamed ScheduledJobService to ScheduledJobRunner
Original commit: elastic/x-pack-elasticsearch@ab504fe3d9
Now when a cluster gets updated with a new license,
the license uid and type will be logged as info. Making
it easier to verify the current license mode of the cluster.
Original commit: elastic/x-pack-elasticsearch@c205cb42ae
Make jobId and from/size mutually exclusive options.
This approach has the main properties of not allowing an invalid Request to be built, and alerting the user if they set an incorrect configuration. It has the downside that PageParams can be null so the consumer will have to check for it. Since jobId could be null before, this seemed acceptable.
Original commit: elastic/x-pack-elasticsearch@106dcdf61a
Adds a delete_list endpoint. If a list is currently in use by a job, it is not allowed to be deleted
Original commit: elastic/x-pack-elasticsearch@7d9a984b3a
There were some test failures as LicenseServiceClusterTests was not properly setting minimum_master_nodes while having autoMinMasterNodes disabled.
Original commit: elastic/x-pack-elasticsearch@2030db2424
This avoids the confusing situation that a there is no allocation when a job hasn't been opened yet. Now it complains about the fact that the job status is closed.
Original commit: elastic/x-pack-elasticsearch@3159dc6954
* Collapse ElasticsearchBulkDeleter into JobDataDeleter
* Add blocking delete to JobDataDeleter
* Delete interim results only after all the results are parsed.
* Remove unused deleteModelSizeStats and deleteModelDebugOutput methods.
Document missing javadoc tags
Original commit: elastic/x-pack-elasticsearch@1997541673
* A job now has the following statuses: OPENING, OPENED, CLOSING, CLOSED and FAILED.
* The open job and close job APIs wait until the job gets into a OPENED or CLOSED state.
* The post data api no longer lazily opens a job and fails if the job has not been opened.
* When a job gets into a failed state also the reason is recorded in the allocation.
* Removed pause and resume APIs.
* Made `max_running_jobs` setting dynamically updatedable.
Original commit: elastic/x-pack-elasticsearch@3485ec5317
A companion PR to https://github.com/elastic/elasticsearch/pull/21846 where the above methods were removed. See ES PR for details.
With the concurrent starting the issues with licenses and time freeze became more apparent and I had to apply my suggestion to only freeze time once the license has been applied (as opposed to freeze on node start up). Since this also means that a node that starts up after the cluster time has been frozen need to also immediately freeze, it felt natural to use a `ServiceDisruptionScheme`. Although the name doesn't really make sense here, it all has all the logic we need.
Original commit: elastic/x-pack-elasticsearch@5641742f60
This commit migrates the actual authentication calls that are performed by internal realms
to be asynchronous so that we do not execute blocking calls on network threads. The realms
that used LDAP were especially bad as they can issue many different requests and/or open
multiple connections per authentication attempt.
The LdapRealm now uses the ThreadPool to fork a thread for authentication. This is necessary
because a LDAP bind operation is not allowed to be performed asynchronously and must be done
in a blocking manner. After the bind request has completed, all other requests can be done
asynchrnously. The LdapUtils class now provides asynchronous methods for searching and also
includes the handling of referral following as automated following is not supported by
UnboundID when using the asynchronous calls.
Original commit: elastic/x-pack-elasticsearch@dfb259e498
* [Watcher] HistoryStore should hide secrets
Some watcher actions like Jira stores the whole HTTP request in case of a failure. This is very helpful when something goes wrong, but it has to hide the password used for Basic authentication otherwise it will appear in the watch record in the .watcher-history index. In general, I would consider better to always hide secrets when storing something in the history index.
This commit removes the XContentParams "hide secrets" set at the JiraIssue so that it is now set at the HistoryStore level.
It also fixes test failures that can be reproduced with:
gradle :x-plugins:elasticsearch:test -Dtests.seed=BA730F93E726AECC -Dtests.class=org.elasticsearch.xpack.notification.jira.JiraIssueTests -Dtests.method="testToXContent" -Dtests.security.manager=true -Dtests.locale=es -Dtests.timezone=Asia/Choibalsan
Original commit: elastic/x-pack-elasticsearch@d686b3b3ad
The local exporter previously fired off asynchronous requests every time a cluster state was
observed that did not contain all of the required items for monitoring. This change adds a
flag so that monitoring can wait for the pending requests to complete before retrying. This
will reduce the number of duplicated log messages as well.
Additionally, the log message for adding modern aliases now contains the name of the indices.
Closeselastic/elasticsearch#3756
Original commit: elastic/x-pack-elasticsearch@727a0adfbe
* Add test to read model size stats
* Most recent model_size_stats document should have the name ‘model_size_stats’
Original commit: elastic/x-pack-elasticsearch@e192d4c34d
* Allow POST with body to get records
* Allow records endpoint to accept POST requests with body
* CategoryDefinition can accept POST requests with body parameters
Original commit: elastic/x-pack-elasticsearch@2edb7a9c47
Colin Goodheart-Smithe
Martijn van Groningen
David Kyle
Daniel Mitterdorfer
Lee Hinman
Nik Everett
Jay Modi
David Roberts
jaymode
Tim Vernum
Jason Tedor
javanna
Zachary Tong
Tim B
Tanguy Leroux
Alexander Reelsen
polyfractal
Chris Earle
Chris Earle
Christoph Büscher
Luca Cavanna
Adrien Grand
Igor Motov
Yannick Welsch
Dimitrios Athanasiou
Areek Zillur
Ryan Ernst
Daniel Mitterdorfer
Simon Willnauer
Boaz Leskes
Jim Ferenczi