This deletes tests getting results from MlJobIT since
such tests already exist in a form that is simpler to
understand and maintain in the YAML suite.
Original commit: elastic/x-pack-elasticsearch@b708e24877
This commit means that newly created ML state indices will have a single
type named "doc", and newly persisted state documents will have type
"doc" too.
Retrieving state is only supported for type "doc".
When deleting state, documents with the old types are deleted in addition
to those with type "doc". This means jobs created by the beta can be fully
deleted.
Relates elastic/x-pack-elasticsearch#668
Original commit: elastic/x-pack-elasticsearch@29c07d40f1
This commit adds an internal project call ml-cpp-snapshot which when
built will pull the ml cpp zip file from the prelert bucket. The GET
request has retries added to handle the dynamic aws creds eventual
consistency.
Original commit: elastic/x-pack-elasticsearch@1bba7d0f08
This commit cleans up the check for SSL with client authentication when a PKI realm is enabled by
moving it from the realm to a actual bootstrap check.
A bug was found during this cleanup in the check for transport profiles and that is also fixed in
this commit.
relates elastic/x-pack-elasticsearch#420
Original commit: elastic/x-pack-elasticsearch@3aa6a3edc0
The commit that converted the results index into single type
broke the search for fetching results for renormalization.
This commit fixes that.
Original commit: elastic/x-pack-elasticsearch@1ca7517adc
REST endpoints that support GET and POST need
to also support source parsing. As these
endpoints can accept a body but some clients
do not allow doing a GET with a request body,
elasticsearch has support for parsing via a
source URI parameter. This commit adds source
handling to all such endpoints.
relates elastic/x-pack-elasticsearch#1204
Original commit: elastic/x-pack-elasticsearch@3949ea31fe
Previously we used to normalize records with their buckets. This required
nested scrolls: an outer scroll over buckets, then a nested scroll for
records in each bucket. This was fragile.
The new approach is to simply scroll first through buckets, then through
records. This is made possible because we no longer store max_record_score
on buckets nor bucket anomaly_score on records.
While making these changes I noticed that the PerPartitionMaxProbabilities
class was redundant (because it was storing max_record_score in the case of
per-partition normalization), so I removed it. I also removed a redundant
Map from the Bucket class and fixed its equals() and hashCode() methods.
relates elastic/x-pack-elasticsearch#1115
Original commit: elastic/x-pack-elasticsearch@efbee63573
Deleting a job issues 2 cluster state updates.
The first marks the job as deleted.
The second actually removes the job.
Both check that there is no datafeed referring to the job.
If a put datafeed request arrives between those 2 cluster
state updates, the datafeed gets created and the final
job cluster state update fails. This means we end up with
both the job and the datafeed, but the job's results and
state have been deleted.
This commit changes the behaviour so that the put
datafeed request fails for a job that is marked as deleted
as this scenario is avoiding partially executing an action.
relates elastic/x-pack-elasticsearch#1510
Original commit: elastic/x-pack-elasticsearch@76fa0f0b1a
This commit changes all results to use the single doc type.
All searches were adjusted to work without the need to specify
type which ensures BWC with 5.4 results.
Additional work is needed to put the new doc mapping in indices
created with 5.4 but it will be done in separate PR.
Relates elastic/x-pack-elasticsearch#668
Original commit: elastic/x-pack-elasticsearch@041c88ac2d
* Reduced a longish timeout to a shorter one, as a watch should be
executed in a HTTP test.
* Ensured that the TimeThrottleIntegration tests only query for own
watches in the watch history, also use random names for watch ids
* HipChatServiceTests configured deprecated logging package, so it was
not possible to follow the HTTP calls to the hipchat service endpoint.
relates elastic/x-pack-elasticsearch#1514
Relates elastic/x-pack-elasticsearch#1515
Original commit: elastic/x-pack-elasticsearch@adb492e4e9
* Remove sequenceNum from anomaly records and influencers
* Generate unqiue IDs without sequence numbers
* Remove more instances of sequence_num
* Handle parsing sequnce_num from v5.4
Original commit: elastic/x-pack-elasticsearch@e60b206daf
This commit simplifies the WatchStatsTests to use a MockScriptPlugin.
The latch script engine previously depended on a static instance of the
engine to contain the latches. These are now moved to statics of the
test class itself.
Original commit: elastic/x-pack-elasticsearch@4170cd1bd3
This commit enables security to work with an index named .security (as
it could before) OR an alias named .security that points to a concrete
index by a different name that has the security index. This prepares
the ability to migrate from a 5.x to 6.x security index that allows
changing and re-indexing the underlying security index while maintaining
a .security alias that points to the underlying updated index.
relates elastic/x-pack-elasticsearch#1216
Original commit: elastic/x-pack-elasticsearch@9fee12e5a0
Suppress many job/datafeed errors if a node is known to be shutting down. Also, ensure started datafeeds and open jobs don't end up stopped/failed due to errors as the shutdown progresses, as this would prevent them automatically relocating to a different node.
relates elastic/x-pack-elasticsearch#1114
Original commit: elastic/x-pack-elasticsearch@e56a7dbea1
Set job to failed if autodetect manager fails closing, fix force closing of jobs that hang in closing
state, set timeout when waiting for clusterstate update, disallow closing of failed jobs with normal
close
relates elastic/x-pack-elasticsearch#1453
Original commit: elastic/x-pack-elasticsearch@493cf85e22
Aliases might be contained in requests that check index permissions
to disable caches etc. This commit preserves permissions for
aliases as well.
Original commit: elastic/x-pack-elasticsearch@233195aeba
The authentication object was changed in 5.4.0 in that it was conditionally signed depending on
the version and other factors. A bug was introduced however that causes the authentication to
actually get written with the version of the node it is being sent to even if that version is
greater than the version of the current node, which causes rolling upgrades to fail.
Original commit: elastic/x-pack-elasticsearch@a718ff8a52
* Add sort parameter for get buckets
* Add secondary sort by time
* Use default values from actions in rest requests
Original commit: elastic/x-pack-elasticsearch@a530c0bed6
This commit ensures the old 5.x index templates are removed
using the existing plugin hook, instead of the self written part.
Original commit: elastic/x-pack-elasticsearch@6faf08d98d
This commit adds compatibility checks while opening a job:
- Checks that jobs without versions (< 5.5) are not opened
- Checks that jobs with incompatible types are not opened
Original commit: elastic/x-pack-elasticsearch@a3adab733e
* Hide ML actions for tribe node client
* Remove unused parameters
* Enable ML actions and rest endpoints for the transport client
* Create the ML components for the transport client
* Add ml transport client tests
Original commit: elastic/x-pack-elasticsearch@509007ca29
* Remove repeated calls to validateAndReturnJobTask
* Wait for closing job
* Refactor resolving job ids
* More close job unit tests
* Don’t finalise closing jobs twice
Original commit: elastic/x-pack-elasticsearch@20616d6f0a
The LocalExporterTests.testLocalExporterFlush() test was removed in elastic/x-pack-elasticsearch#835 when
LocalExporterTests was changed. This test verified that export exceptions are
thrown when monitoring documents are exported using the Monitoring Bulk API but
the underlying monitoring indices are closed.
This commit reintroduces this test, but as a REST test this time.
relates elastic/x-pack-elasticsearch#416
Original commit: elastic/x-pack-elasticsearch@0a42f9a1be
This is the xpack side of elastic/elasticsearch#24447. The one caveat is
there are a number of places within the xpack api that use
BytesReference to pass down to templates. These are convert to encode
into a BytesArray when necessary, so as not to require changing all of
those apis here at once, but they should all be convert to String as
well.
Original commit: elastic/x-pack-elasticsearch@8399b9d8c3
For user tokens, we were storing the expiration time as a date with the date time format in the
mappings. Occasionally, we would get CI failures due to parsing the date and having an invalid
format. This change simplifies the user tokens to simply use an Instant and convert it to the epoch
millis when we index the document. This eliminates the need for a date formatter and should ensure
we no longer have issues with parsing the dates.
Additionally, the TokenAuthIntegTests#testExpireMultipleTimes could fail if the token was expired
but the approximation for the expiration time was after the current time. In order to resolve this
we get the exact expiration time from the token and use that in the test.
relates elastic/x-pack-elasticsearch#1255
Original commit: elastic/x-pack-elasticsearch@d4bf53e7bc
When we update a model snapshot we need to write it back to
the exact index where we read it from. This is necessary
for rollover as otherwise we would end up with two different
versions of the same snapshot.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@b5d1ab38a7
Support the resolution of remote index names, including those that contain wildcards in the cluster name or index part)
Specifically these work:
- `GET /remote*:foo/_search`
- `GET /*:foo/_search`
- `GET /*:foo,*/_search`
- `GET /remote:*/_search`
- `GET /*:*/_search`
This change assumes that every user is allowed to attempt a cross-cluster search against any remote index, and the actual authorisation of indices happens on the remote nodes. Thus ` GET /*:foo/_search` will expand to search the `foo` index on every registered remote without consideration of the roles and privileges that the user has on the source cluster.
Original commit: elastic/x-pack-elasticsearch@b45041aaa3
As this does not require any reindexing this is easy to fix by just
changing the watch history template.
In addition the old templates are deleted on start up and the new ones
are instantiated.
Original commit: elastic/x-pack-elasticsearch@7e1ad495ad
I broke this by adding more randomization to core's randomTimeValue
method. Now it produces valid time values that don't round trip
properly over the XContent API. This change causes the test to skip
trying to round trip time these sorts of time values.
Original commit: elastic/x-pack-elasticsearch@dcdd588bdb
This is in preparation of introducing a write alias.
It adjusts all requests to persist results to do so
using a method that returns the write alias (even though
it currently returns the same as the read alias).
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@1358dd8dcf
This commit updates the RunAsIntegTests to randomly assign the superuser role to the user that
is authenticating with the cluster but the request is being run as a different user. This provides
additional validation that the authorization errors are actually coming from the user the request
is running as and not due to the authenticating user's privileges.
Original commit: elastic/x-pack-elasticsearch@c6360d13e6
Each job introduces new fields to the results index matching
the analysis terms. When the job is created, mappings for those
are added explicitly. However, when rollover is introduced, that
will not be the case. This commit prepares for that by adding
dynamic mapping of new fields as keyword.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@8f6cd09a71
This was the behaviour in Shield 2.x, but it was accidentally changed during migration to X-Pack 5.x
Original commit: elastic/x-pack-elasticsearch@de0bf5e688
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.
Original commit: elastic/x-pack-elasticsearch@4e447874f6
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.
Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.
This commit moves the validation of job configurations to the first node that
handles the action.
Fixeselastic/x-pack-kibana#1172
Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.
Original commit: elastic/x-pack-elasticsearch@29db55a1fe
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.
relates elastic/x-pack-elasticsearch#1236
Original commit: elastic/x-pack-elasticsearch@2f1100b64a
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.
Relates to elastic/x-pack-elasticsearch#1299
Original commit: elastic/x-pack-elasticsearch@dfef87e757
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.
relates elastic/x-pack-elasticsearch#1278
Original commit: elastic/x-pack-elasticsearch@f6c93d776c
* [ML] Not an error to close a job twice
* Error if job is opening
* Address review comments
* Test closed job isn’t resolved
Original commit: elastic/x-pack-elasticsearch@7da7b24c08
Originally we used to try to get the native code version even when ML was disabled.
However, this has proved to be an annoyance in cases where people running on
unsupported platforms have disabled ML. This commit means we'll only try to get
the native code version if ML is enabled on a node.
Original commit: elastic/x-pack-elasticsearch@778d6708d2
This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.
This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).
Original commit: elastic/x-pack-elasticsearch@2cadb5939d
This is a task towards allowing rollover.
Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.
This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.
Relates elastic/x-pack-elasticsearch#827
Original commit: elastic/x-pack-elasticsearch@b2796e9b08
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).
This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.
Original commit: elastic/x-pack-elasticsearch@def472cf2e
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys
Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested
Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
This introduces a role-mapping API to X-Pack security.
Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index
- A custom expression language (in JSON) for expressing the mapping rules
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings
Original commit: elastic/x-pack-elasticsearch@d9972ed1da
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.
Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
JobDataDeleter handles the deletion logic for 3 cases:
1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results
The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.
This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.
relates elastic/x-pack-elasticsearch#821
Original commit: elastic/x-pack-elasticsearch@7da91332bd
Nik Everett
David Roberts
David Kyle
Dimitris Athanasiou
Ryan Ernst
Jay Modi
Alexander Reelsen
Andrew Cholakian
Jason Tedor
Ali Beyad
Hendrik Muhs
Simon Willnauer
Tim Vernum
Jack Conradson
Tanguy Leroux
Igor Motov
Tim Vernum
Jim Ferenczi
javanna
jaymode
Chris Earle
Adrien Grand
Luca Cavanna
Tim Brooks