* Use well-defined IDs for records and influencers
Removes the reliance on ES autogenerated UUIDs for all types that will
be renormalized
* Address some review comments
Original commit: elastic/x-pack-elasticsearch@85fde8b957
* Make ModelDebugOutput a result type
* Delete unused ElasticsearchBatchedModelDebugOutputIterator
* Add result_type field to ModelDebugOutput
* Address review comments
Original commit: elastic/x-pack-elasticsearch@a48e4cd946
The start scheduler api call will run until the scheduler has completed. Either when lookback only scheduler completes or the scheduler has been stopped.
The start scheduler api will first update the scheduler status from STOPPED to STARTED on master node and then start running the scheduler.
Once a scheduled job completes it updates the scheduler status from STARTED to STOPPED and then the start schedule api returns.
The STARTING and STOPPING statuses are no longer used, so have been removed.
The stop scheduler api is a sugar api that uses the task list and cancel apis stop the scheduler.
Renamed ScheduledJobService to ScheduledJobRunner
Original commit: elastic/x-pack-elasticsearch@ab504fe3d9
Make jobId and from/size mutually exclusive options.
This approach has the main properties of not allowing an invalid Request to be built, and alerting the user if they set an incorrect configuration. It has the downside that PageParams can be null so the consumer will have to check for it. Since jobId could be null before, this seemed acceptable.
Original commit: elastic/x-pack-elasticsearch@106dcdf61a
Adds a delete_list endpoint. If a list is currently in use by a job, it is not allowed to be deleted
Original commit: elastic/x-pack-elasticsearch@7d9a984b3a
This avoids the confusing situation that a there is no allocation when a job hasn't been opened yet. Now it complains about the fact that the job status is closed.
Original commit: elastic/x-pack-elasticsearch@3159dc6954
* Collapse ElasticsearchBulkDeleter into JobDataDeleter
* Add blocking delete to JobDataDeleter
* Delete interim results only after all the results are parsed.
* Remove unused deleteModelSizeStats and deleteModelDebugOutput methods.
Document missing javadoc tags
Original commit: elastic/x-pack-elasticsearch@1997541673
* A job now has the following statuses: OPENING, OPENED, CLOSING, CLOSED and FAILED.
* The open job and close job APIs wait until the job gets into a OPENED or CLOSED state.
* The post data api no longer lazily opens a job and fails if the job has not been opened.
* When a job gets into a failed state also the reason is recorded in the allocation.
* Removed pause and resume APIs.
* Made `max_running_jobs` setting dynamically updatedable.
Original commit: elastic/x-pack-elasticsearch@3485ec5317
* Add test to read model size stats
* Most recent model_size_stats document should have the name ‘model_size_stats’
Original commit: elastic/x-pack-elasticsearch@e192d4c34d
* Allow POST with body to get records
* Allow records endpoint to accept POST requests with body
* CategoryDefinition can accept POST requests with body parameters
Original commit: elastic/x-pack-elasticsearch@2edb7a9c47
* Make persist DataCounts a non-blocking operation
* Add trace and debug level logging to the persist data counts action listener.
Remove dead code from test
Original commit: elastic/x-pack-elasticsearch@84bbfa880a
The changes the way the Elasticsearch plugin module gets the cpp code when running the bundlePlugin task. Instead of manually going into the cpp builds output and grabbing the folders it depends on the buildZip task of the :cpp project and grabs the resource that task builds. This means that it does not matter if the cpp buildZip task is changed as long as it results in a zip being generated.
Original commit: elastic/x-pack-elasticsearch@257e5df2e7
Make sure CountDownLatch gets removed when it is no longer needed
Also add CountDownLatch is it is missing when we ack a flush id,
we may ack before we wait for it
Original commit: elastic/x-pack-elasticsearch@83a993b9ad
* Persist DataCounts every 10 seconds from a thread pool scheduled task.
Also rework the isReportingBoundary function changing the function as the boundary changes
* Remove overloaded constructors from StatusReporter and DummyStatusReporter
* Persist dataCounts in a background thread from the status reporter
* Use generic threadpool
Original commit: elastic/x-pack-elasticsearch@f00c1067aa
* Renamed REST path parameters to snake case.
And added missing parameter descriptions
* Document endpoint which accept body parameters
* Endpoints that support body parameters must also accept POST
Original commit: elastic/x-pack-elasticsearch@2cad2e8af6
The problem is that the Java security manager can open the pipe and then
quickly close it, leading to a need to reconnect from the C++ side.
Original commit: elastic/x-pack-elasticsearch@772b57f443
it can happen we encounter more then one cluster states with no usage index and causing this class to send two or more create index requests, but only one request will succeed.
Original commit: elastic/x-pack-elasticsearch@524a1dda61
Also moved all creation logic that is required to run at cluster startup into PrelertInitializationService.
Original commit: elastic/x-pack-elasticsearch@453ba3efa3
* Add result_type field to bucket
* Query and delete buckets/records by result_type
* Add a filter to the ElasticsearchBatchedDocumentsIterator subclasses for result_type:bucket
* De-type Influencers, BucketInfluencers and Category Definitions
* Revert de-typing CategoryDefinition
* Resolve merge errors after rebase
Original commit: elastic/x-pack-elasticsearch@65605432e8
In c++ the results are built all together under a bucket hierarchy.
This buckets was written out and java would read it and split the
bucket into its parts: the bucket itself, its records and its
influencers.
During the migration, the bucket started being persisted as a whole,
including its records and influencers.
This commit is changing this by modifying the way results are written
in c++. This way, the java and c++ results writing/reading are in sync.
To achieve this, the change involved writing records and influencers as
top level results from c++. In addition, they are written as an array
object in order to allow the java side to persist them in a bulk
request.
* Fix bucket counting in results processor
Original commit: elastic/x-pack-elasticsearch@feadf3f887
Elasticsearch will now attempt to spawn a controller daemon called simply
`controller` when it starts up, but ours was called `prelert_controller`.
For consistency it makes sense to remove the prefix from all our programs.
Original commit: elastic/x-pack-elasticsearch@d06714b231
* Make close job a master node action
The close job action now:
- Is a master node action
- Sets the job status to CLOSING
- Waits for the job status to change to CLOSED before it responds
JobLifeCycleService picks up on a job status change to CLOSING and
closes the job. At the end, it sets the job status to CLOSED.
* Assert job status is closed after close in integration test
This also correctly passes an ActionListener to JobManager.setJobStatus
in order to ensure the job status request has completed and to properly
propagate failures.
Original commit: elastic/x-pack-elasticsearch@1546c77fca
The PID used needs to be the PID of autodetect's parent proecss. The parent
is of course the controller daemon rather than the JVM.
Original commit: elastic/x-pack-elasticsearch@607481e1e2
* Added a setting that determines the maximum number of jobs that can run on a single node.
* Fail to start autodetect process if a user attempts to start more jobs than is allowed on a single node.
* Prevent concurrent data write, flush and close operation to the autodetect process.
Original commit: elastic/x-pack-elasticsearch@aca15fd51c
QueryPage now requires a ParseField in the constructor, which is used for the name of the array of results. I considered putting an enum of different field names in QueryPage itself, but it seemed better to keep the field name local to each respective object.
hitCount was also renamed to count, and getters updated appropriately.
Finally, added a static helper to throw the ResourceNotFoundException, just to make life easier/more consistent.
Closeselastic/elasticsearch#359
Original commit: elastic/x-pack-elasticsearch@9ba42ad4a1
Putting the various endpoints back to their plural form (and adjusting Java class names as necessary).
Original commit: elastic/x-pack-elasticsearch@5c8f3c7341
Also uploads the pack zip to the nas instead of the elasticsearch plugin.
The cpp build can be disabled with `-Pxpack.cpp.build=false`
Original commit: elastic/x-pack-elasticsearch@1efb1b2e7e
* Add scheduler status filter to jobs endpoint
* For scheduled jobs set the initial scheduler state
* Add status filter to job endpoint
Original commit: elastic/x-pack-elasticsearch@c7ed1627e2
In the same vein of GetBucket/GetBuckets and GetCategory/GetCategories.
This one was a bit more involved, since previously GetJobs didn't support
configuring the various metrics. So now all metrics can be configured when
requesting both a single job, or a range of jobs. It also unifies the request
body handling between the two and adds POST handlers.
And similar to the other refactorings, the SingleDoc response is gone in
favor of always returning an array of hits.
Original commit: elastic/x-pack-elasticsearch@ac47bb9bf6
These classes are no longer needed. They used to be
necessary in order to handle compressed data and sending
data to multiple jobs. Compressed data is now handled
by elasticsearch and multi-job upload is no longer supported.
Original commit: elastic/x-pack-elasticsearch@08a9f29855
* Remove DataCounts and ModelSizeStats from Job
* Delete unused ElasticsearchJobDetailsMapper
Conflicts:
elasticsearch/src/main/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapper.java
elasticsearch/src/test/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapperTests.java
* Add missing mappings for DataCounts
Fixes yaml test failures
* Add test to assert revert a model snapshot sets DataCounts.latest_record_timestamp
* Resolve merge errors
* Add NORELEASE for persisting dataCounts in cluster update
Original commit: elastic/x-pack-elasticsearch@46099d4db6
This change moves the Java code from `/java/apps/engine-node` to `/elasticsearch` so it matches the structure required by X-Pack. This will make it easier to get the gradle build working correct and building the full Pack zip.
Original commit: elastic/x-pack-elasticsearch@2fd6539e85
David Roberts
Zachary Tong
David Kyle
Martijn van Groningen
Dimitris Athanasiou
Colin Goodheart-Smithe