Commit Graph

263 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe bfb72d0a96 Adds task to upload cpp distribution zip
Original commit: elastic/x-pack-elasticsearch@c4fee26b37
2016-12-01 14:19:00 +00:00
Martijn van Groningen 6fee7a21b5 applied feedback
Original commit: elastic/x-pack-elasticsearch@579fd037a4
2016-12-01 15:02:19 +01:00
Martijn van Groningen c387a3636d Make auditor index messages asyn in the background, the auditor will not let the calling thread wait.
Also collapsed the the interface into the only implementation.

Original commit: elastic/x-pack-elasticsearch@c68f77627e
2016-12-01 15:02:19 +01:00
Colin Goodheart-Smithe e5e039973e Changes build to only grab cpp zip if it was built
Original commit: elastic/x-pack-elasticsearch@24fc48fe86
2016-12-01 12:16:47 +00:00
David Kyle 53adc100ad Refactor deleting old results (elastic/elasticsearch#431)
* Refactor deleting old results

* Reinstate quiet logging when deleting interim results

Original commit: elastic/x-pack-elasticsearch@01ea95469c
2016-12-01 10:32:41 +00:00
David Kyle 2fdf848df5 Add jobid to job response (elastic/elasticsearch#434)
* Add jobId to job info responses

* Remove getJobId() accessor from job - use getId()

Original commit: elastic/x-pack-elasticsearch@faacef1217
2016-12-01 10:30:37 +00:00
Colin Goodheart-Smithe f0a968292a Makes elasticsearch plugin generation depend on cpp (elastic/elasticsearch#433)
The changes the way the Elasticsearch plugin module gets the cpp code when running the bundlePlugin task. Instead of manually going into the cpp builds output and grabbing the folders it depends on the buildZip task of the :cpp project and grabs the resource that task builds. This means that it does not matter if the cpp buildZip task is changed as long as it results in a zip being generated.

Original commit: elastic/x-pack-elasticsearch@257e5df2e7
2016-12-01 09:49:03 +00:00
Zachary Tong ee54258908 Convert RevertModel response from SingleDoc to simple, inlined model (elastic/elasticsearch#423)
Convert response from SingleDoc to simple, inlined model

Original commit: elastic/x-pack-elasticsearch@6231195b47
2016-11-30 09:52:06 -05:00
Zachary Tong 7c9f65231a Convert PutModelDescription response from SingleDoc to simple, inlined model (elastic/elasticsearch#424)
Original commit: elastic/x-pack-elasticsearch@1d28285e77
2016-11-30 09:48:49 -05:00
Dimitrios Athanasiou 211d787f33 Remove mappings for influencer fields that overlap with bucket or record
Original commit: elastic/x-pack-elasticsearch@36acf9f31b
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou 9dda1ee21a Remove redundant repetition of partition scores in mappings
Original commit: elastic/x-pack-elasticsearch@4194a16529
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou 75c36207b4 Also rename epochStart/epochEnd to start/end in BucketsQueryBuilder
Original commit: elastic/x-pack-elasticsearch@e2eaca7ba5
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou d0c0e6904e Move RestGetInfluencersAction under results package
Original commit: elastic/x-pack-elasticsearch@281d432732
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou ec5aa34d17 Rename epochStart/epochEnd to start/end in result query builders
Original commit: elastic/x-pack-elasticsearch@f4fdd64278
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou 2898e3c421 Make start and end optional params in results endpoints
Original commit: elastic/x-pack-elasticsearch@0ab5da04f9
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou 9d2ce12624 Set type to influencer while getting influencers
Original commit: elastic/x-pack-elasticsearch@c65365e3ee
2016-11-30 14:00:42 +00:00
Dimitrios Athanasiou 5455b5bbad Fix result mappings
Original commit: elastic/x-pack-elasticsearch@e63209af98
2016-11-30 14:00:42 +00:00
David Kyle 1c98d59db0 Bug 401 Fix NPE when POSTing empty JSON object to /jobs (elastic/elasticsearch#420)
Original commit: elastic/x-pack-elasticsearch@fa6db52c8c
2016-11-30 10:19:03 +00:00
Martijn van Groningen fd743cbfc6 Simplify flush listener by using computeIfAbsent(...)
Make sure CountDownLatch gets removed when it is no longer needed
Also add CountDownLatch is it is missing when we ack a flush id,
we may ack before we wait for it

Original commit: elastic/x-pack-elasticsearch@83a993b9ad
2016-11-30 10:04:42 +01:00
Zachary Tong 7f6907da8b Convert GetList from SingleDocument to QueryPage (elastic/elasticsearch#408)
Related to elastic/elasticsearch#314

Original commit: elastic/x-pack-elasticsearch@40702a3d5b
2016-11-29 13:22:54 -05:00
Martijn van Groningen fb2bd73bc1 Let close autodetect wait for other operations to complete.
Original commit: elastic/x-pack-elasticsearch@de517f4fba
2016-11-29 18:05:56 +01:00
David Kyle f88216eaa5 Persist DataCounts every 10 seconds from a thread pool scheduled task. (elastic/elasticsearch#388)
* Persist DataCounts every 10 seconds from a thread pool scheduled task.

Also rework the isReportingBoundary function changing the function as the boundary changes

* Remove overloaded constructors from StatusReporter and DummyStatusReporter

* Persist dataCounts in a background thread from the status reporter

* Use generic threadpool

Original commit: elastic/x-pack-elasticsearch@f00c1067aa
2016-11-29 16:37:47 +00:00
David Kyle 688b5cc202 Renamed REST path parameters to snake case. (elastic/elasticsearch#367)
* Renamed REST path parameters to snake case.

And added missing parameter descriptions

* Document endpoint which accept body parameters

* Endpoints that support body parameters must also accept POST

Original commit: elastic/x-pack-elasticsearch@2cad2e8af6
2016-11-29 16:04:08 +00:00
David Roberts c99ee42f0e Fix a stream parsing edge case (elastic/elasticsearch#413)
BytesReference doesn't like size 0 slices

Original commit: elastic/x-pack-elasticsearch@20ef0d2c1f
2016-11-29 13:52:34 +00:00
David Kyle 5d2cc13797 Fix long lines failing check and parsing tests (elastic/elasticsearch#415)
Original commit: elastic/x-pack-elasticsearch@0711a9c919
2016-11-29 13:50:56 +00:00
David Roberts 62a0f64012 Fix Windows named pipes with Java security manager (elastic/elasticsearch#410)
The problem is that the Java security manager can open the pipe and then
quickly close it, leading to a need to reconnect from the C++ side.

Original commit: elastic/x-pack-elasticsearch@772b57f443
2016-11-29 12:38:06 +00:00
David Kyle 576a591d3b Make the per partition max anomaly scores a result type (elastic/elasticsearch#411)
Original commit: elastic/x-pack-elasticsearch@002a1d7623
2016-11-29 11:52:37 +00:00
Martijn van Groningen 78cd60048c log ResourceAlreadyExistsException differently,
it can happen we encounter more then one cluster states with no usage index and causing this class to send two or more create index requests, but only one request will succeed.

Original commit: elastic/x-pack-elasticsearch@524a1dda61
2016-11-29 10:21:04 +01:00
Martijn van Groningen 0cba57194c test: also wait for paused state
Original commit: elastic/x-pack-elasticsearch@8ffec23a21
2016-11-28 22:43:54 +01:00
Zachary Tong 2b2307a82b Use jobId_timestamp_bucketSpan as ID for bucket (elastic/elasticsearch#375)
Removes the reliance on ES autogenerated UUIDs and instead uses `{jobId}_{timestamp}_{bucketSpan}`

Original commit: elastic/x-pack-elasticsearch@3cd774edd8
2016-11-28 13:59:47 -05:00
Martijn van Groningen b526d7920d change test to wait allocation has been added (which is added in the background)
Original commit: elastic/x-pack-elasticsearch@6b60d37fb5
2016-11-28 18:13:31 +01:00
Martijn van Groningen 2cbc415f38 mute test
Original commit: elastic/x-pack-elasticsearch@adc69f682e
2016-11-28 18:00:05 +01:00
Martijn van Groningen ee132337b5 Update the job status only once when closing (or pausing) the analytical process.
Also changed the the post data api to change the job status to running when the analytical process is started.

Closes elastic/elasticsearch#319

Original commit: elastic/x-pack-elasticsearch@b38d52d849
2016-11-28 16:43:21 +01:00
Martijn van Groningen 2e78706a3f Create usage index upon startup.
Also moved all creation logic that is required to run at cluster startup into PrelertInitializationService.

Original commit: elastic/x-pack-elasticsearch@453ba3efa3
2016-11-28 15:02:30 +01:00
David Kyle 3362d5c965 Remove average processing time mapping (elastic/elasticsearch#402)
Original commit: elastic/x-pack-elasticsearch@97fdeaf748
2016-11-28 12:48:14 +00:00
David Kyle 39fe1b7b09 Remove System.out.println call from test.
Fails the gradle check task

Original commit: elastic/x-pack-elasticsearch@daace999a7
2016-11-28 11:08:28 +00:00
David Kyle 02a94ce729 Detype Results (elastic/elasticsearch#384)
* Add result_type field to bucket

* Query and delete buckets/records by result_type

* Add a filter to the ElasticsearchBatchedDocumentsIterator subclasses for result_type:bucket

* De-type Influencers, BucketInfluencers and Category Definitions

* Revert de-typing CategoryDefinition

* Resolve merge errors after rebase

Original commit: elastic/x-pack-elasticsearch@65605432e8
2016-11-28 10:47:17 +00:00
Dimitris Athanasiou 37cd03ad4d Split records and influencers from bucket (elastic/elasticsearch#389)
In c++ the results are built all together under a bucket hierarchy.
This buckets was written out and java would read it and split the
bucket into its parts: the bucket itself, its records and its
influencers.

During the migration, the bucket started being persisted as a whole,
including its records and influencers.

This commit is changing this by modifying the way results are written
in c++. This way, the java and c++ results writing/reading are in sync.

To achieve this, the change involved writing records and influencers as
top level results from c++. In addition, they are written as an array
object in order to allow the java side to persist them in a bulk
request.

* Fix bucket counting in results processor

Original commit: elastic/x-pack-elasticsearch@feadf3f887
2016-11-25 17:54:24 +00:00
David Roberts 9286ef2304 Add a missing field to the mappings for modelSnapshot documents (elastic/elasticsearch#396)
This fixes one of the problems of elastic/elasticsearch#394

Original commit: elastic/x-pack-elasticsearch@ea627767d2
2016-11-25 17:19:37 +00:00
David Roberts 16b91c9d0f Add job ID to NativeAutodetectProcess log messages (elastic/elasticsearch#392)
Original commit: elastic/x-pack-elasticsearch@5d3b03910e
2016-11-25 16:34:57 +00:00
Martijn van Groningen 8b1b035965 Wait upto 30 seconds for flush to complete Instead of waiting 5 times 6 seconds.
Original commit: elastic/x-pack-elasticsearch@4766e1e903
2016-11-25 15:35:01 +01:00
David Roberts 2ef240c20b Remove last remnants of bucket/record parent/child relationship (elastic/elasticsearch#387)
The majority of this change was done in PR elastic/elasticsearch#300 - this is just a tidy-up

Original commit: elastic/x-pack-elasticsearch@c9886f0592
2016-11-25 11:08:42 +00:00
Martijn van Groningen 25fb2a4b24 renamed test
Original commit: elastic/x-pack-elasticsearch@eb5e2738cd
2016-11-25 11:21:12 +01:00
David Roberts ba8b60818f StatusReporter log messages should include job ID
Original commit: elastic/x-pack-elasticsearch@841af87535
2016-11-25 10:16:46 +00:00
David Roberts 7fe64bed59 Upgrade to Elasticsearch 5.2 SNAPSHOT
Original commit: elastic/x-pack-elasticsearch@cf0822d27d
2016-11-24 22:55:19 +00:00
David Roberts d1bf5d83ad Remove prelert_ prefix from programs to fit what Elasticsearch expects (elastic/elasticsearch#332)
Elasticsearch will now attempt to spawn a controller daemon called simply
`controller` when it starts up, but ours was called `prelert_controller`.
For consistency it makes sense to remove the prefix from all our programs.

Original commit: elastic/x-pack-elasticsearch@d06714b231
2016-11-24 22:21:17 +00:00
Martijn van Groningen a627e6621c Re-enabled disabled scheduler tests. Main issues seemed to be that the state was sometimes set back from STOPPED to STOPPING and stop scheduler would close the job without updating the status.
Also when executing lookback / realtime searches keep `Future` instances around, so that we can cancel the opertion when a job gets closed.

Closes elastic/elasticsearch#381

Original commit: elastic/x-pack-elasticsearch@9773ff3810
2016-11-24 21:50:31 +01:00
Martijn van Groningen 04968d3ee6 Make persisters node level services and remove the notion of a job logger,
the job id should always be included into the log message itself

Original commit: elastic/x-pack-elasticsearch@7dc6464a9a
2016-11-24 18:09:21 +01:00
Dimitris Athanasiou 26e3ca9155 Make close job a master node action (elastic/elasticsearch#362)
* Make close job a master node action

The close job action now:

 - Is a master node action
 - Sets the job status to CLOSING
 - Waits for the job status to change to CLOSED before it responds

JobLifeCycleService picks up on a job status change to CLOSING and
closes the job. At the end, it sets the job status to CLOSED.

* Assert job status is closed after close in integration test

This also correctly passes an ActionListener to JobManager.setJobStatus
in order to ensure the job status request has completed and to properly
propagate failures.

Original commit: elastic/x-pack-elasticsearch@1546c77fca
2016-11-24 14:59:25 +00:00
David Roberts 3f0b13cda9 Fix the PID used for autodetect process license validation (elastic/elasticsearch#380)
The PID used needs to be the PID of autodetect's parent proecss.  The parent
is of course the controller daemon rather than the JVM.

Original commit: elastic/x-pack-elasticsearch@607481e1e2
2016-11-24 13:51:54 +00:00