Commit Graph

223 Commits

Author SHA1 Message Date
Martijn van Groningen 8b1b035965 Wait upto 30 seconds for flush to complete Instead of waiting 5 times 6 seconds.
Original commit: elastic/x-pack-elasticsearch@4766e1e903
2016-11-25 15:35:01 +01:00
David Roberts 2ef240c20b Remove last remnants of bucket/record parent/child relationship (elastic/elasticsearch#387)
The majority of this change was done in PR elastic/elasticsearch#300 - this is just a tidy-up

Original commit: elastic/x-pack-elasticsearch@c9886f0592
2016-11-25 11:08:42 +00:00
Martijn van Groningen 25fb2a4b24 renamed test
Original commit: elastic/x-pack-elasticsearch@eb5e2738cd
2016-11-25 11:21:12 +01:00
David Roberts ba8b60818f StatusReporter log messages should include job ID
Original commit: elastic/x-pack-elasticsearch@841af87535
2016-11-25 10:16:46 +00:00
David Roberts 7fe64bed59 Upgrade to Elasticsearch 5.2 SNAPSHOT
Original commit: elastic/x-pack-elasticsearch@cf0822d27d
2016-11-24 22:55:19 +00:00
David Roberts d1bf5d83ad Remove prelert_ prefix from programs to fit what Elasticsearch expects (elastic/elasticsearch#332)
Elasticsearch will now attempt to spawn a controller daemon called simply
`controller` when it starts up, but ours was called `prelert_controller`.
For consistency it makes sense to remove the prefix from all our programs.

Original commit: elastic/x-pack-elasticsearch@d06714b231
2016-11-24 22:21:17 +00:00
Martijn van Groningen a627e6621c Re-enabled disabled scheduler tests. Main issues seemed to be that the state was sometimes set back from STOPPED to STOPPING and stop scheduler would close the job without updating the status.
Also when executing lookback / realtime searches keep `Future` instances around, so that we can cancel the opertion when a job gets closed.

Closes elastic/elasticsearch#381

Original commit: elastic/x-pack-elasticsearch@9773ff3810
2016-11-24 21:50:31 +01:00
Martijn van Groningen 04968d3ee6 Make persisters node level services and remove the notion of a job logger,
the job id should always be included into the log message itself

Original commit: elastic/x-pack-elasticsearch@7dc6464a9a
2016-11-24 18:09:21 +01:00
Dimitris Athanasiou 26e3ca9155 Make close job a master node action (elastic/elasticsearch#362)
* Make close job a master node action

The close job action now:

 - Is a master node action
 - Sets the job status to CLOSING
 - Waits for the job status to change to CLOSED before it responds

JobLifeCycleService picks up on a job status change to CLOSING and
closes the job. At the end, it sets the job status to CLOSED.

* Assert job status is closed after close in integration test

This also correctly passes an ActionListener to JobManager.setJobStatus
in order to ensure the job status request has completed and to properly
propagate failures.

Original commit: elastic/x-pack-elasticsearch@1546c77fca
2016-11-24 14:59:25 +00:00
David Roberts 3f0b13cda9 Fix the PID used for autodetect process license validation (elastic/elasticsearch#380)
The PID used needs to be the PID of autodetect's parent proecss.  The parent
is of course the controller daemon rather than the JVM.

Original commit: elastic/x-pack-elasticsearch@607481e1e2
2016-11-24 13:51:54 +00:00
Martijn van Groningen b8856eea54 added awaitfix annotations
Original commit: elastic/x-pack-elasticsearch@e490816285
2016-11-24 13:58:53 +01:00
Dimitris Athanasiou d8b6ecfb31 Write record influencers as they are expected in java (elastic/elasticsearch#379)
Original commit: elastic/x-pack-elasticsearch@d016684866
2016-11-24 12:35:24 +00:00
Dimitrios Athanasiou 2a46837296 Fix get records yaml test
Original commit: elastic/x-pack-elasticsearch@9553fd0b52
2016-11-24 11:46:00 +00:00
Martijn van Groningen aa53e7177a wait for the allocation to be added before sending data
Original commit: elastic/x-pack-elasticsearch@504a75a2f2
2016-11-24 11:50:14 +01:00
Dimitrios Athanasiou 02c755bfbe Fix AnomalyRecord.Type and remove setParent call
Original commit: elastic/x-pack-elasticsearch@3ffe114ac2
2016-11-24 10:43:54 +00:00
Martijn van Groningen 4562ec9d6c adding more logging
Original commit: elastic/x-pack-elasticsearch@5c20d662f0
2016-11-24 09:11:53 +01:00
Martijn van Groningen 118abf963b * Added dedicated TP for scheduler and interacting with autodetect process. This capped at the number of threads required to run autodetect process times maximum number of jobs allowed to run on a node.
* Added a setting that determines the maximum number of jobs that can run on a single node.
* Fail to start autodetect process if a user attempts to start more jobs than is allowed on a single node.
* Prevent concurrent data write, flush and close operation to the autodetect process.

Original commit: elastic/x-pack-elasticsearch@aca15fd51c
2016-11-23 17:17:47 +01:00
Zachary Tong 17b3224e03 Parameterize the result field name in QueryPage (elastic/elasticsearch#364)
QueryPage now requires a ParseField in the constructor, which is used for the name of the array of results. I considered putting an enum of different field names in QueryPage itself, but it seemed better to keep the field name local to each respective object.

hitCount was also renamed to count, and getters updated appropriately.

Finally, added a static helper to throw the ResourceNotFoundException, just to make life easier/more consistent.

Closes elastic/elasticsearch#359 

Original commit: elastic/x-pack-elasticsearch@9ba42ad4a1
2016-11-23 11:12:39 -05:00
Dimitris Athanasiou 9fc3c77905 Allow job delete only when job is not running (elastic/elasticsearch#357)
Original commit: elastic/x-pack-elasticsearch@f2959fe2ba
2016-11-23 16:00:36 +00:00
Zachary Tong 4dc20467cb Re-pluralize Endpoints :) (elastic/elasticsearch#363)
Putting the various endpoints back to their plural form (and adjusting Java class names as necessary).

Original commit: elastic/x-pack-elasticsearch@5c8f3c7341
2016-11-23 10:34:49 -05:00
David Roberts b11e5dbf4a Gradle task bundlePack needs to depend on cpp:strip (elastic/elasticsearch#368)
Otherwise the ES plugin can be bundled before the C++ is built

Original commit: elastic/x-pack-elasticsearch@079a59efdf
2016-11-23 14:00:08 +00:00
Colin Goodheart-Smithe b295d764a6 Creates a cpp gradle module to control the cpp build (elastic/elasticsearch#361)
Also uploads the pack zip to the nas instead of the elasticsearch plugin.

The cpp build can be disabled with `-Pxpack.cpp.build=false`

Original commit: elastic/x-pack-elasticsearch@1efb1b2e7e
2016-11-23 11:22:04 +00:00
David Kyle d5bb1f603b Add scheduler status filter to jobs endpoint (elastic/elasticsearch#350)
* Add scheduler status filter to jobs endpoint

* For scheduled jobs set the initial scheduler state

* Add status filter to job endpoint


Original commit: elastic/x-pack-elasticsearch@c7ed1627e2
2016-11-23 10:00:21 +00:00
Martijn van Groningen 423a9cf7b2 write start array only after opening output and
write end array only after closing process

Original commit: elastic/x-pack-elasticsearch@b549ec3552
2016-11-23 08:23:57 +01:00
Martijn van Groningen f8569ca353 no need to check if an index exists (and then fail) prior to removing,
the delete index api will return an index not found error if that index has already been removed

Original commit: elastic/x-pack-elasticsearch@52000532be
2016-11-22 12:08:52 +01:00
Martijn van Groningen fe7d0d32ff Use ES' TP to start reading from the persist state.
Renamed StateReader to StateProcessor.
Folded parsing package into output package.

Original commit: elastic/x-pack-elasticsearch@c027843ec4
2016-11-22 11:29:13 +01:00
Martijn van Groningen 5214de23a1 bug fix: create empty response in post data action for serialization
added getter for data counts
and initialize resetStart / resetEnd with empty strings

Original commit: elastic/x-pack-elasticsearch@d4925506e6
2016-11-22 10:44:14 +01:00
Colin Goodheart-Smithe 15c9e8baa4 Adds ability to build a pack zip in gradle (elastic/elasticsearch#356)
Original commit: elastic/x-pack-elasticsearch@87d6750fc5
2016-11-22 08:29:13 +00:00
Zachary Tong 15e8cf7bcb Consolidate GetJob/GetJobs (elastic/elasticsearch#342)
In the same vein of GetBucket/GetBuckets and GetCategory/GetCategories.

This one was a bit more involved, since previously GetJobs didn't support 
configuring the various metrics. So now all metrics can be configured when 
requesting both a single job, or a range of jobs. It also unifies the request 
body handling between the two and adds POST handlers.

And similar to the other refactorings, the SingleDoc response is gone in 
favor of always returning an array of hits.

Original commit: elastic/x-pack-elasticsearch@ac47bb9bf6
2016-11-21 16:31:01 -05:00
Colin Goodheart-Smithe c8ef5c64de Adds kibana plugin to gradle build (elastic/elasticsearch#354)
This change moves the kibana app to a `kibana` directory at the root of the repository and adds gradle tasks to build the kibana app as part of `gradle check`. The kibana build can be muted by running `gradle check -Pxpack.kibana.build=false`

Original commit: elastic/x-pack-elasticsearch@f6ebb2d62b
2016-11-21 17:56:55 +00:00
Dimitris Athanasiou 63e40d9c72 Remove DataStreamer and DataStreamerThread (elastic/elasticsearch#353)
These classes are no longer needed. They used to be
necessary in order to handle compressed data and sending
data to multiple jobs. Compressed data is now handled
by elasticsearch and multi-job upload is no longer supported.

Original commit: elastic/x-pack-elasticsearch@08a9f29855
2016-11-21 16:30:10 +00:00
David Kyle be0b8575c1 Remove averageBucketProcessingTime from Job (elastic/elasticsearch#351)
Original commit: elastic/x-pack-elasticsearch@60703caa1b
2016-11-21 16:30:01 +00:00
David Roberts c6d52cd949 Fix a knock-on effect of removing JobLogs
Original commit: elastic/x-pack-elasticsearch@501fcda045
2016-11-21 15:49:26 +00:00
David Roberts c1d8b31b0e Remove redundant JobLogs class (elastic/elasticsearch#348)
Now we no longer have a log directory per job we don't need the functionality
to delete those directories.

Fixes elastic/elasticsearch#346

Original commit: elastic/x-pack-elasticsearch@a18beb0519
2016-11-21 15:36:50 +00:00
David Roberts f9dde66678 Add a simple license validation argument to the C++ processes (elastic/elasticsearch#339)
NOT the controller process, as this cannot take arguments when
started as a daemon by Elasticsearch

Closes elastic/elasticsearch#253

Original commit: elastic/x-pack-elasticsearch@14ea155caa
2016-11-21 11:26:10 +00:00
Martijn van Groningen b11f238791 fix test bug
Original commit: elastic/x-pack-elasticsearch@9f1584e48f
2016-11-21 11:53:03 +01:00
David Kyle 7c66c3a553 Remove dataCounts and modelSizeStats from Job (elastic/elasticsearch#333)
* Remove DataCounts and ModelSizeStats from Job

* Delete unused ElasticsearchJobDetailsMapper

Conflicts:
	elasticsearch/src/main/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapper.java
	elasticsearch/src/test/java/org/elasticsearch/xpack/prelert/job/persistence/ElasticsearchJobDetailsMapperTests.java

* Add missing mappings for DataCounts

Fixes yaml test failures

* Add test to assert revert a model snapshot sets DataCounts.latest_record_timestamp

* Resolve merge errors

* Add NORELEASE for persisting dataCounts in cluster update

Original commit: elastic/x-pack-elasticsearch@46099d4db6
2016-11-21 10:18:18 +00:00
Colin Goodheart-Smithe 617a1b65d2 remove warning
Original commit: elastic/x-pack-elasticsearch@73fae7a8fe
2016-11-21 09:28:42 +00:00
Colin Goodheart-Smithe f9de2fdc74 Remove unused headers and variables
Original commit: elastic/x-pack-elasticsearch@c720e2f8cc
2016-11-18 16:57:00 +00:00
Colin Goodheart-Smithe 06df439db0 Moves Java code to fit x-pack style structure
This change moves the Java code from `/java/apps/engine-node` to `/elasticsearch` so it matches the structure required by X-Pack. This will make it easier to get the gradle build working correct and building the full Pack zip.

Original commit: elastic/x-pack-elasticsearch@2fd6539e85
2016-11-18 16:35:00 +00:00
David Roberts 85b5220c59 cppClean task should remove old cppdistribution contents
The `git clean` we used to do at the beginning of the build used to take
care of this, but due to long paths on Windows we can no longer use this

Original commit: elastic/x-pack-elasticsearch@3210c7507b
2016-11-18 13:44:41 +00:00
Martijn van Groningen a77f6df3f2 attempt to always download latest snapshot
Original commit: elastic/x-pack-elasticsearch@869760c9df
2016-11-18 09:34:18 +01:00
David Roberts 155ee948ef Strip the C++ binaries when built via gradle to reduce download size (elastic/elasticsearch#323)
By default this isn't done when building directly with make, so developers
should still have debugging symbols in binaries they build themselves

Original commit: elastic/x-pack-elasticsearch@64cd92431b
2016-11-17 19:34:39 +00:00
Colin Goodheart-Smithe 83f2997ee0 Add infrastructure to the build for documentation (elastic/elasticsearch#320)
Original commit: elastic/x-pack-elasticsearch@7efa8c3f98
2016-11-17 14:07:20 +00:00
David Roberts cd04814cfb Change the Jenkins build entry point from shell script to gradle (elastic/elasticsearch#317)
Other related changes:
1) The Windows build is now done on Windows Server 2012r2
2) The Windows build is now done using Visual Studio 2013 Express (closes elastic/elasticsearch#256)
3) set_env.sh is only used with the C++ build
4) It is no longer necessary to set PRELERT_SRC_HOME if building via gradle -
   this environment variable IS still required for make, but gradle sets
   PRELERT_SRC_HOME before calling make
5) No build tools are loaded from the Prelert NAS
6) The version number for C++ components is now picked up from the same
   gradle.properties as the version number for Java components (closes elastic/elasticsearch#37)
7) "make test" now propagates error returns even when PRELERT_KEEP_GOING is
   defined - this means gradle only needs to check the return code of make
8) The C++ processes now print out Elasticsearch BV as the copyright owner

Jenkins implications:
1) Jenkins installs gradle 2.13 as part of the build process
2) Jenkins now does the build by executing these gradle tasks: cppAll, pluginAll
3) Jenkins must define the JAVA_HOME environment variables
4) The Jenkins slave running on Windows now has a standard setup, i.e. using
   cmd for its shell instead of Git bash
5) 64 bit Git for Windows is used on Windows build slaves instead of 32 bit
   msysgit

Original commit: elastic/x-pack-elasticsearch@01c863bfd0
2016-11-17 13:12:26 +00:00
Colin Goodheart-Smithe c6dadacb29 Changes the build to track elasticsearch 5.1.0-SNAPSHOT (elastic/elasticsearch#248)
Original commit: elastic/x-pack-elasticsearch@af689298f7
2016-11-07 09:27:22 +00:00
David Roberts 3033674607 Fix some Jenkins build issues (elastic/elasticsearch#221)
1) The C++ 3rd party libraries need to be copied to cppdistribution
   before gradle assemble runs
2) Use gradle check instead of gradle test in the build
3) We don't need Maven or pbzip2 any more
4) Make new_version.sh script update the right files
5) Keep a copy of the built plugin after the build

Outstanding TODOs:

1) Versioning needs to be brought in line with Elasticsearch
2) We are building a plugin per platform rather than one containing
   all platforms - we need a step that runs when all platforms have
   completed to create a single plugin
3) Strip the C++ binaries so the uploads aren't so big

Original commit: elastic/x-pack-elasticsearch@0cf32e134f
2016-11-02 15:41:12 +00:00
David Roberts f08faff040 Removed old Java projects (elastic/elasticsearch#208)
They'll soon cease to work at all when $PRELERT_HOME is removed

Any remaining code that needs to be ported can be copied from the analytics-dev branch

Original commit: elastic/x-pack-elasticsearch@1f2d97b429
2016-11-01 10:19:17 +00:00
Colin Goodheart-Smithe 68ecc10ea7 Adds ide build gradle config (elastic/elasticsearch#203)
Makes the ides work the same as the main ES project

Original commit: elastic/x-pack-elasticsearch@3ca9d78ea9
2016-10-31 13:05:32 +00:00
David Roberts 3ea5e62e98 Upgrade the analytics-dev branch to use GA Elasticsearch and Kibana 5.0 (elastic/elasticsearch#188)
Disabled the build of the incomplete engine-node project as it no longer
builds with the GA versions

Original commit: elastic/x-pack-elasticsearch@e07820f228
2016-10-28 11:16:35 +01:00