Commit Graph

4844 Commits

Author SHA1 Message Date
Martijn van Groningen c6763489d5 Moved job lifecycle over to persistent tasks
Also replaced the DELETING status from JobState with a boolean flag on Job. The state of a job is now stored inside a persistent task in cluster state. Jobs that aren't running don't have a persistent task, so I moved that notion of being deleted to the job config itself.

Original commit: elastic/x-pack-elasticsearch@b0ed82124d
2017-02-09 16:20:34 +01:00
Colin Goodheart-Smithe af2486b834 Adds named pipes access for windows to security policy (elastic/elasticsearch#4927)
Windows requires named pipes to be created at a specific location so we need to allow access to this special location in the security policy for the machine learning plugin to run on X-Pack

Original commit: elastic/x-pack-elasticsearch@3079e88713
2017-02-09 15:09:31 +00:00
David Roberts de5e65b687 Allow github.token to work on Windows (elastic/elasticsearch#4925)
Original commit: elastic/x-pack-elasticsearch@95c8b79af1
2017-02-09 14:09:51 +00:00
Jay Modi 8c2ce2c504 [Monitoring] _xpack/monitoring/_bulk action support content stream (elastic/elasticsearch#4916)
This commit marks the monitoring rest bulk action as supporting a content stream. This endpoint takes the same format as the normal bulk endpoint so we need to also accept the newline delimited JSON content type header here.

Closes elastic/elasticsearch#4913

Original commit: elastic/x-pack-elasticsearch@a312cd1256
2017-02-09 06:27:52 -05:00
Martijn van Groningen 5185e06631 [Watcher] Removed not needed usages of `com.google.common.*` classes.
Original commit: elastic/x-pack-elasticsearch@a8dea17a90
2017-02-09 10:40:30 +01:00
Alexander Reelsen 2838946d8b Watcher: Add REST test for execute watch API
to ensure that a 404 is returned, when the watch does not exist.

Relates elastic/elasticsearch#4919

Original commit: elastic/x-pack-elasticsearch@494cd50804
2017-02-09 09:20:54 +01:00
Colin Goodheart-Smithe fe933b027e temporarily increase vault wait
This is temporary fix to allow more time for the AWS credentials to be progagated by the AWS API. Following this we will implement a retry mechanism to keep trying to find the dependency for an amount of time

Original commit: elastic/x-pack-elasticsearch@a300b8698b
2017-02-08 18:15:14 +00:00
Colin Goodheart-Smithe 2f5865756b Migrates Machine Learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@611f3590bb
2017-02-08 17:28:40 +00:00
Colin Goodheart-Smithe b26594cf00 Adds post-migration Elasticsearch changes
Original commit: elastic/x-pack-elasticsearch@f842107076
2017-02-08 16:58:57 +00:00
Colin Goodheart-Smithe 1da752152a Migrates machine learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@9ad22980b8
2017-02-08 16:58:56 +00:00
Colin Goodheart-Smithe e761b76765 Migrates QA files
Original commit: elastic/x-pack-elasticsearch@ac651f51d4
2017-02-08 16:58:55 +00:00
Colin Goodheart-Smithe 1b2381d355 Migrates Elasticsearch files
Original commit: elastic/x-pack-elasticsearch@370af88d14
2017-02-08 16:58:55 +00:00
Colin Goodheart-Smithe fa7a82a945 Removes files no longer needed
Original commit: elastic/x-pack-elasticsearch@8f197075a3
2017-02-08 16:58:55 +00:00
Martijn van Groningen 14a677396e [TEST] Reduce size of large documents to be more heap memory friendly in xpack build
Original commit: elastic/x-pack-elasticsearch@d3864a5021
2017-02-08 17:45:50 +01:00
Alexander Reelsen 1ba5f8fb30 Watcher: Enable with STANDARD license (elastic/elasticsearch#4838)
In order to have monitoring use alerts on cloud, we need to
enable watcher when the STANDARD license is in place. For
more information, please refer to the referenced issue.

Closes elastic/elasticsearch#4766

Original commit: elastic/x-pack-elasticsearch@9d5547274c
2017-02-08 17:38:00 +01:00
Zachary Tong 1591003c7d Fix rest of mocking issues, remove awaitsFix
Original commit: elastic/x-pack-elasticsearch@d5e876e867
2017-02-08 10:57:17 -05:00
David Kyle 6e929fb290 Fix test
Original commit: elastic/x-pack-elasticsearch@0e656d8906
2017-02-08 15:52:23 +00:00
Zachary Tong 99ce9be6ca Fix (some) mocking issues due to upstream changes.
Two suites marked as awaitsFix while being worked on.

Original commit: elastic/x-pack-elasticsearch@06eb352b1e
2017-02-08 10:36:15 -05:00
Colin Goodheart-Smithe 30b6425b3a Convert ml-cpp repo to be part of elasticsearch-extra (elastic/elasticsearch#890)
This means we can reference the local build from within the prelert-legacy build script and build it directly

Original commit: elastic/x-pack-elasticsearch@14024841ab
2017-02-08 14:39:22 +00:00
Zachary Tong 91883ad57b Upstream fixes: use getter methods instead of (now) private members
Original commit: elastic/x-pack-elasticsearch@80786e4f84
2017-02-08 09:37:10 -05:00
David Kyle 9dc4a2f31c Online updates to the running autodetect process (elastic/elasticsearch#886)
* Methods to update the running process with new settings

* Task to update the running autodetect process

* Don’t start process update task if not config specified

Original commit: elastic/x-pack-elasticsearch@4364b141b5
2017-02-08 14:19:24 +00:00
Simon Willnauer 0e779b41de Followup from elastic/elasticsearchelastic/elasticsearch#23042 (elastic/elasticsearch#4895)
This change accomodates for the renamings done in elastic/elasticsearchelastic/elasticsearch#23042

Original commit: elastic/x-pack-elasticsearch@c290c8ecc4
2017-02-08 14:40:17 +01:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Boaz Leskes aadbe81767 TransportService.connectToNode should validate remote node ID (elastic/elasticsearch#4866)
companion PR for elastic/elasticsearchelastic/elasticsearch#22828

Original commit: elastic/x-pack-elasticsearch@2465a03ebc
2017-02-07 22:11:54 +02:00
Tim Brooks b59ead91e0 Remove usages for forbidden apis or suppress (elastic/elasticsearch#4850)
This is related to elastic/elasticsearchelastic/elasticsearch#22964. Methods that could
open socket connections are being made forbidden apis. This commit
either replaces usages with URL.openStream() with a call to
FileSystemUtils.openFileURLStream(URL url) (in the case of a file url)
or adds SuppressForbidden annotations.

Original commit: elastic/x-pack-elasticsearch@93b1b11375
2017-02-07 12:38:21 -06:00
David Roberts 639c02a45e Change variable name
Original commit: elastic/x-pack-elasticsearch@5576ec2196
2017-02-07 17:16:45 +00:00
Jay Modi bd04b30acd security: transport always uses TLS (elastic/elasticsearch#4738)
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.

For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.

Closes elastic/elasticsearch#4332

Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
2017-02-07 11:39:31 -05:00
David Roberts 3eec3ab42a Increase time allowed for large state test (elastic/elasticsearch#871)
Previously it would fail on some old/slow development machines

Closes elastic/elasticsearch#805

Original commit: elastic/x-pack-elasticsearch@6f182ed125
2017-02-07 15:39:59 +00:00
David Roberts af10f880fb Allow vault to work on Windows (elastic/elasticsearch#878)
Original commit: elastic/x-pack-elasticsearch@a404f4793a
2017-02-07 14:59:55 +00:00
Dimitris Athanasiou 15160e41a2 Fix datafeed with date_histogram aggregation (elastic/elasticsearch#876)
date_histogram buckets return the key as a DateTime object.
This PR checks if the key is DateTime and returns the epoch
millis when suitable.

Fixes elastic/elasticsearch#869

Original commit: elastic/x-pack-elasticsearch@8e39760dad
2017-02-07 14:45:02 +00:00
Dimitris Athanasiou 678ae53596 Make flush wait to completion (elastic/elasticsearch#875)
Flush has the contract that when it is done results are up-to-date.
Thus, it adds no value to have it timeout. In most cases, the request
should be pretty responsive apart from when it advances time forward.
In the latter scenario, it could force results to be calculated for a
long period of time which could take long. The one use case for this
is the datafeeds and there is no issue with waiting flush to finish.

This PR changes flush to always wait to completion. However, it adds
checking that the c++ process is alive every second, to avoid long
waits in vain when something has gone horribly wrong.

Fixes elastic/elasticsearch#826

Original commit: elastic/x-pack-elasticsearch@de421ab843
2017-02-07 14:28:01 +00:00
Colin Goodheart-Smithe 0c64c22883 Fixed vault URL for ci build
Original commit: elastic/x-pack-elasticsearch@c9cb05bf0e
2017-02-07 12:32:38 +00:00
Colin Goodheart-Smithe 9ed50211d1 fixed CI env variable for vault
Original commit: elastic/x-pack-elasticsearch@623ea83217
2017-02-07 12:28:00 +00:00
Colin Goodheart-Smithe 7dc4adf238 Adds vault access to build to get aws creds (elastic/elasticsearch#874)
After this change the build requires a github.token file in the root directory of the repository so that it can authenticate with the Vault service to get AWS credentials to download the ml-cpp artifacts

Original commit: elastic/x-pack-elasticsearch@630efadef8
2017-02-07 12:22:34 +00:00
Dimitris Athanasiou ccb9ab5717 Fix time field extraction after upstream change (elastic/elasticsearch#873)
Elasticsearch changed doc_values of date fields to return a
joda DateTime object. Thus, we need to call getMillis() to extract
the epoch millis value.

Original commit: elastic/x-pack-elasticsearch@b992882af5
2017-02-07 12:04:01 +00:00
Tim Vernum 8820902f0c [ci] Print git revision of elasticsearch to ci log (elastic/elasticsearch#4782)
The output `Checked out Elasticsearch revision: ...` shows which revision of elasticsearch was included in the CI build. This makes it easier to diagnose and resolve x-pack build failures.

Original commit: elastic/x-pack-elasticsearch@8c39aafc2d
2017-02-07 10:36:49 +11:00
Tim Vernum 286d62f00e Don't clear realm-cache during security upgrade on basic license (elastic/elasticsearch#4806)
If a user has a basic license, but previously had a full-featured license (e.g. a trial license that expired) then they may have .security index that needs to be migrated even though security is not allowed under their license.

This change makes the clearing of the realm-cache conditional on the license state. If X-Pack is running on a license that does not allow auth, then, when the `logstash_system` user is disabled as part of an upgrade migration, the cache is not cleared.

This change also fix a bug whereby a mapping update could take place even if a data migration was in progress, which could cause the `logstash_system` user to be temporarily enabled when it ought not be.

Original commit: elastic/x-pack-elasticsearch@f272e2b19f
2017-02-07 08:56:04 +11:00
Zachary Tong 0b71e015d8 Remove incorrect/unused parameter
Original commit: elastic/x-pack-elasticsearch@4f33186b5c
2017-02-06 14:50:10 -05:00
Zachary Tong a1a5d590b6 Integrate DBQ into job deletion process (elastic/elasticsearch#691)
A JobStorageDeletionTask is created, which supervises the physical deletion of the job.  This
task is a child of the DeleteJob action.  After the DBQ finishes, the normal flow
resumes (physical index deleted, job removed from CS)

Original commit: elastic/x-pack-elasticsearch@5d6f694408
2017-02-06 14:34:36 -05:00
Alexander Reelsen 752356916d Tests: Fix failing test
Caused by removing search QUERY_AND_FETCH

Original commit: elastic/x-pack-elasticsearch@18070b40bb
2017-02-06 19:40:15 +01:00
Simon Willnauer 14e97565fa Remove obsolet search type QUERY_AND_FETCH
Original commit: elastic/x-pack-elasticsearch@10329462bd
2017-02-06 18:26:01 +01:00
Igor Motov a0b37a2510 Replace List with Map in PersistentTasksInProgress
Store currently running persistent tasks in a map instead of a list.

Original commit: elastic/x-pack-elasticsearch@f383b0bbed
2017-02-06 12:18:42 -05:00
Boaz Leskes ccf3ac9c82 fix compilation issue caused by https://github.com/elastic/elasticsearch/pull/22984
Original commit: elastic/x-pack-elasticsearch@6e4d902deb
2017-02-06 16:48:44 +01:00
Dimitrios Athanasiou aa86d57487 Include start/end in log message for starting datafeed
Original commit: elastic/x-pack-elasticsearch@7b88bb27c1
2017-02-06 13:49:03 +00:00
David Roberts f594030c9e Fix some mappings on the .ml indexes (elastic/elasticsearch#870)
Closes elastic/elasticsearch#814

Original commit: elastic/x-pack-elasticsearch@206efacc4c
2017-02-06 12:26:03 +00:00
Suyog Rao 77c477f575 [Monitoring UI] Add Logstash persistent queue stats (elastic/elasticsearch#4775)
* Add persistent queue charts to Logstash "Advanced" view.

Original commit: elastic/x-pack-elasticsearch@62fac9f469
2017-02-06 16:56:10 +05:30
David Kyle 3f9741b85f Make config and result objects with dates human readable (elastic/elasticsearch#863)
Original commit: elastic/x-pack-elasticsearch@9c0c306741
2017-02-06 09:46:21 +00:00
David Roberts 50c4090541 Remove timeout setting from Job (elastic/elasticsearch#866)
This setting was related to auto-close, and Jobs no longer auto-close.

Closes elastic/elasticsearch#832

Original commit: elastic/x-pack-elasticsearch@fef81f9c3b
2017-02-06 09:39:55 +00:00
Jason Tedor 65ddba8585 Avoid certgen status logger error message
Today when running the certgen command, an error message is output by
the status logger. This is due to a logger instance being created before
logging is configured. This happens because the class initializer for
CertificateTool runs the class initializer for ParseField which creates
a deprecation logger which creates a logger. This commit fixes this
issue by wrapping the parser in another class so that we can defer class
initialization until it's actually needed, thus deferring creating the
logger instance until after logging is initialized.

Relates elastic/elasticsearch#4831

Original commit: elastic/x-pack-elasticsearch@00f978c878
2017-02-04 16:07:36 -05:00
Tim Brooks 6675f6bc8d Add missing doPrivileged blocks for connect ops (elastic/elasticsearch#4815)
This is related to elastic/elasticsearchelastic/elasticsearch#22116. There were a few
places where x-pack opens socket connects that were missed in PR elastic/elasticsearch#4701.
This commit adds the doPrivileged blocks.

Original commit: elastic/x-pack-elasticsearch@32bfa087f0
2017-02-03 14:48:07 -06:00