After authenticating with vault it can take time for the credentials to be propagated by the AWS API. previously we would just blindly wait for 10 seconds and then try to continue. This change introduces a retry loop where we will do a `headBucket` request every 0.5 seconds until the bucket is accessible or until we have tried 15 times. This means the build is only held up for at most 0.5 seconds after the bucket is accessible.
This is a step towards the final solution since the authentication with vault still happens on every build in the configuraiton phase. A subsequent change will be made to move this out of the configuration phase so that it only runs when the dependencies are required.
Original commit: elastic/x-pack-elasticsearch@ab3abba1ea
Also replaced the DELETING status from JobState with a boolean flag on Job. The state of a job is now stored inside a persistent task in cluster state. Jobs that aren't running don't have a persistent task, so I moved that notion of being deleted to the job config itself.
Original commit: elastic/x-pack-elasticsearch@b0ed82124d
Windows requires named pipes to be created at a specific location so we need to allow access to this special location in the security policy for the machine learning plugin to run on X-Pack
Original commit: elastic/x-pack-elasticsearch@3079e88713
This commit marks the monitoring rest bulk action as supporting a content stream. This endpoint takes the same format as the normal bulk endpoint so we need to also accept the newline delimited JSON content type header here.
Closeselastic/elasticsearch#4913
Original commit: elastic/x-pack-elasticsearch@a312cd1256
This is temporary fix to allow more time for the AWS credentials to be progagated by the AWS API. Following this we will implement a retry mechanism to keep trying to find the dependency for an amount of time
Original commit: elastic/x-pack-elasticsearch@a300b8698b
In order to have monitoring use alerts on cloud, we need to
enable watcher when the STANDARD license is in place. For
more information, please refer to the referenced issue.
Closeselastic/elasticsearch#4766
Original commit: elastic/x-pack-elasticsearch@9d5547274c
This means we can reference the local build from within the prelert-legacy build script and build it directly
Original commit: elastic/x-pack-elasticsearch@14024841ab
* Methods to update the running process with new settings
* Task to update the running autodetect process
* Don’t start process update task if not config specified
Original commit: elastic/x-pack-elasticsearch@4364b141b5
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.
Adds a bootstrap check that the above setting must be set to `false` if security is enabled.
Adds docs for the new setting and bootstrap.
Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.
For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)
Closes: elastic/elasticsearch#4333
Original commit: elastic/x-pack-elasticsearch@db64564093
This is related to elastic/elasticsearchelastic/elasticsearch#22964. Methods that could
open socket connections are being made forbidden apis. This commit
either replaces usages with URL.openStream() with a call to
FileSystemUtils.openFileURLStream(URL url) (in the case of a file url)
or adds SuppressForbidden annotations.
Original commit: elastic/x-pack-elasticsearch@93b1b11375
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.
For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.
Closeselastic/elasticsearch#4332
Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
Flush has the contract that when it is done results are up-to-date.
Thus, it adds no value to have it timeout. In most cases, the request
should be pretty responsive apart from when it advances time forward.
In the latter scenario, it could force results to be calculated for a
long period of time which could take long. The one use case for this
is the datafeeds and there is no issue with waiting flush to finish.
This PR changes flush to always wait to completion. However, it adds
checking that the c++ process is alive every second, to avoid long
waits in vain when something has gone horribly wrong.
Fixeselastic/elasticsearch#826
Original commit: elastic/x-pack-elasticsearch@de421ab843
Elasticsearch changed doc_values of date fields to return a
joda DateTime object. Thus, we need to call getMillis() to extract
the epoch millis value.
Original commit: elastic/x-pack-elasticsearch@b992882af5
If a user has a basic license, but previously had a full-featured license (e.g. a trial license that expired) then they may have .security index that needs to be migrated even though security is not allowed under their license.
This change makes the clearing of the realm-cache conditional on the license state. If X-Pack is running on a license that does not allow auth, then, when the `logstash_system` user is disabled as part of an upgrade migration, the cache is not cleared.
This change also fix a bug whereby a mapping update could take place even if a data migration was in progress, which could cause the `logstash_system` user to be temporarily enabled when it ought not be.
Original commit: elastic/x-pack-elasticsearch@f272e2b19f
A JobStorageDeletionTask is created, which supervises the physical deletion of the job. This
task is a child of the DeleteJob action. After the DBQ finishes, the normal flow
resumes (physical index deleted, job removed from CS)
Original commit: elastic/x-pack-elasticsearch@5d6f694408
Today when running the certgen command, an error message is output by
the status logger. This is due to a logger instance being created before
logging is configured. This happens because the class initializer for
CertificateTool runs the class initializer for ParseField which creates
a deprecation logger which creates a logger. This commit fixes this
issue by wrapping the parser in another class so that we can defer class
initialization until it's actually needed, thus deferring creating the
logger instance until after logging is initialized.
Relates elastic/elasticsearch#4831
Original commit: elastic/x-pack-elasticsearch@00f978c878
This is related to elastic/elasticsearchelastic/elasticsearch#22116. There were a few
places where x-pack opens socket connects that were missed in PR elastic/elasticsearch#4701.
This commit adds the doPrivileged blocks.
Original commit: elastic/x-pack-elasticsearch@32bfa087f0
* Removed getPersistStream() method from this interface and let the NativeAutodetectProcess implementation deal with this. The persist stream is an implementation detail and BlackHoleAutodetectProcess doesn't deal with this too.
* Replaced getProcessOutStream() method with readAutodetectResults() method. This method now returns a `Iterator<AutodetectResult>` instead of an inputstream. This makes the BlackHoleAutodetectProcess and future mocked implementations easier.
Original commit: elastic/x-pack-elasticsearch@086e7b40ab
* Reintroduce chunking to improve data extractor performance
Performing a sorted search/scroll over a period of time that matches
a lot of documents is very expensive because for each page all
documents are traversed.
The solution is to chunk the search time and perform separate
search/scrolls for each chunk.
This commit is introducing a new `chung` config in `datafeed_config`
whose mode can be set to either of AUTO, OFF, MANUAL, with the latter
allowing to specify an explicit chunk size.
When set to AUTO, a heuristic is used in order to determine the chunk
size. The heuristic is based on estimating the time interval within
which we expect `scroll_size` documents and then taking the 10x multiple
of that. Based on benchmarking, this method gives a dramatic performance
increase. For example, for the citizens dataset it improved the ingest
rate from 0.33M docs / minute to 13.6M docs / minute. Farequote is now
done in ~1 second.
Finally, note that when `chunk` is not specified, it defaults to AUTO
when aggregations are not set and to OFF otherwise. This is because
the chunk size heuristic does not lend itself great for aggregations
where one needs to chunk based on the cardinality of buckets rather
than simply time.
Relates to elastic/elasticsearch#734
Original commit: elastic/x-pack-elasticsearch@a738e86d21
Colin Goodheart-Smithe
Martijn van Groningen
David Roberts
Jay Modi
Alexander Reelsen
Zachary Tong
David Kyle
Simon Willnauer
Tim Vernum
Boaz Leskes
Tim Brooks
Dimitris Athanasiou
Zachary Tong
Igor Motov
Suyog Rao
Jason Tedor