https://github.com/elastic/elasticsearch/pull/12908
Moved output to verbose level and additionally changed the plugin info output format
Before:
```shell
PluginInfo{name='cloud-aws', description='The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.', site=false, jvm=true, classname=org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin, isolated=true, version='2.1.0-SNAPSHOT'}
```
After:
```shell
- Plugin information:
Name: cloud-aws
Description: The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.
Site: false
Version: 2.1.0-SNAPSHOT
JVM: true
* Classname: org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin
* Isolated: true
```
Fixes#12907
If we don't wait we might retrieve the cluster state before second node
was added when we try to add the delegate in which the discovery node for
node_2 is null.
When a single node starts up it will first elect itself as master and then tries to
recover the cluster state or, if there is none, initialize an empty one and publish it.
Until it has done that, the cluster state will contain a global block and
requests might fail with
SERVICE_UNAVAILABLE/1/state not recovered / initialized
We need to wait for green.
* Automatic package repository creation for debian and rpm repositories using deb-s3 and rpm-s3 tools
* Fixing paths in email for repositories
* Add manual verification step for maven staging repo
* Do not create release directory in /tmp, because we might loose it on VMs
* Removed unused download-s3 script
* Add signage check for RPM
* Removed download-s3.py/upload-s3.py, as they are unused
Closes#13209
This commit fixes a test bug in
o.e.a.s.b.n.TransportBroadcastByNodeActionTests. Namely, the randomized
test allowed for the creation of cluster states that allocated indices
having zero shards. This ultimately surfaced in a
NoSuchElementException when attempting to iterate over the nonexistent
shards. The fix is merely to draw the random number of shards from 1 to
10 instead of 0 to 10.
This commit replaces the usage of LoadedCache with a simple CHM and calls
to computeIfAbsent and adds LoadingCache and CacheLoader to forbidden APIs
Relates to #13224
* Added BWC indices
* Added snapshot version to Version.java
* Fixed create_bwc_index to use localhost instead of localhost and 127.0.0.1 (problem with ipv4/6 setup)
This commit removes and now forbids all uses of
Function, Charsets, Collections2 across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.
Relates #13224
When publishing a new cluster state, the master will send it to all the node of the cluster, noting down how many *master* nodes responded successfully. The nodes do not yet process the new cluster state, but rather park it in memory. As soon as at least minimum master nodes have ack-ed the cluster state change, it is committed and a commit request is sent to all the node that responded so far (and will respond in the future). Once receiving the commit requests the nodes continue to process the cluster state change as they did before this change.
A few notable comments:
1. For this change to have effect, min master nodes must be configured.
2. All basic cluster state validation is done in the first phase of publish and is thus now part of `ShardOperationResult`
3. A new `COMMIT_TIMEOUT` settings is introduced, dictating how long a master should wait for nodes to ack the first phase. Unlike `PUBLISH_TIMEOUT`, if waiting for a commit times out, the cluster state change will be rejected.
4. Failing to achieve a min master node of acks, will cause the master to step down as it clearly doesn't have enough active followers.
5. Previously there was a short window between the moment a master lost it's followers and it stepping down because of node fault detection failures. In this short window, the master could process any change (but fail to publish it). This PR closes this gap to 0.
6. A dedicated pending cluster states queue was added to keep pending non-comitted cluster states and manage the logic around processing committed cluster states. See #13303 for details.
Closes#13062 , Closes#13303
This commit allows to refresh the info service in a blocking fashion
which allows tests to prevent installing listeners alltogether and
makes the class easier to test. Installing a listnener is always subject
to concurrent modifications where the listener might be called mulitple times
or with stale information which causes tests to fail.
Java 8 allows for method references which in-turn will cause
compile errors if a method is not visible while reflection fails late
and maybe too late. We can now register Request instances via FooRequest::new
instead of passing FooRequest.class and call it's ctor via reflection.
This test sometimes fails because the first node is elected as master and waits 30s for incoming joins but in the meanwhile the 3 other nodes form a cluster on their side. The index will be created and its shards allocated on these 3 nodes, then the test checks for the number of shards on each node (it should be 2 or 3) but because the first node has not fully join the cluster yet one node will have 5 shards.
closes#13305
1. FileSystem wrapping code is broken, thats why you get providermismatch exception!
Instead of fixing this, it SuppressesForbidden!!!!
2. Because it uses SuppressForbidden on the test, the whole thing is lenient, it uses java.io.File for example!
3. Of course it fails consistently on windows because it can't remove files, because it leaks file handles (locks)
like a sieve since it does not close node environment. With correct wrapping this is always detected by e.g.
our leak detection FS. Instead of fixing the leak, it assumesFalse(WINDOWS) !!!!!
I do not know how this snuck past me, but I need this fixed to remove setAccessible.
During the second phase of recovery, replayed transaction log entries may need to wait on mapping changes that have not yet propagated to the target node. Currently we correctly replay the operation at a later stage, but we acknowledge the replay request before actually performing the work.
Example failure: http://build-us-00.elastic.co/job/es_feature_two_phase_pub/859/Closes#13535
If the machine is very slow this test fails if the delta of the unallocaiton
timestamp and the last scheduled delay is greater than the scheduled delay time.
Today this is really horrible, and we have a PR sent to fix it, but nobody
does anything: https://github.com/aws/aws-sdk-java/pull/432
With java 9, we cannot even grant the permission, this kind of sheistiness is not allowed,
and s3 repository is completely broken.
The problem is their code is still broken, and won't handle neither SecurityException (our PR)
nor the new InaccessibleObjectException they will get from java 9.
We use a really hacky hack to deliver an exception that their code catches (IllegalAccessException) instead.
This means s3 repository is working on java 9, and we close off access to sun.security.ssl completely
Don't worry, I will fix the rest. But some of those remaining will need a lucene upgrade,
we need to add a getter or two for tests to do things cleanly.
In addition to being a big security problem, setAccessible is a risk
for java 9 migration. We need to clean up our code so we can ban it
and eventually enforce this with security manager for third-party code, too,
or we may have problems.
Instead of using setAccessible, use the correct modifier (e.g. public).
TODO: ban in tests
TODO: ban in security manager at runtime
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableSortedMap across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.
Relates #13224
Instead of asking blob store to create output for posting blob content, this change provides that content of the blob to the blob store for writing. This will significantly simplify the interface for S3 and Azure plugins.
This commit replaces:
* com.google.common.util.concurrent.ListenableFuture
* com.google.common.util.concurrent.SettableFuture
* com.google.common.util.concurrent.Futures
* com.google.common.util.concurrent.MoreExecutors
And forbits its usage via forbidden APIs. This is one of
many steps in the eventual removal of Guava as a dependency.
Relates to #13224
Christoph Büscher
Lee Hinman
Nik Everett
Britta Weber
Britta Weber
Alexander Reelsen
Simon Willnauer
Jason Tedor
Robert Muir
Boaz Leskes
Simon Willnauer
Tanguy Leroux
Robert Muir
Igor Motov