Commit Graph

264 Commits

Author SHA1 Message Date
Jason Tedor 371cb9a8ce
Remove Log4j 1.2 API as a dependency ()
We had this as a dependency for legacy dependencies that still needed
the Log4j 1.2 API. This appears to no longer be necessary, so this
commit removes this artifact as a dependency.

To remove this dependency, we had to fix a few places where we were
accidentally relying on Log4j 1.2 instead of Log4j 2 (easy to do, since
both APIs were on the compile-time classpath).

Finally, we can remove our custom Netty logger factory. This was needed
when we were on Log4j 1.2 and handled logging in our own unique
way. When we migrated to Log4j 2 we could have dropped this
dependency. However, even then Netty would still pick up Log4j 1.2 since
it was on the classpath, thus the advantage to removing this as a
dependency now.
2019-05-30 16:08:07 -04:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time () 2019-05-30 10:29:42 -07:00
Armin Braun 116b050cc6
Cleanup Bulk Delete Exception Logging () ()
* Cleanup Bulk Delete Exception Logging

* Follow up to 
* Collect all failed blob deletes and add them to the exception message
* Remove logging of blob name list from caller exception logging
2019-05-28 11:00:28 +02:00
Armin Braun 44bf784fe1
Add Infrastructure to Run 3rd Party Repository Tests () ()
* Add Infrastructure to Run 3rd Party Repository Tests

* Add infrastructure to run third party repository tests using our standard JUnit infrastructure
* This is a prerequisite of 
2019-05-28 10:46:22 +02:00
Armin Braun c4f44024af
Remove Delete Method from BlobStore () ()
* Remove Delete Method from BlobStore ()
* The delete method on the blob store was used almost nowhere and just duplicates the delete method on the blob containers
  * The fact that it provided for some recursive delete logic (that did not behave the same way on all implementations) was not used and not properly tested either
2019-05-27 12:24:20 +02:00
Armin Braun aad33121d8
Async Snapshot Repository Deletes () ()
Motivated by slow snapshot deletes reported in e.g.  and the fact that these likely are a contributing factor to repositories accumulating stale files over time when deletes fail to finish in time and are interrupted before they can complete.

* Makes snapshot deletion async and parallelizes some steps of the delete process that can be safely run concurrently via the snapshot thread poll
   * I did not take the biggest potential speedup step here and parallelize the shard file deletion because that's probably better handled by moving to bulk deletes where possible (and can still be parallelized via the snapshot pool where it isn't). Also, I wanted to keep the size of the PR manageable.
* See https://github.com/elastic/elasticsearch/pull/39656#issuecomment-470492106
* Also, as a side effect this gives the `SnapshotResiliencyTests` a little more coverage for master failover scenarios (since parallel access to a blob store repository during deletes is now possible since a delete isn't a single task anymore).
* By adding a `ThreadPool` reference to the repository this also lays the groundwork to parallelizing shard snapshot uploads to improve the situation reported in 
2019-04-26 15:36:09 +02:00
Armin Braun 23b3741618
Remove Exists Check from S3 Repository Deletes () ()
* The check doesn't add much if anything practically, since the S3 repository is eventually consistent and we only log the non-existence of a blob anyway
  * We don't do the check on writes for this very reason and documented it as such
  * Removing the check saves one API call per single delete speeding up the deletion process and lowering costs
2019-04-25 18:25:03 +02:00
Armin Braun c4e84e2b34
Add Bulk Delete Api to BlobStore () ()
* Adds Bulk delete API to blob container
* Implement bulk delete API for S3
* Adjust S3Fixture to accept both path styles for bulk deletes since the S3 SDK uses both during our ITs
* Closes 
2019-04-16 17:19:05 +02:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test () ()
* Replace usages RandomizedTestingTask with built-in Gradle Test ()

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 ()
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
Alpar Torok 35d96c22c0 Fix 3rd pary S3 tests ()
* Fix 3rd pary S3 tests

This is allready excluded on line 186, by doing this again here, the
other exclusion from arround that line are removed causing the tests to
fail.

* Fix blacklisting with the fixture
2019-03-29 08:04:16 +02:00
Alpar Torok 524e0273ae Testclusters: convert plugin repository-s3 ()
* Add support for setting and keystore settings
* system properties and env var config
* use testclusters for repository-s3
* Some cleanup of the build.gradle file for plugin-s3
* add runner {} to rest integ test task
2019-03-27 08:40:16 +02:00
Armin Braun 65732d707f
Add Support for S3 Intelligent Tiering () ()
* Add support for S3 intelligent tiering
* Closes 
2019-03-04 10:32:37 +01:00
Jason Tedor 224600f370
Bump jackson-databind version for AWS SDK ()
This commit bumps the jackson-databind version for discovery-ec2 and
repository-s3 to 2.8.11.3.
2019-02-20 13:04:50 -05:00
Henning Andersen 00a26b9dd2 Blob store compression fix ()
Blob store compression was not enabled for some of the files in
snapshots due to constructor accessing sub-class fields. Fixed to
instead accept compress field as constructor param. Also fixed chunk
size validation to work.

Deprecated repositories.fs.compress setting as well to be able to unify
in a future commit.
2019-02-20 09:24:41 +01:00
Jay Modi 54dbf9469c
Update httpclient for JDK 11 TLS engine ()
The apache commons http client implementations recently released
versions that solve TLS compatibility issues with the new TLS engine
that supports TLSv1.3 with JDK 11. This change updates our code to
use these versions since JDK 11 is a supported JDK and we should
allow the use of TLSv1.3.
2019-01-30 14:24:29 -07:00
Colin Goodheart-Smithe 21e392e95e
Removes typed calls from YAML REST tests ()
This PR attempts to remove all typed calls from our YAML REST tests. The PR adds include_type_name: false to create index requests that use a mapping and also to put mapping requests. It also removes _type from index requests where they haven't already been removed. The PR ignores tests named *_with_types.yml since this are specifically testing typed API behaviour.

The change also includes changing the test harness to add the type _doc to index, update, get and bulk requests that do not specify the document type when the test is running against a mixed 7.x/6.x cluster.
2019-01-30 16:32:58 +00:00
Armin Braun 57823c484f
Streamline S3 Repository- and Client-Settings ()
* Make repository settings override static settings
* Cache clients according to settings
   * Introduce custom implementations for the AWS credentials here to be able to use them as part of a hash key
2019-01-30 06:22:31 +01:00
Armin Braun 899dfc38bc
Fix S3 Repository ITs When Docker is not Available ()
* Disable Minio fixture and tests that require it when fixtures are disabled or Docker is not available
* Relates 
2019-01-25 22:55:29 +01:00
Armin Braun be6bdab346
Use TestFixturesPlugin to Run Minio in Tests ()
* Use TestFixturesPlugin to Run Minio in Tests

* Closes 
* Closes 
2019-01-25 12:56:40 +01:00
Armin Braun d7fe4e57fe
Make Minio Setup more Reliable ()
* Retry starting Minio five times in case we run into a race between finding the free port and starting it up
* Closes 
2019-01-23 19:05:25 +01:00
Alpar Torok a7c3d5842a
Split third party audit exclusions by type () 2019-01-07 17:24:19 +02:00
Armin Braun 617e294133
SNAPSHOT: Make Atomic Blob Writes Mandatory ()
* With  introducing atomic writes to HDFS repository we can enforce atomic write capabilities on this interface
* The overrides on the other three cloud implementations are ok because:
   * https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html states that "Amazon S3 never adds partial objects; if you receive a success response, Amazon S3 added the entire object to the bucket."
   * https://cloud.google.com/storage/docs/consistency states that GCS has strong read-after-write consistency
   * https://docs.microsoft.com/en-us/rest/api/storageservices/put-block#remarks Azure has the concept of committing blobs, so there's no partial content here either
* Relates 
2019-01-07 12:11:19 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task ()
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Ryan Ernst 37b3fc383f
Build: Use explicit deps on test tasks for check ()
This commit moves back to use explicit dependsOn for test tasks on
check. Not all tasks extending RandomizedTestingTask should be run by
check directly.
2018-12-06 14:13:49 -08:00
Tanguy Leroux 6186ccf83e
[Tests] Fix third party tests with Gradle 5.0 ()
* [Tests] Fix third party tests with Gradle 5.0

* apply feedback
2018-12-06 16:05:05 +01:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 ()
Closes 

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Alpar Torok 59b0900174
Upgrade to Gradle 5.0 () 2018-12-05 14:06:11 +02:00
Yannick Welsch 2970abfce9
Add read-only repository verification ()
Adds a verification mode for read-only repositories. It also makes the extra bucket check on
repository creation obsolete, which fixes .
2018-11-23 14:45:05 +01:00
Jernej Klancic baf33b3162 Removes AbstractComponent from several classes ()
Removes inhertiting from AbstractComponent for some classes (mostly
in the plugins module).

Relates to 
2018-11-16 20:50:18 +01:00
Christoph Büscher 09cac321e7
Upgrade to Joda 2.10.1 ()
This version contains a bugfix that allows us to reenable one of our muted tests
in DateTimeUnitTests.

Closes 
2018-11-12 10:02:41 +01:00
Armin Braun 02b4e28534
Add S3 Setting to Force Path Type Access ()
* SNAPSHOTS: Use Path Style Access in S3

* Use path style access pattern to fix 
* closes 
2018-11-09 05:07:26 +01:00
Andy Bristol eec357ebde [test] quote base_path in repository tests 2018-11-01 13:01:53 -07:00
Nik Everett e28509fbfe
Core: Less settings to AbstractComponent ()
Stop passing `Settings` to `AbstractComponent`'s ctor. This allows us to
stop passing around `Settings` in a *ton* of places. While this change
touches many files, it touches them all in fairly small, mechanical
ways, doing a few things per file:
1. Drop the `super(settings);` line on everything that extends
`AbstractComponent`.
2. Drop the `settings` argument to the ctor if it is no longer used.
3. If the file doesn't use `logger` then drop `extends
AbstractComponent` from it.
4. Clean up all compilation failure caused by the `settings` removal
and drop any now unused `settings` isntances and method arguments.

I've intentionally *not* removed the `settings` argument from a few
files:
1. TransportAction
2. AbstractLifecycleComponent
3. BaseRestHandler

These files don't *need* `settings` either, but this change is large
enough as is.

Relates to 
2018-10-31 21:23:20 -04:00
Nik Everett 086ada4c08
Core: Drop settings member from AbstractComponent ()
Drops the `Settings` member from `AbstractComponent`, moving it from the
base class on to the classes that use it. For the most part this is a
mechanical change that doesn't drop `Settings` accesses. The one
exception to this is naming threads where it switches from an invocation
that passes `Settings` and extracts the node name to one that explicitly
passes the node name.

This change doesn't drop the `Settings` argument from
`AbstractComponent`'s ctor because this change is big enough as is.
We'll do that in a follow up change.
2018-10-30 16:10:38 -04:00
Nik Everett 9f87fdc7ab
Drop deprecationLogger from AbstractComponent ()
Drops the `deprecationLogger` from `AbstractComponent`, moving it to
places where we need it. This saves us from building a bunch of
`DeprecationLogger`s that we don't need.

Relates to 
2018-10-26 15:40:16 -04:00
Armin Braun 65edec0d42
TEST: Stablize Minio Free Port Search ()
* Binding to `0` gives us free ports that are assigned sequentially by Linux making collisions much less likely compared to manually finding a free port in a range
* Closes 
2018-10-26 12:55:21 +02:00
Alpar Torok 59536966c2
Add a new "contains" feature ()
The contains syntax was added in  but the skips were not properly
put in place.
The java runner has the feature so the tests will run as part of the
build, but language clients will be able to support it at their own
pace.
2018-10-25 08:50:50 +03:00
Alpar Torok 795d57b4f9
Auto configure all test tasks ()
With this change, we apply the common test config automatically to all
newly created tasks instead of opting in specifically.

For plugin authors using the plugin externally this means that the
configuration will be applied to their RandomizedTestingTasks as well.

The purpose of the task is to simplify setup and make it easier to
change projects that use the `test` task but actually run integration
tests to use a task called `integTest` for clarity, but also because
we may want to configure and run them differently.
E.x. using different levels of concurrency.
2018-10-24 16:05:50 +03:00
Nik Everett 6c07d105f3
Amazon: Wrap at 140 columns ()
Applies our standard column wrapping to the `discovery-ec2` and
`repository-s3` plugins.
2018-10-18 09:09:09 -04:00
Jason Tedor 99681f91f8
Use more precise does S3 bucket exist method ()
We are using a deprecated method for checking if an S3 bucket
exists. This deprecated method has a limitation that it can not
distinguish between invalid credentials and a lack of permissions. This
commit switches to using a method that correctly surfaces if invalid
credentials are supplied when checking for the existence of a bucket.
2018-09-28 10:05:04 -04:00
Jason Tedor 839a677557
Do not override named S3 client credentials ()
In cases when mixed secure S3 client credentials and insecure S3 client
credentials were used (that is, those defined on the repository), we
were overriding the credentials from the repository using insecure
settings to all the repositories. This commit fixes this by not mixing
up repositories that use insecure settings with those that use secure
settings.
2018-09-19 16:18:54 -04:00
Tanguy Leroux 7e195c2912
Update AWS SDK to 1.11.406 in repository-s3 () 2018-09-12 15:27:57 +02:00
Alpar Torok 2cc611604f
Run Third party audit with forbidden APIs CLI (part3/3) ()
The new implementation is functional equivalent with the old, ant based one.
It parses task standard error to get the missing classes and violations in the same way.
I considered re-using ForbiddenApisCliTask but Gradle makes it hard to build inheritance with tasks that have task actions , since the order of the task actions can't be controlled.
This inheritance isn't dully desired either as the third party audit task is much more opinionated and we don't want to expose some of the configuration.
We could probably extract a common base class without any task actions, but probably more trouble than it's worth.

Closes 
2018-08-28 10:03:30 +03:00
Yannick Welsch cd0de16089
[TEST] Select free port for Minio ()
Minio does not support dynamic ports. The workaround here is to scan for a free port first. This is
not foolproof, but as we don't expect too many of these builds to run at once on the same machine,
this should do the trick.

Closes 
Closes 
2018-08-14 16:36:37 +02:00
Vladimir Dolzhenko a2547168d2
fixes broken build for third-party-tests ()
fixes broken build repository-s3 for third-party-tests
2018-07-25 16:34:30 +02:00
Vladimir Dolzhenko 46709f1406
fixes broken build for third-party-tests ()
fixes broken build for third-party-tests ()
Relates  / Closes infra/issues/6085
2018-07-24 14:09:41 +02:00
Vladimir Dolzhenko 7c0fc209bf
ECS Task IAM profile credentials ignored in repository-s3 plugin ()
ECS Task IAM profile credentials ignored in repository-s3 plugin ()

Closes 
2018-07-19 12:54:38 +02:00
Vladimir Dolzhenko 8235b254ab
Add EC2 credential test for repository-s3 ()
Add EC2 credential test for repository-s3

Relates to 
2018-07-18 12:18:00 +02:00
Vladimir Dolzhenko b1bf643e41
lazy snapshot repository initialization ()
lazy snapshot repository initialization
2018-07-13 20:05:49 +02:00