Commit Graph

67 Commits

Author SHA1 Message Date
Ali Beyad c4ae23f5d8 Enables implementations of the BlobContainer interface to (#19749)
conform with the requirements of the writeBlob method by
throwing a FileAlreadyExistsException if attempting to write
to a blob that already exists. This change means implementations
of BlobContainer should never overwrite blobs - to overwrite a
blob, it must first be deleted and then can be written again.

Closes #15579
2016-08-02 09:48:21 -04:00
Ali Beyad 9f88a8194a Merge pull request #19706 from elastic/enhancement/snapshot-blob-handling
More resilient blob handling in snapshot repositories
2016-08-01 12:03:53 -04:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes #19079

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Ali Beyad 630218a16f Change the BlobContainer interface to throw a NoSuchFileFoundException
for reads and deletes if the blob does not exist.
2016-07-22 13:49:25 -04:00
gfyoung dfcdadb59f Added HdfsBlobStoreContainer tests
Added BlobContainer tests for HDFS storage
and caught a bug at the same time in which
deleteBlob was not raising an IOException
when the blobName did not exist.
2016-07-22 13:48:45 -04:00
gfyoung d98fd36dad Added deleteBlob IOException test 2016-07-22 13:48:45 -04:00
gfyoung b02a6da8fd Properly raise IOException for Azure, Fs, Hdfs, and S3 2016-07-22 13:48:45 -04:00
gfyoung 0620a3d6c2 Raised IOException on deleteBlob
Closes gh-18530.
2016-07-22 13:48:45 -04:00
Ali Beyad 19d0dbcd17 Removes waiting for yellow cluster health upon index (#19460)
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).

Relates #19450
2016-07-15 17:18:34 -04:00
Ryan Ernst e6be4af014 Plugins: Add RepositoryPlugin interface for registering snapshot repositories
Repository plugins currently use a lot of custom classes like
RepositoryName and RepositorySettings in order to use guice to construct
repository implementations. But repositories now only really need their
settings to be constructed. Anything else they need (eg a cloud client)
can be constructed within the plugin, instead of via guice.

This change makes repository plugins use the new pull model. It removes
guice from the construction of Repository objects (no more child
injectors) and also from all repository plugins.
2016-07-08 00:10:03 -07:00
Ryan Ernst dd7be74bcf Plugins: Simplified repository api for snapshot/restore
The api for snapshot/restore was split up between two interfaces,
Repository and IndexShardRepository. There was also complex
initialization and injection between the two. However, there is always a
one to one relationship between the two.

This change moves the IndexShardRepository api into Repository, as well
as updates the API so as not to require any services to be injected for
sublcasses.
2016-07-06 17:09:30 -07:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
javanna 598c36128e Revert "Raised IOException on deleteBlob (#18815)"
This reverts commit d24cc65cad as it seems to be causing test failures.
2016-07-01 11:00:32 +02:00
gfyoung d24cc65cad Raised IOException on deleteBlob (#18815)
Raise IOException on deleteBlob if the blob doesn't exist

This commit raises an IOException on BlobContainer#deleteBlob
if the blob does not exist, in conformance with the BlobContainer
interface contract.  Each implementation of BlobContainer now
conforms to this contract (file system, S3, Azure, HDFS).  This 
commit also contains blob container tests for each of the 
repository implementations.

Closes #18530
2016-06-30 23:00:10 -04:00
Ryan Ernst a4503c2aed Plugins: Remove name() and description() from api
In 2.0 we added plugin descriptors which require defining a name and
description for the plugin. However, we still have name() and
description() which must be overriden from the Plugin class. This still
exists for classpath plugins. But classpath plugins are mainly for
tests, and even then, referring to classpath plugins with their class is
a better idea. This change removes name() and description(), replacing
the name for classpath plugins with the full class name.
2016-06-15 17:12:22 -07:00
Ali Beyad 0efac76f01 Clarify the semantics of the BlobContainer interface
This commit clarifies the behavior that must be adhered to by any
implementors of the BlobContainer interface.  This is done through
expanded Javadocs.

Closes #18157
Closes #15580
2016-05-31 19:22:55 -04:00
Adrien Grand 42526ac28e Remove Settings.settingsBuilder.
We have both `Settings.settingsBuilder` and `Settings.builder` that do exactly
the same thing, so we should keep only one. I kept `Settings.builder` since it
has my preference but also it is the one that we use in examples of the Java API.
2016-04-08 18:10:02 +02:00
Simon Willnauer 5008694ba1 Remove support for legacy checksums
Elasticsearch 5.0 doesn't support indices wiht legacy checksums anymore.
The last time we write legacy checksums was in 1.3.0 which was based
on lucene 4.9 already which means that all files have CRC32 checksums.
All indices that Elasticsearch can read today must be written with
lucene version >= 4.8 anyway so we can drop this layer of backwards
compatibility entirely.

Since we are close to upgrading to Lucene 6.0 we should get rid of this
in a more contiained change than the lucene upgrade.
2016-03-03 22:58:18 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Robert Muir f14a21639c add cleanups from simon 2015-12-23 18:15:33 -05:00
Robert Muir 7abd051734 better containing of hadoop for actual blobstore operations 2015-12-22 12:07:37 -05:00
Robert Muir 91dcc9e073 tidy up 2015-12-22 00:28:53 -05:00
Robert Muir 9573bb9f15 make sure BlobStore.close always triggers ACE on any access afterwards 2015-12-22 00:21:03 -05:00
Robert Muir a04268e42e reorder checks 2015-12-21 23:52:16 -05:00
Robert Muir a587ba110c add some safety around repository 2015-12-21 23:48:22 -05:00
Ryan Ernst d104d6d652 Refactor hdfs unit tests to be simple and check every configuration
error condition
2015-12-21 19:38:54 -08:00
Ryan Ernst af7d6b629c Change hdfs unit tests to be a single node test instead of integ test 2015-12-21 18:32:28 -08:00
Robert Muir c54d53c8d5 streamline these classes a bit 2015-12-21 21:11:06 -05:00
Robert Muir 795869c345 remove filecontextfactory 2015-12-21 20:55:17 -05:00
Robert Muir 956281f039 remove shitton of permissions 2015-12-21 20:26:13 -05:00
Ryan Ernst d0e9306413 Remove reading node settings as defaults for hdfs repository settings 2015-12-21 16:25:55 -08:00
Robert Muir 3c07a427dc fix exc handling 2015-12-21 19:06:56 -05:00
Ryan Ernst 26eaa16a89 Remove "additional config" from hdfs repositories 2015-12-21 15:55:13 -08:00
Robert Muir 7065639a26 add test for listing 2015-12-21 17:25:15 -05:00
Robert Muir 5ebcf183e5 tests 2015-12-21 17:02:50 -05:00
Robert Muir b8524bdb11 add tests 2015-12-21 16:16:24 -05:00
Robert Muir 3a2464b80e improve build logic on windows without native libraries 2015-12-21 15:37:34 -05:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00
Robert Muir deaf8884e9 Fix exc handling 2015-12-21 13:04:22 -05:00
Robert Muir 3ffd1a5219 final 2015-12-21 12:54:33 -05:00
Robert Muir f81b12e327 minimize accessiblity, remove unused threadpool 2015-12-21 12:39:40 -05:00
Robert Muir f67390e0c8 in the plugin: guard against HADOOP_HOME in environment on any platform.
hdfs fixture: minihdfs works on windows now, if things are properly set
but our test fixture still cannot launch this on windows.
2015-12-21 02:21:53 -05:00
Robert Muir 53530f1243 remove hacks, test fixtures are clean before each execution 2015-12-20 22:23:30 -05:00
Robert Muir 04966bcc3e contain and improve hack 2015-12-20 21:02:03 -05:00
Robert Muir a37417085d blind stab at unit test issues on windows 2015-12-20 18:31:55 -05:00
Robert Muir ee546ff655 try to get windows working 2015-12-20 17:10:01 -05:00
Robert Muir 2347e3c373 Get forbidden apis passing again, this needs to be investigated 2015-12-20 16:17:17 -05:00