Commit Graph

75 Commits

Author SHA1 Message Date
Andrew Gaul 3c9f66b8e9 Avoid lower-case l literal suffix
Readers can confuse this with 1.  Found via error-prone.  Fixed via:

find -name \*.java | xargs sed -i 's/\( [0-9][0-9]*\)l/\1L/g'
find -name \*.java | xargs sed -i 's/\(([0-9][0-9]*\)l/\1L/g'
2016-10-23 14:24:35 -07:00
Andrew Gaul d1508b2d23 Make inner class static
Found via error-prone.
2016-10-23 06:39:46 -07:00
Zack Shoylev 00f7ee0738 jclouds was not properly retrying on an expect: 100-continue PUT request that requires re-athentication because of java protocol code
JCLOUDS-1179 This fix also addresses the same problem with other providers

No currently working tests because of https://github.com/square/okhttp/issues/675

This is a common problem in other tools as well https://curl.haxx.se/mail/lib-2004-08/0002.html
2016-10-13 12:17:19 -05:00
Zack Shoylev 4bbca9edf9 More fixes to parallel download resource cleanup 2016-09-14 14:36:44 -05:00
Zack Shoylev de68c2a1b0 Some handles were not properly closed 2016-09-07 16:28:00 -05:00
Zack Shoylev 05c05e3de2 Allows users to download large files efficiently and directly to disk. 2016-09-02 14:05:26 -05:00
Zack Shoylev 42079e1392 Parallel upload for BaseBlobStore 2016-07-15 04:15:17 -05:00
Zack Shoylev 6bff97b6d3 Changes the upload behavior to parallel, a TODO 2016-07-01 11:26:56 -05:00
Andrew Gaul fff12293f9 JCLOUDS-1125: Add missing expected exception 2016-06-23 23:06:13 -07:00
Andrew Gaul 5fec2346a6 JCLOUDS-1028: Configure idempotent methods
Enable POST for Atmos, S3, and Swift.
2016-06-15 16:35:22 -07:00
Andrew Gaul 0bd2959410 JCLOUDS-1125: portable list multipart uploads
Only Azure, B2, and S3 support this operation.  Some MultipartUpload
fields become nullable.
2016-06-13 16:26:43 -07:00
Zack Shoylev c1ce819f61 updateTemporaryUrlKey test 2016-06-07 17:37:39 -05:00
IgnacioCalvo 7d2f1e8b4d JCLOUDS-1080: Force JSON format for requests
Also make all fields optional.
2016-03-17 20:40:51 -07:00
Zack Shoylev f8ebbfdf8a Fixes an intermittently failing swift live test. 2016-03-08 12:38:13 -06:00
Andrew Gaul dd670f3f78 JCLOUDS-1064: More helpful error for null length 2016-03-06 19:08:26 -08:00
Zack Shoylev 8bbcfed312 Ensures the Swift container list call requests JSON as per spec 2016-02-18 18:48:55 -06:00
Andrew Gaul 7eb46cce36 JCLOUDS-651: Swift support for conditional copies 2016-02-16 16:29:54 -08:00
Andrew Gaul 293d3f864e Convert CopyOptions into an AutoValue
This commit requires an interface change since AutoValue lacks support
for Optional and uses Nullable annotations instead.
2016-02-16 16:29:54 -08:00
Andrew Gaul 2bd0550110 Add support for Swift conditional copy 2016-02-09 16:34:48 -08:00
Reijhanniel Jearl Campos d83282ca53 JCLOUDS-691: Remove 404 fallbacks of PUT and POST calls from OpenStack APIs 2016-02-04 12:10:06 +01:00
Andrew Gaul 6360023f09 Remove unnecessary static imports
Found via error-prone.
2016-01-29 09:14:55 -08:00
Andrew Gaul 02b2e80b27 Delete single- and multi-part objects in Swift
Due to Swift's wonky MPU implementation we must try the multi-part
delete first followed by a single-part delete if it fails.  Previously
delete would leave orphan subobjects.  Fixes andrewgaul/s3proxy#92.
2016-01-26 21:22:54 -08:00
Andrew Gaul 68ff250c38 Return response from StaticLargeObjectApi.delete 2016-01-25 13:26:03 -08:00
Andrew Gaul b63f74a6eb JCLOUDS-948: Swift Cache-Control support
Not supported.
2016-01-15 13:22:15 -08:00
Rita Zhang a12ce5a8b7 Add support for AzureBlobStore.getBlobAccess and RegionScopedSwiftBlobStore.getBlobAccess.
They are currently throwing UnsupportedOperationException exception.
2016-01-05 15:33:50 -08:00
Andrew Gaul 91b202f4ea JCLOUDS-844: Swift putBlob ACLs 2015-12-22 00:46:34 -08:00
Andrew Gaul 4abd5dd16f Revert "Delete parts when removing Swift multipart objects"
This reverts commit 3c1588527d.
2015-12-08 10:08:22 +08:00
Andrew Gaul 3c1588527d Delete parts when removing Swift multipart objects
Swift implements multi-part upload with user-visible parts and an
explicit manifest.  When deleting an MPU blob it can delete only the
manifest or both the manifest and parts.  For consistency with other
providers, we now do the latter in the portable abstraction.  Swift
ignores the multipart-manifest=delete parameter for single-part
objects.  Fixes andrewgaul/s3proxy#92.
2015-11-17 23:06:33 -08:00
Andrew Gaul 22c789da9e JCLOUDS-894: Use prefix when listing MPU on Swift
Previously we only used the blob name and not the blob
name/slo/timestamp/part size cookie which yield extra parts when
listing an MPU with parts from a previous MPU.  Listing using the
stricter prefix gives the expected results.
Fixes andrewgaul/s3proxy#91.
2015-11-13 12:15:29 -08:00
Oliver Gondža edde721937 JCLOUDS-962: Make all @Provider methods final 2015-09-02 23:54:44 +02:00
Timur Alperovich 2c417c83a6 JCLOUDS-992: Handle directories as blobs in Swift. 2015-08-20 12:42:10 -07:00
Andrew Gaul e2b6950462 JCLOUDS-872: Swift copy object and append metadata
The existing method replaces metadata.
2015-08-11 13:26:56 -07:00
Andrew Gaul 4f32e863ad JCLOUDS-883: Replace metadata during Swift copy
Previously Swift copy would append metadata instead of replacing it.
2015-08-06 11:26:45 -07:00
Andrew Gaul 86f47394bf Correct FindBugs violations 2015-07-31 14:53:58 -07:00
Ka-Hing Cheung 2c578d7bcb marker should be null when there are no more items 2015-07-21 15:46:58 -07:00
Ka-Hing Cheung ad38e48205 range get of manifest object has no etag 2015-07-14 17:20:14 -07:00
Timur Alperovich 6ec11fd6ec JCLOUDS-929: Implement delimiter support in Swift.
The patch adds the delimiter support in the openstack-swift API. As
part of the change, the subdirectory support in results is introduced.
This occurs when a prefix and delimiter options are set and there are
subdirectories present in the listing (i.e. multiple objects under the
same prefix/delimiter). In this case, Swift will return a list of
"subdir" objects (similar to CommonPrefixes in S3), which need to be
treated differently.
2015-07-13 14:34:59 -07:00
Timur Alperovich a29d75a5d1 JCLOUDS-929: Implement generic delimiter support.
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
2015-07-13 14:34:58 -07:00
Zack Shoylev 65b7bdf282 Removes CopyObjectException in favor of jclouds-standard blobstore exceptions 2015-07-09 18:28:14 -05:00
Timur Alperovich 86af0753bf Swift: do not assume only 1 container in tests.
Openstack Swift ContainerApiLiveTest testListWithOptions assumes that
there are no other containers and that ContainerApiLiveTest container
will _always_ be the first container.

The patch changes the behavior of the test to instead pull out the
container from the returned list.
2015-07-07 18:05:20 -07:00
Timur Alperovich 6d27fbb18a JCLOUDS-930: Handle subdir entries correctly.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).

When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
   delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
   will result in results that include common prefixes (subdir)

In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.

This is caught in the testDirectory integration test.

The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
2015-07-07 17:55:50 -07:00
Timur Alperovich 1cb0822972 JCLOUDS-930: Add prefix option to OpenStack Swift.
Plumbs the prefix option to the openstack-swift provider. In the
process, the support for the recursive option is modified to avoid
setting the _path_ parameter, but rather use the delimiter if required
(setting the delimiter is sufficient for a non-recursive listing).
2015-06-29 17:31:56 -07:00
Timur Alperovich 7c7d2e2df6 JCLOUDS-930: Add the prefix option.
Add a prefix option to the ListContainerOptions class.
2015-06-29 17:31:55 -07:00
Ka-Hing Cheung e0bafd4c10 list MPU parts with prefix listing instead of marker
we now use segment conventions that allow us to do more
efficient listing
2015-06-23 20:06:10 -07:00
Ka-Hing Cheung b8b20c16f5 use segment names similar to those created by python-swiftclient
Currently swift creates blob-1, blob-2... blob-n for multipart
upload segments, which are very common names that can easily
collide with normal objects. This changes the naming convention
to that of python-swiftclient (the `swift' command line).

python-swiftclient also uses a different segment container by
default, that's not implemented by this patch
2015-06-19 17:34:40 -07:00
Andrew Gaul b0dfae1092 Use a configurable timeout for awaitConsistency
This allows setting to zero for strongly-consistency implementations
of s3 and swift stores.
2015-06-15 15:45:17 -07:00
Ka-Hing Cheung 320742c6b5 send HEAD instead of GET for blobMetadata 2015-06-11 17:33:51 -07:00
Ka-Hing Cheung 6ed95d8968 fix range get for manifest object
swift doesn't return the etag if the client is doing a range
get of a manifest (SLO or DLO), this fixes SwiftObject so
etag is optional
2015-06-10 13:38:55 -07:00
Andrew Gaul 4f6af13329 JCLOUDS-894: Swift portable MPU improvements 2015-06-08 11:20:09 -07:00
Andrew Gaul 9a02157a7f JCLOUDS-894: Add portable multipart upload for Swift 2015-06-06 16:32:16 -07:00