Commit Graph

709 Commits

Author SHA1 Message Date
Andrew Gaul a79499464a JCLOUDS-1400: Test PUT signed URL Content-Type 2018-04-12 22:44:43 -07:00
Alin Dreghiciu bc78739ea9 JCLOUDS-1392: Do not filter non BLOBs during list 2018-03-08 22:39:58 -08:00
andreaturli 91bb9e8d4c initial commit for maintenance branch 2.1.x 2018-02-21 17:21:11 +01:00
andreaturli 9273bd56ad Apache jclouds 2.1.0-rc3 release 2018-02-16 11:20:21 +01:00
Andrew Gaul eb5db026da JCLOUDS-1368: Fix off-by-one in slicing algorithm 2018-02-08 21:00:33 -08:00
Andrew Gaul 8cd68a3503 JCLOUDS-1369: improving slicing with many parts
Previously with GCS and its maximum 32 parts, the slicing algorithm
would sliced 3.2 GB blobs into (31) 32 MB parts and (1) 2.3 GB part.
With this algorithm it creates (31) 100 MB parts and (1) smaller part.
2018-01-09 12:56:00 -08:00
Andrew Gaul d05be89614 JCLOUDS-1368: Correct use of slicing algorithm
MultipartUploadSlicingAlgorithm creates multiple equal-sized parts and
a remaining amount.  BaseBlobStore.putMultipartBlob used this
interface incorrectly, which could create more parts than intended
since the remaining size could be larger than the part size.  This
manifested with Google Cloud Storage which only allows 32 parts.
2018-01-09 12:56:00 -08:00
Yuval Kashtan 17cef5652f [JCLOUDS-1363] - Fix case sensitivity of blobstore metadata
HTTP headers are case insensitive by nature (see RFC 2616). When addUserMetadataTo check if this is indeed a user metadata header, it must properly ignore case.
The fix make sure that both metadataPrefix and the header key are compared with toLowerCase().

This solves issue with minio metadata read
2017-12-07 08:52:50 +01:00
Andrew Gaul d29b79674d JCLOUDS-1358: Fix zero byte InputStream test input
Do not use ByteSource.empty() since it is backed by a
ByteArrayInputStream which supports reset.
2017-11-21 10:50:56 -08:00
Andrew Gaul e473d7df6a Tighten up test expectations
Follow on to 8b94febfeb.
2017-11-11 07:45:23 -08:00
Andrew Gaul 26570db680 Include HTTP status code in illegal range error 2017-11-10 22:49:45 -08:00
Andrew Gaul 8b94febfeb Throw consistent exception on invalid range read
Previously S3, Swift, and local blobstores threw a generic
IllegalArgumentException for this uncommon error.  Instead
consistently throw HttpResponseException.
2017-11-10 21:39:47 -08:00
Timur Alperovich 343897d6d8 Use different content on overwrite.
When testing blobstore overwrite behavior, jclouds should use a blob
with different content (but same name).
2017-11-06 17:22:32 -08:00
Andrew Gaul 83d10e8194 JCLOUDS-1337: restore tier from stub blob
This allows completeMultipartUpload to propagate the tier set in
initiateMultipartUpload.
2017-11-03 09:52:42 -07:00
Andrew Gaul 7fbef10d57 JCLOUDS-1337: Include tier in object listing
This requires hoisting Tier from BlobMetadata to StorageMetadata.
2017-11-02 16:24:34 -07:00
Andrew Gaul 9a0b51a4d4 Include trailing slash in prefix test 2017-10-30 18:31:40 -07:00
Andrew Gaul af05e24ae9 Implement partial prefix support for Atmos
Atmos only supports listing by directories while other blobstores
allow listing via arbitrary prefixes.  Allow requests which list
directories via both prefix and delimiter = "/" to succeed instead of
failing all requests.  Also change a test which specified recursive to
instead be delimiter = "/".  Fixes gaul/s3proxy#244.
2017-10-30 17:51:35 -07:00
Ignasi Barrera f7b74d95c9 Exclude tier tests on OSX 2017-10-25 08:30:49 +02:00
Andrew Gaul f513bf7c40 JCLOUDS-1337: Atmos putBlob portable storage tiers 2017-10-23 21:31:48 -07:00
Andrew Gaul 6158b60954 JCLOUDS-1337: Portable storage tiers 2017-10-11 12:04:02 -07:00
Andrew Gaul 7704893650 Remove Inject on constructors for abstract classes
This is not meaningful since these classes cannot be instantiated.
Found via error-prone.
2017-09-07 23:37:24 -07:00
Andrew Gaul 517323fbd5 JCLOUDS-1225: Address Guava 18 MoreExecutors changes
Fixed with:

find -name \*.java | xargs sed -i 's/sameThreadExecutor/newDirectExecutorService/g'
2017-08-22 17:14:22 -07:00
Andrew Gaul a493e2ca2f JCLOUDS-1225: Address Guava 18 Objects changes
Fixed with:

find -name \*.java | xargs sed -i 's/Objects.[Tt]oStringHelper/More&/g'
find -name \*.java | xargs sed -i 's/Objects.firstNonNull/More&/g'
find -name \*.java | xargs sed -i 's/^\(import com.google.common.base.\)\(Objects.*\)/\1More\2\n\1\2/g'
find -name \*.java | xargs java -jar google-java-format-1.3-all-deps.jar -i --fix-imports-only --skip-sorting-imports
2017-08-22 17:12:53 -07:00
Chaithanya Ganta d6038487f7 JCLOUDS-1327: Add tests for zero-length blobs 2017-08-06 13:51:07 -07:00
Timur Alperovich d07c4a215e Handle empty delimiter/prefix in FS store.
When delimiter/prefix is an empty string, jclouds filesystem blobstore
should treat them as not being set.
2017-08-03 14:37:12 -07:00
Andrew Gaul 0bc935dd57 Remove clojure bindings
These have not seen any development in many years.
2017-07-10 11:39:11 -07:00
Andrew Gaul e446b5b8b4 JCLOUDS-1111: Force overwriting Atmos objects
This avoids a racy delete then write in the portable abstraction.
2017-07-08 15:50:04 -07:00
Andrew Gaul 2b49bf3618 JCLOUDS-1299: Handle % in names during Swift list
uriBuilder.appendPath assumes an encoded path.
2017-05-16 17:29:02 -07:00
Andrew Gaul 2e7de9f850 JCLOUDS-1271: Deprecate signRemoveBlob
URL signing should always use authentication parameters instead of
headers yet no provider supports DELETEs with the former.
2017-04-19 18:08:52 -07:00
Andrew Gaul 97643823fc Emit marker in PageSet toString 2017-03-18 18:55:12 -07:00
Andrew Gaul e058973abc Ensure SOME_CONSTANTS are static final
Found via error-prone.
2017-03-17 16:59:41 -07:00
Andrew Gaul 87eda15085 Return more metadata from listMultipartUpload
Include Content-Length and Last-Modified.
2017-03-08 20:35:24 -08:00
Andrew Gaul 95c4011982 Include part size in listMultipartUpload result
Clients use this during completeMultipartUpload.
2016-11-15 20:30:50 -08:00
Ignasi Barrera d290705419 Next development version 2.1.0-SNAPSHOT 2016-11-15 01:12:59 +01:00
Ignasi Barrera 6a3a37f4e0 Apache jclouds 2.0.0-rc3 release 2016-11-09 22:30:45 +01:00
Andrew Gaul 69aa5d6426 JCLOUDS-1065: Deprecate fake directory support
Superseded by prefix and delimiter support.
2016-10-26 21:30:52 -07:00
Andrew Gaul 3c9f66b8e9 Avoid lower-case l literal suffix
Readers can confuse this with 1.  Found via error-prone.  Fixed via:

find -name \*.java | xargs sed -i 's/\( [0-9][0-9]*\)l/\1L/g'
find -name \*.java | xargs sed -i 's/\(([0-9][0-9]*\)l/\1L/g'
2016-10-23 14:24:35 -07:00
Shri Javadekar 7826d22d30 JCLOUDS-1161: Make AWSS3BlobRequestSignerV4 the default signer.
Added new live tests and fixed some unit tests accordingly.
2016-10-18 20:06:34 -07:00
Andrew Gaul 50e6d44433 JCLOUDS-945: fix local blobstore marker handling
Previously using prefix markers would not correctly find the next key.
Add test for this behavior.
2016-10-12 22:22:28 -07:00
Andrew Gaul 3b42bafe25 Add streaming methods to ForwardingBlobStore
Follow on to 05c05e3de2.
2016-09-04 22:16:40 -07:00
Zack Shoylev 05c05e3de2 Allows users to download large files efficiently and directly to disk. 2016-09-02 14:05:26 -05:00
Zack Shoylev 42079e1392 Parallel upload for BaseBlobStore 2016-07-15 04:15:17 -05:00
Andrew Gaul 4874a1eb18 JCLOUDS-1137: Handle TOCTOU during blobMetadata
A similar issue exists when getting a blob payload when a caller
simultaneously removes the blob.
2016-07-11 22:02:15 -07:00
Zack Shoylev 6bff97b6d3 Changes the upload behavior to parallel, a TODO 2016-07-01 11:26:56 -05:00
Andrew Gaul 984b6ae8fb Handle null ETag in copyBlob and conditional get
Mac OS X and Docker aufs do not support xattr and thus have null
ETags.  Fixes andrewgaul/s3proxy#143.
2016-06-28 09:55:53 -07:00
Andrew Gaul 6df0472ab5 JCLOUDS-1125: local blobstore list MPUs 2016-06-23 21:45:04 -07:00
Andrew Gaul a845471b88 Create Payload per thread in testPutBlobParallel
Fixes a regression from 45bcc3ce22.
2016-06-23 05:26:21 -07:00
Andrew Gaul 45bcc3ce22 Use in-memory payload for testPutBlobParallel
Also simplify parts of the test.
2016-06-22 22:50:22 -07:00
Andrew Gaul de04c69141 Consistently name test container names
Previously jclouds allocated the initial container pool as
$USERNAME-blobstore-1, 2, etc. and subsequent containers with
$USERNAME-blobstore-$RANDOM.  Use only the former instead.
2016-06-22 22:50:22 -07:00
Andrew Gaul 7a1c3a7b06 Remove listed multipart parts in abort
Previously we unconditionally removed all possible part names.
2016-06-21 10:01:00 -07:00