Commit Graph

722 Commits

Author SHA1 Message Date
Andrew Gaul 05573fa108 Fix TOCTOU issue when getting blob during removal
Fixes gaul/s3proxy#318.
2019-11-09 11:25:24 +09:00
Ignasi Barrera ec0ef7c9ff Next development version 2.1.4-SNAPSHOT 2019-10-13 17:13:46 +02:00
Ignasi Barrera 82b3a053e0 Apache jclouds 2.1.3-rc1 release 2019-10-13 17:10:14 +02:00
Andrew Gaul d6c22a0d3b Next development version 2.1.3-SNAPSHOT 2019-02-02 15:57:53 -08:00
Andrew Gaul a6029fffa2 Apache jclouds 2.1.2-rc1 release 2019-02-02 15:55:55 -08:00
Andrew Gaul 7bf9c474c6 JCLOUDS-1371: JCLOUDS-1488: list optimize prefix
Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.
2019-01-29 17:46:49 -08:00
Andrew Gaul 1f392212bd JCLOUDS-1366: JCLOUDS-1472: Fix InputStream MPU
Previously jclouds attempted to slice non-repeatable InputStream
Payloads in order to upload sequentially.  This never worked due to
mutating the single stream via skip and close.  Also backfill test
which spuriously succeeded.
2019-01-04 15:43:49 -08:00
Joe Meiring e118e58567 Fix for FileSystem blob store clearContainer with options 2019-01-04 14:18:07 -08:00
Joe Meiring 9ede9d3ddc Removed unneeded check for prefix in clearContainer 2018-11-02 12:19:59 -07:00
Ignasi Barrera 2dec150888 Next development version 2.1.2-SNAPSHOT 2018-08-09 12:45:47 +02:00
Ignasi Barrera 0c79252f64 Apache jclouds 2.1.1-rc1 release 2018-08-09 12:42:41 +02:00
Jesse Glick 8d2808e309 [JCLOUDS-1422] Ensure that LocalBlobStore.list honors !recursive even when a prefix is defined. 2018-06-06 22:49:19 -07:00
Jesse Glick 3cb8d4f87c Include the delimiter field in ListContainerOptions.toString. 2018-06-06 22:49:18 -07:00
Andrew Gaul a79499464a JCLOUDS-1400: Test PUT signed URL Content-Type 2018-04-12 22:44:43 -07:00
Alin Dreghiciu bc78739ea9 JCLOUDS-1392: Do not filter non BLOBs during list 2018-03-08 22:39:58 -08:00
andreaturli 91bb9e8d4c initial commit for maintenance branch 2.1.x 2018-02-21 17:21:11 +01:00
andreaturli 9273bd56ad Apache jclouds 2.1.0-rc3 release 2018-02-16 11:20:21 +01:00
Andrew Gaul eb5db026da JCLOUDS-1368: Fix off-by-one in slicing algorithm 2018-02-08 21:00:33 -08:00
Andrew Gaul 8cd68a3503 JCLOUDS-1369: improving slicing with many parts
Previously with GCS and its maximum 32 parts, the slicing algorithm
would sliced 3.2 GB blobs into (31) 32 MB parts and (1) 2.3 GB part.
With this algorithm it creates (31) 100 MB parts and (1) smaller part.
2018-01-09 12:56:00 -08:00
Andrew Gaul d05be89614 JCLOUDS-1368: Correct use of slicing algorithm
MultipartUploadSlicingAlgorithm creates multiple equal-sized parts and
a remaining amount.  BaseBlobStore.putMultipartBlob used this
interface incorrectly, which could create more parts than intended
since the remaining size could be larger than the part size.  This
manifested with Google Cloud Storage which only allows 32 parts.
2018-01-09 12:56:00 -08:00
Yuval Kashtan 17cef5652f [JCLOUDS-1363] - Fix case sensitivity of blobstore metadata
HTTP headers are case insensitive by nature (see RFC 2616). When addUserMetadataTo check if this is indeed a user metadata header, it must properly ignore case.
The fix make sure that both metadataPrefix and the header key are compared with toLowerCase().

This solves issue with minio metadata read
2017-12-07 08:52:50 +01:00
Andrew Gaul d29b79674d JCLOUDS-1358: Fix zero byte InputStream test input
Do not use ByteSource.empty() since it is backed by a
ByteArrayInputStream which supports reset.
2017-11-21 10:50:56 -08:00
Andrew Gaul e473d7df6a Tighten up test expectations
Follow on to 8b94febfeb.
2017-11-11 07:45:23 -08:00
Andrew Gaul 26570db680 Include HTTP status code in illegal range error 2017-11-10 22:49:45 -08:00
Andrew Gaul 8b94febfeb Throw consistent exception on invalid range read
Previously S3, Swift, and local blobstores threw a generic
IllegalArgumentException for this uncommon error.  Instead
consistently throw HttpResponseException.
2017-11-10 21:39:47 -08:00
Timur Alperovich 343897d6d8 Use different content on overwrite.
When testing blobstore overwrite behavior, jclouds should use a blob
with different content (but same name).
2017-11-06 17:22:32 -08:00
Andrew Gaul 83d10e8194 JCLOUDS-1337: restore tier from stub blob
This allows completeMultipartUpload to propagate the tier set in
initiateMultipartUpload.
2017-11-03 09:52:42 -07:00
Andrew Gaul 7fbef10d57 JCLOUDS-1337: Include tier in object listing
This requires hoisting Tier from BlobMetadata to StorageMetadata.
2017-11-02 16:24:34 -07:00
Andrew Gaul 9a0b51a4d4 Include trailing slash in prefix test 2017-10-30 18:31:40 -07:00
Andrew Gaul af05e24ae9 Implement partial prefix support for Atmos
Atmos only supports listing by directories while other blobstores
allow listing via arbitrary prefixes.  Allow requests which list
directories via both prefix and delimiter = "/" to succeed instead of
failing all requests.  Also change a test which specified recursive to
instead be delimiter = "/".  Fixes gaul/s3proxy#244.
2017-10-30 17:51:35 -07:00
Ignasi Barrera f7b74d95c9 Exclude tier tests on OSX 2017-10-25 08:30:49 +02:00
Andrew Gaul f513bf7c40 JCLOUDS-1337: Atmos putBlob portable storage tiers 2017-10-23 21:31:48 -07:00
Andrew Gaul 6158b60954 JCLOUDS-1337: Portable storage tiers 2017-10-11 12:04:02 -07:00
Andrew Gaul 7704893650 Remove Inject on constructors for abstract classes
This is not meaningful since these classes cannot be instantiated.
Found via error-prone.
2017-09-07 23:37:24 -07:00
Andrew Gaul 517323fbd5 JCLOUDS-1225: Address Guava 18 MoreExecutors changes
Fixed with:

find -name \*.java | xargs sed -i 's/sameThreadExecutor/newDirectExecutorService/g'
2017-08-22 17:14:22 -07:00
Andrew Gaul a493e2ca2f JCLOUDS-1225: Address Guava 18 Objects changes
Fixed with:

find -name \*.java | xargs sed -i 's/Objects.[Tt]oStringHelper/More&/g'
find -name \*.java | xargs sed -i 's/Objects.firstNonNull/More&/g'
find -name \*.java | xargs sed -i 's/^\(import com.google.common.base.\)\(Objects.*\)/\1More\2\n\1\2/g'
find -name \*.java | xargs java -jar google-java-format-1.3-all-deps.jar -i --fix-imports-only --skip-sorting-imports
2017-08-22 17:12:53 -07:00
Chaithanya Ganta d6038487f7 JCLOUDS-1327: Add tests for zero-length blobs 2017-08-06 13:51:07 -07:00
Timur Alperovich d07c4a215e Handle empty delimiter/prefix in FS store.
When delimiter/prefix is an empty string, jclouds filesystem blobstore
should treat them as not being set.
2017-08-03 14:37:12 -07:00
Andrew Gaul 0bc935dd57 Remove clojure bindings
These have not seen any development in many years.
2017-07-10 11:39:11 -07:00
Andrew Gaul e446b5b8b4 JCLOUDS-1111: Force overwriting Atmos objects
This avoids a racy delete then write in the portable abstraction.
2017-07-08 15:50:04 -07:00
Andrew Gaul 2b49bf3618 JCLOUDS-1299: Handle % in names during Swift list
uriBuilder.appendPath assumes an encoded path.
2017-05-16 17:29:02 -07:00
Andrew Gaul 2e7de9f850 JCLOUDS-1271: Deprecate signRemoveBlob
URL signing should always use authentication parameters instead of
headers yet no provider supports DELETEs with the former.
2017-04-19 18:08:52 -07:00
Andrew Gaul 97643823fc Emit marker in PageSet toString 2017-03-18 18:55:12 -07:00
Andrew Gaul e058973abc Ensure SOME_CONSTANTS are static final
Found via error-prone.
2017-03-17 16:59:41 -07:00
Andrew Gaul 87eda15085 Return more metadata from listMultipartUpload
Include Content-Length and Last-Modified.
2017-03-08 20:35:24 -08:00
Andrew Gaul 95c4011982 Include part size in listMultipartUpload result
Clients use this during completeMultipartUpload.
2016-11-15 20:30:50 -08:00
Ignasi Barrera d290705419 Next development version 2.1.0-SNAPSHOT 2016-11-15 01:12:59 +01:00
Ignasi Barrera 6a3a37f4e0 Apache jclouds 2.0.0-rc3 release 2016-11-09 22:30:45 +01:00
Andrew Gaul 69aa5d6426 JCLOUDS-1065: Deprecate fake directory support
Superseded by prefix and delimiter support.
2016-10-26 21:30:52 -07:00
Andrew Gaul 3c9f66b8e9 Avoid lower-case l literal suffix
Readers can confuse this with 1.  Found via error-prone.  Fixed via:

find -name \*.java | xargs sed -i 's/\( [0-9][0-9]*\)l/\1L/g'
find -name \*.java | xargs sed -i 's/\(([0-9][0-9]*\)l/\1L/g'
2016-10-23 14:24:35 -07:00