f292408af4
JCLOUDS-948: S3 Cache-Control support
...
Deprecate older S3-specific Cache-Control mechanism.
2016-01-15 13:22:14 -08:00
8bddbb496a
JCLOUDS-480: AWS S3 v4 signature
...
This includes support for chunked uploads.
2016-01-13 15:08:18 -08:00
c20fcb8cda
Do not compare MPU ETag contents
...
AWS S3 MPU ETag are hashes of the part ETag headers but some
implementations, specifically S3Proxy with the filesystem provider,
represent multi-part objects as a single object. Remove these checks
since they add nothing.
2016-01-12 15:36:37 -08:00
85951e2af9
JCLOUDS-844: S3 putBlob ACLs
2015-12-22 00:46:34 -08:00
86491bc607
JCLOUDS-844: Portable putBlob ACLs
2015-12-22 00:46:33 -08:00
ffa7c5c587
Update via canned ACLs instead of XML ACLs for S3
...
These are simpler than the full XML API and better supported by
non-AWS S3 implementations, e.g., Ceph, S3Proxy. Further this makes
the provider more consistent when creating a bucket or object which
only supports setting canned ACLs.
2015-12-19 19:28:03 -08:00
c2ba0bef32
Allow updating S3 canned ACLs
...
These are simpler than the full XML API and better supported by
non-AWS S3 implementations, e.g., Ceph, S3Proxy.
2015-12-19 19:28:03 -08:00
4c8f758e3a
Loosen restrictions for S3 bucket naming
...
US Standard and most S3 clones allow more bucket names than DNS
allows.
2015-12-19 15:10:01 -08:00
750c5d5316
JCLOUDS-1042: Remove unneeded ACL swizzling
...
The previous commit shows that we do not need this.
2015-12-18 23:28:39 -08:00
b47c50b7a5
Correct malformed format strings
...
Found via error-prone.
2015-12-12 08:53:08 +08:00
721a0dbd3c
Add support for standard_ia storage class
...
This enables storage with lower at-rest prices, higher request prices,
and lower availability. Also rework existing reduced redundancy
parsing into a more generic storage class. More background on
standard_ia:
https://aws.amazon.com/blogs/aws/aws-storage-update-new-lower-cost-s3-storage-option-glacier-price-reduction/
2015-11-16 14:04:51 -08:00
4829bbbd2c
JCLOUDS-964: S3 multipart copy
2015-10-14 23:17:40 -07:00
a0b95ea7a3
JCLOUDS-217: Fix encoding handling in S3.
...
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
2015-09-08 15:19:02 -07:00
edde721937
JCLOUDS-962: Make all @Provider methods final
2015-09-02 23:54:44 +02:00
a2d8dfcb27
JCLOUDS-992: Handle directories as blobs in S3.
2015-08-20 12:42:11 -07:00
d86a6c9ee1
Correct Checkstyle violations
2015-07-31 15:28:25 -07:00
86f47394bf
Correct FindBugs violations
2015-07-31 14:53:58 -07:00
fe13b07233
JCLOUDS-929: Implement delimiter support in S3.
...
Plumb the delimiter option to the S3 API.
2015-07-13 14:34:59 -07:00
a29d75a5d1
JCLOUDS-929: Implement generic delimiter support.
...
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
2015-07-13 14:34:58 -07:00
c409c19ff3
JCLOUDS-930: Plumb prefix support down to S3.
...
Plumbs the ListContainerOptions.prefix setting down to the S3 API.
2015-06-29 17:31:56 -07:00
7c7d2e2df6
JCLOUDS-930: Add the prefix option.
...
Add a prefix option to the ListContainerOptions class.
2015-06-29 17:31:55 -07:00
61b06329d8
create java-xmlbuilder.version property for maven
...
add java-xmlbuilder to dependencyManagement so downstream project don't need to specify the version
2015-06-29 08:47:54 +02:00
ae157991ba
JCLOUDS-894: Add portable multipart upload for S3
2015-06-06 16:32:16 -07:00
00734ee755
JCLOUDS-894: Move MultipartUploadSlicingAlgorithm to core
2015-06-06 16:32:16 -07:00
5095590d60
JCLOUDS-894: Do not add unnecessary ETag quotes
...
Previously
AWSS3BlobIntegrationLiveTest.testMultipartUploadMultipleParts failed.
2015-05-26 11:19:58 -07:00
9128ae515f
JCLOUDS-894: Expose S3 multipart operations
2015-05-01 13:32:35 -07:00
ca543f3cdd
Add S3 list multipart parts
...
TODO: pagination, etc.?
2015-05-01 13:32:35 -07:00
21a422bc71
Fix testCopyIfModifiedSince for AWS S3
2015-04-10 23:11:21 -07:00
a43dcece16
JCLOUDS-651: S3 copy object content metadata
2015-04-08 18:07:28 -07:00
b85068ca7f
JCLOUDS-651: Add S3 server-side copyBlob
2015-04-02 15:36:53 -07:00
f469cf8b51
Next development version 2.0.0-SNAPSHOT
2015-03-24 10:52:01 +01:00
bb41ed4341
Apache jclouds 1.9.0 release
2015-03-24 10:35:42 +01:00
85637ccee2
Fix ParseS3ErrorFromXmlContentTest
...
References JCLOUDS-794.
2015-02-25 18:29:17 -08:00
d7e25d9cc6
Use injected metadata instead of new instance
...
This addresses a regression from JCLOUDS-794.
2015-02-25 17:24:44 -08:00
6c8be12703
JCLOUDS-732: S3 portable object ACLs
2015-02-21 10:28:02 -08:00
2553b09bb4
Build S3 multi-delete request with an XML library
...
This commit addresses test failures in
AWSS3BlobIntegrationLiveTest.deleteMultipleObjects.
2015-02-15 10:25:23 -08:00
ecc8548fbd
JCLOUDS-144: S3 multi-blob delete
2015-02-15 01:06:03 -08:00
4bb319a0cc
JCLOUDS-820: Support multi-delete for generic S3
...
Tested against AWS and DreamObjects. This commit only moves and
renames code.
2015-02-13 06:23:16 -08:00
ba2f8ac232
JCLOUDS-258: Support MPU for generic S3
...
Tested against AWS-S3 and DreamObjects. This commit only moves and
renames code although some classes are duplicated for deprecation
purposes.
2015-02-12 17:59:33 -08:00
ec3f035487
JCLOUDS-660: S3 portable container ACLs
2015-02-12 17:26:48 -08:00
eacfc8fdf1
Handle removal during iteration
...
Previously this threw UnsupportedOperationException due to
findGrantsForGrantee returning an immutable view.
2015-02-12 17:26:47 -08:00
8a2c821e13
JCLOUDS-794: Use bogus URL for generic S3 endpoint
...
This prevents users from accidentally connecting to AWS.
2015-02-04 13:42:03 -08:00
affad9fa3b
JCLOUDS-96: JCLOUDS-351: Fix S3 copy if tests
...
Use correct options in testCopyIfUnmodifiedSince, use looser time
bounds, and ensure that tests throw HttpResponseException when
expected.
2015-01-24 10:56:21 -08:00
c770ffbeb0
Use sorted set to order relative paths correctly
2015-01-22 11:37:46 -08:00
fae097e144
JCLOUDS-654: Add object size to StorageMetadata
...
This allows callers to read the content length during container
listing. Tested against: atmosonline, aws-s3, azureblob, filesystem,
and transient. Intentionally not implemented for legacy swift
provider.
2015-01-09 22:34:56 -08:00
d3c1e2eab4
Use AutoService to generate service loader metadata
2015-01-09 11:22:12 -08:00
90a6bb19f4
Finished partial fix in 91f405c9fe.
2014-11-16 10:37:31 -08:00
3980cd791b
Revert "JCLOUDS-653: Address Guava 18 deprecations"
...
This reverts commit 95cfc0d950
2014-10-24 10:45:58 -07:00
b9525a0877
JCLOUDS-40 unasync Fallback
2014-10-06 13:14:59 -07:00
0a236f59ad
JCLOUDS-153 remove IO Executor and usage of it.
2014-10-06 07:56:31 -07:00
100d43360a
JCLOUDS-150 clear remaining async stuff from aws
2014-10-05 22:32:27 -07:00
044223efcb
Rather than rely on or work around JRE behavior based, lock S3ClientMockTest using OkHttp.
2014-10-05 20:14:45 -07:00
99e217b720
Move off confusing test names.
2014-10-05 17:22:18 -07:00
cacc986dc4
JCLOUDS-40 Remove AsyncBlobStore references from s3 api
2014-10-05 13:18:12 -07:00
dfb583b67a
JCLOUDS-40 remove all implementations of AsyncBlobStore except Submission in preparation for complete removal.
2014-10-05 08:49:38 -07:00
b6497556f6
JCLOUDS-150 add SubmissionAsyncBlobStore; unasync s3 and aws-s3
2014-10-03 13:18:29 -07:00
58438d7d1e
Address Checkstyle violations
2014-09-04 17:24:27 -07:00
0b89b1fd10
Ensure that Content-MD5 has 128 bits
...
Update many tests Content-MD5 headers and ETag responses.
2014-09-04 16:57:48 -07:00
195998b3b6
Miscellaneous FindBugs fixes
2014-08-30 14:33:10 -07:00
df996f3231
Set Content-MD5 only if ETag has correct format
...
Previously this check was more lenient than it should have been.
2014-08-28 21:50:57 -07:00
1b824afbfc
Remove redundant modifiers from interfaces
2014-08-26 17:34:13 -07:00
480f0e883d
Prevent instantiation of utility classes
...
Also prevent extension.
2014-08-26 12:23:36 -07:00
80a4430035
JCLOUDS-679: Use HEAD for S3 bucketExists
...
This method costs 0.4 cents per 10,000 requests instead of 0.5 cents
per 1,000 requests:
http://aws.amazon.com/s3/pricing/
API reference:
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketHEAD.html
2014-08-20 12:44:20 -07:00
e711275fb1
Prefer isEmpty() for collections rather than size()
2014-08-20 11:07:48 -07:00
95cfc0d950
JCLOUDS-653: Address Guava 18 deprecations
...
Mostly renaming Objects.toStringHelper to MoreObjects.toStringHelper.
2014-08-08 11:12:21 -07:00
59462747e7
Reset StringBuilder instead of creating new ones
...
This avoids unneeded garbage, especially during XML parsing. Replaced
with:
find -name \*.java | xargs sed -i 's/^\( *[^ ]*\) = new StringBuilder();$/\1.setLength(0);/'
2014-08-06 11:22:49 -07:00
e3d9851216
Up to 2.0.0-SNAPSHOT after the 1.8.0 release
2014-08-05 09:10:24 +02:00
eba727fef7
'iso8601DateParseWithOptionalTZ' -> 'iso8601DateOrSecondsDateParse' in DateService
...
Follow-up to d57bbebe
2014-07-28 12:20:49 -04:00
d57bbebe7e
Try iso8601SecondsDateParse if iso8601DateParse fails.
...
S3 compatible blobStores sometimes return date in the format:
"2014-07-23T20:53:17+0000" instead of the more common
"2014-07-23T18:09:39.944Z". This caused jclouds to barf with an
IllegalArgumentException.
This commit tries to parse both the formats for S3. The exception
is thrown if both fail.
Added unit tests for the same.
2014-07-26 22:19:54 -07:00
fb60d76704
JCLOUDS-622: Remove most vestiges of InputSupplier
...
Guava 16 deprecated InputSupplier and a future release will remove it.
2014-07-16 16:07:59 -07:00
05c37c2c77
Consistently use Square's mockwebserver
...
Google mockwebserver merged into OkHttp as per:
https://code.google.com/p/mockwebserver/
2014-07-03 14:39:07 -04:00
9cdd53b0b7
JCLOUDS-546: Remove Javadoc @author annotations
...
Annotations removed with:
find -name \*.java | xargs sed -i /@author/d
Empty Javadoc removed with multiple iterations of:
find -name \*.java | xargs sed -i -z 's/ \*\n \*\// \*\//'
find -name \*.java | xargs sed -i -z 's/ \* \n \*\// \*\//'
find -name \*.java | xargs sed -i -z 's/\/\*\*\n \*\/\n//'
2014-06-07 21:44:54 -07:00
70cf3e1f7c
Make constants final classes instead of interfaces
...
This commit prohibits implementation of the empty interface and
instantiation of the class. Refer to _Effective Java_ item 19 for
more background.
2014-05-16 09:45:52 -07:00
8d51ad6f87
Prefer Guava HttpHeaders over JAX-RS
...
jclouds code contains more instances of the former and Guava has
additional constants like CONTENT_MD5.
2014-05-15 01:55:51 -07:00
67c2b5f6b9
Enable whitespace around Checkstyle rule
2014-05-08 18:31:47 -07:00
2d88f5164e
Enforce that all commas have trailing whitespace
2014-05-08 18:28:08 -07:00
c1ee11d9bd
Updating to 1.8.0-SNAPSHOT
2013-12-17 11:31:56 -08:00
828d8790c2
Enforce no unused imports via Checkstyle
...
Removed with:
mvn checkstyle:checkstyle --quiet -Dcheckstyle.output.file=/dev/stdout -Dcheckstyle.output.format=plain | tac | while read i; do echo $i | sed -n 's/\([^:]*\):\([^:]*\):.*/sed -i \2d \1/p' | bash; done
2013-12-11 17:27:43 -08:00
4cef85d1cd
Address Checkstyle violations in tests
2013-12-09 14:40:07 -08:00
cb98a47173
Correct jcloud -> jclouds typos
...
Corrected with:
find -name pom.xml | xargs sed -i 's/jcloud\>/jclouds/'
2013-12-01 11:21:26 -08:00
6c5effcdb1
Address error-prone empty statement warnings
2013-11-05 16:22:44 -08:00
17ce72794c
Enable RedundantModifier for Checkstyle
2013-10-31 10:23:30 -07:00
53146fb7bb
[JCLOUD 301]: Make increased use of Invokable params cache
...
This patch moves the Invokable Parameter cache to Reflection2 and adds
a convenience method for it to allow it to be shared by multiple
callers. The subsequent ability of S3Utils to use this cache results
in a ~40% improvement in performance for generating signed GETs and
PUTs for S3. This commit also converts a few others calls to
Invokable.getParameters() but the observed benefit from those was
small in microbenchmarks.
2013-10-14 17:44:03 -07:00
f1fc63ab76
Add whitespace around keywords and braces
2013-09-30 09:39:29 -07:00
06b8cd7ac5
JCLOUDS-305. Disable S3 virtual host buckets for generic S3
...
Not all S3-compatible providers support virtual host buckets and thus
we should disable this feature by default. Continue to enable virtual
host buckets for AWS-S3 which supports this although this feature
suffers from DNS settling issues. Ran ran integration tests against
AWS-S3 and Scality using its S3 API.
2013-09-27 16:22:42 -07:00
e8ef5c0665
[JCLOUDS-301] Reduce reflection overhead of Invokable.getParameters()
...
By caching the results from Invokable.getParameters(), this commit
improves request signing performance (GETs and PUTs) for S3 by >
3X. These performance problems were seen in production and diagnosed
using the YourKit profiler.
2013-09-27 07:49:09 -07:00
5f8961723f
Remove BlobStore Map abstractions
...
These provide little end-user benefit and make evolving the core API
more difficult. We deprecated these in 1.6.0.
2013-09-25 20:46:31 -07:00
15d9abead0
Skip testDirectory on S3
...
This test cannot succeed due to the way directories are implemented as
prefix pseudo-files in S3.
2013-08-27 23:01:04 -07:00
f6b5b67b5b
JCLOUDS-200: Consolidate all feature-specific in the AWS provider
...
Follow-up to c64c7423
2013-07-26 15:03:12 -04:00
c64c7423cd
AWS-S3 configurable temporary signed URL support
...
Introduces AWSS3BlobRequestSigner, which reuses the
RequestAuthorizeSignature filter for most of the heavy lifting.
Other implementation details based on [1].
Tested with AWSS3BlobSignerLiveTest, in particular,
testSign(Get|Put)UrlWithTime.
Closes JCLOUDS-200
[1] http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html
2013-07-24 15:22:45 -07:00
75385ee57d
JCLOUDS-100. Add AWSServerErrorRetryHandler, with an increased back
...
off time.
2013-06-17 16:48:15 -07:00
64e9a4e4c6
Remove Strings2.replaceAll(String, char, String)
...
String.replace(String, String) serves this purpose better.
2013-06-03 14:41:18 -07:00
ed2f18a1af
JCLOUDS-31. updated to ASF headers via mvn com.mycila.maven-license-plugin:maven-license-plugin:format
2013-05-16 21:30:37 -07:00
fa10fc4838
JCLOUDS-31. remove malformed, duplicate, or unnecessary headers
2013-05-16 21:30:32 -07:00
446ac67884
JCLOUDS-30. Switch to 2 space indents for POM files.
2013-05-13 12:04:43 -07:00
259b04ff1c
JCLOUDS-16. Switch groupId to org.apache.jclouds, change parent POM, remove old distributionManagement content, add ASF repos.
2013-05-13 12:04:42 -07:00
fff2fa6e05
Address Checkstyle violations
...
This commit requires temporarily disabling some checks.
2013-04-17 16:21:21 -07:00
282f3e2a37
Trying to get the test to *pass* now
2013-04-10 13:04:44 +02:00
52301e0a86
Fixing S3ClientMockTest hanging on Java 7 since 843ed57
2013-04-10 12:03:34 +02:00
f56da43db1
deprecated blobstore interfaces that imply complex internal code
2013-04-09 14:37:34 -07:00
Andrew Gaul
Zhao Jin
Timur Alperovich
Oliver Gondža
Andrea Turli
Ignasi Barrera
Jeremy Daggett
Adrian Cole
Adrian Cole
Andrew Phillips
Shri Javadekar
Andrew Bayer
Niraj Tolia
Diwaker Gupta
Andrew Gaul
Andrew Phillips