248 Commits

Author SHA1 Message Date
Andrew Gaul
13f32b28c9 Lazily open InputStream during complete MPU
Previously the filesystem provider could exhaust file descriptors by
eagerly opening up to 10,000 parts.  This partially undoes
JCLOUDS-1367.
2019-01-29 21:54:47 -08:00
Andrew Gaul
29eec441e9 JCLOUDS-1371: JCLOUDS-1488: list optimize prefix
Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.
2019-01-29 17:39:51 -08:00
Joe Meiring
a36c9dcef0 Fix for FileSystem blob store clearContainer with options 2019-01-04 14:17:00 -08:00
Andrew Gaul
f9cebd59f8 Add error handling for missing xattr
Previously this prevented listing inside a directory when using file
systems like HFS and NFS.  References gaul/s3proxy#279.
2019-01-04 11:41:32 -08:00
Andrew Gaul
b7d59e3fe9 Correct precondition
Follow on to 1ae735bb7d8fa4d512e744f79582477efdfd7f26.
2018-11-14 14:53:05 -08:00
Andrew Gaul
1ae735bb7d Do not allow options in filesystem clearContainer
This requires some additional logic to clean up empty directories.
Test regression from 22ce5484a412bc06ef62995675c07e7a85f66bdf.
2018-11-14 14:42:30 -08:00
Timur Alperovich
896e99df09 Filesystem: Fix the MPU ETags to match S3.
Prior commit introduced a bug in the computation of the MPU ETag value,
where it was concatenating strings, rather than operating on the bytes
of the integer value.
2018-11-02 12:25:17 -07:00
Timur Alperovich
539a9854c1 JCLOUDS-1450: Use S3-style ETags for MPUs.
S3 uses a different ETag for multipart uploads (MPUs) than regular
objects. The ETag consists of the md5 hash of the concatenated ETags of
individual parts followed by the number of parts (separated by "-").

The patch changes the LocalBlobStore's implementation of
CompleteMultipartUpload to set the S3-style ETag before calling
putBlob() and return that ETag to the caller.

To simplify testing, a new protected method with a default NOOP
implementation is added to the BaseBlobIntegrationTest. It allows
providers to further verify MPUs (i.e. ensuring the correct ETag has
been stored alongside the object).
2018-10-24 13:25:00 -07:00
Ignasi Barrera
3839432757
Configure Java 7 language level 2018-07-11 09:42:18 -07:00
Andrew Gaul
46a9f574e2 Fix some orphaned format strings
These lack values to format with.  Found via error-prone 2.3.0.
2018-04-19 08:43:46 -07:00
Andrew Gaul
90fac3b683 JCLOUDS-1271: Remove BlobStore.signRemoveBlob
Most providers never supported this functionality and the portable
abstraction should not have included it.
2018-03-09 20:52:12 -08:00
andreaturli
d17487b223 fix typo in project.version
- from 2.2.0-SNAPSHOST to 2.2.0-SNAPSHOT
2018-02-21 16:45:43 +01:00
andreaturli
a5db565581 Next development version 2.2.0-SNAPSHOST 2018-02-16 11:23:12 +01:00
andreaturli
9273bd56ad Apache jclouds 2.1.0-rc3 release 2018-02-16 11:20:21 +01:00
Andrew Gaul
18eb7f3d38 Handle "file" and "file/" collision
Fixes gaul/s3proxy#240.
2018-01-09 22:26:53 -08:00
Andrew Gaul
52c92a9eb5 Propagate access denied when creating container
Previously this method only reported whether it created a container or
not and callers could not determine whether there was an error or if
the container already existed.  References gaul/s3proxy#122.
2018-01-09 12:44:24 -08:00
Ignasi Barrera
f7b74d95c9 Exclude tier tests on OSX 2017-10-25 08:30:49 +02:00
Andrew Gaul
5448b92ee8 JCLOUDS-1337: only set fs tier when non-null
Fixes regression from 61d4be87467afd830208db9bbaa60df36eff8323.
2017-10-17 23:14:09 -07:00
Andrew Gaul
61d4be8746 JCLOUDS-1337: filesystem putBlob portable storage tiers 2017-10-12 11:15:14 -07:00
Tomas Tulka
6452960c72 fix for deleting an invalid file
If the file for delete is invalid (typically a wrong filename) an IOException will be thrown.
2017-05-10 01:02:09 -07:00
Geert Vanheusden
fbf62437c7 JCLOUDS-1218: filesystem get blob without xattrs
Some filesystems, notably Docker and Mac OS X, do not support xattr
which causes an IOException when getting a blob.
2017-03-01 13:41:30 -08:00
Ignasi Barrera
4509cdecbb Fix container presence check in filesystem provider 2017-02-01 17:42:28 +01:00
Andrew Gaul
32bb2db06e Propagate error on non-existent container or key 2017-01-18 11:20:47 -08:00
mildis
61d0920f19 WindowsEveryonePrincipal : update regex for FR
"Everyone" in a french Windows is "Tout le monde".
Thus, the line.split(" ") returns only "Tout" and putBlob() throws an exception.
The modified regex search for the first two consecutive white char to return the name.
Windows's whoami returns a bunch of 0x20 between the name and the type.
2017-01-10 13:11:33 -06:00
Ignasi Barrera
d290705419 Next development version 2.1.0-SNAPSHOT 2016-11-15 01:12:59 +01:00
Ignasi Barrera
6a3a37f4e0 Apache jclouds 2.0.0-rc3 release 2016-11-09 22:30:45 +01:00
Zack Shoylev
78b3120e5e filesystem: Modify the backslash only on Windows 2016-10-26 19:53:55 -05:00
Zack Shoylev
fc91f64290 Ensure that jclouds consistently uses /, works with \\, and uses File.separator on the filesystem level. 2016-10-25 17:23:32 -05:00
Francois Rigault
dacd9825f0 Allow overriding getUserDefinedFileAttributeView
Today the filesystem blobstore provider use the extended attributes of the
filesystem to store metadata. This is not always desirable as some filesystems
(such as NFSv3) do not support extended metadata.  The current source code does
not allow to change this easily. getUserDefinedFileAttributeView could easily
be overriden for this purpose and allow extensions that customize where the
metadata should be stored.

See also JCLOUDS-658
2016-09-01 11:19:38 -05:00
Iván Lomba
79e95c5a4c JCLOUDS-482: Add support for arbitrary CPU and RAM
This is a combination of 16 commits:

* First approach to ArbitraryCpuRamTemplateBuilderImpl
* Several fixes: refactoring some names, format, identation problems, some missing license headers and generateId method
* Refactored parse utility
* Added GoogleComputeEngineArbitraryCpuRamTemplateBuilderImpl to support GCE custom machine URI
* extracted hardware creation to automaticHardwareForCpuAndRam method
* Fixed ide automatic asterisk imports
* correcting WIP base case PR according to comments
* added machineTypeUriToHardware to set custom hardware in nodes
* fix checkstyle violations and other PR comments
* Set the providerId to custom machineType URI and fix adding node log
* Arbitrary hardware tests added to BaseTemplateBuilderLiveTest and GoogleComputeEngineTemplateBuilderLiveTest
* Added two more tests to BaseTemplateBuilderLiveTest
* Move repeated constants to TestUtils to reuse code
* Fix full path in the Hardware id and URI
* Add custom hardware tests to BaseComputeServiceLiveTest and GCEServiceLiveTest
* Change customHardware test to use buildTemplate and fix identation
2016-08-16 02:44:04 +02:00
Andrew Gaul
4874a1eb18 JCLOUDS-1137: Handle TOCTOU during blobMetadata
A similar issue exists when getting a blob payload when a caller
simultaneously removes the blob.
2016-07-11 22:02:15 -07:00
Andrew Gaul
6df0472ab5 JCLOUDS-1125: local blobstore list MPUs 2016-06-23 21:45:04 -07:00
Andrew Gaul
0bd2959410 JCLOUDS-1125: portable list multipart uploads
Only Azure, B2, and S3 support this operation.  Some MultipartUpload
fields become nullable.
2016-06-13 16:26:43 -07:00
Andrew Gaul
62410d31ae Handle missing containers in LocalBlobStore.list
Also fix FilesystemStorageStrategyImpl.getContainerMetadata to return
null on missing container as jdbc and transient already do.
2016-05-06 21:36:28 -07:00
Andrew Gaul
6371235568 Null out tmpFile to avoid remove after rename 2016-02-05 12:35:27 -08:00
Andrew Gaul
6cefc99cf0 Always delete tmpFile in filesystem putBlob
Previously we would only do this on the exceptional path.  Also throw
IOException on failed rename for consistency.
2016-02-05 00:52:37 -08:00
Andrew Gaul
9feeee8347 Set xattr before rename to make blob create atomic
Previously concurrent operations could expose an object before put had
completed.  Note that the temporary file is still exposed to the
client.
2016-02-04 23:14:54 -08:00
Andrew Gaul
de333e8b3c Check Content-Length in local blobstore putBlob 2016-02-03 22:03:48 -08:00
Andrew Gaul
86e947dded Revert "Always return ETag from filesystem object store"
This reverts commit 496e27f1afa32b90d0656c3f21387cf68a30cb31.  This
commit imposed too much computational overhead when listing blobs.
2016-01-28 23:04:12 -08:00
Andrew Gaul
a8f5401f6c Enable filesystem service tests 2016-01-28 22:42:05 -08:00
Andrew Gaul
7b3a300d51 Skip testBlobAccess on filesytem
Follow on to 52dc1a3ccfc6413d3350bc8f1d092cda8d7a25d3.
2016-01-20 23:47:06 -08:00
Andrew Gaul
52dc1a3ccf Add BaseBlobIntegrationTest.testSetBlobAccess
Similar to BaseContainerIntegrationTest.testSetContainerAccess.
2016-01-19 14:40:55 -08:00
Andrew Gaul
82ad05e98e JCLOUDS-948: Filesystem Cache-Control support 2016-01-15 04:43:39 -08:00
Andrew Gaul
37f307ecd0 Skip testSetContainerAccess for local blobstores
These lack a public HTTP endpoint needed for this test.
2015-12-20 01:14:33 -08:00
Ignasi Barrera
90423d7b9b Fix filesystem api in OSX 2015-11-24 15:38:10 +01:00
Zack Shoylev
ab25fc7259 putblob fails when file exists 2015-11-23 16:01:46 -06:00
Ka-Hing Cheung
ca70d44143 don't remove directory if it's a blob
the filesystem blob store was always removing a/ when a/b was
removed, even if the client explicitly created a/. this patch
fixes that
2015-11-20 17:27:19 -08:00
Andrew Gaul
f706d9c13a JCLOUDS-835: Atomically put filesystem blobs
Write to a temporary file and rename to the desired destination to
better match object store semantics.
2015-11-20 14:44:02 -08:00
Zack Shoylev
2efcb2c5a9 Fixes a windows locale bug with the "Everyone" principal 2015-11-19 11:50:15 -06:00
Zack Shoylev
41ce90ec36 Fixes tests failing on windows filesystems Makes windows behavior more consistent, especially for deletes 2015-11-11 21:01:13 -06:00