Commit Graph

10626 Commits

Author SHA1 Message Date
davidsloan 17fd80cd5a
JCLOUDS-1557 - Azure local server support
Co-authored-by: David Sloan <david.sloan@lenses.io>
2020-12-08 11:14:03 +09:00
roded 3a7e41f4e2 JCLOUDS-1559: add Charset to Json.fromJson InputStream methods 2020-12-05 20:46:11 +09:00
gurkerl83 94f09325ba Remove scope declaration from deps management
The default scope of imports is "compile." There is no reason to add this declaration to the GSON import declaration. Also, scope declarations are required to be defined in the main dependency declaration section, not in dependency management, where dependency information gets managed, which is valid across all of its children, e.g., version numbers.
2020-10-26 19:58:41 +09:00
gurkerl83 7b5c2ae529 Lowering the GSON version
Lowering the GSON version from 2.8.6 (latest) to 2.8.5, to make sure no side effects get introduced on a JDK level - for reference https://github.com/google/gson/issues/1601. Another feature is in its making to introduce a conditional build profile to support JDK 11 and above; here, a switch to the latest version of GSON gets provided.
2020-10-26 19:58:41 +09:00
gurkerl83 9215bfcb70 In the final state of this feature, a rebase on Master was executed. In resolving a merge problem with the Maven project file "JClouds Project," an important instruction got overwritten, to generate test jars for each module. This modification re-adds this ability for all modules. Counter versa, defining this build step repeatedly, e.g., in the api/oauth module, is no longer required. Also, correct a typo, add groupId.
Note: Previously, the maven jar plugin contained a configuration embedded in each module's generated manifest files. The configuration got relocated to the project/bnd.bnd file in a previous commit, and gets handled through the bnd plugin.
2020-10-26 19:58:41 +09:00
gurkerl83 083bd50122 In the final state of this feature a rebase on Master was executed. In resolving a merge problem with the Maven project file "JClouds Project", an important change already introduced in Master was overlooked. The library Guice was already updated to the latest version 4.2.3 in Master, and the original version number 3.0 was accidentally added to the Maven project file. This modification removes the version number 3.0 from the configuration. 2020-10-26 19:58:41 +09:00
gurkerl83 6a99821136 Re-Enable the build for all modules. Increase version of bnd plugin to the latest. 2020-10-26 19:58:41 +09:00
gurkerl83 32f6c4d50f Remove the OSGi configuration from each module. The approach of defining OSGi configuration through common properties and serving them to the bundle plugin gets no longer used; instead, OSGi configuration gets defined in each module's dedicated bnd file. 2020-10-26 19:58:41 +09:00
gurkerl83 7a9cd345a6 Onboard bnd-configuration files, one per module 2020-10-26 19:58:41 +09:00
gurkerl83 8ac994c2b5 Integrate GSON library in Clouds Core Bundle Final
In the last commit (last section of squashed commit), the GSON library was integrated into the JClouds core module using maven-bundle plugins include resource instruction. Building OSGi instruction variables from the respective modules show a weakness when resources such as script builder shell scripts are required to be integrated into the bundle but not provide a dedicated variable declaration for the resource section.

The following commit demonstrates a change in strategy in declaration and integration of OSGi metadata.

- Replace old bundle-plugin with newest bnd-plugin (bundle-plugin uses bnd-plugin internally)
- Move OSGi metadata declarations from a maven variable passing strategy into dedicated bnd.bnd files
+ Cleaner pom files, no bundle packaging
+ Intellisense / Autocomplete support for .bnd files in terms of package exports etc.

For demonstration, the overall OSGi adjustments are limited to project, core, script builder, compute, blob store, and load balancer because most custom OSGi metadata is defined here.

Note: Other modules are currently disabled from build because some feedback is needed first.

Make GSON integration work.
To understand the changes, see the core modules' bnd file. GSON internal packages also define a version. Both already exported and new export declarations are fused. The global JClouds core module exports defined the entire set of GSON packages available.

Some minor modifications were made in the module project; replace maven jar plugin with a minified version of the declaration, outsourced in projects bnd file.
2020-10-26 19:58:41 +09:00
gurkerl83 d82868cc47 Replace embedded and repackaged GSON library
Replace substituted GSON package names with those provided from the vendor.
Reduce OSGi-metadata declaration of core-module because the artificial package org.jclouds.json.gson.internal was removed.
Remove the Gson module its children Gson bundle, and Gson shaded.
Remove duplication conflict and check-style rules due to the removal of the internal Gson module.
Add maven repository where a custom version of the Gson library gets hosted, which exports all packages.

Remove particular repository

Remove the declaration of the repository that serves a custom build GSON version. The build uses GSON in its original form of the vendor, which gets distributed through the standard distribution channel. The identifiers for group, artifact, and version correspond to the latest stable release of GSON.

Integrate GSON library in Clouds Core Bundle

The change contained in the commit puts the GSON library into the classpath of the JClouds core module.
After several tests with Karaf and Karaf JClouds, especially if the Maven identifier matches the original GSON library, there are only a limited number of ways to keep the deployment effort low.

Specifically, Karaf has a set of predefined Maven repositories that can be easily customized. The order in which a particular repository is resolved into the customized GSON library is more difficult. In normal OSGi applications, which do not have such a management function, I imagine this configuration to be more complicated.  Sure, a unique identifier would help, but then we are back to step 1.

Although I honestly don't like to see this kind of approach in a codebase I'm working with, there are not many alternatives to the main aspect of deployment mentioned above.

Maybe the approach can still ease the problem in the short term. In a further mid-term step, however, this problem must be addressed in general.
2020-10-26 19:58:41 +09:00
Andrew Gaul 1cd28c93c4 Remove unintended executable permissions
Fixed via:

find -executable -not -type d -name \*.java -exec chmod -x {} \;
2020-10-19 13:13:34 +09:00
Andrew Gaul 31a3e5b5df JCLOUDS-1498: Upgrade to Guice 4.2.3
Release notes:

https://github.com/google/guice/wiki/Guice423
2020-09-26 15:48:32 +09:00
Tamas Cservenak d65047c87b
JCLOUDS-1552: Do not attempt to parse empty payload (#82) 2020-09-21 15:46:25 +02:00
Andrew Gaul 3ea2cce5f2 Optimize MultiBlobInputStream.read()
Previously this allocated a byte array for every call.
2020-08-01 18:37:17 +09:00
Andrew Gaul 9c21bf2cc2 JCLOUDS-1367: Do not open open InputStream in copyBlob
This avoids an unneeded call to ByteStreams2.toByteArrayAndClose in
getBlob for range requests and elides a large memory allocation.
2020-08-01 18:37:04 +09:00
Andrew Gaul 8de7b696e1 Store transient Blob data with ByteArrayPayload
This avoids a race condition due to sharing the same Closer instance
and unbounded growth of its Closeable Deque.  References
gaul/s3proxy#303.
2020-07-12 05:13:48 +09:00
Jean-Noël Rouvignac 9e4c7a16da
JCLOUDS-1542 follow-up: check whether isAccessible() is already set (#79)
AccessibleObject.canAccess() would be much much better,
but this API can only be used from Java 9 onward.
2020-07-05 23:33:07 +02:00
Jean-Noël Rouvignac 238c975078
JCLOUDS-1542 Java 11 warns of illegal reflective access (#76)
With Java 11, an illegal reflective access is output for the google cloud storage blobstore.

    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.jclouds.reflect.Reflection2$1 (file:/.../jclouds-core.jar) to constructor java.lang.String(char[],int,int,java.lang.Void)
    WARNING: Please consider reporting this to the maintainers of org.jclouds.reflect.Reflection2$1
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release

Indeed, JClouds calls `setAccessible(true)` on the package protected constructor
     `java.lang.String(char[],int,int,java.lang.Void)`.
Investigating the code, it turns out it is looking for constructors annotated with any of:
* java.beans.ConstructorProperties
* org.jclouds.json.SerializedNames
* com.google.inject.Inject

But `String` being defined in `java.base` module, it is impossible
     that it will be annotated with any of these annotation.

This commit is complementary to JClouds commit db4e4af931 .

&nbsp;

Reflection2.java:
Do not call `setAccessible(true)` on core java constructors and methods.

&nbsp;

For reference, here is the stacktrace of this illegal access warning:

    java.lang.String.<init>(String.java:3208)
    java.lang.String.<init>(String.java:251)
    java.util.StringJoiner.compactElts(StringJoiner.java:250)
    java.util.StringJoiner.toString(StringJoiner.java:173)
    jdk.internal.module.IllegalAccessLogger.loudWarning(IllegalAccessLogger.java:339)
    jdk.internal.module.IllegalAccessLogger.log(IllegalAccessLogger.java:288)
    jdk.internal.module.IllegalAccessLogger.log(IllegalAccessLogger.java:261)
    jdk.internal.module.IllegalAccessLogger.logIfOpenedForIllegalAccess(IllegalAccessLogger.java:226)
    java.lang.reflect.AccessibleObject.logIfOpenedForIllegalAccess(AccessibleObject.java:366)
    java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:325)
    java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
    java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:189)
    java.lang.reflect.Constructor.setAccessible(Constructor.java:182)
    org.jclouds.reflect.Reflection2$1.load(Reflection2$1.java:157)
    org.jclouds.reflect.Reflection2$1.load(Reflection2$1.java:153)
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache$LoadingValueReference.java:3529)
    com.google.common.cache.LocalCache$Segment.loadSync(LocalCache$Segment.java:2278)
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache$Segment.java:2155)
    com.google.common.cache.LocalCache$Segment.get(LocalCache$Segment.java:2045)
    com.google.common.cache.LocalCache.get(LocalCache.java:3953)
    com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
    com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache$LocalLoadingCache.java:4960)
    org.jclouds.reflect.Reflection2.get(Reflection2.java:346)
    org.jclouds.reflect.Reflection2.constructors(Reflection2.java:100)
    org.jclouds.json.internal.NamingStrategies$AnnotationConstructorNamingStrategy.getDeserializer(NamingStrategies$AnnotationConstructorNamingStrategy.java:271)
    org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory.create(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:125)
    com.google.gson.Gson.getAdapter(Gson.java:458)
    org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
    org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
    org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
    com.google.gson.Gson.getAdapter(Gson.java:458)
    com.google.gson.Gson.toJson(Gson.java:696)
    com.google.gson.Gson.toJson(Gson.java:683)
    com.google.gson.Gson.toJson(Gson.java:638)
    com.google.gson.Gson.toJson(Gson.java:618)
    org.jclouds.json.internal.GsonWrapper.toJson(GsonWrapper.java:65)
    org.jclouds.oauth.v2.functions.ClaimsToAssertion.apply(ClaimsToAssertion.java:59)
    org.jclouds.oauth.v2.functions.ClaimsToAssertion.apply(ClaimsToAssertion.java:43)
    org.jclouds.rest.internal.RestAnnotationProcessor.getParamValue(RestAnnotationProcessor.java:829)
    org.jclouds.rest.internal.RestAnnotationProcessor.getFormParamKeyValues(RestAnnotationProcessor.java:847)
    org.jclouds.rest.internal.RestAnnotationProcessor.addFormParams(RestAnnotationProcessor.java:435)
    org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:258)
    org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:137)
    org.jclouds.rest.internal.InvokeHttpMethod.toCommand(InvokeHttpMethod.java:189)
    org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:85)
    org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
    org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
    org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
    org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
    com.sun.proxy.$Proxy49.authorize(Unknown Source)
    org.jclouds.oauth.v2.filters.JWTBearerTokenFlow$AuthorizeToken.load(JWTBearerTokenFlow$AuthorizeToken.java:84)
    org.jclouds.oauth.v2.filters.JWTBearerTokenFlow$AuthorizeToken.load(JWTBearerTokenFlow$AuthorizeToken.java:68)
    com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache$LoadingValueReference.java:3529)
    com.google.common.cache.LocalCache$Segment.loadSync(LocalCache$Segment.java:2278)
    com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache$Segment.java:2155)
    com.google.common.cache.LocalCache$Segment.get(LocalCache$Segment.java:2045)
    com.google.common.cache.LocalCache.get(LocalCache.java:3953)
    com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
    com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache$LocalLoadingCache.java:4960)
    com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache$LocalLoadingCache.java:4966)
    org.jclouds.oauth.v2.filters.JWTBearerTokenFlow.filter(JWTBearerTokenFlow.java:99)
    org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:90)
    org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
    org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
    org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
    org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
    com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
    com.sun.proxy.$Proxy54.simpleUpload(Unknown Source)
    org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.uploadMultipartPart(GoogleCloudStorageBlobStore.java:422)
    org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
    org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
    com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
    com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
    com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor$Worker.java:628)
    java.lang.Thread.run(Thread.java:834)
2020-06-26 19:55:32 +02:00
Sam Ottenhoff 8762fbaf8e JCLOUDS-1473 add INTELLIGENT_TIERING enum 2020-06-25 09:11:33 +09:00
Andrew Gaul 49a54ea9ca Disable doclint during Jenkins build
Now that we require Java 8 we do not need this workaround.
2020-06-25 08:37:25 +09:00
Andrew Gaul 62767a1461 JCLOUDS-1333: JCLOUDS-1334: JCLOUDS-1470: Require Java 8 and Guava 22
This allows compatibility with Guava 29.  Also unwind some older
workarounds.
2020-06-25 08:11:30 +09:00
Jean-Noël Rouvignac 20b1394f36 JCLOUDS-1491 Jclouds uses a deprecated version of Guava to support Azure storage
DnsNameValidator.java uses a deprecated guava APIs in code that is used
 to support Azure cloud storage. When forcing the use of more recent guava
 versions, the code fails with NoSuchFieldError.

However, CharMatcher.JAVA_LETTER_OR_DIGIT has been removed in guava 26.0,
 and CharMatcher.javaLetterOrDigit() should be used instead since guava
 19.0. Note that CharMatcher.javaLetterOrDigit() was immediately
 deprecated in Guava 26.0, and java.lang.Character.isLetterOrDigit(int)
 should be used instead.

This commit replaces the use of this deprecated API by
 java.lang.Character.isLetterOrDigit(int).
 It is no worse than the previous code.

(If I understand correctly, updating the guava version is a challenge due to
dependencies on Apache Karaf anyway)
2020-06-03 09:14:41 +09:00
Andrew Gaul 6e6f8ebf77 JCLOUDS-912: JCLOUDS-1547: GCS InputStream single-part upload
Previously this provider worked around a RestAnnotationProcessor quirk
by using multi-part uploads for InputStream payloads.  Instead work
around the quirk another way which allows a single-part upload.  This
allows inclusion of the Content-MD5 header during object creation.
Backfill tests with both ByteSource and InputStream inputs.
2020-05-31 17:48:31 +09:00
Andrew Gaul 08a16c95fb JCLOUDS-1546: Support GCS Archive storage class
Also change portable abstraction mapping for Tier.ARCHIVE.
2020-05-09 12:01:41 +09:00
roded d220b245d7 JCLOUDS-1543: remove unused imports from FetchBlobMetadataTest.java (#70)
Co-authored-by: Roded Bahat <roded.bahat@model9.io>
2020-04-22 10:32:54 +02:00
Fritz Elfert 286fe5cba0
Fix toolchain setup on jenkins (#68) 2020-04-22 09:56:36 +02:00
Roded Bahat 5ac92111c4 JCLOUDS-1543: change FetchBlobMetadata to retain original blob order 2020-04-17 19:24:15 +09:00
Fritz Elfert 89e1713c7c
Enable docs in travis build (#69) 2020-04-17 08:26:50 +02:00
Fritz Elfert 413ee30720
Fix javadoc generation on JDK >= 8 (#67) 2020-04-16 13:23:31 +02:00
Timur Alperovich 8d0e9dc962 Hash the content for fs MPU ETag if no xattr.
If there is no extended attribute support in the file system, the blobs
will not have their associated ETags available. In that case, the file
system blob store should rehash the content while producing the combined
blob and return the expected S3-style ETag.
2020-04-04 18:42:59 +09:00
ikky888 69ca45720d
JCLOUDS-1541: Add Middle East (Bahrain) region to the AWS EC2 and S3 providers list 2020-04-04 10:48:01 +09:00
Xavier BOURGOUIN d6702e5ee0 Fix BlobMetadata null size when using ApacheHCHttp module
JClouds is apparently exclusively using the Payload object from the HTTP
response to fill in the size of the BlobMetadata (when calling
blobStore.blobMetadata(...) ) - adapt this driver accordingly otherwise
we systematically get null size BlobMetadata out of it.
2020-03-08 22:15:43 +09:00
Colm O hEigeartaigh 99ef891e76 JCLOUDS-1540 - Update Snakeyaml 2020-03-03 17:41:33 +09:00
majaseremet ca5190636a
JCLOUDS-1533 - Fix upload with SAS token when blob name contains slash (#61) 2020-02-17 15:28:33 +01:00
Ian Springer 3db2939885 JCLOUDS-1538: fix typo in rfc1123SimpleDateFormat (#60) 2020-01-24 09:34:50 -05:00
Colm O hEigeartaigh b96158e6ed JCLOUDS-1532 - Update SSHJ + JSCH (#57) 2019-12-03 17:17:06 +01:00
Markus Alexander Kuppe 11d05ac9bf JCLOUDS-1529: Do not export org.jclouds.json.gson.internal (#56)
Do not export packages matching org.jclouds.json.gson.internal*
because it causes a use constraint violation with OSGi between
jclouds-core and jclouds-gson.

https://issues.apache.org/jira/browse/JCLOUDS-1529
2019-11-28 10:34:16 +01:00
Colm O hEigeartaigh 3a338da682 JCLOUDS-1528 - Use TLS instead of SSL in SSLContext.getInstance (#55) 2019-11-25 14:54:48 +01:00
Colm O hEigeartaigh 18da28af9f Downgrade BouncyCastle to 1.60, which works with Oracle JDK 1.7 (#54) 2019-11-18 14:54:30 +01:00
Colm O hEigeartaigh 30830bfa4e JCLOUDS-1526 - Update BouncyCastle dependency (#53) 2019-11-15 16:03:40 +01:00
Colm O hEigeartaigh 5f08ee2f1a JCLOUDS-1525 - Update xmlbuilder dependency (#52) 2019-11-15 16:03:28 +01:00
Andrew Gaul febc9b73df Fix TOCTOU issue when getting blob during removal
Fixes gaul/s3proxy#318.
2019-11-09 12:08:40 +09:00
Xavier BOURGOUIN 316b74db94 Fix infinite socket timeout when using ApacheHCHttpCommandExecutorServiceModule (#51)
Because the Java type BasicHttpParams wasn't matching the one returned
by provider "newBasicHttpParams" (which is returning a HttpParams type), it was configuring the Apache HTTP client with default settings (including connection and socket timeout), ignoring the ones of JClouds which were must safer in this regard (JCloudsones are currently 60sec for both, while Apache HC ones are -1, so infinite). It also prevented to override those default/infinite timeout settings with JClouds ones.
2019-10-23 11:44:58 +02:00
Ignasi Barrera f5b29c7028 Next development version 2.3.0-SNAPSHOT 2019-10-21 10:32:43 +02:00
Ignasi Barrera 7221844fac Apache jclouds 2.2.0-rc1 release 2019-10-21 10:32:43 +02:00
roded 6d389b0d86 JCLOUDS-1520: change UntrustedSSLContextSupplier to return the same SSLContext (#49)
Using the same SSLContext prevents consistent cache misses on the JVM's KeepAliveCache
when attempting to reuse TLS connections.
2019-10-16 09:48:47 +02:00
Ignasi Barrera 0f93525814
Add TravisCI file for pull requests (#50)
* Add TravisCI file for pull requests

* Fix env var config

* Oracle JDK not available in the default travis env

* Add license header

* Do not use Maven toolchains

* Less verbose build

* Further reduce output
2019-10-14 10:25:06 +02:00
Ignasi Barrera a2cbdd3385 Etag is not supported in OSX 2019-10-13 10:32:50 +02:00
Xavier BOURGOUIN c445547ea8 Fix null content-length header on HEAD requests
https://issues.apache.org/jira/projects/JCLOUDS/issues/JCLOUDS-1505
2019-10-10 09:59:38 -07:00