The previous site plugin utilizes several plugins. In about half of the cases, the respective version of a plugin as described in the outside plugin section. The following list enumerates those plugins and my reasoning to remove them safely from the section plugins.
- maven-project-info-reports-plugin (remove - because no reports get generated)
- maven-javadoc-plugin (stays - even when a profile doc exists, which provides a dedicated setup.) We have to discuss the removal to avoid any side effects.
- emma-maven-plugin (stays - my impression is that it is a duplicated declaration)
- maven-jxr-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- maven-surefire-report-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- apache-rat-plugin (stays - decision is straightforward. We have to discuss a restructure because the plugin gets mentioned three times; I guess one can get removed.
- maven-pmd-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- spotbugs-maven-plugin (stays - beacuse of its extended configuration section - discuss a removal)
The maven site-plugin uses the declaration "site" in the section "distribution management," which points to a git repository named "jclouds-maven-site." This git repository got never relocated during the apache migration; also, it is ancient. My impression is that the site deployment is outdated and got used in the early days of JClouds.
The JClouds project module upgrades the okhttp server library and related dependencies such as mockwebserver from 2.2.0 to 3.14.9. The vendor switched the groupId declaration from com.squareup.okhttp to com.squareup.okhttp3.
Adjust imports and api calls for newer okhttp vers
All OkHttp components get shipped with any OSGi metadata. Tracking the progress of that issue from version 5 onwards, OkHttp finally provides OSGi support. Because JClouds OkHttp dependencies got renewed from 2.2.0 to the latest 3 (3.14.9), the respective bundles from Apache Servicemix got used in a deployment scenario. On the other hand, the problem with Servicemix is that there is no full match of published OkHttp dependencies to the respective bundle equivalent.
https://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.okhttp
My test shows that none of the version 3 bundles work because of a missing package dependency "javax.annotation.meta". Sure higher version of 9 strips the javax.annotation module entirely, but the meta package is not there in JDK 8 either. As you can see from the link above, only two versions of 3.14 got published in the service mix; it is not clear to which version the bundle equivalents relate. From my understanding, it is much more important to use identical versions of OkHttp dependencies used in JClouds.
To utilize the original OKHttp libraries, consumer projects have to ensure those libraries get loaded using OSGi helper tools such as the wrap protocol provided by Apache Karaf or similar.
To reduce OSGi noise, I removed the OSGi import range of okio dependency entirely. A declaration like that does not make sense; otherwise, a set of Okhttp ranges must also be defined.
Note: The wildcard (*) declaration provides a way out, which allows any version of a package used in JClouds but itself does not give any OSGi metadata. Hopefully, I do not confuse you too much.
- Mock-Webserver v2.2.0 - Vendor, defined dependency bcprov-jdk15on > Previously, the dependency got excluded; otherwise, it conflicted with the existing bouncy castle version used in JClouds.
- Mock-Webserver v3.14.9 - The vendor does not define a bouncy castle dependency anymore; instead, it moved to the okhttp-tls module.
- Introduce the okhttp-tls module for tests in the core module. The following APIs (Docker, Elastic-Stack) and providers Softlayer extend their respective MockTest from BaseMockWebServerTest. The mock base test is located in the test jar of JClouds core module.
- Due to conflicting bouncy castle classes in the classpath, those which get loaded from the okhttp-tls module, and those defined in JClouds bouncy castle module, the bouncy castle dependency of okhttp-tls has to get skipped for the two APIs and providers mentioned.
Side note:
The JClouds GAE driver module also requires the new okhttp-tls dependency because of the following chain of inheritance.
Different from the situation above, the bouncy castle classes of the okhttp-tls got not excluded.
GaeHttpCommandExecutorServiceIntegrationTest -> BaseHttpCommandExecutorServiceIntegrationTest -> BaseMockWebServerTest
The reason for this is unknown to me.
In the present feature, the two properties
- guava.osgi.import and
- okhttp.version
have not been attached to the build process (Bnd plugin), but have been recorded as comments only.
Before the resolution method is considered, the problem of Guava and its close coupling with Guice must be examined.
The Guice project possesses, differently to GSON no explicit configuration, how its dependencies are regarded in an OSGi execution environment.
Guice / Guava
- 4.1.0 / 19.0
- 4.2.0 / [23.6,24)
- 4.2.1 / [25.1,26)
- 4.2.3 / [27.1,28) => used
- 5.0.0.BETA-1 / [27.1,28)
The version of Guice built into JClouds is 4.2.3. Increasing to this version number was not the subject of the current branch.
To avoid backward compatibility problems between Guice and Guava (e.g. Guice 4.2.3 -> Guava 24) this means that the version of Guava integrated in JClouds must be at least version 27.1. Guice puts this in its dependencies.
In my opinion, consumers (JClouds, as well as consumers of JClouds) should take this minimum barrier into account.
Resolution
To understand the resolution, please consider the following previously unmerged feature.
https://github.com/apache/jclouds/pull/86
In order to add a Guava verison range to the bundles rolled out by JClouds, the following import rule is added in the BND Configuration of the Project Module.
Import Package: \
com.google.common.*;version="[27.1,30.1.0)", \
okio.*;version="[1.2.0,1.3)", \
*
In the output of each JClouds module that uses Guava (and there are some) the configuration to the range version is observed. The bnd built into the JClouds project modules acts as a base of other bnd configurations, and should also apply in the Lab and Co repositories. Adjusted artefacts only become valid when a release is available or the build of these repositories is triggered again.
Output - Core Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.cache;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
...
com.google.gson;version="[2.8,3)",
...
Output - Blobstore Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
com.google.common.hash;version="[27.1,30.1.0)",
...
com.google.inject;version="[1.4,2)",
...
Oki version range
The okhttp dependency requires okio version 1.2. The ok* libraries respectively the dependencies used in these libraries do not have OSGi instructions. For this reason, the previously integrated configuration is used.
Local tests show that the import directive can also be set globally, see above. An explicit specification in the bnd file of the JClouds okHTTP module is not useful because this JClouds module also uses Guava.
Output - OkHTTP Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
okio;version="[1.2.0,1.3)",
com.google.inject;version="[1.4,2)",
...
Note:
In the following feature, the version of the "ok" framework has been significantly increased. Extensive adjustments are also performed.
At this point, it should be noted that the import policy in the okio will probably have to be removed again because standalone OSGi metadata may be provided.
Increase of OSGi dependencies core and compendium (now cmpn) from 4.2 to 6.0.
Previously it was possible to run JClouds in OSGi environments from version 4.
An essential aspect to use JClouds in an OSGi environment requires so-called feature sets. These can be generated manually or automatically - see JClouds-Karaf project. Since there have been significant changes in the structure and behaviour of Karaf in the meantime, an adaptation is appropriate.
Breaking change - probably not, as no other APIs of core and compendium are used than up to now.
OSGi - V4.2 - Karaf 2.2.x - to 2.2.9 (Current Status - not active)
OSGi - V6.0 - Karaf 4.1.x - 4.2 (Current Status - active)
In the source code of JClouds cloudsigma as a keyword gets not used anywhere. In the JClouds-Lab project, cloudSigma is available both as an API and multiple provider modules. Because the project module from JClouds serves as a parent module for all JClouds-Labs modules, it seems reasonable to maintain those rules in the JClouds project module.
After inspecting the JClouds Lab source code, group artifact combinations of
<groupId>org.apache.jclouds.api</groupId>
<artifactId>cloudsigma</artifactId>
..., respectively
<groupId>org.apache.jclouds.provider</groupId><artifactId>cloudsigma-lvs</artifactId>
..., are not available.
Cloudsigma in Lab uses the following group artifact combinations, all with a "2" prefix.
<groupId>org.apache.jclouds.labs</groupId
<artifactId>cloudsigma (2)</artifactId>
<groupId>org.apache.jclouds.labs</groupId>
<artifactId>cloudsigma (2) -hnl</artifactId>
Loading those bundles into an OSGi runtime, a runtime collision happens because the API exports the identical packages as the provider modules.
Although this was the case in a previous version, it has since been corrected.
e7885359a7
This commit removes the exception handling.
Remove javax libraries inject and annotation. In JDK 8 those libraries are provided by the JDK. Younger JDK versions > 11 exclude them again to make the JDK leaner. Supporting younger JDK versions means integrating more younger libraries maintained by the Jakarta project.
- Increase version of Guava dependency from 22.0 to 27.1-jre.
Note: In an OSGi runtime, Guice version 4.2.3 expects Guava with an exact version of 27.1-jre. For now, higher versions of Guava are not possible.
- Separate version numbers from dependencies and plugins
Following is a description of the purpose of the feature.
New contributors find it difficult to understand the function and intent of different plugins integrated into JClouds.
The primary goal is to simplify and streamline the overall setup.
On closer examination of the integrated plugins you will notice that,
- different plugins are used for the same or a similar goal
- the development activity of used plugins has been terminated
- modified integration requirements through Maven. The development of used plugins is progressing. Versions of the plugins integrated into JClouds are not up to date. Newer plugin versions require an adjustment to how integration into a project is done.
Basically the setup of the plugins used in the Maven project POM was done a long time ago. The project POM file of the JClouds project serves as a parent for all modules in JClouds.
The first step is to consolidate the used plugins.
The same aspects apply to the build profiles used in the project. On closer look these must be considered obsolete. A deconstruction of declared build profiles is aimed at.
In order to support newer runtime environments, for example JDK11 and above, we aim to integrate build profiles.
In the respective build profile the libraries which are no longer contained in the respective target JDK but needed in JClouds are to be integrated.
In the course of this rebuild, a property-based way of declaring the version number of dependencies and plugins will be implemented.
* animal sniffer should be on java18, just like `<jdk.version>`
* Only use XMLBuilder's elem() and text() methods to have similar looking code
* Remove unnecessary call to XMLBuilder's up() because the returned value is never used
* Simplify code
* Deduplicate code
* Make the code more explicit by returning the rootBuilder
This project, the aws-lab version of Apache JClouds, share the exact build instructions as the primary Apache JClouds project with all its modules.
Apache JClouds is shifting its strategy in handling OSGi configuration. Instead of using the Maven Bundle Plugin, a wrapper of the BND plugin, the BND plugin gets used directly.
- Remove the OSGi configuration from each module. The configuration gets served to the BND through dedicated configuration / bnd files.
- Onboard bnd-configuration files, one per module.
Ignore bnd files in rat plugin
The default scope of imports is "compile." There is no reason to add this declaration to the GSON import declaration. Also, scope declarations are required to be defined in the main dependency declaration section, not in dependency management, where dependency information gets managed, which is valid across all of its children, e.g., version numbers.
Lowering the GSON version from 2.8.6 (latest) to 2.8.5, to make sure no side effects get introduced on a JDK level - for reference https://github.com/google/gson/issues/1601. Another feature is in its making to introduce a conditional build profile to support JDK 11 and above; here, a switch to the latest version of GSON gets provided.
Note: Previously, the maven jar plugin contained a configuration embedded in each module's generated manifest files. The configuration got relocated to the project/bnd.bnd file in a previous commit, and gets handled through the bnd plugin.
In the last commit (last section of squashed commit), the GSON library was integrated into the JClouds core module using maven-bundle plugins include resource instruction. Building OSGi instruction variables from the respective modules show a weakness when resources such as script builder shell scripts are required to be integrated into the bundle but not provide a dedicated variable declaration for the resource section.
The following commit demonstrates a change in strategy in declaration and integration of OSGi metadata.
- Replace old bundle-plugin with newest bnd-plugin (bundle-plugin uses bnd-plugin internally)
- Move OSGi metadata declarations from a maven variable passing strategy into dedicated bnd.bnd files
+ Cleaner pom files, no bundle packaging
+ Intellisense / Autocomplete support for .bnd files in terms of package exports etc.
For demonstration, the overall OSGi adjustments are limited to project, core, script builder, compute, blob store, and load balancer because most custom OSGi metadata is defined here.
Note: Other modules are currently disabled from build because some feedback is needed first.
Make GSON integration work.
To understand the changes, see the core modules' bnd file. GSON internal packages also define a version. Both already exported and new export declarations are fused. The global JClouds core module exports defined the entire set of GSON packages available.
Some minor modifications were made in the module project; replace maven jar plugin with a minified version of the declaration, outsourced in projects bnd file.
Replace substituted GSON package names with those provided from the vendor.
Reduce OSGi-metadata declaration of core-module because the artificial package org.jclouds.json.gson.internal was removed.
Remove the Gson module its children Gson bundle, and Gson shaded.
Remove duplication conflict and check-style rules due to the removal of the internal Gson module.
Add maven repository where a custom version of the Gson library gets hosted, which exports all packages.
Remove particular repository
Remove the declaration of the repository that serves a custom build GSON version. The build uses GSON in its original form of the vendor, which gets distributed through the standard distribution channel. The identifiers for group, artifact, and version correspond to the latest stable release of GSON.
Integrate GSON library in Clouds Core Bundle
The change contained in the commit puts the GSON library into the classpath of the JClouds core module.
After several tests with Karaf and Karaf JClouds, especially if the Maven identifier matches the original GSON library, there are only a limited number of ways to keep the deployment effort low.
Specifically, Karaf has a set of predefined Maven repositories that can be easily customized. The order in which a particular repository is resolved into the customized GSON library is more difficult. In normal OSGi applications, which do not have such a management function, I imagine this configuration to be more complicated. Sure, a unique identifier would help, but then we are back to step 1.
Although I honestly don't like to see this kind of approach in a codebase I'm working with, there are not many alternatives to the main aspect of deployment mentioned above.
Maybe the approach can still ease the problem in the short term. In a further mid-term step, however, this problem must be addressed in general.
With Java 11, an illegal reflective access is output for the google cloud storage blobstore.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jclouds.reflect.Reflection2$1 (file:/.../jclouds-core.jar) to constructor java.lang.String(char[],int,int,java.lang.Void)
WARNING: Please consider reporting this to the maintainers of org.jclouds.reflect.Reflection2$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Indeed, JClouds calls `setAccessible(true)` on the package protected constructor
`java.lang.String(char[],int,int,java.lang.Void)`.
Investigating the code, it turns out it is looking for constructors annotated with any of:
* java.beans.ConstructorProperties
* org.jclouds.json.SerializedNames
* com.google.inject.Inject
But `String` being defined in `java.base` module, it is impossible
that it will be annotated with any of these annotation.
This commit is complementary to JClouds commit db4e4af931 .
Reflection2.java:
Do not call `setAccessible(true)` on core java constructors and methods.
For reference, here is the stacktrace of this illegal access warning:
java.lang.String.<init>(String.java:3208)
java.lang.String.<init>(String.java:251)
java.util.StringJoiner.compactElts(StringJoiner.java:250)
java.util.StringJoiner.toString(StringJoiner.java:173)
jdk.internal.module.IllegalAccessLogger.loudWarning(IllegalAccessLogger.java:339)
jdk.internal.module.IllegalAccessLogger.log(IllegalAccessLogger.java:288)
jdk.internal.module.IllegalAccessLogger.log(IllegalAccessLogger.java:261)
jdk.internal.module.IllegalAccessLogger.logIfOpenedForIllegalAccess(IllegalAccessLogger.java:226)
java.lang.reflect.AccessibleObject.logIfOpenedForIllegalAccess(AccessibleObject.java:366)
java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:325)
java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:280)
java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:189)
java.lang.reflect.Constructor.setAccessible(Constructor.java:182)
org.jclouds.reflect.Reflection2$1.load(Reflection2$1.java:157)
org.jclouds.reflect.Reflection2$1.load(Reflection2$1.java:153)
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache$LoadingValueReference.java:3529)
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache$Segment.java:2278)
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache$Segment.java:2155)
com.google.common.cache.LocalCache$Segment.get(LocalCache$Segment.java:2045)
com.google.common.cache.LocalCache.get(LocalCache.java:3953)
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache$LocalLoadingCache.java:4960)
org.jclouds.reflect.Reflection2.get(Reflection2.java:346)
org.jclouds.reflect.Reflection2.constructors(Reflection2.java:100)
org.jclouds.json.internal.NamingStrategies$AnnotationConstructorNamingStrategy.getDeserializer(NamingStrategies$AnnotationConstructorNamingStrategy.java:271)
org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory.create(DeserializationConstructorAndReflectiveTypeAdapterFactory.java:125)
com.google.gson.Gson.getAdapter(Gson.java:458)
org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.createBoundField(ReflectiveTypeAdapterFactory.java:117)
org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:166)
org.jclouds.json.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:102)
com.google.gson.Gson.getAdapter(Gson.java:458)
com.google.gson.Gson.toJson(Gson.java:696)
com.google.gson.Gson.toJson(Gson.java:683)
com.google.gson.Gson.toJson(Gson.java:638)
com.google.gson.Gson.toJson(Gson.java:618)
org.jclouds.json.internal.GsonWrapper.toJson(GsonWrapper.java:65)
org.jclouds.oauth.v2.functions.ClaimsToAssertion.apply(ClaimsToAssertion.java:59)
org.jclouds.oauth.v2.functions.ClaimsToAssertion.apply(ClaimsToAssertion.java:43)
org.jclouds.rest.internal.RestAnnotationProcessor.getParamValue(RestAnnotationProcessor.java:829)
org.jclouds.rest.internal.RestAnnotationProcessor.getFormParamKeyValues(RestAnnotationProcessor.java:847)
org.jclouds.rest.internal.RestAnnotationProcessor.addFormParams(RestAnnotationProcessor.java:435)
org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:258)
org.jclouds.rest.internal.RestAnnotationProcessor.apply(RestAnnotationProcessor.java:137)
org.jclouds.rest.internal.InvokeHttpMethod.toCommand(InvokeHttpMethod.java:189)
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:85)
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
com.sun.proxy.$Proxy49.authorize(Unknown Source)
org.jclouds.oauth.v2.filters.JWTBearerTokenFlow$AuthorizeToken.load(JWTBearerTokenFlow$AuthorizeToken.java:84)
org.jclouds.oauth.v2.filters.JWTBearerTokenFlow$AuthorizeToken.load(JWTBearerTokenFlow$AuthorizeToken.java:68)
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache$LoadingValueReference.java:3529)
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache$Segment.java:2278)
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache$Segment.java:2155)
com.google.common.cache.LocalCache$Segment.get(LocalCache$Segment.java:2045)
com.google.common.cache.LocalCache.get(LocalCache.java:3953)
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3976)
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache$LocalLoadingCache.java:4960)
com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache$LocalLoadingCache.java:4966)
org.jclouds.oauth.v2.filters.JWTBearerTokenFlow.filter(JWTBearerTokenFlow.java:99)
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:90)
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
com.sun.proxy.$Proxy54.simpleUpload(Unknown Source)
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.uploadMultipartPart(GoogleCloudStorageBlobStore.java:422)
org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:415)
org.jclouds.blobstore.internal.BaseBlobStore$BlobUploader.call(BaseBlobStore.java:402)
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor$Worker.java:628)
java.lang.Thread.run(Thread.java:834)
DnsNameValidator.java uses a deprecated guava APIs in code that is used
to support Azure cloud storage. When forcing the use of more recent guava
versions, the code fails with NoSuchFieldError.
However, CharMatcher.JAVA_LETTER_OR_DIGIT has been removed in guava 26.0,
and CharMatcher.javaLetterOrDigit() should be used instead since guava
19.0. Note that CharMatcher.javaLetterOrDigit() was immediately
deprecated in Guava 26.0, and java.lang.Character.isLetterOrDigit(int)
should be used instead.
This commit replaces the use of this deprecated API by
java.lang.Character.isLetterOrDigit(int).
It is no worse than the previous code.
(If I understand correctly, updating the guava version is a challenge due to
dependencies on Apache Karaf anyway)
Previously this provider worked around a RestAnnotationProcessor quirk
by using multi-part uploads for InputStream payloads. Instead work
around the quirk another way which allows a single-part upload. This
allows inclusion of the Content-MD5 header during object creation.
Backfill tests with both ByteSource and InputStream inputs.
gurkerl83
korlov42
Andrew Gaul
Jean-Noël Rouvignac
davidsloan
roded
Tamas Cservenak
Sam Ottenhoff
Jean-Noël Rouvignac
roded
Fritz Elfert
Roded Bahat