The issue happens if a cloud provider returns lowercase metadata headers, for example: "x-object-meta-apiversion" instead of "X-Object-Meta-ApiVersion"
In that case, BlobStore.blobMetadata(CONTAINER, PATH).getUserMetadata()
incorrectly returns an empty map.
This happens because the code is looking for the exact String "-Meta-" (case-sensitive).
This checkin allows to handle metadata headers of any case, and also adds a unit test for that situation.
Currently, jclouds assumes that the ListBuckets response follows a
specific order: the <Owner> tag, followed by the <Buckets> tag. Within
the <Owner> tag, jclouds further assumes that the <ID> must occur before
the <DisplayName> tag. If the XML body does not adhere to this order,
the parser throws a NullPointerException.
DigitalOcean spaces does not adhere to this order and returns the
<DisplayName> tag before the <ID> tag. The patch changes the parser to
not depend on the order of the tags.
XMLBuilder is a very nice fluent API for building XML documents.
However, it is only used by a tiny portion of jclouds: the s3 api. The
use of the XMLBuilder class requires a dependency to
com.jamesmurty.utils:java-xmlbuilder jar and a transitive
dependency to the net.iharder:base64 jar (superseded by
java.util.Base64 in java 8). They are 18kb each approximately and they
not OSGi compatible. They are not huge, but they represent more API
surface and more things to change when trying to use jclouds in an OSGi
context (they need to be replaced by OSGi compatible bundles like
org.apache.servicemix.bundles.java-xmlbuilder).
This commit replaces the use of XMLBuilder by a direct use of the
javax.xml and org.w3c.dom APIs.
I hope retesting will be minimal, and most of this code is covered by
unit tests.
Previously, the plugin "emma" was integrated/used for maven site deployment. The following commits removed site deployment entirely.
For reference
- e02503e1cb
- 9158ab0b0e
The project requires at least JDK version 8. The specification to disable the liniting process in the JavaDoc process as of JDK version 8 is thus doubly specified and fulfilled by default; an explicit specification is no longer required. Version 3 of the JavaDoc plugin removes the "additional param" configuration attribute. The "doclint=false" configuration attribute disables the lint operation of the plugin.
Summary:
- Remove the profile "disable-doclint"
- Bump version of JavaDoc plugin to the latest > 3
- Remove the JavaDoc plugin's declaration from the build plugins used for default builds (building without a profile)
- Bump version of source plugin to latest > 3
- Remove unnecessary profile "doc" from the jclouds pom artifact
Remove distribution profiles from the JClouds parent, main, and core modules. Each profile defines assembly descriptors, but none of those files are provided by the project anymore.
The maven assembly plugin relies on those descriptors to dictate its execution.
The estimation is that individual maven artifacts, one per module, get deployed instead of a compressed deployment unit.
In that case, it is enough to use mavens deploy command in combination with distribution management configuration pushing individual jars to the artifact repositories.
According to the Jenkins configuration, only the maven profiles src and doc are necessary to execute in the build process. This commit removes the dedicated jenkins profile.
The site profile defines various enforcer execution rules, banned-dependencies, java (java and maven version). Those declarations in combination with the standard maven properties for the JDK version seem misleading and duplicated. Also, there is no indicator that the site profile is ever defined for a maven build.
In the last three commits, the site reporting feature got gradually decommissioned. This commit removes any remaining fragments in this regard. Remove the section distribution management from the profile clouds-project. The comment about incorrect links generated might have resulted from multiple invocations of the check style plugin. Previously the check style plugin got utilized at multiple sections (reporting section and profiles) and probably overlapping maven lifecycles.
In the last two commits, the site reporting feature got gradually decommissioned. The maven plugin project-info-reports got removed in the previous commit, which gets controlled by the reporting section. Because this workflow is out of use, this commit removes the reporting section.
The previous site plugin utilizes several plugins. In about half of the cases, the respective version of a plugin as described in the outside plugin section. The following list enumerates those plugins and my reasoning to remove them safely from the section plugins.
- maven-project-info-reports-plugin (remove - because no reports get generated)
- maven-javadoc-plugin (stays - even when a profile doc exists, which provides a dedicated setup.) We have to discuss the removal to avoid any side effects.
- emma-maven-plugin (stays - my impression is that it is a duplicated declaration)
- maven-jxr-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- maven-surefire-report-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- apache-rat-plugin (stays - decision is straightforward. We have to discuss a restructure because the plugin gets mentioned three times; I guess one can get removed.
- maven-pmd-plugin (remove - only version declaration, with no further config, not utilized anywhere)
- spotbugs-maven-plugin (stays - beacuse of its extended configuration section - discuss a removal)
The maven site-plugin uses the declaration "site" in the section "distribution management," which points to a git repository named "jclouds-maven-site." This git repository got never relocated during the apache migration; also, it is ancient. My impression is that the site deployment is outdated and got used in the early days of JClouds.
All OkHttp components get shipped with any OSGi metadata. Tracking the progress of that issue from version 5 onwards, OkHttp finally provides OSGi support. Because JClouds OkHttp dependencies got renewed from 2.2.0 to the latest 3 (3.14.9), the respective bundles from Apache Servicemix got used in a deployment scenario. On the other hand, the problem with Servicemix is that there is no full match of published OkHttp dependencies to the respective bundle equivalent.
https://mvnrepository.com/artifact/org.apache.servicemix.bundles/org.apache.servicemix.bundles.okhttp
My test shows that none of the version 3 bundles work because of a missing package dependency "javax.annotation.meta". Sure higher version of 9 strips the javax.annotation module entirely, but the meta package is not there in JDK 8 either. As you can see from the link above, only two versions of 3.14 got published in the service mix; it is not clear to which version the bundle equivalents relate. From my understanding, it is much more important to use identical versions of OkHttp dependencies used in JClouds.
To utilize the original OKHttp libraries, consumer projects have to ensure those libraries get loaded using OSGi helper tools such as the wrap protocol provided by Apache Karaf or similar.
To reduce OSGi noise, I removed the OSGi import range of okio dependency entirely. A declaration like that does not make sense; otherwise, a set of Okhttp ranges must also be defined.
Note: The wildcard (*) declaration provides a way out, which allows any version of a package used in JClouds but itself does not give any OSGi metadata. Hopefully, I do not confuse you too much.
- Mock-Webserver v2.2.0 - Vendor, defined dependency bcprov-jdk15on > Previously, the dependency got excluded; otherwise, it conflicted with the existing bouncy castle version used in JClouds.
- Mock-Webserver v3.14.9 - The vendor does not define a bouncy castle dependency anymore; instead, it moved to the okhttp-tls module.
- Introduce the okhttp-tls module for tests in the core module. The following APIs (Docker, Elastic-Stack) and providers Softlayer extend their respective MockTest from BaseMockWebServerTest. The mock base test is located in the test jar of JClouds core module.
- Due to conflicting bouncy castle classes in the classpath, those which get loaded from the okhttp-tls module, and those defined in JClouds bouncy castle module, the bouncy castle dependency of okhttp-tls has to get skipped for the two APIs and providers mentioned.
Side note:
The JClouds GAE driver module also requires the new okhttp-tls dependency because of the following chain of inheritance.
Different from the situation above, the bouncy castle classes of the okhttp-tls got not excluded.
GaeHttpCommandExecutorServiceIntegrationTest -> BaseHttpCommandExecutorServiceIntegrationTest -> BaseMockWebServerTest
The reason for this is unknown to me.
In the present feature, the two properties
- guava.osgi.import and
- okhttp.version
have not been attached to the build process (Bnd plugin), but have been recorded as comments only.
Before the resolution method is considered, the problem of Guava and its close coupling with Guice must be examined.
The Guice project possesses, differently to GSON no explicit configuration, how its dependencies are regarded in an OSGi execution environment.
Guice / Guava
- 4.1.0 / 19.0
- 4.2.0 / [23.6,24)
- 4.2.1 / [25.1,26)
- 4.2.3 / [27.1,28) => used
- 5.0.0.BETA-1 / [27.1,28)
The version of Guice built into JClouds is 4.2.3. Increasing to this version number was not the subject of the current branch.
To avoid backward compatibility problems between Guice and Guava (e.g. Guice 4.2.3 -> Guava 24) this means that the version of Guava integrated in JClouds must be at least version 27.1. Guice puts this in its dependencies.
In my opinion, consumers (JClouds, as well as consumers of JClouds) should take this minimum barrier into account.
Resolution
To understand the resolution, please consider the following previously unmerged feature.
https://github.com/apache/jclouds/pull/86
In order to add a Guava verison range to the bundles rolled out by JClouds, the following import rule is added in the BND Configuration of the Project Module.
Import Package: \
com.google.common.*;version="[27.1,30.1.0)", \
okio.*;version="[1.2.0,1.3)", \
*
In the output of each JClouds module that uses Guava (and there are some) the configuration to the range version is observed. The bnd built into the JClouds project modules acts as a base of other bnd configurations, and should also apply in the Lab and Co repositories. Adjusted artefacts only become valid when a release is available or the build of these repositories is triggered again.
Output - Core Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.cache;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
...
com.google.gson;version="[2.8,3)",
...
Output - Blobstore Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
com.google.common.hash;version="[27.1,30.1.0)",
...
com.google.inject;version="[1.4,2)",
...
Oki version range
The okhttp dependency requires okio version 1.2. The ok* libraries respectively the dependencies used in these libraries do not have OSGi instructions. For this reason, the previously integrated configuration is used.
Local tests show that the import directive can also be set globally, see above. An explicit specification in the bnd file of the JClouds okHTTP module is not useful because this JClouds module also uses Guava.
Output - OkHTTP Module
Import package:
com.google.common.base;version="[27.1,30.1.0)",
com.google.common.collect;version="[27.1,30.1.0)",
okio;version="[1.2.0,1.3)",
com.google.inject;version="[1.4,2)",
...
Note:
In the following feature, the version of the "ok" framework has been significantly increased. Extensive adjustments are also performed.
At this point, it should be noted that the import policy in the okio will probably have to be removed again because standalone OSGi metadata may be provided.
Increase of OSGi dependencies core and compendium (now cmpn) from 4.2 to 6.0.
Previously it was possible to run JClouds in OSGi environments from version 4.
An essential aspect to use JClouds in an OSGi environment requires so-called feature sets. These can be generated manually or automatically - see JClouds-Karaf project. Since there have been significant changes in the structure and behaviour of Karaf in the meantime, an adaptation is appropriate.
Breaking change - probably not, as no other APIs of core and compendium are used than up to now.
OSGi - V4.2 - Karaf 2.2.x - to 2.2.9 (Current Status - not active)
OSGi - V6.0 - Karaf 4.1.x - 4.2 (Current Status - active)
In the source code of JClouds cloudsigma as a keyword gets not used anywhere. In the JClouds-Lab project, cloudSigma is available both as an API and multiple provider modules. Because the project module from JClouds serves as a parent module for all JClouds-Labs modules, it seems reasonable to maintain those rules in the JClouds project module.
After inspecting the JClouds Lab source code, group artifact combinations of
<groupId>org.apache.jclouds.api</groupId>
<artifactId>cloudsigma</artifactId>
..., respectively
<groupId>org.apache.jclouds.provider</groupId><artifactId>cloudsigma-lvs</artifactId>
..., are not available.
Cloudsigma in Lab uses the following group artifact combinations, all with a "2" prefix.
<groupId>org.apache.jclouds.labs</groupId
<artifactId>cloudsigma (2)</artifactId>
<groupId>org.apache.jclouds.labs</groupId>
<artifactId>cloudsigma (2) -hnl</artifactId>
Loading those bundles into an OSGi runtime, a runtime collision happens because the API exports the identical packages as the provider modules.
Although this was the case in a previous version, it has since been corrected.
e7885359a7
This commit removes the exception handling.
Remove javax libraries inject and annotation. In JDK 8 those libraries are provided by the JDK. Younger JDK versions > 11 exclude them again to make the JDK leaner. Supporting younger JDK versions means integrating more younger libraries maintained by the Jakarta project.
- Increase version of Guava dependency from 22.0 to 27.1-jre.
Note: In an OSGi runtime, Guice version 4.2.3 expects Guava with an exact version of 27.1-jre. For now, higher versions of Guava are not possible.
- Separate version numbers from dependencies and plugins
Following is a description of the purpose of the feature.
New contributors find it difficult to understand the function and intent of different plugins integrated into JClouds.
The primary goal is to simplify and streamline the overall setup.
On closer examination of the integrated plugins you will notice that,
- different plugins are used for the same or a similar goal
- the development activity of used plugins has been terminated
- modified integration requirements through Maven. The development of used plugins is progressing. Versions of the plugins integrated into JClouds are not up to date. Newer plugin versions require an adjustment to how integration into a project is done.
Basically the setup of the plugins used in the Maven project POM was done a long time ago. The project POM file of the JClouds project serves as a parent for all modules in JClouds.
The first step is to consolidate the used plugins.
The same aspects apply to the build profiles used in the project. On closer look these must be considered obsolete. A deconstruction of declared build profiles is aimed at.
In order to support newer runtime environments, for example JDK11 and above, we aim to integrate build profiles.
In the respective build profile the libraries which are no longer contained in the respective target JDK but needed in JClouds are to be integrated.
In the course of this rebuild, a property-based way of declaring the version number of dependencies and plugins will be implemented.
* animal sniffer should be on java18, just like `<jdk.version>`
* Only use XMLBuilder's elem() and text() methods to have similar looking code
* Remove unnecessary call to XMLBuilder's up() because the returned value is never used
* Simplify code
* Deduplicate code
* Make the code more explicit by returning the rootBuilder
The default scope of imports is "compile." There is no reason to add this declaration to the GSON import declaration. Also, scope declarations are required to be defined in the main dependency declaration section, not in dependency management, where dependency information gets managed, which is valid across all of its children, e.g., version numbers.
Lowering the GSON version from 2.8.6 (latest) to 2.8.5, to make sure no side effects get introduced on a JDK level - for reference https://github.com/google/gson/issues/1601. Another feature is in its making to introduce a conditional build profile to support JDK 11 and above; here, a switch to the latest version of GSON gets provided.
Note: Previously, the maven jar plugin contained a configuration embedded in each module's generated manifest files. The configuration got relocated to the project/bnd.bnd file in a previous commit, and gets handled through the bnd plugin.
In the last commit (last section of squashed commit), the GSON library was integrated into the JClouds core module using maven-bundle plugins include resource instruction. Building OSGi instruction variables from the respective modules show a weakness when resources such as script builder shell scripts are required to be integrated into the bundle but not provide a dedicated variable declaration for the resource section.
The following commit demonstrates a change in strategy in declaration and integration of OSGi metadata.
- Replace old bundle-plugin with newest bnd-plugin (bundle-plugin uses bnd-plugin internally)
- Move OSGi metadata declarations from a maven variable passing strategy into dedicated bnd.bnd files
+ Cleaner pom files, no bundle packaging
+ Intellisense / Autocomplete support for .bnd files in terms of package exports etc.
For demonstration, the overall OSGi adjustments are limited to project, core, script builder, compute, blob store, and load balancer because most custom OSGi metadata is defined here.
Note: Other modules are currently disabled from build because some feedback is needed first.
Make GSON integration work.
To understand the changes, see the core modules' bnd file. GSON internal packages also define a version. Both already exported and new export declarations are fused. The global JClouds core module exports defined the entire set of GSON packages available.
Some minor modifications were made in the module project; replace maven jar plugin with a minified version of the declaration, outsourced in projects bnd file.
Replace substituted GSON package names with those provided from the vendor.
Reduce OSGi-metadata declaration of core-module because the artificial package org.jclouds.json.gson.internal was removed.
Remove the Gson module its children Gson bundle, and Gson shaded.
Remove duplication conflict and check-style rules due to the removal of the internal Gson module.
Add maven repository where a custom version of the Gson library gets hosted, which exports all packages.
Remove particular repository
Remove the declaration of the repository that serves a custom build GSON version. The build uses GSON in its original form of the vendor, which gets distributed through the standard distribution channel. The identifiers for group, artifact, and version correspond to the latest stable release of GSON.
Integrate GSON library in Clouds Core Bundle
The change contained in the commit puts the GSON library into the classpath of the JClouds core module.
After several tests with Karaf and Karaf JClouds, especially if the Maven identifier matches the original GSON library, there are only a limited number of ways to keep the deployment effort low.
Specifically, Karaf has a set of predefined Maven repositories that can be easily customized. The order in which a particular repository is resolved into the customized GSON library is more difficult. In normal OSGi applications, which do not have such a management function, I imagine this configuration to be more complicated. Sure, a unique identifier would help, but then we are back to step 1.
Although I honestly don't like to see this kind of approach in a codebase I'm working with, there are not many alternatives to the main aspect of deployment mentioned above.
Maybe the approach can still ease the problem in the short term. In a further mid-term step, however, this problem must be addressed in general.
i831992
Andrew Gaul
Timur Alperovich
Jean-Noël Rouvignac
gurkerl83
korlov42
davidsloan
roded
Tamas Cservenak