...lets see if that helps things to build, though it does look like a
mvn repo change.
key point: dependencies are the same as before, but protobuf can be excluded
by downstream projects and all RPC code will still work.
Change-Id: I681b75d3dfe5cd5d0e6852ee5d73a63e28a3b8d0
* new org.apache.hadoop.ipc.internal for internal only classes
* with a ShadedProtobufHelper in there which has shaded protobuf refs
only, so guaranteed not to need protobuf-2.5 on the CP
* findbugs, protobuf source patch etc
* can spec new export policies for the protobuf jar in hadoop-common
and other places it is referenced. hadoop-common back @compile
Change-Id: I61a99d2fd673259ab50d000f28a29e8a38aaf1b1
The methods which were overloaded with shaded/unshaded protobuf paramters
now have @Private ones with different names. the existing overloaded
methods are still there but deprecated, in case things use them.
* removed the extra imports needed before this change went in
* protobuf2.scope is the name for the scope var
+added a test for exception extraction, so yetus stops complaining
I am thinking of merging this in to 3.3.5 with the scope set to "compile",
as we get today. That way
* the overloading changes are in
* anyone who wants to cut their own release without protobuf2.5 can do it
Change-Id: I3423720c7047c63f7c9797d6e386774bff10b21a
The option protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.
* all declarations except those in yarn-csi are updated
* those modules which don't compile without their own explicit
import (hadoop-hdfs-client, hadoop-hdfs-rbf)
It's actually interesting to see where/how that compile fails
hadoop-hdfs-client: ClientNamenodeProtocolTranslatorPB
hadoop-hdfs-rbf:RouterAdminProtocolTranslatorPB
both with "class file for com.google.protobuf.ServiceException not found",
even though *neither class uses it*
what they do have is references to ProtobufHelper.getRemoteException(),
which is overloaded to both the shaded ServiceException and the original one
Hypothesis: the javac overload resolution needs to look at the entire
class hierarchy before it can decide which one to use.
Proposed: add a new method
ioe extractException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
and move our own code to it. Without the overloading the classes should not
be needed
Change-Id: I70354abfe3f1fdc03c418dac88e60f8cc4929a33
POM and LICENSE fixup of transient dependencies
* Exclude hadoop-cloud-storage imports which come in with hadoop-common
* Add explicit import of hadoop's org.codehaus.jettison declaration
to hadoop-aliyun
* Tune aliyun jars imports
* Update LICENSE-binary for the current set of libraries.
Contributed by Steve Loughran
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners
Contributed by Steve Loughran
Addresses CVE-2021-37533, which *only* relates to FTP.
Applications not using the ftp:// filesystem, which, as
anyone who has used it will know is very minimal and
so rarely used, is not a critical part of the project.
Furthermore, the FTP-related issue is at worst information leakage
if someone connects to a malicious server.
This is a due diligence PR rather than an emergency fix.
Contributed by Steve Loughran
Updates okhttp3 and okio so their transitive dependency on Kotlin
stdlib is free from recent CVEs.
okhttp3:okhttp => 4.10.0
okio:okio => 3.2.0
kotlin stdlib => 1.6.20
kotlin CVEs fixed:
CVE-2022-24329
CVE-2020-29582
Contributed by PJ Fanning.
This addresses an issue where the plugin's default classpath for executing tests fails to include org.junit.platform.launcher.core.LauncherFactory.
Contributed by: Steve Vaughan Jr
Fixes CVE-2018-7489 in shaded jackson.
+Add more commands in testing.md
to the CLI tests needed when qualifying
a release
Contributed by Steve Loughran
This downgrades jackson from the version switched to in
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.
The 2.12.7 release still contains the fix for CVE-2020-36518.
Contributed by PJ Fanning
Update the dependencies of the LDAP libraries used for testing:
ldap-api.version = 2.0.0
apacheds.version = 2.0.0.AM26
Contributed by Colm O hEigeartaigh.
part of HADOOP-18103.
Add support for multiple ranged vectored read api in PositionedReadable.
The default iterates through the ranges to read each synchronously,
but the intent is that FSDataInputStream subclasses can make more
efficient readers especially in object stores implementation.
Also added implementation in S3A where smaller ranges are merged and
sliced byte buffers are returned to the readers. All the merged ranged are
fetched from S3 asynchronously.
Contributed By: Owen O'Malley and Mukund Thakur
Steve Loughran
hchaverr
nao
Viraj Jasani
Szilard Nemeth
PJ Fanning
Ayush Saxena
Murali Krishna
Steve Vaughan
Ashutosh Gupta
Willi Raschkowski
Hexiaoqiao
slfan1989
Colm O hEigeartaigh
Wei-Chiu Chuang
Igor Dvorzhak
Mukund Thakur