the build failed with two modules missing the shaded protobuf lib
either: old hadoop-common was picked up or somehow maven wasn't including
it. added one more declaration of dependency
Change-Id: Ifb5b4f45d745a7ad7ab0636cb1f5d1262b4d98fa
* new org.apache.hadoop.ipc.internal for internal only classes
* with a ShadedProtobufHelper in there which has shaded protobuf refs
only, so guaranteed not to need protobuf-2.5 on the CP
* findbugs, protobuf source patch etc
* can spec new export policies for the protobuf jar in hadoop-common
and other places it is referenced. hadoop-common back @compile
Change-Id: I61a99d2fd673259ab50d000f28a29e8a38aaf1b1
The methods which were overloaded with shaded/unshaded protobuf paramters
now have @Private ones with different names. the existing overloaded
methods are still there but deprecated, in case things use them.
* removed the extra imports needed before this change went in
* protobuf2.scope is the name for the scope var
+added a test for exception extraction, so yetus stops complaining
I am thinking of merging this in to 3.3.5 with the scope set to "compile",
as we get today. That way
* the overloading changes are in
* anyone who wants to cut their own release without protobuf2.5 can do it
Change-Id: I3423720c7047c63f7c9797d6e386774bff10b21a
The option protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.
* all declarations except those in yarn-csi are updated
* those modules which don't compile without their own explicit
import (hadoop-hdfs-client, hadoop-hdfs-rbf)
It's actually interesting to see where/how that compile fails
hadoop-hdfs-client: ClientNamenodeProtocolTranslatorPB
hadoop-hdfs-rbf:RouterAdminProtocolTranslatorPB
both with "class file for com.google.protobuf.ServiceException not found",
even though *neither class uses it*
what they do have is references to ProtobufHelper.getRemoteException(),
which is overloaded to both the shaded ServiceException and the original one
Hypothesis: the javac overload resolution needs to look at the entire
class hierarchy before it can decide which one to use.
Proposed: add a new method
ioe extractException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
and move our own code to it. Without the overloading the classes should not
be needed
Change-Id: I70354abfe3f1fdc03c418dac88e60f8cc4929a33
Removed JUnit APIs from WebServicesTestUtils and TestContainerLogsUtils.
They are used by MapReduce modules as well as YARN modules, so the
APIs need to be removed to upgrade the JUnit version on a per-module basis.
Also, this effectively reverts the prior fix in #5209 because it didn't actually
fix the issue.
Declares its compatibility with Spark's dynamic
output partitioning by having the stream capability
"mapreduce.job.committer.dynamic.partitioning"
Requires a Spark release with SPARK-40034, which
does the probing before deciding whether to
accept/rejecting instantiation with
dynamic partition overwrite set
This feature can be declared as supported by
any other PathOutputCommitter implementations
whose algorithm and destination filesystem
are compatible.
None of the S3A committers are compatible.
The classic FileOutputCommitter is, but it
does not declare itself as such out of our fear
of changing that code. The Spark-side code
will automatically infer compatibility if
the created committer is of that class or
a subclass.
Contributed by Steve Loughran.
This downgrades jackson from the version switched to in
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.
The 2.12.7 release still contains the fix for CVE-2020-36518.
Contributed by PJ Fanning
* HADOOP-18321.Fix when to read an additional record from a BZip2 text file split
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com> and Reviewed by Akira Ajisaka.
Steve Loughran
Szilard Nemeth
Ayush Saxena
Viraj Jasani
Tamas Domok
Nikita Eshkeev
Ashutosh Gupta
skysiders
Akira Ajisaka
slfan1989
Szilard Nemeth
Steve Vaughan
wangteng13
Samrat
PJ Fanning
sreeb-msft
9uapaw
Masatake Iwasaki
cfg1234
Christian Bartolomäus