The option protobuf.scope defines whether the protobuf 2.5.0
dependency is marked as provided or not.
* all declarations except those in yarn-csi are updated
* those modules which don't compile without their own explicit
import (hadoop-hdfs-client, hadoop-hdfs-rbf)
It's actually interesting to see where/how that compile fails
hadoop-hdfs-client: ClientNamenodeProtocolTranslatorPB
hadoop-hdfs-rbf:RouterAdminProtocolTranslatorPB
both with "class file for com.google.protobuf.ServiceException not found",
even though *neither class uses it*
what they do have is references to ProtobufHelper.getRemoteException(),
which is overloaded to both the shaded ServiceException and the original one
Hypothesis: the javac overload resolution needs to look at the entire
class hierarchy before it can decide which one to use.
Proposed: add a new method
ioe extractException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
and move our own code to it. Without the overloading the classes should not
be needed
Change-Id: I70354abfe3f1fdc03c418dac88e60f8cc4929a33
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations.
ignoredNodes list is only used on hedged read codepath
Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
POM and LICENSE fixup of transient dependencies
* Exclude hadoop-cloud-storage imports which come in with hadoop-common
* Add explicit import of hadoop's org.codehaus.jettison declaration
to hadoop-aliyun
* Tune aliyun jars imports
* Update LICENSE-binary for the current set of libraries.
Contributed by Steve Loughran
Followup to the original HADOOP-18582.
Temporary path cleanup is re-enabled for -append jobs
as these will create temporary files when creating or overwriting files.
Contributed by Ayush Saxena
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.
Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.
Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.
Contributed by Steve Loughran
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners
Contributed by Steve Loughran
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging
Contributed by Steve Vaughan Jr
Steve Loughran
Varun Saxena
ZanderXu
Viraj Jasani
Tom
Szilard Nemeth
slfan1989
rdingankar
Simbarashe Dzinamarira
Owen O'Malley
Ayush Saxena
hchaverr
susheel-gupta
nao
zhtttylz
Arnout Engelen
Bryan Beaudreault
Mehakmeet Singh
hfutatzhanghb
Ankit Saurabh
Zita Dombi
SimhadriGovindappa
GuoPhilipse
Tamas Domok
Steve Vaughan