0d4f9c7789
Contributed by Thomas Marquardt DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency bug in the Azure Storage Java SDK that can cause the results of a list blobs operation to appear empty. This causes the Filesystem listStatus and similar APIs to return empty results. This has been seen in Spark work loads when jobs use more than one executor core. See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK. TESTS: A new test was added to validate the fix. All tests are passing: $mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify Tests run: 231, Failures: 0, Errors: 0, Skipped: 4 Tests run: 588, Failures: 0, Errors: 0, Skipped: 12 Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify Tests run: 37, Failures: 0, Errors: 0, Skipped: 0 Tests run: 407, Failures: 0, Errors: 0, Skipped: 34 Tests run: 151, Failures: 0, Errors: 0, Skipped: 19 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24 |
||
---|---|---|
dev-support | ||
hadoop-assemblies | ||
hadoop-build-tools | ||
hadoop-client | ||
hadoop-cloud-storage-project | ||
hadoop-common-project | ||
hadoop-dist | ||
hadoop-hdfs-project | ||
hadoop-mapreduce-project | ||
hadoop-maven-plugins | ||
hadoop-minicluster | ||
hadoop-project | ||
hadoop-project-dist | ||
hadoop-tools | ||
hadoop-yarn-project | ||
.gitattributes | ||
.gitignore | ||
BUILDING.txt | ||
Jenkinsfile | ||
LICENSE.txt | ||
NOTICE.txt | ||
README.txt | ||
pom.xml | ||
start-build-env.sh |
README.txt
For the latest information about Hadoop, please visit our website at: http://hadoop.apache.org/core/ and our wiki, at: http://wiki.apache.org/hadoop/ This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See <http://www.wassenaar.org/> for more information. The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code. The following provides more details on the included cryptographic software: Hadoop Core uses the SSL libraries from the Jetty project written by mortbay.org.