Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,014 Commits 326 Branches 371 Tags 1.5 GiB
Java 92.7%
C++ 2.9%
C 1.9%
JavaScript 1.2%
Shell 0.5%
Other 0.5%
Go to file
Sneha Vijayarajan d166420302
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 		2020-10-14 22:29:13 +00:00
.github HADOOP-15184. Add GitHub pull request template. (#1419) 2019-09-11 11:10:11 +09:00
dev-support HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 2020-10-06 17:15:17 +01:00
hadoop-assemblies Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
hadoop-build-tools HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
hadoop-client-modules YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars (#2075) 2020-06-17 09:29:49 +05:30
hadoop-cloud-storage-project Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
hadoop-common-project HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 2020-10-12 17:38:42 -07:00
hadoop-dist Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
hadoop-hdfs-project HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 2020-10-14 17:28:02 -05:00
hadoop-mapreduce-project MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 2020-10-12 11:09:08 -07:00
hadoop-maven-plugins Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
hadoop-minicluster Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
hadoop-project HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 2020-10-13 17:56:38 +09:00
hadoop-project-dist HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 2020-10-06 17:15:17 +01:00
hadoop-tools HADOOP-17215: Support for conditional overwrite. 2020-10-14 22:29:13 +00:00
hadoop-yarn-project YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:59:25 +00:00
licenses HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
licenses-binary HADOOP-15993. Upgrade Kafka to 2.4.0 in hadoop-kafka module. (#1796) 2020-01-09 16:24:58 +09:00
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:50:10 +09:00
BUILDING.txt HADOOP-17125. Use snappy-java in SnappyCodec (#2297) 2020-10-06 17:15:17 +01:00
Jenkinsfile HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 2020-08-18 12:11:23 +09:00
LICENSE-binary HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027) 2020-05-22 11:20:16 +09:00
LICENSE.txt YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-09 01:25:10 +00:00
NOTICE-binary HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
NOTICE.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
README.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
pom.xml Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
start-build-env.sh HADOOP-16849. start-build-env.sh behaves incorrectly when username is numeric only. Contributed by Jihyun Cho. 2020-02-12 14:06:23 +09:00

README.txt 

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   https://cwiki.apache.org/confluence/display/HADOOP/