Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HADOOP-16775. DistCp reuses the same temp file within the task for different files. 

Contributed by Amir Shenavandeh.

This avoids overwrite consistency issues with S3 and other stores

Change-Id: Ic4d05ef3397e963ba28fd9f775bb362b0da36ad9
This commit is contained in:
Steve Loughran 2020-03-13 19:33:28 +00:00
parent 4e1021975b
commit caec6a1945
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
View File

@ -231,7 +231,9 @@ public class RetriableFileCopyCommand extends RetriableCommand {
Path root = target.equals(targetWorkPath) ? targetWorkPath.getParent()
: targetWorkPath;
Path tempFile = new Path(root, ".distcp.tmp." +
context.getTaskAttemptID().toString());
context.getTaskAttemptID().toString() +
"." + String.valueOf(System.currentTimeMillis()));
LOG.info("Creating temp file: " + tempFile);
return tempFile;
}