Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,885 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7885 Commits

Author SHA1 Message Date
Jian He 852ffea84e Missing file for YARN-1879 
(cherry picked from commit 4a78a75228)
 2014-10-20 21:15:30 -07:00
Jian He 23b4af488e Missing file for YARN-2701 
(cherry picked from commit 4fa1fb3193)
 2014-10-20 20:53:28 -07:00
Jian He c7db30522c YARN-2701. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 
(cherry picked from commit 2839365f23)
 2014-10-20 20:53:28 -07:00
Vinod Kumar Vavilapalli debb3686d3 YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 
commit ba7f31c2ee is the corresponding trunk commit, this is a slightly different patch for branch-2.
(cherry picked from commit 3326fba382)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-20 20:47:49 -07:00
Zhijie Shen 2af7f48625 YARN-2582. Fixed Log CLI and Web UI for showing aggregated logs of LRS. Contributed Xuan Gong. 
(cherry picked from commit e90718fa5a)
 2014-10-20 13:43:03 -07:00
Zhijie Shen 6d78dc3ee4 YARN-2673. Made timeline client put APIs retry if ConnectException happens. Contributed by Li Lu. 
(cherry picked from commit 89427419a3)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2014-10-20 12:31:33 -07:00
Karthik Kambatla cdf265de4f HADOOP-11194. Ignore .keep files (kasha) 
(cherry picked from commit d5084b9fa3)
 2014-10-20 11:30:13 -04:00
Ivan Mitic a78dd99d67 Revert "MAPREDUCE-5911. Terasort TeraOutputFormat does not check for output directory existance. Contributed by Bruno P. Kinoshita." 
This reverts commit 71e05b1900.
 2014-10-19 23:43:38 -04:00
Ivan Mitic 71e05b1900 MAPREDUCE-5911. Terasort TeraOutputFormat does not check for output directory existance. Contributed by Bruno P. Kinoshita. 
(cherry picked from commit 7bbda6ef92)
 2014-10-19 19:15:55 -04:00
Vinod Kumar Vavilapalli 88b7e5ca65 YARN-2504. Enhanced RM Admin CLI to support management of node-labels. Contribyted by Wangda Tan. 
(cherry picked from commit 8256766498)
 2014-10-18 12:09:02 -07:00
Vinod Kumar Vavilapalli 24b0529a9f YARN-2676. Enhanced Timeline auth-filter to support proxy users. Contributed by Zhijie Shen. 
(cherry picked from commit 39063cd36f)
 2014-10-17 22:04:14 -07:00
Tsz-Wo Nicholas Sze 0ee6ddb782 HDFS-7260. Change DFSOutputStream.MAX_PACKETS to be configurable. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
 2014-10-17 18:34:23 -07:00
Vinod Kumar Vavilapalli ce42210651 YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing test-failures: added a dummy in-memory labels-manager. Contributed by Wangda Tan. 
(cherry picked from commit e9c66e8fd2)
 2014-10-17 18:27:34 -07:00
Jian He 4d03915be9 YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit c3de2412eb)
 2014-10-17 16:36:45 -07:00
arp 9183dadb73 HDFS-6921. Add LazyPersist flag to FileStatus. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsLocatedFileStatus.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 2014-10-17 16:00:53 -07:00
cnauroth 2e98379bcb HDFS-7090. Use unbuffered writes when persisting in-memory replicas. Contributed by Xiaoyu Yao. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-10-17 16:00:53 -07:00
cnauroth 1b6d115ccf HDFS-7112. LazyWriter should use either async IO or one thread per physical disk. Contributed by Xiaoyu Yao. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-10-17 16:00:53 -07:00
arp 137f0b8cbf HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
 2014-10-17 16:00:53 -07:00
arp e6457a0e08 HDFS-7129. Metrics to track usage of memory for writes. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:53 -07:00
arp d61daff9bf HDFS-7159. Use block storage policy to set lazy persist preference. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 2014-10-17 16:00:52 -07:00
arp ba31a9503c HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (Contributed by Tsz Wo Nicholas Sze) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:52 -07:00
arp 3481f5d4f3 HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:52 -07:00
arp 05cfd68863 HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (Contributed by Tsz Wo Nicholas Sze) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:52 -07:00
arp 9de4d1897a HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
 2014-10-17 16:00:52 -07:00
arp a59ce50627 HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:52 -07:00
arp e9eebaa012 HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2014-10-17 16:00:52 -07:00
arp 144198892f HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp ebdbb5ec24 HDFS-7091. Add forwarding constructor for INodeFile for existing callers. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
 2014-10-17 16:00:51 -07:00
arp d32ce5794a HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp b7062e8d3e HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp 7d168d9ff9 HDFS-7079. Few more unit test fixes for HDFS-6581. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp d3ccbd8e65 HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp 6cc206b8ad HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:51 -07:00
arp 2fdd9f51ce HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:50 -07:00
arp bcec6582cf HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
 2014-10-17 16:00:50 -07:00
arp 186d195033 HDFS-6977. Delete all copies when a block is deleted from the block space. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:50 -07:00
arp 34f28eaff9 HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:50 -07:00
arp 4609ba0dc8 HDFS-6950. Add Additional unit tests for HDFS-6581. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
 2014-10-17 16:00:50 -07:00
arp 754ba3a42d HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2014-10-17 16:00:50 -07:00
arp d973656e5d HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:50 -07:00
arp 2079d4b72e HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:49 -07:00
arp 15372df48d HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:49 -07:00
arp c2721edc50 HDFS-6927. Initial unit tests for Lazy Persist files. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 16:00:49 -07:00
arp 8e01a89b61 HDFS-6926. DN support for saving replicas to persistent storage and evicting in-memory replicas. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2014-10-17 16:00:49 -07:00
arp 34d0088bf9 HDFS-6925. DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
 2014-10-17 16:00:49 -07:00
arp 8c3c0ec977 HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
 2014-10-17 16:00:49 -07:00
arp 1d9d29ad40 HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and edit logs. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
 2014-10-17 16:00:48 -07:00
arp d09f316520 HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
 2014-10-17 16:00:34 -07:00
Vinod Kumar Vavilapalli db2bfed839 HADOOP-11207. Enhanced common DelegationTokenAuthenticationHandler to support proxy-users on Delegation-token management operations. Contributed by Zhijie Shen. 
(cherry picked from commit 1137557816)
 2014-10-17 15:57:39 -07:00
Jason Lowe 8217665297 MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in client. Contributed by Rohith 
(cherry picked from commit 209b1699fc)
 2014-10-17 19:53:35 +00:00