Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,539 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7539 Commits

Author SHA1 Message Date
arp e4ef0023b8 HDFS-7079. Few more unit test fixes for HDFS-6581. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:02 -07:00
arp 5f86ce7ea9 HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:02 -07:00
arp 14edbc9419 HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp 6906eccec4 HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp b8a2eb793a HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
 2014-10-17 13:42:01 -07:00
arp 3abf34af9b HDFS-6977. Delete all copies when a block is deleted from the block space. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp 21046d8310 HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp 225ffdb6d8 HDFS-6950. Add Additional unit tests for HDFS-6581. (Contributed by Xiaoyu Yao) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
 2014-10-17 13:42:01 -07:00
arp 43ea145c39 HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2014-10-17 13:42:01 -07:00
arp d0e6dd88fb HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp 08cc08adb3 HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:01 -07:00
arp 0aaa2b9ba7 HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:00 -07:00
arp 49a72e1360 HDFS-6927. Initial unit tests for Lazy Persist files. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
 2014-10-17 13:42:00 -07:00
arp 3b262caf11 HDFS-6926. DN support for saving replicas to persistent storage and evicting in-memory replicas. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2014-10-17 13:42:00 -07:00
arp 193528c84d HDFS-6925. DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
 2014-10-17 13:42:00 -07:00
arp 3b7d4715a1 HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
 2014-10-17 13:42:00 -07:00
arp 183f4a4dfb HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and edit logs. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
 2014-10-17 13:42:00 -07:00
arp b1d085e7e3 HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
 2014-10-17 13:42:00 -07:00
Jason Lowe 8885b75d78 MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in client. Contributed by Rohith 
(cherry picked from commit 209b1699fc)
 2014-10-17 19:53:17 +00:00
Jian He 79428e021f YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks 
(cherry picked from commit a6aa6e42ca)
 2014-10-17 10:55:04 -07:00
Vinod Kumar Vavilapalli 99ce4277a8 YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan. 
(cherry picked from commit abae63caf9)
 2014-10-17 08:59:22 -07:00
Vinayakumar B 6f19ba7688 HDFS-7252. small refinement to the use of isInAnEZ in FSNamesystem. (Yi Liu via vinayakumarb) 
(cherry picked from commit 368743140d)
 2014-10-17 18:05:16 +05:30
Vinayakumar B 7c87c2b42e HDFS-7242. Code improvement for FSN#checkUnreadableBySuperuser. (Contributed by Yi Liu) 
(cherry picked from commit 1c3ff0b7c8)
 2014-10-17 17:49:37 +05:30
Vinayakumar B fea721daf9 HDFS-6995. Block should be placed in the client's 'rack-local' node if 'client-local' node is not available (vinayakumarb) 
Adding missed test file

(cherry picked from commit cba1f9e389)
 2014-10-17 17:34:54 +05:30
Jian He 10110b6eae Moved YARN-2682 from 2.6 to 2.7 in CHANGES.txt 
(cherry picked from commit 72093fd8cb)
 2014-10-16 19:03:29 -07:00
Jian He a78f1f8da8 YARN-2682. Updated WindowsSecureContainerExecutor to not use DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() instead. Contributed by Zhihai Xu 
(cherry picked from commit 0fd0ebae64)
 2014-10-16 18:15:43 -07:00
Ravi Prakash a85767f376 HADOOP-11182. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak) Fix CHANGES.txt 2014-10-16 16:53:38 -07:00
Jian He caba212f12 YARN-2621. Simplify the output when the user doesn't have the access for getDomain(s). Contributed by Zhijie Shen 
(cherry picked from commit 233d446be1)
 2014-10-16 16:38:00 -07:00
Steve Loughran fddbf52caa YARN-2689 TestSecureRMRegistryOperations failing on windows: secure ZK won't start 2014-10-16 14:21:49 -07:00
Tsz-Wo Nicholas Sze aef8dbde91 HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support FSIMAGE_NAME_OPTIMIZATION. 2014-10-15 20:58:01 -07:00
Tsz-Wo Nicholas Sze 603293da92 HDFS-7208. NN doesn't schedule replication when a DN storage fails. Contributed by Ming Ma 2014-10-15 20:49:54 -07:00
Vinod Kumar Vavilapalli 76cf2250de YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. 
(cherry picked from commit b3056c266a)
 2014-10-15 18:48:36 -07:00
Vinod Kumar Vavilapalli e8e3a36213 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.

(cherry picked from commit f2ea555ac6)
 2014-10-15 18:34:19 -07:00
Ravi Prakash 08eeb3e5b6 HADOOP-11181. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak) 
(cherry picked from commit 466f08792f)
 2014-10-15 15:52:46 -07:00
Jian He f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Jason Lowe 025b6c13e9 MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling. Contributed by Gera Shegalov 
(cherry picked from commit f19771a24c)
 2014-10-15 17:51:53 +00:00
Jing Zhao e68fef4d81 HDFS-7185. The active NameNode will not accept an fsimage sent from the standby during rolling upgrade. Contributed by Jing Zhao. 2014-10-15 10:28:02 -07:00
Jason Lowe 1179a0cb07 MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting for maps. Contributed by Siqi Li 
(cherry picked from commit b9edad6403)
 2014-10-15 15:54:57 +00:00
Haohui Mai 3ee0a5892b HDFS-7190. Bad use of Preconditions in startFileInternal(). Contributed by Dawson Choong. 2014-10-14 23:42:36 -07:00
Zhijie Shen bec165e163 YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. 
(cherry picked from commit 1220bb72d4)
 2014-10-14 21:56:44 -07:00
Haohui Mai 174138ec26 HDFS-7201. Fix typos in hdfs-default.xml. Contributed by Dawson Choong. 2014-10-14 17:36:23 -07:00
Zhijie Shen 8aeda20b25 HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(cherry picked from commit cdce88376a)
 2014-10-14 11:37:28 -07:00
Jing Zhao 6c3ac6a485 HDFS-7228. Add an SSD policy into the default BlockStoragePolicySuite. Contributed by Jing Zhao. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
 2014-10-14 10:33:51 -07:00
Haohui Mai 262fef119e HADOOP-11198. Fix typo in javadoc for FileSystem#listStatus(). Contributed by Li Lu. 2014-10-13 20:43:17 -07:00
Tsz-Wo Nicholas Sze dd0857fd70 HDFS-7237. The command "hdfs namenode -rollingUpgrade" throws ArrayIndexOutOfBoundsException. 2014-10-13 18:30:21 -07:00
Aaron T. Myers f3132eee10 HADOOP-11176. KMSClientProvider authentication fails when both currentUgi and loginUgi are a proxied user. Contributed by Arun Suresh. 
(cherry picked from commit 0e57aa3bf6)
 2014-10-13 18:10:23 -07:00
Karthik Kambatla 88455173e8 YARN-2566. DefaultContainerExecutor should pick a working directory randomly. (Zhihai Xu via kasha) 
(cherry picked from commit cc93e7e683)
 2014-10-13 16:32:42 -07:00
Karthik Kambatla a1116b56a4 YARN-2641. Decommission nodes on -refreshNodes instead of next NM-RM heartbeat. (Zhihai Xu via kasha) 
(cherry picked from commit da709a2eac)
 2014-10-13 16:24:00 -07:00
Jian He 31a4bf7321 Missing Changes.txt for YARN-2308 
(cherry picked from commit 178bc505da)
 2014-10-13 14:20:18 -07:00
Jian He 536254f5e7 YARN-2308. Changed CapacityScheduler to explicitly throw exception if the queue 
to which the apps were submitted is changed across RM restart. Contributed by Craig Welch & Chang Li
(cherry picked from commit f9680d9a16)
 2014-10-13 14:16:12 -07:00