69aacf19c1
HDFS-7112. LazyWriter should use either async IO or one thread per physical disk. Contributed by Xiaoyu Yao.
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2014-10-17 13:42:03 -07:00
ae8c9cdb18
HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
2014-10-17 13:42:03 -07:00
c865c93dc1
HDFS-7129. Metrics to track usage of memory for writes. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:03 -07:00
6916d41fe6
HDFS-7159. Use block storage policy to set lazy persist preference. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
2014-10-17 13:42:03 -07:00
ca6ac1997f
HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (Contributed by Tsz Wo Nicholas Sze)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:03 -07:00
081244fc8b
HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:03 -07:00
6bb27f0ff6
HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (Contributed by Tsz Wo Nicholas Sze)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:03 -07:00
69828a9bf0
HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestStorageMover.java
2014-10-17 13:42:02 -07:00
70f70f845f
HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
bf1b84abe1
HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
2014-10-17 13:42:02 -07:00
f6903ca945
HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
a973b2f55e
HDFS-7091. Add forwarding constructor for INodeFile for existing callers. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
2014-10-17 13:42:02 -07:00
f193d96627
HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
d5940dfe4b
HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
e4ef0023b8
HDFS-7079. Few more unit test fixes for HDFS-6581. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
5f86ce7ea9
HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:02 -07:00
14edbc9419
HDFS-6978. Directory scanner should correctly reconcile blocks on RAM disk. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
6906eccec4
HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
b8a2eb793a
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
2014-10-17 13:42:01 -07:00
3abf34af9b
HDFS-6977. Delete all copies when a block is deleted from the block space. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
21046d8310
HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
225ffdb6d8
HDFS-6950. Add Additional unit tests for HDFS-6581. (Contributed by Xiaoyu Yao)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
2014-10-17 13:42:01 -07:00
43ea145c39
HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
2014-10-17 13:42:01 -07:00
d0e6dd88fb
HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
08cc08adb3
HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:01 -07:00
0aaa2b9ba7
HDFS-6929. NN periodically unlinks lazy persist files with missing replicas from namespace. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:00 -07:00
49a72e1360
HDFS-6927. Initial unit tests for Lazy Persist files. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
2014-10-17 13:42:00 -07:00
3b262caf11
HDFS-6926. DN support for saving replicas to persistent storage and evicting in-memory replicas. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
2014-10-17 13:42:00 -07:00
193528c84d
HDFS-6925. DataNode should attempt to place replicas on transient storage first if lazyPersist flag is received. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
2014-10-17 13:42:00 -07:00
3b7d4715a1
HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java
2014-10-17 13:42:00 -07:00
183f4a4dfb
HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and edit logs. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileAttributes.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeLayoutVersion.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
2014-10-17 13:42:00 -07:00
b1d085e7e3
HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal)
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StorageType.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
hadoop-hdfs-project/hadoop-hdfs/src/main/proto/hdfs.proto
2014-10-17 13:42:00 -07:00
8885b75d78
MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in client. Contributed by Rohith
...
(cherry picked from commit 209b1699fc
2014-10-17 19:53:17 +00:00
79428e021f
YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks
...
(cherry picked from commit a6aa6e42ca
2014-10-17 10:55:04 -07:00
99ce4277a8
YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan.
...
(cherry picked from commit abae63caf9
2014-10-17 08:59:22 -07:00
6f19ba7688
HDFS-7252. small refinement to the use of isInAnEZ in FSNamesystem. (Yi Liu via vinayakumarb)
...
(cherry picked from commit 368743140d
2014-10-17 18:05:16 +05:30
7c87c2b42e
HDFS-7242. Code improvement for FSN#checkUnreadableBySuperuser. (Contributed by Yi Liu)
...
(cherry picked from commit 1c3ff0b7c8
2014-10-17 17:49:37 +05:30
fea721daf9
HDFS-6995. Block should be placed in the client's 'rack-local' node if 'client-local' node is not available (vinayakumarb)
...
Adding missed test file
(cherry picked from commit cba1f9e389
2014-10-17 17:34:54 +05:30
10110b6eae
Moved YARN-2682 from 2.6 to 2.7 in CHANGES.txt
...
(cherry picked from commit 72093fd8cb
2014-10-16 19:03:29 -07:00
a78f1f8da8
YARN-2682. Updated WindowsSecureContainerExecutor to not use DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() instead. Contributed by Zhihai Xu
...
(cherry picked from commit 0fd0ebae64
2014-10-16 18:15:43 -07:00
a85767f376
HADOOP-11182. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak) Fix CHANGES.txt
2014-10-16 16:53:38 -07:00
caba212f12
YARN-2621. Simplify the output when the user doesn't have the access for getDomain(s). Contributed by Zhijie Shen
...
(cherry picked from commit 233d446be1
2014-10-16 16:38:00 -07:00
fddbf52caa
YARN-2689 TestSecureRMRegistryOperations failing on windows: secure ZK won't start
2014-10-16 14:21:49 -07:00
aef8dbde91
HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support FSIMAGE_NAME_OPTIMIZATION.
2014-10-15 20:58:01 -07:00
603293da92
HDFS-7208. NN doesn't schedule replication when a DN storage fails. Contributed by Ming Ma
2014-10-15 20:49:54 -07:00
76cf2250de
YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan.
...
(cherry picked from commit b3056c266a
2014-10-15 18:48:36 -07:00
e8e3a36213
YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan.
...
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.
(cherry picked from commit f2ea555ac6
2014-10-15 18:34:19 -07:00
08eeb3e5b6
HADOOP-11181. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak)
...
(cherry picked from commit 466f08792f
2014-10-15 15:52:46 -07:00
f93d2ea27e
YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA
2014-10-15 15:28:26 -07:00
025b6c13e9
MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling. Contributed by Gera Shegalov
...
(cherry picked from commit f19771a24c
2014-10-15 17:51:53 +00:00
cnauroth
arp
Jason Lowe
Jian He
Vinod Kumar Vavilapalli
Vinayakumar B
Ravi Prakash
Steve Loughran
Tsz-Wo Nicholas Sze
Ravi Prakash