Colin Patrick Mccabe
adc3405321
HDFS-7552. Change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe)
...
(cherry picked from commit a4876c130f
)
(cherry picked from commit 22f72c7946
)
2015-09-09 14:59:31 -07:00
yliu
0adcfe6c7e
HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu)
...
(cherry picked from commit 146db49f7f
)
2015-09-08 22:57:36 -07:00
Jian He
2b526ba757
YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe
...
(cherry picked from commit 7a445fcfab
)
(cherry picked from commit 703fa1b141
)
2015-09-08 22:57:35 -07:00
Xuan
d59bf81e08
YARN-3999. RM hangs on draing events. Contributed by Jian He
...
(cherry picked from commit 3ae716fa69
)
(cherry picked from commit 2ebdf5bfce
)
2015-09-08 22:57:35 -07:00
Jonathan Eagles
6ed2486c7e
YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles)
...
(cherry picked from commit 3cd02b9522
)
(cherry picked from commit 899df5bce0
)
2015-09-08 22:57:34 -07:00
Jian He
92742b4402
YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R
...
(cherry picked from commit 258623ff8b
)
(cherry picked from commit 1d1e7682c9
)
2015-09-08 22:57:28 -07:00
cnauroth
c3928b3068
HDFS-7570. SecondaryNameNode need twice memory when calling reloadFromImageFile. Contributed by zhaoyunjiong.
...
(cherry picked from commit 85aec75ce5
)
(cherry picked from commit 27eecaef10
)
2015-09-08 17:21:58 -07:00
Vinod Kumar Vavilapalli
33537078a8
HDFS-8384. Allow NN to startup if there are files having a lease but are not under construction. Contributed by Jing Zhao.
2015-09-08 16:45:38 -07:00
Colin Patrick Mccabe
a976acc02d
HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe)
...
(cherry picked from commit a4d9acc51d
)
(cherry picked from commit 9264b7e119
)
2015-09-08 16:18:30 -07:00
Colin Patrick Mccabe
2e86d1f254
HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe)
...
(cherry picked from commit 7b424f938c
)
(cherry picked from commit cbd11681ce
)
2015-09-08 16:12:45 -07:00
Colin Patrick Mccabe
4363145128
HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe)
...
(cherry picked from commit 75a326aaff
)
(cherry picked from commit 06552a15d5
)
2015-09-08 15:32:17 -07:00
Tsz-Wo Nicholas Sze
8ed162bcbf
HDFS-7980. Incremental BlockReport will dramatically slow down namenode startup. Contributed by Walter Su
...
(cherry picked from commit 4e1f2eb395
)
2015-09-07 11:46:33 -07:00
cnauroth
c76f5b3188
HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay.
...
(cherry picked from commit 860b8373c3
)
(cherry picked from commit 33648268ce
)
2015-09-07 11:09:20 -07:00
Jian He
2336264900
YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan
...
(cherry picked from commit f489a4ec96
)
(cherry picked from commit d817fbb34d
)
2015-09-06 14:15:33 -07:00
Jian He
ee2b6bc248
YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan
...
(cherry picked from commit 18a594257e
)
(cherry picked from commit 1be2d64ddd
)
2015-09-06 11:54:40 -07:00
Jian He
637e7f9e39
YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan
...
(cherry picked from commit c1957fef29
)
(cherry picked from commit 3ddafaa7c8
)
2015-09-05 21:07:51 -07:00
Jian He
4c94f07140
YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan
...
(cherry picked from commit 21d80b3dd9
)
(cherry picked from commit c0b1311a93
)
2015-09-05 20:54:20 -07:00
Jian He
d9281fbbab
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan
...
(cherry picked from commit 86358221fc
)
(cherry picked from commit cabf97ae4f
)
2015-09-05 20:54:20 -07:00
Jian He
b0ad553841
YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan
...
(cherry picked from commit 6f9fe76918
)
(cherry picked from commit 61b4116b4b
)
2015-09-05 20:54:19 -07:00
Jian He
419e18cb37
YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena
...
(cherry picked from commit dd57c2047b
)
(cherry picked from commit c61e8a7bfa
)
2015-09-05 20:54:19 -07:00
Jian He
88f022da24
YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan
...
(cherry picked from commit fdf042dfff
)
(cherry picked from commit 411836b74c
)
2015-09-05 20:54:18 -07:00
Wangda Tan
2073fc0f84
Add missing test file of YARN-3733
...
(cherry picked from commit 405bbcf68c
)
(cherry picked from commit 344b750915
)
2015-09-03 17:43:03 -07:00
Wangda Tan
85d92721a4
YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda)
...
(cherry picked from commit ebd797c48f
)
(cherry picked from commit 78d626fa89
)
2015-09-03 17:43:01 -07:00
Jian He
f1b35ffd4c
YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch
...
(cherry picked from commit c53420f583
)
(cherry picked from commit 4931600030
)
2015-09-03 17:40:24 -07:00
Colin Patrick Mccabe
3bd9b7459b
HADOOP-12280. Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe)
...
(cherry picked from commit 6f83274afc
)
(cherry picked from commit e92107b18f
)
2015-09-03 15:02:33 -07:00
Jian He
d8ddfea491
HADOOP-11932. MetricsSinkAdapter may hang when being stopped. Contributed by Brahma Reddy Battula
...
(cherry picked from commit f59612edd7
)
(cherry picked from commit 5950c1f6f8
)
2015-09-03 14:45:04 -07:00
Jason Lowe
ca7fe71000
YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt
...
(cherry picked from commit 32e490b6c0
)
(cherry picked from commit c31e3ba921
)
(cherry picked from commit 07d31d4c0808a169f4770187d655f38aa105255c)
2015-09-03 14:40:20 -07:00
Jason Lowe
fe5877a49e
YARN-3850. NM fails to read files from full disks which can lead to container logs being lost and other issues. Contributed by Varun Saxena
...
(cherry picked from commit 40b256949a
)
(cherry picked from commit 0221d19f4e
)
(cherry picked from commit 87d2204f28f192a964c04a5fa1e2e31644d74b59)
2015-09-03 14:35:01 -07:00
Harsh J
34739fc91e
HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1389006 13f79535-47bb-0310-9956-ffa450edef68
(cherry picked from commit ac31d6a448
)
Conflicts:
hadoop-common-project/hadoop-common/CHANGES.txt
(cherry picked from commit 55427fb66c
)
2015-09-03 14:30:35 -07:00
Jason Lowe
f21fb808f1
YARN-3832. Resource Localization fails on a cluster due to existing cache directories. Contributed by Brahma Reddy Battula
...
(cherry picked from commit 8d58512d6e
)
(cherry picked from commit 15b1800b12
)
(cherry picked from commit 38400507e3352d83c2a1f364de137366249b7983)
2015-09-03 14:26:43 -07:00
Vinayakumar B
4f53c98ca4
HADOOP-7139. Allow appending to existing SequenceFiles (Contributed by kanaka kumar avvaru)
...
(cherry picked from commit 295d678be8
)
(cherry picked from commit 80697e4f32
)
(cherry picked from commit b3546b6034
)
Conflicts:
hadoop-common-project/hadoop-common/CHANGES.txt
(cherry picked from commit e9c8d8c58516aa64589cd44e9e5dd0a23ba72a17)
2015-09-03 14:15:23 -07:00
Jason Lowe
193d8d3667
YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. Contributed by Rohith Sharmaks
...
(cherry picked from commit e13b671aa5
)
(cherry picked from commit 752caa95a4
)
(cherry picked from commit 13c4db632b0e7f19dcfa883c2492431c2c7d0799)
2015-09-03 14:09:16 -07:00
Vinayakumar B
a2434dcf2a
HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina)
...
(cherry picked from commit 54f83d9bd9
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java
(cherry picked from commit 8090a6ee63
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
(cherry picked from commit 04a7000c8d955a134d86967f4b4622ea920e1ea7)
2015-09-03 14:06:09 -07:00
Wangda Tan
ae0fac3efa
YARN-3725. App submission via REST API is broken in secure mode due to Timeline DT service address is empty. (Zhijie Shen via wangda)
...
(cherry picked from commit 5cc3fced95
)
(cherry picked from commit a3734f67d3
)
(cherry picked from commit 9ccc22e2ac89990f3e7997f1d89594523c66e76a)
2015-09-03 13:50:14 -07:00
Xuan
1c6a287bf5
YARN-2900. Application (Attempt and Container) Not Found in AHS results
...
in Internal Server Error (500). Contributed by Zhijie Shen and Mit Desai
(cherry picked from commit 06f8e9caba
)
(cherry picked from commit 4fee8b3202
)
(cherry picked from commit 6c7b625138ce3b262a8c8aa28077074b553638ed)
2015-09-03 13:45:06 -07:00
Jing Zhao
46b9393cab
HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. Contributed by Ming Ma.
...
(cherry picked from commit 7817674a3a
)
(cherry picked from commit 17fb442a4c
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
(cherry picked from commit fad2a062ddbb955a42dd5a90d64781617287f8df)
2015-09-03 13:40:45 -07:00
cnauroth
563dbd29eb
HDFS-8431. hdfs crypto class not found in Windows. Contributed by Anu Engineer.
...
(cherry picked from commit 50eeea1300
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd
(cherry picked from commit 25db341278
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd
(cherry picked from commit 77a10e76e99c14cd26ebb3664304f6ed9cc7bf65)
2015-09-03 12:59:35 -07:00
Zhijie Shen
42ce052585
YARN-3700. Made generic history service load a number of latest applications according to the parameter or the configuration. Contributed by Xuan Gong.
...
(cherry picked from commit 54504133f4
)
(cherry picked from commit 839f81a632
)
(cherry picked from commit 058380d9ef35f35e8c624fb8783eac0904c4d1f5)
2015-09-03 12:59:33 -07:00
Zhijie Shen
0f33fcd507
YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, attempts and containers. Contributed by Robert Kanter.
...
(cherry picked from commit 3648cb57c9
)
(cherry picked from commit 53d6c91df9
)
2015-09-03 12:58:42 -07:00
Kihwal Lee
fb096b97cc
HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts.
...
(cherry picked from commit 8860e352c3
)
(cherry picked from commit 536b9ee6d6
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
(cherry picked from commit 2d5e60fa12a62463cd54f1b6b0fcb2ccdbd82c42)
2015-09-03 12:32:49 -07:00
Kihwal Lee
b3ca4dfaff
HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah.
...
(cherry picked from commit 2d4ae3d18b
)
(cherry picked from commit f264a5aeed
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java
(cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
2015-09-03 12:19:21 -07:00
Xuan
7b1a71a7ad
YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang
...
(cherry picked from commit b0ad644083
)
(cherry picked from commit 802676e1be
)
(cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)
2015-09-03 11:54:23 -07:00
Jason Lowe
778da79e6f
YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du
...
(cherry picked from commit 711d77cc54
)
(cherry picked from commit a81ad81461
)
(cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)
2015-09-03 11:50:30 -07:00
Tsuyoshi Ozawa
6c0ad99b9d
MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du.
...
(cherry picked from commit f4e2b3cc0b
)
(cherry picked from commit fb5b0ebb45
)
(cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)
2015-09-03 11:46:47 -07:00
Colin Patrick Mccabe
51d5fc62c2
HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe)
...
(cherry picked from commit baf794dc40
)
(cherry picked from commit 5f3d967aae
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
(cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)
2015-09-03 11:43:27 -07:00
Kihwal Lee
bbed4e67a9
HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula.
...
(cherry picked from commit 6f622672b6
)
(cherry picked from commit 802a5775f3
)
(cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)
2015-09-03 11:38:46 -07:00
Xiaoyu Yao
08006b84f8
HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao)
...
(cherry picked from commit 0100b15501
)
(cherry picked from commit e68e8b3b5c
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java
(cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)
2015-09-02 15:31:46 -07:00
Jason Lowe
41601eae6b
MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov
...
(cherry picked from commit 7dc3c1203d
)
(cherry picked from commit 87c2d915f1
)
(cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)
2015-09-02 15:27:47 -07:00
Gera Shegalov
c90f025e90
HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera)
...
(cherry picked from commit f343f8657e
)
(cherry picked from commit 58970d69de
)
2015-09-02 15:16:43 -07:00
Vinod Kumar Vavilapalli
262275d561
MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe.
...
(cherry picked from commit 9fc32c5c4d
)
(cherry picked from commit 32dc13d907
)
(cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3)
2015-09-02 15:12:21 -07:00