Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,918 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7918 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze 8ed162bcbf HDFS-7980. Incremental BlockReport will dramatically slow down namenode startup. Contributed by Walter Su 
(cherry picked from commit 4e1f2eb395)
 2015-09-07 11:46:33 -07:00
cnauroth c76f5b3188 HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop. Contributed by Larry McCay. 
(cherry picked from commit 860b8373c3)
(cherry picked from commit 33648268ce)
 2015-09-07 11:09:20 -07:00
Jian He 2336264900 YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan 
(cherry picked from commit f489a4ec96)

(cherry picked from commit d817fbb34d)
 2015-09-06 14:15:33 -07:00
Jian He ee2b6bc248 YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan 
(cherry picked from commit 18a594257e)

(cherry picked from commit 1be2d64ddd)
 2015-09-06 11:54:40 -07:00
Jian He 637e7f9e39 YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan 
(cherry picked from commit c1957fef29)

(cherry picked from commit 3ddafaa7c8)
 2015-09-05 21:07:51 -07:00
Jian He 4c94f07140 YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan 
(cherry picked from commit 21d80b3dd9)

(cherry picked from commit c0b1311a93)
 2015-09-05 20:54:20 -07:00
Jian He d9281fbbab YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 86358221fc)

(cherry picked from commit cabf97ae4f)
 2015-09-05 20:54:20 -07:00
Jian He b0ad553841 YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan 
(cherry picked from commit 6f9fe76918)

(cherry picked from commit 61b4116b4b)
 2015-09-05 20:54:19 -07:00
Jian He 419e18cb37 YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 
(cherry picked from commit dd57c2047b)

(cherry picked from commit c61e8a7bfa)
 2015-09-05 20:54:19 -07:00
Jian He 88f022da24 YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 
(cherry picked from commit fdf042dfff)

(cherry picked from commit 411836b74c)
 2015-09-05 20:54:18 -07:00
Wangda Tan 2073fc0f84 Add missing test file of YARN-3733 
(cherry picked from commit 405bbcf68c)
(cherry picked from commit 344b750915)
 2015-09-03 17:43:03 -07:00
Wangda Tan 85d92721a4 YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda) 
(cherry picked from commit ebd797c48f)
(cherry picked from commit 78d626fa89)
 2015-09-03 17:43:01 -07:00
Jian He f1b35ffd4c YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch 
(cherry picked from commit c53420f583)

(cherry picked from commit 4931600030)
 2015-09-03 17:40:24 -07:00
Colin Patrick Mccabe 3bd9b7459b HADOOP-12280. Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) 
(cherry picked from commit 6f83274afc)
(cherry picked from commit e92107b18f)
 2015-09-03 15:02:33 -07:00
Jian He d8ddfea491 HADOOP-11932. MetricsSinkAdapter may hang when being stopped. Contributed by Brahma Reddy Battula 
(cherry picked from commit f59612edd7)

(cherry picked from commit 5950c1f6f8)
 2015-09-03 14:45:04 -07:00
Jason Lowe ca7fe71000 YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt 
(cherry picked from commit 32e490b6c0)

(cherry picked from commit c31e3ba921)
(cherry picked from commit 07d31d4c0808a169f4770187d655f38aa105255c)
 2015-09-03 14:40:20 -07:00
Jason Lowe fe5877a49e YARN-3850. NM fails to read files from full disks which can lead to container logs being lost and other issues. Contributed by Varun Saxena 
(cherry picked from commit 40b256949a)

(cherry picked from commit 0221d19f4e)
(cherry picked from commit 87d2204f28f192a964c04a5fa1e2e31644d74b59)
 2015-09-03 14:35:01 -07:00
Harsh J 34739fc91e HADOOP-8151. Error handling in snappy decompressor throws invalid exceptions. Contributed by Matt Foley. (harsh) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1389006 13f79535-47bb-0310-9956-ffa450edef68
(cherry picked from commit ac31d6a448)

Conflicts:

	hadoop-common-project/hadoop-common/CHANGES.txt

(cherry picked from commit 55427fb66c)
 2015-09-03 14:30:35 -07:00
Jason Lowe f21fb808f1 YARN-3832. Resource Localization fails on a cluster due to existing cache directories. Contributed by Brahma Reddy Battula 
(cherry picked from commit 8d58512d6e)

(cherry picked from commit 15b1800b12)
(cherry picked from commit 38400507e3352d83c2a1f364de137366249b7983)
 2015-09-03 14:26:43 -07:00
Vinayakumar B 4f53c98ca4 HADOOP-7139. Allow appending to existing SequenceFiles (Contributed by kanaka kumar avvaru) 
(cherry picked from commit 295d678be8)
(cherry picked from commit 80697e4f32)
(cherry picked from commit b3546b6034)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt

(cherry picked from commit e9c8d8c58516aa64589cd44e9e5dd0a23ba72a17)
 2015-09-03 14:15:23 -07:00
Jason Lowe 193d8d3667 YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. Contributed by Rohith Sharmaks 
(cherry picked from commit e13b671aa5)

(cherry picked from commit 752caa95a4)
(cherry picked from commit 13c4db632b0e7f19dcfa883c2492431c2c7d0799)
 2015-09-03 14:09:16 -07:00
Vinayakumar B a2434dcf2a HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina) 
(cherry picked from commit 54f83d9bd9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java

(cherry picked from commit 8090a6ee63)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

(cherry picked from commit 04a7000c8d955a134d86967f4b4622ea920e1ea7)
 2015-09-03 14:06:09 -07:00
Wangda Tan ae0fac3efa YARN-3725. App submission via REST API is broken in secure mode due to Timeline DT service address is empty. (Zhijie Shen via wangda) 
(cherry picked from commit 5cc3fced95)
(cherry picked from commit a3734f67d3)
(cherry picked from commit 9ccc22e2ac89990f3e7997f1d89594523c66e76a)
 2015-09-03 13:50:14 -07:00
Xuan 1c6a287bf5 YARN-2900. Application (Attempt and Container) Not Found in AHS results 
in Internal Server Error (500). Contributed by Zhijie Shen and Mit Desai

(cherry picked from commit 06f8e9caba)
(cherry picked from commit 4fee8b3202)
(cherry picked from commit 6c7b625138ce3b262a8c8aa28077074b553638ed)
 2015-09-03 13:45:06 -07:00
Jing Zhao 46b9393cab HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. Contributed by Ming Ma. 
(cherry picked from commit 7817674a3a)
(cherry picked from commit 17fb442a4c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java

(cherry picked from commit fad2a062ddbb955a42dd5a90d64781617287f8df)
 2015-09-03 13:40:45 -07:00
cnauroth 563dbd29eb HDFS-8431. hdfs crypto class not found in Windows. Contributed by Anu Engineer. 
(cherry picked from commit 50eeea1300)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 25db341278)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 77a10e76e99c14cd26ebb3664304f6ed9cc7bf65)
 2015-09-03 12:59:35 -07:00
Zhijie Shen 42ce052585 YARN-3700. Made generic history service load a number of latest applications according to the parameter or the configuration. Contributed by Xuan Gong. 
(cherry picked from commit 54504133f4)
(cherry picked from commit 839f81a632)
(cherry picked from commit 058380d9ef35f35e8c624fb8783eac0904c4d1f5)
 2015-09-03 12:59:33 -07:00
Zhijie Shen 0f33fcd507 YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, attempts and containers. Contributed by Robert Kanter. 
(cherry picked from commit 3648cb57c9)
(cherry picked from commit 53d6c91df9)
 2015-09-03 12:58:42 -07:00
Kihwal Lee fb096b97cc HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. 
(cherry picked from commit 8860e352c3)

(cherry picked from commit 536b9ee6d6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java

(cherry picked from commit 2d5e60fa12a62463cd54f1b6b0fcb2ccdbd82c42)
 2015-09-03 12:32:49 -07:00
Kihwal Lee b3ca4dfaff HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. 
(cherry picked from commit 2d4ae3d18b)

(cherry picked from commit f264a5aeed)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java

(cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
 2015-09-03 12:19:21 -07:00
Xuan 7b1a71a7ad YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang 
(cherry picked from commit b0ad644083)
(cherry picked from commit 802676e1be)
(cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)
 2015-09-03 11:54:23 -07:00
Jason Lowe 778da79e6f YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du 
(cherry picked from commit 711d77cc54)

(cherry picked from commit a81ad81461)
(cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)
 2015-09-03 11:50:30 -07:00
Tsuyoshi Ozawa 6c0ad99b9d MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. 
(cherry picked from commit f4e2b3cc0b)
(cherry picked from commit fb5b0ebb45)
(cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)
 2015-09-03 11:46:47 -07:00
Colin Patrick Mccabe 51d5fc62c2 HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) 
(cherry picked from commit baf794dc40)

(cherry picked from commit 5f3d967aae)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm

(cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)
 2015-09-03 11:43:27 -07:00
Kihwal Lee bbed4e67a9 HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 6f622672b6)

(cherry picked from commit 802a5775f3)
(cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)
 2015-09-03 11:38:46 -07:00
Xiaoyu Yao 08006b84f8 HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) 
(cherry picked from commit 0100b15501)
(cherry picked from commit e68e8b3b5c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java

(cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)
 2015-09-02 15:31:46 -07:00
Jason Lowe 41601eae6b MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov 
(cherry picked from commit 7dc3c1203d)

(cherry picked from commit 87c2d915f1)
(cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)
 2015-09-02 15:27:47 -07:00
Gera Shegalov c90f025e90 HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera) 
(cherry picked from commit f343f8657e)
(cherry picked from commit 58970d69de)
 2015-09-02 15:16:43 -07:00
Vinod Kumar Vavilapalli 262275d561 MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. 
(cherry picked from commit 9fc32c5c4d)
(cherry picked from commit 32dc13d907)
(cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3)
 2015-09-02 15:12:21 -07:00
Karthik Kambatla 6ade6b5051 YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) 
(cherry picked from commit 47279c3228)
(cherry picked from commit 4045c41afe)
(cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930)
 2015-09-02 15:03:51 -07:00
Xuan 9af5b1dcd0 YARN-3024. LocalizerRunner should give DIE action when all resources are 
localized. Contributed by Chengbing Liu

(cherry picked from commit 0d6bd62102)
(cherry picked from commit a7696b3fbf)
(cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be)
 2015-09-02 14:52:06 -07:00
Colin Patrick Mccabe 8770f1f039 HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) 
(cherry picked from commit a0e0a63209)
(cherry picked from commit 788b76761d)
 2015-09-02 14:38:26 -07:00
Colin Patrick Mccabe 181281cfca HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) 
(cherry picked from commit a8898445dc)
(cherry picked from commit 3d0385c3c8)
 2015-09-02 14:37:51 -07:00
Colin Patrick Mccabe 9df4b7c10e HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 
(cherry picked from commit bc9cb3e271)
(cherry picked from commit c71b54fa32)
 2015-09-02 14:20:17 -07:00
Steve Loughran e0e93e9f8d HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 
(cherry picked from commit a6a5d1d6b5)
 2015-09-02 14:01:53 -07:00
Gera Shegalov 736ebeca33 HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera) 
(cherry picked from commit 6d2cf9fbbd)
(cherry picked from commit 1544c63602)
 2015-09-02 11:46:47 -07:00
Robert Kanter c253ac68eb MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 
(cherry picked from commit d50e8f0928)
(cherry picked from commit 756c254293)
(cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)
 2015-09-02 11:40:32 -07:00
Karthik Kambatla 2f66dff7ce MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) 
(cherry picked from commit c66c3ac6bf)
(cherry picked from commit f4d6c5e337)
(cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)
 2015-09-02 11:32:08 -07:00
Wangda Tan e081593042 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
(cherry picked from commit 3316cd4357)
(cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)
 2015-09-02 11:28:22 -07:00
Wangda Tan 61f2ddb125 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
(cherry picked from commit e7cbecddc3)
(cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)
 2015-09-02 11:14:35 -07:00