Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,798 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7798 Commits

Author SHA1 Message Date
Jason Lowe 193d8d3667 YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. Contributed by Rohith Sharmaks 
(cherry picked from commit e13b671aa5)

(cherry picked from commit 752caa95a4)
(cherry picked from commit 13c4db632b0e7f19dcfa883c2492431c2c7d0799)
 2015-09-03 14:09:16 -07:00
Vinayakumar B a2434dcf2a HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina) 
(cherry picked from commit 54f83d9bd9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java

(cherry picked from commit 8090a6ee63)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

(cherry picked from commit 04a7000c8d955a134d86967f4b4622ea920e1ea7)
 2015-09-03 14:06:09 -07:00
Wangda Tan ae0fac3efa YARN-3725. App submission via REST API is broken in secure mode due to Timeline DT service address is empty. (Zhijie Shen via wangda) 
(cherry picked from commit 5cc3fced95)
(cherry picked from commit a3734f67d3)
(cherry picked from commit 9ccc22e2ac89990f3e7997f1d89594523c66e76a)
 2015-09-03 13:50:14 -07:00
Xuan 1c6a287bf5 YARN-2900. Application (Attempt and Container) Not Found in AHS results 
in Internal Server Error (500). Contributed by Zhijie Shen and Mit Desai

(cherry picked from commit 06f8e9caba)
(cherry picked from commit 4fee8b3202)
(cherry picked from commit 6c7b625138ce3b262a8c8aa28077074b553638ed)
 2015-09-03 13:45:06 -07:00
Jing Zhao 46b9393cab HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. Contributed by Ming Ma. 
(cherry picked from commit 7817674a3a)
(cherry picked from commit 17fb442a4c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java

(cherry picked from commit fad2a062ddbb955a42dd5a90d64781617287f8df)
 2015-09-03 13:40:45 -07:00
cnauroth 563dbd29eb HDFS-8431. hdfs crypto class not found in Windows. Contributed by Anu Engineer. 
(cherry picked from commit 50eeea1300)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 25db341278)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 77a10e76e99c14cd26ebb3664304f6ed9cc7bf65)
 2015-09-03 12:59:35 -07:00
Zhijie Shen 42ce052585 YARN-3700. Made generic history service load a number of latest applications according to the parameter or the configuration. Contributed by Xuan Gong. 
(cherry picked from commit 54504133f4)
(cherry picked from commit 839f81a632)
(cherry picked from commit 058380d9ef35f35e8c624fb8783eac0904c4d1f5)
 2015-09-03 12:59:33 -07:00
Zhijie Shen 0f33fcd507 YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, attempts and containers. Contributed by Robert Kanter. 
(cherry picked from commit 3648cb57c9)
(cherry picked from commit 53d6c91df9)
 2015-09-03 12:58:42 -07:00
Kihwal Lee fb096b97cc HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. 
(cherry picked from commit 8860e352c3)

(cherry picked from commit 536b9ee6d6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java

(cherry picked from commit 2d5e60fa12a62463cd54f1b6b0fcb2ccdbd82c42)
 2015-09-03 12:32:49 -07:00
Kihwal Lee b3ca4dfaff HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. 
(cherry picked from commit 2d4ae3d18b)

(cherry picked from commit f264a5aeed)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java

(cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
 2015-09-03 12:19:21 -07:00
Xuan 7b1a71a7ad YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang 
(cherry picked from commit b0ad644083)
(cherry picked from commit 802676e1be)
(cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)
 2015-09-03 11:54:23 -07:00
Jason Lowe 778da79e6f YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du 
(cherry picked from commit 711d77cc54)

(cherry picked from commit a81ad81461)
(cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)
 2015-09-03 11:50:30 -07:00
Tsuyoshi Ozawa 6c0ad99b9d MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. 
(cherry picked from commit f4e2b3cc0b)
(cherry picked from commit fb5b0ebb45)
(cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)
 2015-09-03 11:46:47 -07:00
Colin Patrick Mccabe 51d5fc62c2 HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) 
(cherry picked from commit baf794dc40)

(cherry picked from commit 5f3d967aae)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm

(cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)
 2015-09-03 11:43:27 -07:00
Kihwal Lee bbed4e67a9 HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 6f622672b6)

(cherry picked from commit 802a5775f3)
(cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)
 2015-09-03 11:38:46 -07:00
Xiaoyu Yao 08006b84f8 HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) 
(cherry picked from commit 0100b15501)
(cherry picked from commit e68e8b3b5c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java

(cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)
 2015-09-02 15:31:46 -07:00
Jason Lowe 41601eae6b MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov 
(cherry picked from commit 7dc3c1203d)

(cherry picked from commit 87c2d915f1)
(cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)
 2015-09-02 15:27:47 -07:00
Gera Shegalov c90f025e90 HADOOP-11491. HarFs incorrectly declared as requiring an authority. (Brahma Reddy Battula via gera) 
(cherry picked from commit f343f8657e)
(cherry picked from commit 58970d69de)
 2015-09-02 15:16:43 -07:00
Vinod Kumar Vavilapalli 262275d561 MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. 
(cherry picked from commit 9fc32c5c4d)
(cherry picked from commit 32dc13d907)
(cherry picked from commit aad56fe3a2f29e73a013b9afa9b44b151a34e0f3)
 2015-09-02 15:12:21 -07:00
Karthik Kambatla 6ade6b5051 YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) 
(cherry picked from commit 47279c3228)
(cherry picked from commit 4045c41afe)
(cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930)
 2015-09-02 15:03:51 -07:00
Xuan 9af5b1dcd0 YARN-3024. LocalizerRunner should give DIE action when all resources are 
localized. Contributed by Chengbing Liu

(cherry picked from commit 0d6bd62102)
(cherry picked from commit a7696b3fbf)
(cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be)
 2015-09-02 14:52:06 -07:00
Colin Patrick Mccabe 8770f1f039 HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) 
(cherry picked from commit a0e0a63209)
(cherry picked from commit 788b76761d)
 2015-09-02 14:38:26 -07:00
Colin Patrick Mccabe 181281cfca HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) 
(cherry picked from commit a8898445dc)
(cherry picked from commit 3d0385c3c8)
 2015-09-02 14:37:51 -07:00
Colin Patrick Mccabe 9df4b7c10e HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 
(cherry picked from commit bc9cb3e271)
(cherry picked from commit c71b54fa32)
 2015-09-02 14:20:17 -07:00
Steve Loughran e0e93e9f8d HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 
(cherry picked from commit a6a5d1d6b5)
 2015-09-02 14:01:53 -07:00
Gera Shegalov 736ebeca33 HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up split calculation (gera) 
(cherry picked from commit 6d2cf9fbbd)
(cherry picked from commit 1544c63602)
 2015-09-02 11:46:47 -07:00
Robert Kanter c253ac68eb MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 
(cherry picked from commit d50e8f0928)
(cherry picked from commit 756c254293)
(cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)
 2015-09-02 11:40:32 -07:00
Karthik Kambatla 2f66dff7ce MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) 
(cherry picked from commit c66c3ac6bf)
(cherry picked from commit f4d6c5e337)
(cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)
 2015-09-02 11:32:08 -07:00
Wangda Tan e081593042 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
(cherry picked from commit 3316cd4357)
(cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)
 2015-09-02 11:28:22 -07:00
Wangda Tan 61f2ddb125 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
(cherry picked from commit e7cbecddc3)
(cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)
 2015-09-02 11:14:35 -07:00
Jing Zhao b7e7896410 HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 
(cherry picked from commit fddd55279d)
(cherry picked from commit 38b031d6ba)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java

(cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452)
 2015-09-02 11:06:04 -07:00
Akira Ajisaka dcc0d4658c MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9)
(cherry picked from commit 3de0bf9a35)
(cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b)
 2015-09-01 21:40:06 -07:00
Vinod Kumar Vavilapalli 752e3da738 YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e)
(cherry picked from commit 1ff3fd33ed)
(cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)
 2015-09-01 21:31:00 -07:00
Kihwal Lee 619f793846 HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee. 
(cherry picked from commit 285b31e75e)

(cherry picked from commit 7e622076d4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

(cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264)
 2015-09-01 21:24:53 -07:00
Arpit Agarwal 31d30e8111 HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) 
(cherry picked from commit f0324738c9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java

(cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b)
 2015-09-01 20:40:09 -07:00
Colin Patrick Mccabe c3f5ea11ec HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) 
(cherry picked from commit 28bebc81db)
(cherry picked from commit a827089905)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

(cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40)
 2015-09-01 20:15:20 -07:00
Junping Du 8a9665a586 MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. 
(cherry picked from commit eccb7d46ef)

(cherry picked from commit cacadea632)
(cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30)
 2015-09-01 20:10:20 -07:00
Kihwal Lee bc8728cd27 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
(cherry picked from commit 04ee18ed48)

(cherry picked from commit c4cedfc1d6)
(cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)
 2015-09-01 19:04:52 -07:00
Andrew Wang 0b1e66f01d HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 
(cherry picked from commit 50ee8f4e67)
(cherry picked from commit 2f46ee50bd)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)
 2015-09-01 18:30:18 -07:00
Tsz-Wo Nicholas Sze 9b105b78d4 HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the datanode. Contributed by Brahma Reddy Battula 
(cherry picked from commit fe693b72de)
(cherry picked from commit 4c648779d69a668f5147b183af2d40d45d1227d2)
 2015-09-01 18:25:02 -07:00
Xuan 914cc8f4a4 YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
(cherry picked from commit cbdcdfad6d)
(cherry picked from commit 61aafdcfa589cbae8363976c745ea528b03f152d)
 2015-09-01 18:14:51 -07:00
Wangda Tan 005d865494 YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7)
(cherry picked from commit 8e142d27cb)
(cherry picked from commit 4d43be3c01b1bc0deb31a9081fca5395d0eb4e0d)
 2015-09-01 17:10:42 -07:00
yliu 553efd7190 HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 
(cherry picked from commit 90164ffd84)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java

(cherry picked from commit 2c9a7461ec2ceba5885e95bc79f8dcbfd198df60)
 2015-09-01 17:06:37 -07:00
Colin Patrick Mccabe 81a445edf8 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 43b41f2241)
(cherry picked from commit 219eb22c15)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java

(cherry picked from commit 03798416bfe27383c52e4d9f632fe9fa168c6e95)
 2015-09-01 16:55:50 -07:00
Jing Zhao 49ef26b5ac HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 
Committed Ming Ma's 2.6 patch.

(cherry picked from commit 7f0bb5d3fe0db2e6b9354c8d8a1b603f2390184f)
 2015-09-01 16:33:30 -07:00
Jonathan Eagles a9bb641d51 YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 
(cherry picked from commit 8180e676ab)
(cherry picked from commit 44aedad5dd)
(cherry picked from commit f4bbf2c8f97d3601132504453f61e472950a433e)
 2015-09-01 16:04:19 -07:00
yliu 2230754f2a HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization. (Sean Busbey via yliu) 
(cherry picked from commit 813c93cb25)
 2015-09-01 15:42:37 -07:00
Colin Patrick Mccabe 9642a861e1 HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 5c1036d598)
(cherry picked from commit eefca23e8c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit c723f3b1bd9eab261ab5edca33c4dae5ce3d0d30)
 2015-09-01 15:35:08 -07:00
Colin Patrick Mccabe 5dbb0325df HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit a17584936c)
(cherry picked from commit 7779f38e68)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit 65ae3e2ff16ce1114a0115ff916837b0173b77f1)
 2015-09-01 15:29:59 -07:00
Zhijie Shen 9005b141a5 YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by Daryn Sharp and Jonathan Eagles. 
(cherry picked from commit d6e05c5ee2)
(cherry picked from commit a94d23762e)
(cherry picked from commit 68e07eb50b872ec8a78923df8f5f640f08a72aa2)
 2015-09-01 15:24:36 -07:00