Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,772 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7772 Commits

Author SHA1 Message Date
Robert Kanter c253ac68eb MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 
(cherry picked from commit d50e8f0928)
(cherry picked from commit 756c254293)
(cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)
 2015-09-02 11:40:32 -07:00
Karthik Kambatla 2f66dff7ce MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) 
(cherry picked from commit c66c3ac6bf)
(cherry picked from commit f4d6c5e337)
(cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)
 2015-09-02 11:32:08 -07:00
Wangda Tan e081593042 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
(cherry picked from commit 3316cd4357)
(cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)
 2015-09-02 11:28:22 -07:00
Wangda Tan 61f2ddb125 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
(cherry picked from commit e7cbecddc3)
(cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)
 2015-09-02 11:14:35 -07:00
Jing Zhao b7e7896410 HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 
(cherry picked from commit fddd55279d)
(cherry picked from commit 38b031d6ba)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java

(cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452)
 2015-09-02 11:06:04 -07:00
Akira Ajisaka dcc0d4658c MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9)
(cherry picked from commit 3de0bf9a35)
(cherry picked from commit c3b2b377f19293797a88f4636f07e2ab8e18c78b)
 2015-09-01 21:40:06 -07:00
Vinod Kumar Vavilapalli 752e3da738 YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e)
(cherry picked from commit 1ff3fd33ed)
(cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)
 2015-09-01 21:31:00 -07:00
Kihwal Lee 619f793846 HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee. 
(cherry picked from commit 285b31e75e)

(cherry picked from commit 7e622076d4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

(cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264)
 2015-09-01 21:24:53 -07:00
Arpit Agarwal 31d30e8111 HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) 
(cherry picked from commit f0324738c9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java

(cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b)
 2015-09-01 20:40:09 -07:00
Colin Patrick Mccabe c3f5ea11ec HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) 
(cherry picked from commit 28bebc81db)
(cherry picked from commit a827089905)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

(cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40)
 2015-09-01 20:15:20 -07:00
Junping Du 8a9665a586 MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. 
(cherry picked from commit eccb7d46ef)

(cherry picked from commit cacadea632)
(cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30)
 2015-09-01 20:10:20 -07:00
Kihwal Lee bc8728cd27 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
(cherry picked from commit 04ee18ed48)

(cherry picked from commit c4cedfc1d6)
(cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)
 2015-09-01 19:04:52 -07:00
Andrew Wang 0b1e66f01d HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 
(cherry picked from commit 50ee8f4e67)
(cherry picked from commit 2f46ee50bd)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)
 2015-09-01 18:30:18 -07:00
Tsz-Wo Nicholas Sze 9b105b78d4 HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the datanode. Contributed by Brahma Reddy Battula 
(cherry picked from commit fe693b72de)
(cherry picked from commit 4c648779d69a668f5147b183af2d40d45d1227d2)
 2015-09-01 18:25:02 -07:00
Xuan 914cc8f4a4 YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
(cherry picked from commit cbdcdfad6d)
(cherry picked from commit 61aafdcfa589cbae8363976c745ea528b03f152d)
 2015-09-01 18:14:51 -07:00
Wangda Tan 005d865494 YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7)
(cherry picked from commit 8e142d27cb)
(cherry picked from commit 4d43be3c01b1bc0deb31a9081fca5395d0eb4e0d)
 2015-09-01 17:10:42 -07:00
yliu 553efd7190 HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 
(cherry picked from commit 90164ffd84)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java

(cherry picked from commit 2c9a7461ec2ceba5885e95bc79f8dcbfd198df60)
 2015-09-01 17:06:37 -07:00
Colin Patrick Mccabe 81a445edf8 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 43b41f2241)
(cherry picked from commit 219eb22c15)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java

(cherry picked from commit 03798416bfe27383c52e4d9f632fe9fa168c6e95)
 2015-09-01 16:55:50 -07:00
Jing Zhao 49ef26b5ac HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 
Committed Ming Ma's 2.6 patch.

(cherry picked from commit 7f0bb5d3fe0db2e6b9354c8d8a1b603f2390184f)
 2015-09-01 16:33:30 -07:00
Jonathan Eagles a9bb641d51 YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 
(cherry picked from commit 8180e676ab)
(cherry picked from commit 44aedad5dd)
(cherry picked from commit f4bbf2c8f97d3601132504453f61e472950a433e)
 2015-09-01 16:04:19 -07:00
yliu 2230754f2a HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization. (Sean Busbey via yliu) 
(cherry picked from commit 813c93cb25)
 2015-09-01 15:42:37 -07:00
Colin Patrick Mccabe 9642a861e1 HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 5c1036d598)
(cherry picked from commit eefca23e8c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit c723f3b1bd9eab261ab5edca33c4dae5ce3d0d30)
 2015-09-01 15:35:08 -07:00
Colin Patrick Mccabe 5dbb0325df HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit a17584936c)
(cherry picked from commit 7779f38e68)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit 65ae3e2ff16ce1114a0115ff916837b0173b77f1)
 2015-09-01 15:29:59 -07:00
Zhijie Shen 9005b141a5 YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by Daryn Sharp and Jonathan Eagles. 
(cherry picked from commit d6e05c5ee2)
(cherry picked from commit a94d23762e)
(cherry picked from commit 68e07eb50b872ec8a78923df8f5f640f08a72aa2)
 2015-09-01 15:24:36 -07:00
Xuan a57ada6c1f YARN-3227. Timeline renew delegation token fails when RM user's TGT is 
expired. Contributed by Zhijie Shen

(cherry picked from commit d1abc5d4fc)
(cherry picked from commit 56c2050ab7)
(cherry picked from commit 780a9b1a98827a692e0ea9fbc92f9d1ab979e3e0)
 2015-09-01 15:21:46 -07:00
Jing Zhao b25491dc45 HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit 24db0812be)
(cherry picked from commit 994dadb9ba)
(cherry picked from commit 0bc5c6495a7feb4365af0ce5fe48fc87b7e1749f)
 2015-09-01 15:17:57 -07:00
Jian He 7ffdf7d105 YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan Gong 
(cherry picked from commit 95bfd087dc)

(cherry picked from commit a5f3fb4dc1)
(cherry picked from commit 27a2f0acb84202cc082090eef7eea57f6e42f9bb)
 2015-09-01 15:12:53 -07:00
Tsuyoshi Ozawa 81417f7572 YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. Contributed by Ryu Kobayashi. 
(cherry picked from commit 1b67209612)
(cherry picked from commit 6660c2f83b)
(cherry picked from commit 6ea859e435e7cd6bc342f67e1551ccb86fbd976f)
 2015-09-01 14:37:21 -07:00
Wangda Tan 8b5bdac98e YARN-3230. Clarify application states on the web UI. (Jian He via wangda) 
(cherry picked from commit ce5bf927c3)
(cherry picked from commit a1963968d2)
(cherry picked from commit 591e261ccf1fb5dd25e87665c8d5c0341ff6fb24)
 2015-09-01 14:34:07 -07:00
yliu bb1b87f3b8 HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static. (Sean Busbey via yliu) 
(cherry picked from commit b569c3ab1c)
 2015-09-01 14:28:19 -07:00
Karthik Kambatla 5a6755cc0f YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client. (Zhihai Xu via kasha) 
(cherry picked from commit 8d88691d16)
(cherry picked from commit 0d62e94887)
(cherry picked from commit 4a5b0e708d42fbff571229a43d1762d1767e2db5)
 2015-09-01 14:06:34 -07:00
Karthik Kambatla dbc5bab9fd YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi Li via kasha) 
(cherry picked from commit 22426a1c9f)
(cherry picked from commit 721d7b5741)
(cherry picked from commit 5dfa25f22a989222e8b3d1013117b3350a48b2c5)
 2015-09-01 13:54:04 -07:00
Vinod Kumar Vavilapalli 2692401dfb YARN-3222. Added the missing CHANGES.txt entry. 
(cherry picked from commit 4620767156)
(cherry picked from commit 015696fb81)
 2015-09-01 13:49:04 -07:00
Jian He db92b09e03 YARN-3222. Fixed NPE on RMNodeImpl#ReconnectNodeTransition when a node is reconnected with a different port. Contributed by Rohith Sharmaks 
(cherry picked from commit b2f1ec312e)

(cherry picked from commit 888a445638)
(cherry picked from commit b78f87825bd593e30b2f2ea76f37c7a4fd673ab2)
 2015-09-01 13:39:35 -07:00
Jing Zhao 6e090bc53d HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. 
(cherry picked from commit b442aeec95)
(cherry picked from commit 6090f51725)
(cherry picked from commit e1af1ac4e91d36b21df18ce5627e1f69f27f0776)
 2015-09-01 13:36:21 -07:00
Jason Lowe a4b8897b30 YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He 
(cherry picked from commit 1a68fc4346)

(cherry picked from commit 257087417e)
(cherry picked from commit 49468108c203bf093acdc93c1798d90c480c3a17)
 2015-09-01 13:32:21 -07:00
Andrew Wang 406ec495bb HDFS-7763. fix zkfc hung issue due to not catching exception in a corner case. Contributed by Liang Xie. 
(cherry picked from commit 7105ebaa9f)
(cherry picked from commit efb7e287f4)
(cherry picked from commit fd70e4db105e140fc3d60042abb3f598c9afd13f)
 2015-09-01 11:30:04 -07:00
cnauroth be4d4c9cbf HDFS-7009. Active NN and standby NN have different live nodes. Contributed by Ming Ma. 
(cherry picked from commit 769507bd7a)
(cherry picked from commit 657a6e389b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java

(cherry picked from commit d5ddc3450f2f49ea411de590ff3de15b5ec4e17c)
 2015-09-01 11:23:26 -07:00
Xuan 4fcf71c1e7 YARN-3238. Connection timeouts to nodemanagers are retried at multiple 
levels. Contributed by Jason Lowe

(cherry picked from commit 92d67ace32)
(cherry picked from commit fefeba4ac8)
(cherry picked from commit d8f02e1c5c3bcc230d942554b2f4cfbc3ed21526)
 2015-09-01 11:19:37 -07:00
cnauroth 342504f790 HADOOP-11604. Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. Contributed by Chris Nauroth. 
(cherry picked from commit 3c5ff0759c)
(cherry picked from commit 187e081d5a)
 2015-09-01 11:10:41 -07:00
Kihwal Lee 02e0b6e306 HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah. 
(cherry picked from commit 7ae5255a16)

(cherry picked from commit b9157f92fc)

Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/site/resources/image-with-zero-block-size.tar.gz
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java

(cherry picked from commit 1faa44d8f4d7b944e99dd0470ea2638c7653a131)
 2015-08-31 18:32:00 -07:00
Kihwal Lee 4ec7b6174d HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is full. Contributed by Ming Ma. 
(cherry picked from commit 685af8a3d0)
(cherry picked from commit 6c01e58619)
 2015-08-31 18:10:00 -07:00
Xuan 95edb6e64f YARN-3207. Secondary filter matches entites which do not have the key 
being filtered for. Contributed by Zhijie Shen

(cherry picked from commit 57db50cbe3)
(cherry picked from commit ba18adbb27)
(cherry picked from commit 9fd18e94849600ec66832df5ae424eeb0116330c)
 2015-08-31 17:44:42 -07:00
Zhijie Shen 28160a0bd6 YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K. 
(cherry picked from commit d5855c0e46)
(cherry picked from commit fd75b8c9ca)
(cherry picked from commit a62891971380e5f8e4a645ed36bd88aa6fe0e47a)
 2015-08-31 17:38:51 -07:00
Vinayakumar B 6154abfeda HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.(Contributed by Vinayakumar B) 
(cherry picked from commit 3d15728ff5)
(cherry picked from commit a1bf7aecf7)
(cherry picked from commit c1e65de57e8ef760586e28cd37397ea9a7ac7944)
 2015-08-31 17:26:28 -07:00
Jian He a703952d39 YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong 
(cherry picked from commit 0af6a99a3f)

(cherry picked from commit 6146680955)
(cherry picked from commit ab654746fbad2da12b24b13425dc9bf17c46b50c)
 2015-08-31 17:17:47 -07:00
Gera Shegalov bb8a1551c3 HADOOP-11506. Configuration variable expansion regex expensive for long values. (Gera Shegalov via gera) 
(cherry picked from commit 644548f201)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java

(cherry picked from commit 3d36d4737c)
 2015-08-31 16:00:32 -07:00
Kihwal Lee a1c3e6a058 HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang 
(cherry picked from commit 843806d03a)

(cherry picked from commit c428d303f6)
(cherry picked from commit 21d8b22feef2fc9f6793d9c157391b142368e588)
 2015-08-31 15:55:07 -07:00
Jian He e917ef3cfe MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly. Contributed by Jason Lowe 
(cherry picked from commit cff05bff1f)

(cherry picked from commit 43b3b43cea)
(cherry picked from commit 013e271036554020e3d26de4fcbb5b6970f5a9c0)
 2015-08-31 15:49:24 -07:00
Jian He 8658945b3a YARN-3103. AMRMClientImpl does not update AMRM token properly. Contributed by Jason Lowe 
(cherry picked from commit 6d2bdbd7da)

(cherry picked from commit 12522fd9cb)
(cherry picked from commit f50f5ad49d3b70448647384fc5f020214cb58f10)
 2015-08-31 15:42:03 -07:00