13282 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
94bc5cdbb3 HDFS-12042. Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption. Contributed by Misha Dmitriev.
(cherry picked from commit bcba844d1144cc334e2babbc34c9d42eac1c203a)
2017-06-30 11:42:51 -07:00
Mingliang Liu
2d69952925 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
(cherry picked from commit 72993b33b704991f2a0bf743f31b164e58a2dabc)
2017-06-29 17:08:51 -07:00
Daniel Templeton
93c5421a43 YARN-6751. Display reserved resources in web UI per queue
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit ec975197799417a1d5727dedc395fe6c15c30eb2)
2017-06-29 16:57:23 -07:00
Daniel Templeton
367f525e81 HADOOP-14611. NetworkTopology.DEFAULT_HOST_LEVEL is unused
(Contributed by Chen Liang via Daniel Templeton)

(cherry picked from commit 5a75f73893567151f525950cc1a15b3f1bfeac26)
2017-06-29 16:56:55 -07:00
John Zhuge
1272ee153b HADOOP-14024. KMS JMX endpoint throws ClassNotFoundException. Contributed by John Zhuge. 2017-06-29 15:54:30 -07:00
Wei-Chiu Chuang
6917c19167 HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy.
(cherry picked from commit 16c8dbde574f49827fde5ee9add1861ee65d4645)
2017-06-29 06:40:24 -07:00
Daniel Templeton
15242786b6 YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description needs spaces in yarn-default.xml
(Contributed by Lori Loberg via Daniel Templeton)

(cherry picked from commit 25d891a784304fcf02f57bc7984c31af45003553)
2017-06-28 17:17:56 -07:00
Junping Du
c5f43feb7c YARN-5311. Document graceful decommission CLI and usage. Contributed by Elek, Marton.
(cherry picked from commit 4e3eebc943835077e3dd0df9e0b9239ae604cb89)
2017-06-28 15:32:49 -07:00
Mingliang Liu
ba76897392 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
(cherry picked from commit 990aa34de23c625163745ebc338483065d955bbe)
2017-06-28 14:20:44 -07:00
Naganarasimha
a5ae7c0cae YARN-6467. CSQueueMetrics needs to update the current metrics for default partition only. Contributed by Manikandan R. 2017-06-29 00:09:30 +05:30
Steve Loughran
44d43a8848
HADOOP-14190. Add more on S3 regions to the s3a documentation.
Contributed by Steve Loughran

(cherry picked from commit ee243e5289212aa2912d191035802ea023367e19)
2017-06-28 10:26:05 +01:00
Xiao Chen
7e66e133dd HADOOP-14515. Addendum. Specifically configure zookeeper-related log levels in KMS log4j.
(cherry picked from commit aff7a3f8eddd55501366a43d2f44f531c6e8ef48)
2017-06-27 23:49:16 -07:00
Akira Ajisaka
2a5a313539
MAPREDUCE-6697. Concurrent task limits should only be applied when necessary. Contributed by Nathan Roberts.
(cherry picked from commit a5c0476a990ec1e7eb34ce2462a45aa52cc1350d)
2017-06-28 10:51:23 +09:00
Mingliang Liu
ce57458ec5 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
(cherry picked from commit 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)
2017-06-27 17:32:56 -07:00
Mingliang Liu
1b1065a163 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
(cherry picked from commit 686a634f01e454537d1a9d531330e60a60233ca4)
2017-06-27 16:49:48 -07:00
Jason Lowe
44b3c6ffaa YARN-6738. LevelDBCacheTimelineStore should reuse ObjectMapper instances. Contributed by Zoltan Haindrich
(cherry picked from commit 63ce1593c5b78eb172773e7498d9c321debe81e8)
2017-06-27 17:13:49 -05:00
Mingliang Liu
ea402c364f HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
(cherry picked from commit bc4dfe9c9cb31f39e8f1532d5c5837faf2e92bde)
2017-06-27 13:20:03 -07:00
Steve Loughran
15d88b77d1
HADOOP-14536. Update azure-storage sdk to version 5.3.0
Contributed by Georgi Chalakov
2017-06-27 15:23:06 +01:00
Akira Ajisaka
a896c0f11e
HDFS-12040. TestFsDatasetImpl.testCleanShutdownOfVolume fails. Contributed by hu xiaodong.
(cherry picked from commit 07defa4c09db6e8c552167019ca0d2444cfc8fe7)
2017-06-27 20:49:56 +09:00
Steve Loughran
3a87591df8 HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu.
(cherry picked from commit 48f4a229a93b65df9e99df2505719dd3538f1188)
2017-06-26 15:22:44 -07:00
Ravi Prakash
ef491f4637 HDFS-11993. Add log info when connect to datanode socket address failed. Contributed by chencan
(cherry picked from commit a9d3412b4ce40f5ab5a18756ede7e0606b653171)
2017-06-26 13:25:32 -07:00
Akira Ajisaka
b59b96deee
HDFS-11992. Replace commons-logging APIs with slf4j in FsDatasetImpl. Contributed by hu xiaodong. 2017-06-27 03:28:09 +09:00
Akira Ajisaka
239b382153
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he.
(cherry picked from commit a594f96fb1155d94fe08ed8f85dca789a9a4c790)
2017-06-26 16:41:44 +09:00
Naganarasimha
92e266c2b6 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-24 23:14:51 +05:30
Arpit Agarwal
4b420e0678 HADOOP-14543. ZKFC should use getAversion() while setting the zkacl. Contributed by Brahma Reddy Battula. 2017-06-23 16:50:56 -07:00
Steve Loughran
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
Contributed by Thomas.

(cherry picked from commit c6bd73c6c5760c3a52203e6a51628587ceec6896)
2017-06-23 21:40:05 +01:00
Yufei Gu
4c59b446e2 YARN-5876. TestResourceTrackerService#testGracefulDecommissionWithApp fails intermittently on trunk. (Robert Kanter via Yufei Gu)
(cherry picked from commit 0b77262890d76b0a3a35fa64befc8a406bc70b27)
2017-06-23 13:30:20 -07:00
Kihwal Lee
0719345909 Addendum for HADOOP-14146. Use commons Base64 instead of jdk8 Base64. 2017-06-23 13:26:29 -05:00
Kihwal Lee
4c6184b41c Addendum for HADOOP-14146 jdk7 support. 2017-06-23 09:57:51 -05:00
Steve Loughran
5858bc70a2
HADOOP-14568. GenericTestUtils#waitFor missing parameter verification.
Contributed by Yiqun Lin

(cherry picked from commit e5db9af8a00ded73ee1c75bb4c0d5d50e15a305c)
2017-06-23 14:41:08 +01:00
Arpit Agarwal
f9f6ef83eb HDFS-12010. TestCopyPreserveFlag fails consistently because of mismatch in access time. Contributed by Mukul Kumar Singh. 2017-06-22 15:43:00 -07:00
Arun Suresh
66f780e6ab YARN-6127. Add support for work preserving NM restart when AMRMProxy is enabled. (Botong Huang via asuresh). 2017-06-22 15:07:38 -07:00
Akira Ajisaka
897d5a4d24
HADOOP-14542. Add IOUtils.cleanupWithLogger that accepts slf4j logger API. Contributed by Chen Liang.
(cherry picked from commit b64951905e64f6fed581c28634be6ed15c190633)
2017-06-22 17:43:33 +09:00
Daryn Sharp
d4115d71b5 HADOOP-14146. KerberosAuthenticationHandler should authenticate with SPN in AP-REQ. Contributed by Daryn Sharp 2017-06-21 11:05:12 -05:00
Yongjun Zhang
ba487ead42 HADOOP-14429. FTPFileSystem#getFsAction always returns FsAction.NONE. (Hongyuan Li via Yongjun Zhang)
(cherry picked from commit 5157f6c46ec342fb650b3c5853061ed1e4a182b6)
2017-06-20 17:07:01 -07:00
Kihwal Lee
8f533ca76b HDFS-11960. Successfully closed files can stay under-replicated. Contributed by Kihwal Lee. 2017-06-20 09:16:37 -05:00
Brahma Reddy Battula
933fad2e17 HADOOP-14533. Size of args cannot be less than zero in TraceAdmin#run as its linkedlist. Contributed by Weisen Han.
(cherry picked from commit 2e9daa2e27536484846d9c5f020c6afaf27932ce)
2017-06-20 20:45:33 +08:00
Akira Ajisaka
35f6c7c9c7
YARN-6713. Fix dead link in the Javadoc of FairSchedulerEventLog.java. Contributed by Weiwei Yang.
(cherry picked from commit 099dfe92a9743bcb7e27506f8877b8b23b0ff95a)
2017-06-20 17:23:06 +09:00
Akira Ajisaka
6b3c13d543
HDFS-11345. Document the configuration key for FSNamesystem lock fairness. Contributed by Erik Krogen.
(cherry picked from commit 2c769167dbdb66c52d2ba7b7193a686444085570)
2017-06-20 16:36:47 +09:00
Xiao Chen
b9bd72c676 HADOOP-14515. Specifically configure zookeeper-related log levels in KMS log4j.
(cherry picked from commit 4cdae5d88ef7851ae284a115868dc4bbe24ae1bd)
2017-06-19 23:12:28 -07:00
Akira Ajisaka
71626fd0b9
HDFS-11736. OIV tests should not write outside 'target' directory. Contributed by Yiqun Lin. 2017-06-20 13:55:42 +09:00
Akira Ajisaka
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:21:12 +09:00
Masatake Iwasaki
b19ca498ae HDFS-11995. HDFS Architecture documentation incorrectly describes writing to a local temporary file. Contributed by Nandakumar.
(cherry picked from commit d954a64730c00346476322743462cde857164177)
2017-06-20 08:09:09 +09:00
Naganarasimha
578318473c YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp
(cherry picked from commit ee89ac84e68d3e181b75c63f74a0444f9d28146f)
2017-06-19 22:45:21 +05:30
Andrew Wang
1ec1aebd35 HDFS-10480. Add an admin command to list currently open files. Contributed by Manoj Govindassamy. 2017-06-19 09:52:07 -07:00
Akira Ajisaka
6d82db40d5
HADOOP-14540. Replace MRv1 specific terms in HostsFileReader. Contributed by hu xiaodong.
(cherry picked from commit 300804595e5b8fd447ecb530fcd0ae5b0e59bb63)
2017-06-19 19:40:19 +09:00
Akira Ajisaka
2d0b6ab65d
HADOOP-14538. Fix TestFilterFileSystem and TestHarFileSystem failures after DistributedFileSystem.append API. Contributed by Lei (Eddy) Xu.
(cherry picked from commit 7ade5124b8b6c52a084ec187c531017eee0f1884)
2017-06-19 19:10:13 +09:00
Naganarasimha
ac058c75a4 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:41:13 +05:30
Masatake Iwasaki
023dcdc73d HADOOP-14424. Add CRC32C performance test. Contributed by LiXin Ge.
(cherry picked from commit 75043d342935828cfe8bdabfab8784c83f797887)
2017-06-19 09:18:02 +09:00
Naganarasimha
3106f256e2 YARN-6517. Fix warnings from Spotbugs in hadoop-yarn-common(addendum). Contributed by Weiwei Yang.
(cherry picked from commit 7582dedad1c73eabdc3eeece0a3a860e7bb33c1a)
2017-06-18 19:56:11 +05:30