13488 Commits

Author SHA1 Message Date
Yufei Gu
4bd417a056 YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu)
(cherry picked from commit 8991f0baec62625c45144e2544066195800ab95b)
2017-08-18 14:45:13 -07:00
Daniel Templeton
78f9dca2cb YARN-6952. Enable scheduling monitor in FS. (Yufei Gu via Daniel Templeton) 2017-08-18 14:40:30 -07:00
Arpit Agarwal
d8d2a221f8 HDFS-11450. HDFS specific network topology classes with storage type info included. Contributed by Chen Liang. 2017-08-18 11:43:17 -07:00
Arpit Agarwal
850593133b Revert "HADOOP-14732. ProtobufRpcEngine should use Time.monotonicNow to measure durations. Contributed by Hanisha Koneru."
This reverts commit bdc425057ebedb8f0988ce4dae18fdcd67c8523f.
2017-08-18 10:16:13 -07:00
bibinchundatt
ecf1a029bb YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su.
(cherry picked from commit e05fa3451db343c0d22496b332910874b6be5b7f)
2017-08-18 20:31:26 +05:30
Steve Loughran
c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted.
Contributed by Thomas Marquardt

(cherry picked from commit c6b4e656b76b68cc1d0dbcc15a5aa5ea23335b7b)
2017-08-18 14:38:04 +01:00
Manoj Govindassamy
a5bc30170c HDFS-12316. Verify HDFS snapshot deletion doesn't crash the ongoing file writes.
(cherry picked from commit 4230872dd66d748172903b1522885b03f34bbf9b)
2017-08-17 16:25:15 -07:00
Sunil G
5511cea86b YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad.
(cherry picked from commit f9a0e2338150f1bd3ba2c29f76979183fd3ed80c)
2017-08-17 15:10:07 +05:30
Naganarasimha
fa3094e436 YARN-6327. Removing queues from CapacitySchedulerQueueManager and ParentQueue should be done with iterator. Contributed by Jonathan Hung.
(cherry picked from commit 0a3aa40fe7878c939dbf4e6b43466595159ff930)
2017-08-17 10:42:00 +05:30
Robert Kanter
9de332d502 MAPREDUCE-6936. Remove unnecessary dependency of hadoop-yarn-server-common from hadoop-mapreduce-client-common (haibochen via rkanter)
(cherry picked from commit ab051bd42ee1d7c4d3b7cc71e6b2734a0955e767)
2017-08-16 16:15:48 -07:00
Mingliang Liu
03d15fa58e HDFS-11430. Separate class InnerNode from class NetworkTopology and make it extendable. Contributed by Tsz Wo Nicholas Sze 2017-08-16 16:06:13 -07:00
Jason Lowe
107e1722d4 MAPREDUCE-6940. Copy-paste error in the TaskAttemptUnsuccessfulCompletionEvent constructor. Contributed by Oleg Danilov 2017-08-16 16:35:41 -05:00
Jason Lowe
cf30380d46 YARN-7020. TestAMRMProxy#testAMRMProxyTokenRenewal is flakey. Contributed by Robert Kanter
(cherry picked from commit 14553061be0a341df3e628dcaf06717b4630b05e)
2017-08-16 13:06:13 -05:00
Steve Loughran
d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded.
Contributed by Thomas Marquardt.
2017-08-16 13:49:47 +01:00
bibinchundatt
67f9968407 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-16 14:52:09 +05:30
Akira Ajisaka
a28a3dc8f2
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka.
(cherry picked from commit 588c190afd49bdbd5708f7805bf6c68f09fee142)
2017-08-16 14:07:08 +09:00
Subru Krishnan
ad46a90872 HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru).
(cherry picked from commit 75dd866bfb8b63cb9f13179d4365b05c48e0907d)
2017-08-15 16:54:48 -07:00
Ravi Prakash
895a35eac0 HDFS-12301. NN File Browser UI: Navigate to a path when enter is pressed
(cherry picked from commit f34646d652310442cb5339aabbbb269f10dfa838)
2017-08-15 15:45:37 -07:00
Steve Loughran
b30522c59e
HADOOP-14662. Update azure-storage sdk to version 5.4.0.
Contributed by Thomas Marquard
2017-08-15 16:27:31 +01:00
Arun Suresh
7b22df3da6 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh)
(cherry picked from commit 4d7be1d8575e9254c59d41460960708e3718503a)
2017-08-14 19:57:34 -07:00
Arpit Agarwal
bdc425057e HADOOP-14732. ProtobufRpcEngine should use Time.monotonicNow to measure durations. Contributed by Hanisha Koneru. 2017-08-14 15:53:43 -07:00
Jason Lowe
56d0e757c7 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne
(cherry picked from commit 5558792894169425bff054364a1ab4c48b347fb9)
2017-08-14 15:34:50 -05:00
Jason Lowe
e5ba40cc98 YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles
(cherry picked from commit 3325ef653d6f364a82dd32485d9ef6d987380ce3)
2017-08-14 14:41:38 -05:00
Subru Krishnan
a70efb6138 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-14 11:03:50 -07:00
Carlo Curino
394573780b YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino)
(cherry picked from commit 28d97b79b69bb2be02d9320105e155eeed6f9e78)
2017-08-14 11:01:56 -07:00
Jian He
630ada1b4e YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:56:39 -07:00
John Zhuge
0efc590e6d HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria.
(cherry picked from commit 7769e9614956283a86eda9e4e69aaa592c0ca960)
2017-08-13 00:27:33 -07:00
John Zhuge
ed4d6aa2c1 HDFS-11303. Hedged read might hang infinitely if read data from all DN failed . Contributed by Chen Zhang, Wei-chiu Chuang, and John Zhuge.
(cherry picked from commit 8b242f09a61a7536d2422546bfa6c2aaf1d57ed6)
2017-08-11 21:55:23 -07:00
Daniel Templeton
7d3cea232d Revert "YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator"
This reverts commit ce0cdc50d9700118139e7907dede274d2db93fda.
2017-08-11 20:52:07 -07:00
Daniel Templeton
ce0cdc50d9 YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator
(Contributed by Larry Lo via Daniel Templeton)

(cherry picked from commit 0996acde6c325667aa19ae0740eb6b40bf4a682a)
2017-08-11 14:51:42 -07:00
Daniel Templeton
11e3673748 YARN-6967. Limit application attempt's diagnostic message size thoroughly
(Contributed by Chengbing Liu via Daniel Templeton)

(cherry picked from commit 65364defb4a633ca20b39ebc38cd9c0db63a5835)
2017-08-11 14:31:11 -07:00
Daniel Templeton
2566e5fcf7 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces
(Contributed by weiyuan via Daniel Templeton)

(cherry picked from commit c7680d4cc4d9302a5b5efcf2467bd32ecea99585)
2017-08-11 14:23:17 -07:00
Karthik Kambatla
f6c74ac5ea YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha)
(cherry picked from commit 8c4b6d16a526610a03ccc85665744ad071e37400)
2017-08-11 12:18:29 -07:00
John Zhuge
e208a4e7ee HADOOP-14260. Configuration.dumpConfiguration should redact sensitive information. Contributed by John Zhuge.
(cherry picked from commit 582648befaf9908159f937d2cc8f549583a3483e)
2017-08-11 10:54:41 -07:00
John Zhuge
76c4aee2e3 HADOOP-13628. Support to retrieve specific property from configuration via REST API. Contributed by Weiwei Yang
(cherry picked from commit 00160f71b6d98244fcb1cb58b2db9fc24f1cd672)
2017-08-11 10:54:30 -07:00
John Zhuge
9ba3268fa3 HADOOP-13588. ConfServlet should respect Accept request header. Contributed by Weiwei Yang
(cherry picked from commit 59d59667a8b1d3fb4a744a41774b2397fd91cbb3)
2017-08-11 10:54:25 -07:00
Steve Loughran
9c6e8ece98
HADOOP-14754. TestCommonConfigurationFields failed: core-default.xml has 2 wasb properties missing in classes.
Contributed by John Zhuge.

(cherry picked from commit d964062f66c0772f4b1a029bfcdff921fbaaf91c)
2017-08-11 10:58:26 +01:00
Wei-Chiu Chuang
304cc1a856 HADOOP-14743. CompositeGroupsMapping should not swallow exceptions. Contributed by Wei-Chiu Chuang.
(cherry picked from commit a8b75466b21edfe8b12beb4420492817f0e03147)
2017-08-10 09:36:21 -07:00
Steve Loughran
96befde480
HADOOP-14183. Remove service loader config file for wasb fs.
Contributed by Esfandiar Manii.

(cherry picked from commit 54356b1e8366a23fff1bb45601efffc743306efc)
2017-08-10 16:49:51 +01:00
Steve Loughran
bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing.
Contributed by Esfandiar Manii.
2017-08-10 16:49:19 +01:00
Akira Ajisaka
1f80e05eb4
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-10 22:14:44 +09:00
Ravi Prakash
c556cdaaf4 MAPREDUCE-6923. Optimize MapReduce Shuffle I/O for small partitions. Contributed by Robert Schmidtke.
(cherry picked from commit ac7d0604bc73c0925eff240ad9837e14719d57b7)
2017-08-09 15:40:57 -07:00
Kihwal Lee
306abf4980 HDFS-12278. LeaseManager operations are inefficient in 2.8. Contributed by Rushabh S Shah.
(cherry picked from commit b5c02f95b5a2fcb8931d4a86f8192caa18009ea9)
2017-08-09 16:48:55 -05:00
Jian He
8b7590057e YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-08-09 11:12:38 -07:00
Kihwal Lee
74bcc8d1f2 HDFS-12157. Do fsyncDirectory(..) outside of FSDataset lock. Contributed by Vinayakumar B. 2017-08-09 09:31:53 -05:00
Haohui Mai
3e2753daa1 HADOOP-14598. Blacklist Http/HttpsFileSystem in FsUrlStreamHandlerFactory. Contributed by Steve Loughran. 2017-08-08 16:33:54 -07:00
Haohui Mai
21d76f710e HDFS-10326. Disable setting tcp socket send/receive buffers for write pipelines. Contributed by Daryn Sharp. 2017-08-08 14:58:44 -07:00
Subru Krishnan
e9ec1e8f87 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-08-08 14:01:22 -07:00
Daniel Templeton
c5b3d80a4b YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code
(Contributed by Angela Wang via Daniel Templeton)

(cherry picked from commit e0c24145d2c2a7d2cf10864fb4800cb1dcbc2977)
2017-08-08 13:36:49 -07:00
Jason Lowe
50dfef7275 MAPREDUCE-6927. MR job should only set tracking url if history was successfully written. Contributed by Eric Badger
(cherry picked from commit 735fce5bec17f4e1799daf922625c475cf588114)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
2017-08-08 14:55:55 -05:00