Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,026 Commits 326 Branches 371 Tags
Commit Graph

18026 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Inigo Goiri
ed59412434 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:26:32 -08:00
Inigo Goiri
380fefc40e Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 0b75c5238b468e0a486a7b7b0d3cf9fc74bc7b22.
 2019-12-20 11:28:17 -08:00
Inigo Goiri
0b75c5238b HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6d95e799542d6409dac30d0418ce8a8)
 2019-12-19 09:48:02 -08:00
Mingliang Liu
109a2f513f
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:55:29 -08:00
Mingliang Liu
cb63c3cc0d
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:53:04 -08:00
Lukas Majercak
049d061001 HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log. 
MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.

(cherry picked from commit c73334a924d2009d136a8ee62278aaac53418a24)
 2019-11-15 13:27:36 -08:00
Duo Zhang
322cef7762 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:57:26 -07:00
Eric Badger
5196b035f7 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c83b9752e772ece7a716a33045752bf)
 2019-10-28 18:17:47 +00:00
Eric Badger
fdd5ce1720 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:18:16 +00:00
Xiaoyu Yao
0b80308b78
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao
121c6cb59e
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 5369a3bd94ee6b49d3aacdee1ca238be0c25e370.
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao
5369a3bd94
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:44:29 -07:00
Eric E Payne
97ee0af90e YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:16:38 +00:00
Jonathan Hung
c98f7cef9f YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
(cherry picked from commit 6baa0d1e4d78c15833257d98bdd2b38806faed9f)
 2019-08-26 16:04:57 -07:00
Shashikant Banerjee
5b064381aa HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ab7509efe7375949db3b819b53e9461624cd2b1e)
 2019-08-19 10:07:16 -07:00
Surendra Singh Lilhore
0cb3738567 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a9812023754ed1b3e5b428e161f0add50)
(cherry picked from commit 6b01effd011ab1f88bdbb9f2dbf9bbb0765984e6)
 2019-08-18 18:18:02 -07:00
Christopher Gregorian
ccb41000f1 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628d370def74e56112aba3a93e97bbf70)
(cherry picked from e4b650f91e7353f98fee2a725490ca45aa81e1a4)
 2019-08-14 13:37:21 -07:00
Eric Badger
db88224e8f YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
 2019-08-13 17:19:57 +00:00
Chao Sun
4c06b2f96c HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d74029b6ae82263739f598ceb25c597dcd)
(cherry picked from d38b617baaf10cb35f3a8eba904b930a142a071e)
(cherry picked from 90447baeb0c82d2602282c7369de2b603e16a93f)
 2019-08-08 10:47:33 -07:00
Erik Krogen
5657e45fb2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
(cherry picked from 016aa139406d1a151fbec3fb8912eb08e0f638d7)
(cherry picked from f6ce2f4a50898b16556089fe18a6717a15570d85)
 2019-08-07 09:34:01 -07:00
Yiqun Lin
ea5330643c HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:05:09 +08:00
Haibo Chen
6109ecf16d YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d5398531835b24d812f6f95094a0e0493e)
(cherry picked from commit 8d357343c4bc9f18e25543583f8f217b8a2f621b)
(cherry picked from commit f943bff2546330bc9eb0914d1a0fa1ef4c709b72)
 2019-08-06 15:08:56 -07:00
Erik Krogen
b0c5151959 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a8071fbdebb72d409ddfcd46e6c63f82aa)
(cherry picked from 69e3ac5cbb2547416b27dbcfb458099c3a951101)
 2019-08-06 09:49:56 -07:00
Weiwei Yang
373e32480b YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
(cherry picked from commit 70efe253f369f579f05d833d45e6a1bae3e41dfe)
 2019-08-05 10:02:33 -07:00
Weiwei Yang
fd977a138a YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b6701c82799a6231103ca7ee1f8b4efde)
(cherry picked from commit 7dd8eafe337a33689ff6f2f947ad152e06dbc55f)
 2019-08-05 10:02:33 -07:00
Wei Yan
ea345de738 YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat. 
(cherry picked from commit 333ef303ff0caf9adfd378652a8f966377901768)
 2019-08-05 10:02:33 -07:00
Wangda Tan
d18ec19ed8 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
(cherry picked from commit 46baafedf13446f5c1b973d667f437bf7158d3c2)
 2019-08-05 10:02:33 -07:00
Eric Badger
01f76b4f45 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3ba302327baf2e98d1e07b953dcbbfd)
 2019-08-02 21:01:15 +00:00
Erik Krogen
055f824bcb HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83c1fb933d409c514d2155e986e4e25b)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263c699932e5a6b60e00893f857713bcf)
 2019-08-02 11:52:26 -07:00
Christopher Gregorian
ec00431eaa HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
(cherry-picked from d4492bdd9edec60c236aff85de50b963097e5a7f)
(cherry-picked from 7b8f08f59e5c8906930ccc67b967b7cfcbd41768)
 2019-07-30 14:46:11 -07:00
Eric E Payne
c04ccf296d YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 2019-07-30 20:00:32 +00:00
Inigo Goiri
998593e7f2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f284942e0ffc2f40bac0286c578d854)
(cherry picked from commit fe40fbbd4d4858c55d081dffed9d68a4f0b8e923)
 2019-07-29 14:39:20 -07:00
Erik Krogen
f7e517ee24 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb6300619670d3e0554c3ba14c264fa0c705b)
(cherry-picked from 5ec34bf43adb498797afd24bb02989d4fec25409)
(cherry-picked from e2aab0efef9d504e4695a9c1df2835ed60f6c0c7)
 2019-07-26 11:43:34 -07:00
Wei-Chiu Chuang
b5303012a0 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-25 10:07:06 -07:00
Erik Krogen
32d80d2201 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen
11fee2d4e1 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen
16a4043f01 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen
2598282293 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen
45cd309a40 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko
380668559f HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun
9c1c73a0d5 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko
451b8b8f16 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun
9aecfdcb52 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chao Sun
6e19a945bc HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Erik Krogen
8b8ec65e65 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang
2e7610a029 HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. 2019-07-25 10:07:05 -07:00
Erik Krogen
7b1e3c49d1 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-25 10:07:05 -07:00
Chen Liang
82f68a47c4 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Erik Krogen
b8df864ab0 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang
8769e6feda HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00