Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,704 Commits 326 Branches 371 Tags
Commit Graph

19704 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Christopher Gregorian
96ea7f0562 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628d370def74e56112aba3a93e97bbf70)
(cherry picked from e4b650f91e7353f98fee2a725490ca45aa81e1a4)
 2019-08-14 13:37:01 -07:00
Szilard Nemeth
aa0631a042 YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 19:04:09 +02:00
Siyao Meng
224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b933b7acfe55081f5bad46f9fe05a14b)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Wei-Chiu Chuang
6966b76230 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
(cherry picked from commit 4784165bb24228d13f4e738e0093ab0dade0bff1)
 2019-08-13 17:15:10 -07:00
Masatake Iwasaki
e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63d89e5c8377a035f93485a7c77c3430)
 2019-08-14 09:07:00 +09:00
Eric Badger
a995e6352f YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)
 2019-08-13 17:16:57 +00:00
Szilard Nemeth
cb91ab73b0 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 
(cherry picked from commit 8b3c6791b13fc57891cf81e83d4b626b4f2932e6)
 2019-08-13 15:47:57 +02:00
Szilard Nemeth
a762a6be29 Revert "YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko" 
This reverts commit b20fd9e21295add7e80f07b471bba5c76e433aed.
Commit is reverted since unnecessary files were added, accidentally.
 2019-08-13 15:47:57 +02:00
Siyao Meng
b040eb91c7 HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a078ce7fb4e2a852e315de5b6410082)
(cherry picked from commit c5f433b11fe7409cb59d86e410bc2f7135cde7ca)
 2019-08-12 17:30:32 -07:00
Szilard Nemeth
9da9b6d58e YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph 
(cherry picked from commit e4b538bbda6dc25d7f45bffd6a4ce49f3f84acdc)
 2019-08-12 15:16:49 +02:00
Szilard Nemeth
148121d889 YARN-9451. AggregatedLogsBlock shows wrong NM http port. Contributed by Prabhu Joseph 
(cherry picked from commit b91099efd6e1fdcb31ec4ca7142439443c9ae536)
 2019-08-12 15:06:48 +02:00
Szilard Nemeth
6b4ded7647 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-08-12 14:03:50 +02:00
Sunil G
58ad5ad493 YARN-9729. [UI2] Fix error message for logs when ATSv2 is offline. Contributed by Zoltan Siegl. 
(cherry picked from commit 1c5b28659fe1310030d14e0be40dcd77b25056d6)
 2019-08-11 11:49:57 +05:30
Yiqun Lin
f2007818fc HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c734bea0eb490f30c908748458c190c3)
(cherry picked from commit a7bc09c596d99812f104483bb752fed9de7b4d32)
 2019-08-09 18:41:18 -07:00
Szilard Nemeth
be9ac8adf9 Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph 
(cherry picked from commit 54ac80176e8487b7a18cd9e16a11efa289d0b7df)
 2019-08-09 13:23:49 +02:00
Sunil G
1c4364cb9a YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid potential misuses. Contributed by Akhil PB. 
(cherry picked from commit acffec7a92be540aa8531dbe06a3ea7bb813ab93)
 2019-08-09 16:06:07 +05:30
Adam Antal
600a61f410 YARN-9124. Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently (#1121) 
(cherry picked from commit cbcada804d119b837ad99de71d7f44cb4629026e)
 2019-08-09 11:44:22 +02:00
Szilard Nemeth
410f7a3069 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak 
(cherry picked from commit e0c21c6da91776caf661661a19c368939c81fcc4)
 2019-08-09 10:25:12 +02:00
Szilard Nemeth
b2f39f81fe YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak 
(cherry picked from commit 742e30b47381ad63e2b2fe63738cd0fe6cbce106)
 2019-08-09 10:05:40 +02:00
Szilard Nemeth
943dfc78d1 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak 
(cherry picked from commit 72d7e570a73989aa18b737c0e642d570a55c6781)
 2019-08-09 09:53:14 +02:00
Wei-Chiu Chuang
0aa847e158 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494c3aea146d7741bf0ad52ce16ad08e6)
(cherry picked from commit 2f2fa3dad6dc62ad0e415fa7892008d47a31ddb4)
 2019-08-08 13:51:23 -07:00
Stephen O'Donnell
163fb887a6 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf6e92be540f5665bb571418b916d789)
(cherry picked from commit a00a3275944bfdfd0849ac50c89e81b1284c8e13)
 2019-08-08 13:37:22 -07:00
Chao Sun
90447baeb0 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d74029b6ae82263739f598ceb25c597dcd)
(cherry picked from d38b617baaf10cb35f3a8eba904b930a142a071e)
 2019-08-08 10:42:27 -07:00
Eric E Payne
b131214685 YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang. 
(cherry picked from commit 3b38f2019e4f8d056580f3ed67ecef591011d7a6)
 2019-08-08 13:08:05 +00:00
Erik Krogen
f6ce2f4a50 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
(cherry picked from 016aa139406d1a151fbec3fb8912eb08e0f638d7)
 2019-08-07 09:33:36 -07:00
Lokesh Jain
d06fca0104 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360e083010f395cb9d3bd46701417e7b7)
(cherry picked from commit f6fe18a282ce0ed729b6b7586c17b2046a612070)
 2019-08-06 15:31:15 -07:00
Haibo Chen
f943bff254 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d5398531835b24d812f6f95094a0e0493e)
(cherry picked from commit 8d357343c4bc9f18e25543583f8f217b8a2f621b)
 2019-08-06 15:01:06 -07:00
Stephen O'Donnell
99af4ce0ac HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 13:53:01 -07:00
Erik Krogen
69e3ac5cbb HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a8071fbdebb72d409ddfcd46e6c63f82aa)
 2019-08-06 09:49:23 -07:00
Wei-Chiu Chuang
a7a348446d Revert "HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell." 
This reverts commit 45ba41570b02c9e830b3e651a1e8004ef5027a94.
 2019-08-06 08:57:00 -07:00
Stephen O'Donnell
45ba41570b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7bc3149bf8b941fd85d7807d9906d27b)
(cherry picked from commit 8414e6354b4f32f2059d7bb651a7fd16c56bfbfb)
 2019-08-06 07:58:23 -07:00
Eric Badger
698e74d097 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3ba302327baf2e98d1e07b953dcbbfd)
 2019-08-02 20:41:26 +00:00
Erik Krogen
da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83c1fb933d409c514d2155e986e4e25b)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Wei-Chiu Chuang
68d618acd2 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
(cherry picked from commit e1659fe0090b478d823f1c8f7b0859021496be05)
 2019-08-02 10:52:05 -07:00
Wei-Chiu Chuang
8501bed9cb HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat. 
(cherry picked from commit e20b19543ba148cf0242884a3e553ad2f39421b3)
(cherry picked from commit 490b18fd4f35a19dc2dd48249cb38a96dbf2f85c)
 2019-08-01 17:36:59 -07:00
Wei-Chiu Chuang
d66ee4bf61 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98a7ab70741a2efc98eff548c1e431c1)
(cherry picked from commit 5c1f91fd9757a5a788afaa2a1d64433cd34df117)
 2019-08-01 15:49:14 -07:00
He Xiaoqiao
5d35393912 HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f86de6f76a3079c2655df9b242fc968edfb17b9d)
(cherry picked from commit 555b56c0421be84efd09221db2b3f26c8b44ed45)
 2019-08-01 14:56:17 -07:00
Szilard Nemeth
639763c561 MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 16:08:14 +02:00
Christopher Gregorian
7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
(cherry-picked from d4492bdd9edec60c236aff85de50b963097e5a7f)
 2019-07-30 14:45:00 -07:00
Eric E Payne
36af8845de YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 42683aef1a694af883c14842bf41f30b91e039f3)
 2019-07-30 19:45:00 +00:00
Akira Ajisaka
4662b13b26
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 
(cherry picked from commit cbfa3f3e988b8a3f142abadfe4f18201b6438ac9)
 2019-07-29 17:39:42 -07:00
Wei-Chiu Chuang
608e7044a5 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388fec17a40856f36821c142be32aa364)
(cherry picked from commit d28e6249583a1bae4b7c59688f9848669b5cc8ff)
 2019-07-29 14:33:08 -07:00
Erik Krogen
4d21310865 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb6300619670d3e0554c3ba14c264fa0c705b)
(cherry-picked from 5ec34bf43adb498797afd24bb02989d4fec25409)
 2019-07-29 14:00:24 -07:00
Yiqun Lin
2e1afa9a86 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b7d40ac832ae3e23b64e8541b62c4bd)
(cherry picked from commit 69f1c8d77b941adf4a279af4e4511e48eebe8d86)
 2019-07-29 06:50:41 -07:00
Inigo Goiri
fe40fbbd4d HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f284942e0ffc2f40bac0286c578d854)
 2019-07-26 15:56:44 -07:00
Erik Krogen
42f10712a5 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit bf4a2258ce280120778b3f04844330df81c967b0.
 2019-07-26 11:36:59 -07:00
Steve Loughran
1b7c7dc860
HADOOP-9844. NPE when trying to create an error message response of SASL RPC 
This closes #55

Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 07530314c2130ecd1525682c59bf51f15b82c024)
 2019-07-26 17:55:00 +09:00
Masatake Iwasaki
bf4a2258ce HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97251267253fa045ba03c4749f95f530)
 2019-07-26 09:49:42 +09:00
pingsutw
1a2aba85d8
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build 
This closes #1089

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ee87e9a42e4ff1f27a6f1e5b7c7de97f8989d9b2)
 2019-07-23 10:36:01 +09:00
Jonathan Hung
3ff2148482 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 2019-07-22 10:54:08 -07:00