Commit Graph

20862 Commits

Author SHA1 Message Date
Adam Antal d5446b3a23 YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… (#1261)
* YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes

* Using {} placeholder, and increasing loglevel if log aggregation failed.

(cherry picked from commit c89bdfacc8)
2019-08-14 17:36:41 +02:00
Szilard Nemeth 4bb238c480 YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. Contributed by Peter Bacsko 2019-08-14 17:16:54 +02:00
Szilard Nemeth 4dc477b606 YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 17:01:41 +02:00
Szilard Nemeth 9a87e74e54 YARN-9134. No test coverage for redefining FPGA / GPU resource types in TestResourceUtils. Contributed by Peter Bacsko 2019-08-14 16:46:34 +02:00
Wei-Chiu Chuang f5661b630a HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng.
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
2019-08-13 17:15:19 -07:00
Masatake Iwasaki d7ca016d63 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
2019-08-14 08:45:10 +09:00
Siyao Meng 88aece4083 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng.
(cherry picked from commit 6ae8bc3a4a)
2019-08-13 16:36:52 -07:00
Eric Badger cec71691be YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan.
(cherry picked from commit 2ac029b949)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
2019-08-13 16:34:29 +00:00
Siyao Meng c5f433b11f HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng.
(cherry picked from commit c92b49876a)
2019-08-12 17:30:18 -07:00
Szilard Nemeth c5aea8ca56 YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph
(cherry picked from commit e4b538bbda)
2019-08-12 15:16:18 +02:00
Szilard Nemeth 844259203f YARN-9451. AggregatedLogsBlock shows wrong NM http port. Contributed by Prabhu Joseph
(cherry picked from commit b91099efd6)
2019-08-12 15:06:16 +02:00
Szilard Nemeth b20fd9e212 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-08-12 14:02:17 +02:00
Da Zhou 330e450397
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
2019-08-12 09:41:52 +08:00
Sunil G 02b4635ff0 YARN-9729. [UI2] Fix error message for logs when ATSv2 is offline. Contributed by Zoltan Siegl.
(cherry picked from commit 1c5b28659f)
2019-08-11 11:49:25 +05:30
Yiqun Lin a7bc09c596 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
2019-08-09 18:41:03 -07:00
Szilard Nemeth 2e6beb1550 Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph
(cherry picked from commit 54ac80176e)
2019-08-09 13:20:10 +02:00
Sunil G 9fb6c6e2a1 YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid potential misuses. Contributed by Akhil PB.
(cherry picked from commit acffec7a92)
2019-08-09 16:07:04 +05:30
Szilard Nemeth 3e9071207a SUBMARINE-57. Add more elaborate message if submarine command is not recognized. Contributed by Adam Antal
(cherry picked from commit e5f4cd0fda)
2019-08-09 12:14:49 +02:00
Adam Antal 4c4f7d9c80 YARN-9124. Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently (#1121)
(cherry picked from commit cbcada804d)
2019-08-09 11:43:30 +02:00
Szilard Nemeth 02d0e54596 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak
(cherry picked from commit e0c21c6da9)
2019-08-09 10:23:10 +02:00
Szilard Nemeth f0dfb8b832 YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak
(cherry picked from commit 742e30b473)
2019-08-09 10:02:35 +02:00
Szilard Nemeth 3bcf44f070 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak
(cherry picked from commit 72d7e570a7)
2019-08-09 09:50:32 +02:00
Wei-Chiu Chuang 2f2fa3dad6 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao.
(cherry picked from commit 6ad9a11494)
2019-08-08 13:51:14 -07:00
Stephen O'Donnell a00a327594 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
2019-08-08 13:37:10 -07:00
Chao Sun d38b617baa HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
2019-08-08 10:15:48 -07:00
Eric E Payne e47c483d9f YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang.
(cherry picked from commit 3b38f2019e)
2019-08-08 12:54:31 +00:00
Erik Krogen 016aa13940 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen.
(cherry picked from 827dbb11e2)
2019-08-07 09:32:32 -07:00
Akira Ajisaka 8e59ff92b8
HADOOP-15908. hadoop-build-tools jar is downloaded from remote repository instead of using from local. Contributed by Oleksandr Shevchenko.
(cherry picked from commit 0b0ba70b35)

Conflicts:
	hadoop-project/pom.xml
2019-08-07 09:33:39 +09:00
Lokesh Jain f6fe18a282 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
2019-08-06 15:31:05 -07:00
Haibo Chen 8d357343c4 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung)
(cherry picked from commit f51702d539)
2019-08-06 14:59:49 -07:00
Inigo Goiri 3e8e1570f6 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao.
(cherry picked from 5740eea081)
2019-08-06 09:26:01 -07:00
Stephen O'Donnell 8414e6354b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
2019-08-06 07:58:12 -07:00
Jianfei Jiang 0c02217d9b
HDFS-14691. Wrong usage hint for hadoop fs command "test".
Contributed by Jianfei Jiang.

Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
(cherry picked from commit 71aad60e51)
2019-08-05 13:09:39 +01:00
Erik Krogen 2634aaaf9a HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira.
(cherry picked from e7a0b8aa83)
2019-08-02 12:00:48 -07:00
Wei-Chiu Chuang e1659fe009 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq.
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
2019-08-02 10:51:55 -07:00
Wei-Chiu Chuang 490b18fd4f HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat.
(cherry picked from commit e20b19543b)
2019-08-01 17:36:48 -07:00
Wei-Chiu Chuang 5c1f91fd97 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun.
(cherry picked from commit 32607dbd98)
2019-08-01 15:49:05 -07:00
He Xiaoqiao a405db8014 HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f86de6f76a)
2019-08-01 14:56:38 -07:00
Szilard Nemeth e611fb878b MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 16:05:28 +02:00
Christopher Gregorian d4492bdd9e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
2019-07-30 14:41:36 -07:00
Eric E Payne 168dc3f258 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan.
(cherry picked from commit 42683aef1a)
2019-07-30 19:19:33 +00:00
Akira Ajisaka f921c72fda
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.
(cherry picked from commit cbfa3f3e98)
2019-07-29 17:39:05 -07:00
Wei-Chiu Chuang d28e624958 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai.
(cherry picked from commit 8053085388)
2019-07-29 14:32:46 -07:00
Yiqun Lin 69f1c8d77b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
(cherry picked from commit 993dc8726b)
2019-07-29 06:50:28 -07:00
Erik Krogen 5ec34bf43a HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.
(cherry-picked from 62efb63006)
2019-07-26 11:39:14 -07:00
Erik Krogen bd5033e6f9 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena."
This reverts commit 2478d333a6.
2019-07-26 11:36:06 -07:00
Steve Loughran 7418764a67
HADOOP-9844. NPE when trying to create an error message response of SASL RPC
This closes #55

Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 07530314c2)
2019-07-26 17:54:32 +09:00
Masatake Iwasaki 2478d333a6 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
2019-07-26 09:48:00 +09:00
S O'Donnell 23b470ee92
HADOOP-16443. Improve help text for setfacl --set option.
Contributed by S O'Donnell.

Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
2019-07-23 10:25:59 +01:00
pingsutw 772cacdbdd
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build
This closes #1089

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ee87e9a42e)
2019-07-23 10:35:36 +09:00