Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,980 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

20980 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 6d93886231 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342)
 2019-08-19 13:11:11 -07:00
HUAN-PING SU abda503040 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2)
 2019-08-19 11:15:37 -07:00
Wei-Chiu Chuang 8b802d6b44 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-19 08:38:06 -07:00
KAI XIE b3c14d4132 HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb2)
 2019-08-18 18:48:21 -07:00
David Mollitor 1b30b3fbe7 HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 901652e26321832ac42ce54daa18ebc1813772b5)
 2019-08-18 18:43:52 -07:00
Surendra Singh Lilhore 6b01effd01 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
 2019-08-18 18:17:41 -07:00
Shashikant Banerjee e89413da88 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959c)
 2019-08-16 19:09:34 -07:00
Siyao Meng c6e4906ffc HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
 2019-08-16 19:08:05 -07:00
Chen Liang d8a6098a96 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang f507bc059d HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang ca15968d09 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-16 10:50:00 -07:00
Chen Liang 22d81f7fd0 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-16 10:49:59 -07:00
Szilard Nemeth a83718f130 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 15:24:44 +02:00
Szilard Nemeth df616370f0 YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko 2019-08-16 11:52:51 +02:00
Szilard Nemeth 8fee3808c5 YARN-9749. TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk. Contributed by Adam Antal 
(cherry picked from commit 2a05e0ff3b)
 2019-08-16 08:52:34 +02:00
Szilard Nemeth e616037d1f YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph 
(cherry picked from commit 1845a83cec)
 2019-08-15 17:16:06 +02:00
Szilard Nemeth faafab9388 MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph 
(cherry picked from commit 3f4f097f1f)
 2019-08-15 17:08:28 +02:00
Christopher Gregorian e4b650f91e HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628)
 2019-08-14 13:33:33 -07:00
Adam Antal d5446b3a23 YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… (#1261) 
* YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes

* Using {} placeholder, and increasing loglevel if log aggregation failed.

(cherry picked from commit c89bdfacc8)
 2019-08-14 17:36:41 +02:00
Szilard Nemeth 4bb238c480 YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. Contributed by Peter Bacsko 2019-08-14 17:16:54 +02:00
Szilard Nemeth 4dc477b606 YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 17:01:41 +02:00
Szilard Nemeth 9a87e74e54 YARN-9134. No test coverage for redefining FPGA / GPU resource types in TestResourceUtils. Contributed by Peter Bacsko 2019-08-14 16:46:34 +02:00
Wei-Chiu Chuang f5661b630a HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
 2019-08-13 17:15:19 -07:00
Masatake Iwasaki d7ca016d63 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2019-08-14 08:45:10 +09:00
Siyao Meng 88aece4083 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng. 
(cherry picked from commit 6ae8bc3a4a)
 2019-08-13 16:36:52 -07:00
Eric Badger cec71691be YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
 2019-08-13 16:34:29 +00:00
Siyao Meng c5f433b11f HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a)
 2019-08-12 17:30:18 -07:00
Szilard Nemeth c5aea8ca56 YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph 
(cherry picked from commit e4b538bbda)
 2019-08-12 15:16:18 +02:00
Szilard Nemeth 844259203f YARN-9451. AggregatedLogsBlock shows wrong NM http port. Contributed by Prabhu Joseph 
(cherry picked from commit b91099efd6)
 2019-08-12 15:06:16 +02:00
Szilard Nemeth b20fd9e212 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-08-12 14:02:17 +02:00
Da Zhou 330e450397
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-12 09:41:52 +08:00
Sunil G 02b4635ff0 YARN-9729. [UI2] Fix error message for logs when ATSv2 is offline. Contributed by Zoltan Siegl. 
(cherry picked from commit 1c5b28659f)
 2019-08-11 11:49:25 +05:30
Yiqun Lin a7bc09c596 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
 2019-08-09 18:41:03 -07:00
Szilard Nemeth 2e6beb1550 Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph 
(cherry picked from commit 54ac80176e)
 2019-08-09 13:20:10 +02:00
Sunil G 9fb6c6e2a1 YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid potential misuses. Contributed by Akhil PB. 
(cherry picked from commit acffec7a92)
 2019-08-09 16:07:04 +05:30
Szilard Nemeth 3e9071207a SUBMARINE-57. Add more elaborate message if submarine command is not recognized. Contributed by Adam Antal 
(cherry picked from commit e5f4cd0fda)
 2019-08-09 12:14:49 +02:00
Adam Antal 4c4f7d9c80 YARN-9124. Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently (#1121) 
(cherry picked from commit cbcada804d)
 2019-08-09 11:43:30 +02:00
Szilard Nemeth 02d0e54596 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak 
(cherry picked from commit e0c21c6da9)
 2019-08-09 10:23:10 +02:00
Szilard Nemeth f0dfb8b832 YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak 
(cherry picked from commit 742e30b473)
 2019-08-09 10:02:35 +02:00
Szilard Nemeth 3bcf44f070 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak 
(cherry picked from commit 72d7e570a7)
 2019-08-09 09:50:32 +02:00
Wei-Chiu Chuang 2f2fa3dad6 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494)
 2019-08-08 13:51:14 -07:00
Stephen O'Donnell a00a327594 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
 2019-08-08 13:37:10 -07:00
Chao Sun d38b617baa HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
 2019-08-08 10:15:48 -07:00
Eric E Payne e47c483d9f YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang. 
(cherry picked from commit 3b38f2019e)
 2019-08-08 12:54:31 +00:00
Erik Krogen 016aa13940 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
 2019-08-07 09:32:32 -07:00
Akira Ajisaka 8e59ff92b8
HADOOP-15908. hadoop-build-tools jar is downloaded from remote repository instead of using from local. Contributed by Oleksandr Shevchenko. 
(cherry picked from commit 0b0ba70b35)

Conflicts:
	hadoop-project/pom.xml
 2019-08-07 09:33:39 +09:00
Lokesh Jain f6fe18a282 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
 2019-08-06 15:31:05 -07:00
Haibo Chen 8d357343c4 YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d539)
 2019-08-06 14:59:49 -07:00
Inigo Goiri 3e8e1570f6 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from 5740eea081)
 2019-08-06 09:26:01 -07:00
Stephen O'Donnell 8414e6354b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
 2019-08-06 07:58:12 -07:00