Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,812 Commits 326 Branches 371 Tags
Commit Graph

20812 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Stephen O'Donnell
9c0d6e1657 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9c3dbff81fdcf243585fd3e9ae59dde)
 2019-08-29 17:38:55 -07:00
Istvan Fajth
7c6fc964fd HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 03049290fe8ef1459a3c8bd05cf198320e6e4973)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNNStorageRetentionManager.java
 2019-08-29 16:55:18 -07:00
Rohith Sharma K S
7616495fb7 YARN-9796. Fix ASF license issue in branch-3.2. Contributed by Prabhu Joseph. 2019-08-29 12:01:38 +05:30
Rohith Sharma K S
81c0809463 YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt. 2019-08-29 09:30:20 +05:30
Eric E Payne
d562050cec YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 2019-08-28 20:53:43 +00:00
Jonathan Hung
f36ccf0ac1 YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
 2019-08-27 15:54:22 -07:00
Akira Ajisaka
2d8799f4bc HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter. 2019-08-27 19:08:39 +00:00
Sneha Vijayarajan
9d722c637e HADOOP-16460: ABFS: fix for Sever Name Indication (SNI) 
Contributed by Sneha Vijayarajan <snvijaya@microsoft.com>
 2019-08-27 19:08:39 +00:00
bilaharith
3b3c0c4b87 HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-27 19:08:38 +00:00
Robert Levas
ce23e971b4 HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations. 
Contributed by Robert Levas.

This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.

Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
 2019-08-27 19:08:29 +00:00
Da Zhou
a6d50a9054 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-27 19:04:16 +00:00
Da Zhou
dd636127e9 HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-27 19:01:21 +00:00
Da Zhou
006ae258b3 HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens. 
Contributed by Da Zhou.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-08-27 19:01:21 +00:00
Jonathan Hung
d255efa5f8 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e641e65deb002a10732d23bb22802c09d)
 2019-08-27 10:30:55 -07:00
Arpit Agarwal
8b12381717
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
 2019-08-26 15:57:41 -07:00
Jonathan Hung
e4249c3202 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
 2019-08-26 15:51:38 -07:00
bibinchundatt
7f20c31e31 YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt. 
(cherry picked from commit d3ce53e5073e35e162f1725836282e4268cd26a5)
 2019-08-26 23:23:49 +05:30
Rohith Sharma K S
ab98f91638 YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang. 2019-08-26 21:13:02 +05:30
Chen Liang
c824653955 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:28:13 -07:00
Erik Krogen
2408c2491f HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce756d25956d33e022100111aa0dd22e)
 2019-08-23 11:48:11 -07:00
Akira Ajisaka
e8b93d04c4
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5d4ff351645e34c0ef09b7ba82f4285)
 2019-08-23 15:28:54 +09:00
Akira Ajisaka
d3e5898712
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4caea0e197568ee28392c46f1ce0d42e1)
 2019-08-23 14:45:57 +09:00
Akira Ajisaka
afb3f329fd
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 
(cherry picked from commit 84b1982060422760702eca6f1ef515c6ad3e85a5)
 2019-08-23 14:40:15 +09:00
Ayush Saxena
b7a66471ca HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:35:48 +05:30
Akira Ajisaka
43b8396b77
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9ddec46d6b7d1336bb88136d8826972e7a)
 2019-08-22 13:01:55 +09:00
Akira Ajisaka
c1c080458b
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy. 
This closes #1243

(cherry picked from commit 34dd9ee36674be670013d4fc3d9b8f5b36886812)
 2019-08-22 11:03:14 +09:00
Wei-Chiu Chuang
933ea6aa4d Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8b802d6b444b95791ba156c7bb307dd584cc9dba.
 2019-08-21 09:00:23 -07:00
Szilard Nemeth
6980f1740f YARN-9217. Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing. Contributed by Peter Bacsko 2019-08-21 16:49:34 +02:00
bibinchundatt
69255fa1b9 YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 
(cherry picked from commit 10ec31d20ee1b6a0b1da915acb6b6ec33f2cd415)
 2019-08-21 13:57:53 +05:30
Surendra Singh Lilhore
03c62c7989 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
 2019-08-20 15:54:46 -07:00
Wei-Chiu Chuang
77fd690c05 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
 2019-08-20 10:37:05 -07:00
Wei-Chiu Chuang
b004d305e9 HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde289719e0b300e11568c5e68c36b5d05)
 2019-08-20 10:33:59 -07:00
Wei-Chiu Chuang
6d93886231 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342f3c8cb8246f011abb9bcb0b6ef3eaa)
 2019-08-19 13:11:11 -07:00
HUAN-PING SU
abda503040 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2760500b7e7d4414a43069ed4a45930)
 2019-08-19 11:15:37 -07:00
Wei-Chiu Chuang
8b802d6b44 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-19 08:38:06 -07:00
KAI XIE
b3c14d4132 HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)
 2019-08-18 18:48:21 -07:00
David Mollitor
1b30b3fbe7 HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 901652e26321832ac42ce54daa18ebc1813772b5)
 2019-08-18 18:43:52 -07:00
Surendra Singh Lilhore
6b01effd01 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a9812023754ed1b3e5b428e161f0add50)
 2019-08-18 18:17:41 -07:00
Shashikant Banerjee
e89413da88 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959ce505f0659e5c69d0ca83a5dce0a7c2)
 2019-08-16 19:09:34 -07:00
Siyao Meng
c6e4906ffc HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b933b7acfe55081f5bad46f9fe05a14b)
 2019-08-16 19:08:05 -07:00
Chen Liang
d8a6098a96 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang
f507bc059d HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang
ca15968d09 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-16 10:50:00 -07:00
Chen Liang
22d81f7fd0 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-16 10:49:59 -07:00
Szilard Nemeth
a83718f130 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 15:24:44 +02:00
Szilard Nemeth
df616370f0 YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko 2019-08-16 11:52:51 +02:00
Szilard Nemeth
8fee3808c5 YARN-9749. TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk. Contributed by Adam Antal 
(cherry picked from commit 2a05e0ff3b5ab3be8654e9e96c6556865ef26096)
 2019-08-16 08:52:34 +02:00
Szilard Nemeth
e616037d1f YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph 
(cherry picked from commit 1845a83cec6563482523d8c34b38c4e36c0aa9df)
 2019-08-15 17:16:06 +02:00
Szilard Nemeth
faafab9388 MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph 
(cherry picked from commit 3f4f097f1f29249c8ad5d59ccefcddf0a66b2e6b)
 2019-08-15 17:08:28 +02:00
Christopher Gregorian
e4b650f91e HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628d370def74e56112aba3a93e97bbf70)
 2019-08-14 13:33:33 -07:00