Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,701 Commits 326 Branches 371 Tags
Commit Graph

21701 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
1c7043b1ff HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04cb8e5dab8410ef64daa0fe799e389fd)
(cherry picked from commit dd1634ec3b676d6c3da18af7da46734584ddd9f6)
 2020-11-03 11:23:02 -08:00
Eric E Payne
31154fdde5 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:33:57 +00:00
Siyao Meng
beb6cf6cc2
YARN-10472. Backport YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars (#2412) 2020-10-29 16:17:07 -07:00
Jim Brennan
63888afdd0 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 20:17:51 +00:00
Sunil G
14a4606a9a HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He. 
(cherry picked from commit f17e067d527b0089124a8ca69b36fb20ed118c55)
 2020-10-29 07:21:31 +05:30
Chen Liang
576f99ba40 HADOOP-17330. Backport HADOOP-16005 NativeAzureFileSystem does not support setXAttr to branch-3.2. Contributed by Sally Zuo. 2020-10-28 15:39:18 -07:00
Jonathan Hung
d0104e72c5 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
(cherry picked from commit f95c0824b01175590fe98e2fba1e5988694a52da)
 2020-10-28 10:38:58 -07:00
Ahmed Hussein
dedf0d42bc
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac43a480a010d40c182c1d4cc5346db8f)
 2020-10-27 20:52:23 +09:00
Ahmed Hussein
244fd5eed2
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d3325e0b454475fa01cc50f68007b0e15c)
 2020-10-27 17:09:20 +09:00
Hui Fei
b32926f110 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827c80ea37ed5207a65831d0b699c8381)
 2020-10-26 22:34:06 +08:00
Ayush Saxena
d2ecaf217f HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626f0332a4a828f312e2007d8c23564ee)
 2020-10-23 17:53:29 -07:00
Wei-Chiu Chuang
87666e2438 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002717a9684d00f0e9dc263ab7eb49246)
(cherry picked from commit c8eb86ee3d4f997b72e87e139e6a4dafa1ff4941)
 2020-10-23 12:06:36 -07:00
Inigo Goiri
7119af2c6d HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a26988fcf5880311067e62a15e05fc1e)
 2020-10-23 09:49:46 -07:00
Vinayakumar B
8c6478b405
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 (#2387) 2020-10-23 11:39:43 +05:30
Kihwal Lee
f363c3b315 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc286db00b385f3280627d2b2063b4e59)
 2020-10-22 19:48:42 -05:00
Kihwal Lee
f3b2d85690 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d6182471df4eba1333737912a32534b)
 2020-10-22 14:59:09 -05:00
Takanobu Asanuma
0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:20 +09:00
Akira Ajisaka
f8b19c9f21
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb4207af2f06f8ef57e83ff9759871d0cb)
 2020-10-19 10:49:57 +09:00
He Xiaoqiao
4a1f1bf5de
Preparing for 3.2.3 development - Addendum for hadoop-project/pom.xml. 2020-10-17 18:14:50 +08:00
Eric Badger
4c61136616 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 18:51:53 +00:00
Kihwal Lee
7e4572e56a HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:17:34 -05:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Akira Ajisaka
a2c1fb7c8c
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-10-16 01:04:45 +09:00
Kihwal Lee
392b26c23b HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d4e9a9804c777c8f845ad41e1280177)
 2020-10-14 18:19:16 -05:00
Akira Ajisaka
69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c22ccd60a8560a80e2d84129b359e630)
 2020-10-13 18:09:14 +09:00
Pranav Bheda
e009afbae8
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 
* update org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.13
* update org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13

(cherry picked from commit be3edd05321a0cbe5e6d60536b4ab6ed34f2cef6)
 2020-10-13 17:57:03 +09:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
Swaroopa Kadam
55f01bda8e
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:09:50 -07:00
Doroszlai, Attila
61563885f8 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:33 -07:00
Jim Brennan
e1c6804ace YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 2020-10-08 21:09:30 +00:00
Jim Brennan
4ef9cf9d71 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:01:38 +00:00
Jim Brennan
ecf91638a8 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:36:51 +00:00
Adam Antal
b7420eb4b0 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:39:14 +02:00
maobaolong
4b1c0e4e5b HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:24:59 -07:00
Wanqiang Ji
578370d6c8
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd41885a43c7bc6e11f0c0d6e5820768aa2c)
 2020-09-29 23:08:30 +09:00
Stephen O'Donnell
71d006130e HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-09-28 12:45:28 -07:00
Eric E Payne
947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
Kihwal Lee
25a51f2f39 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 
(cherry picked from commit dfc268221352880992c55abec9e6b7b73044b0f1)
 2020-09-22 11:12:43 -05:00
S O'Donnell
5f34e3214e HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 2020-09-22 10:34:05 +01:00
zz
0873555f04
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:35:24 -07:00
hemanthboyina
aa582ccc2a HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 10:15:18 +05:30
Akira Ajisaka
875219bc8e
Revert "HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277)" 
This reverts commit ffc101a27be197a63bab29cf916748289e3baa04.
 2020-09-16 16:46:37 +09:00
Wanqiang Ji
ffc101a27b
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 
(cherry picked from commit ce861836918c0c8e6f0294827e82e90edc984ec3)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2020-09-16 16:25:34 +09:00
Jim Brennan
1efb54bd52 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-15 16:27:08 +00:00
Eric E Payne
5b14af6d09 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 16:46:28 +00:00
bibinchundatt
b5d24d646c YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80ad7864b332b60d5a01585fdf00848ee)
 2020-09-08 21:05:26 +00:00
Eric Badger
01ada576f3 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
 2020-09-08 16:14:26 +00:00
Adam Antal
696494d663 YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 
(cherry picked from commit 34fe74da0e9c68173e1de196c496b9cfca029618)
 2020-09-07 12:01:35 +02:00
Ayush Saxena
544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Kitti Nanasi
9b8eaa0572 HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 14282e311be6ffcaddd2f74fa8e67c4e98a32291)
 2020-08-30 10:21:53 +05:30