Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,787 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19787 Commits

Author SHA1 Message Date
Wei-Chiu Chuang d4633fcff9 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 166bee93af.
 2020-11-04 16:18:54 -08:00
Ayush Saxena 76a7c0c0ef HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:57:13 +05:30
Konstantin V Shvachko 079f1bb2f9 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 12:34:01 -08:00
Akira Ajisaka 166bee93af HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04c)
(cherry picked from commit dd1634ec3b)
(cherry picked from commit 1c7043b1ff)
 2020-11-03 11:23:19 -08:00
Eric E Payne 052b9799c0 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 31154fdde5)
 2020-11-02 17:59:58 +00:00
Jonathan Hung 42fab7897a YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743)
(cherry picked from commit f95c0824b0)
(cherry picked from commit d0104e72c5)
 2020-10-28 10:42:02 -07:00
Ahmed Hussein 34915b0373
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4)
 2020-10-27 20:55:16 +09:00
Hui Fei eb909fd726 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:41:24 +08:00
Ayush Saxena 66e4a8b1d7 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626)
(cherry picked from commit d2ecaf217f)
 2020-10-23 17:53:42 -07:00
Wei-Chiu Chuang 7f3645ec5b HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
(cherry picked from commit c8eb86ee3d)
(cherry picked from commit 87666e2438)
 2020-10-23 12:06:47 -07:00
Inigo Goiri 5f4a12c8a3 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a2)
 2020-10-23 09:50:36 -07:00
Kihwal Lee 7a3085d552 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 21:01:09 -05:00
Kihwal Lee 33db7c140b HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d)
(cherry picked from commit f3b2d85690)
 2020-10-22 15:26:34 -05:00
Takanobu Asanuma c808a21550 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:13:59 +09:00
Akira Ajisaka 1af5d59eb2
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb42)
 2020-10-19 10:56:37 +09:00
Takanobu Asanuma 4afb52d759
HADOOP-15814. Maven 3.3.3 unable to parse pom file. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2626f46691)
 2020-10-19 10:56:06 +09:00
Takanobu Asanuma a1a30db5be
HADOOP-15741. [JDK10] Upgrade Maven Javadoc Plugin from 3.0.0-M1 to 3.0.1. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 281c192e7f)
 2020-10-19 10:55:01 +09:00
Eric Badger 50eba43c7f YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:01:50 +00:00
Kihwal Lee ffe6e39c96 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d)
 2020-10-16 11:41:17 -05:00
Kihwal Lee 9aefc1759b HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:18:42 -05:00
Konstantin V Shvachko 2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:52:37 -07:00
Swaroopa Kadam 0d5534430c
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:10:23 -07:00
Doroszlai, Attila dd17772b53 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:43 -07:00
Jim Brennan 7bda43fbd5 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 
(cherry picked from commit e1c6804ace)
 2020-10-08 21:13:39 +00:00
Jim Brennan 3a360b4bf1 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32ba)
 2020-10-08 19:33:16 +00:00
Jim Brennan c36435c033 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8)
 2020-10-06 18:38:23 +00:00
Adam Antal b5e6ae97f4 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760df)
 2020-10-05 10:53:00 +02:00
S O'Donnell 4a8b59df59 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-30 09:39:34 +01:00
Wanqiang Ji cbc946285e
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd4188)
 2020-09-29 23:08:50 +09:00
S O'Donnell 27b2c1ea7b HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 
(cherry picked from commit 5f34e3214e)
 2020-09-22 10:51:16 +01:00
zz 06ff4d1416
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-20 00:07:07 -07:00
hemanthboyina 43b113de69 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 
(cherry picked from commit aa582ccc2a)
 2020-09-17 10:17:30 +05:30
Akira Ajisaka 3c932dd3d5
Revert "HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277)" 
This reverts commit 63c8e39a2c.
 2020-09-16 16:47:08 +09:00
Wanqiang Ji 63c8e39a2c
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 
(cherry picked from commit ce86183691)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64

(cherry picked from commit ffc101a27b)
 2020-09-16 16:26:36 +09:00
Jim Brennan c83a3177dc YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 1efb54bd52)
 2020-09-15 16:37:19 +00:00
Eric E Payne 1a8d9ddd14 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed17)
 2020-09-11 17:07:09 +00:00
bibinchundatt daff39a601 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80a)
 2020-09-08 21:47:00 +00:00
Eric Badger c3dbbd66b9 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d)
(cherry picked from commit 01ada576f3)
 2020-09-08 17:31:04 +00:00
Steve Loughran 5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
sguggilam e1ac832d5d
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-27 00:06:05 -07:00
Mingliang Liu 9f94c9e60d
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit 12fb9e0600.
 2020-08-26 11:24:03 -07:00
Uma Maheswara Rao G 0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:16:47 -07:00
S O'Donnell 0f234667dd HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 16:05:00 +01:00
sguggilam 12fb9e0600
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-25 00:08:20 -07:00
S O'Donnell f40193298b HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b16)
 2020-08-24 13:07:22 +01:00
Chen Liang e54128205c HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 16:28:46 -07:00
Sunil G 2f7e584c21 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 7369fc31b4.
 2020-08-20 19:13:03 +05:30
Sunil G 7369fc31b4 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f442)
 2020-08-19 12:02:40 +05:30
Chao Sun bd3817b8cd
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fefacf2578)
 2020-08-18 12:12:32 +09:00
Kihwal Lee 0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351ccc)
 2020-08-13 19:02:03 -05:00