Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,779 Commits 326 Branches 371 Tags
Commit Graph

19779 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ayush Saxena
66e4a8b1d7 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626f0332a4a828f312e2007d8c23564ee)
(cherry picked from commit d2ecaf217fcde5f900262ef156df9cee5de950d7)
 2020-10-23 17:53:42 -07:00
Wei-Chiu Chuang
7f3645ec5b HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002717a9684d00f0e9dc263ab7eb49246)
(cherry picked from commit c8eb86ee3d4f997b72e87e139e6a4dafa1ff4941)
(cherry picked from commit 87666e243899e4cd706cfdcf037f3c0a623b85de)
 2020-10-23 12:06:47 -07:00
Inigo Goiri
5f4a12c8a3 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a26988fcf5880311067e62a15e05fc1e)
 2020-10-23 09:50:36 -07:00
Kihwal Lee
7a3085d552 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc286db00b385f3280627d2b2063b4e59)
 2020-10-22 21:01:09 -05:00
Kihwal Lee
33db7c140b HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d6182471df4eba1333737912a32534b)
(cherry picked from commit f3b2d85690aeb65da7540f48edd52ca7315d0b02)
 2020-10-22 15:26:34 -05:00
Takanobu Asanuma
c808a21550 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:59 +09:00
Akira Ajisaka
1af5d59eb2
HADOOP-17309. Javadoc warnings and errors are ignored in the precommit jobs. (#2391) 
(cherry picked from commit 3a0922fb4207af2f06f8ef57e83ff9759871d0cb)
 2020-10-19 10:56:37 +09:00
Takanobu Asanuma
4afb52d759
HADOOP-15814. Maven 3.3.3 unable to parse pom file. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2626f46691e1e1ad09967d0931a79b95e308c8b8)
 2020-10-19 10:56:06 +09:00
Takanobu Asanuma
a1a30db5be
HADOOP-15741. [JDK10] Upgrade Maven Javadoc Plugin from 3.0.0-M1 to 3.0.1. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 281c192e7f917545151117fc7e067dc480b93499)
 2020-10-19 10:55:01 +09:00
Eric Badger
50eba43c7f YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:01:50 +00:00
Kihwal Lee
ffe6e39c96 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d4e9a9804c777c8f845ad41e1280177)
 2020-10-16 11:41:17 -05:00
Kihwal Lee
9aefc1759b HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:18:42 -05:00
Konstantin V Shvachko
2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:52:37 -07:00
Swaroopa Kadam
0d5534430c
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:10:23 -07:00
Doroszlai, Attila
dd17772b53 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:43 -07:00
Jim Brennan
7bda43fbd5 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 
(cherry picked from commit e1c6804ace57ce21ba6acd1e81d73019f939e2b2)
 2020-10-08 21:13:39 +00:00
Jim Brennan
3a360b4bf1 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:33:16 +00:00
Jim Brennan
c36435c033 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8d1f546d90e92d076f32cb5681ce944)
 2020-10-06 18:38:23 +00:00
Adam Antal
b5e6ae97f4 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:53:00 +02:00
S O'Donnell
4a8b59df59 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-30 09:39:34 +01:00
Wanqiang Ji
cbc946285e
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd41885a43c7bc6e11f0c0d6e5820768aa2c)
 2020-09-29 23:08:50 +09:00
S O'Donnell
27b2c1ea7b HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 
(cherry picked from commit 5f34e3214e42098c7515f0441697980303aab643)
 2020-09-22 10:51:16 +01:00
zz
06ff4d1416
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-20 00:07:07 -07:00
hemanthboyina
43b113de69 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 
(cherry picked from commit aa582ccc2a2d7fa08cfa1a04d4cfa28c40183f14)
 2020-09-17 10:17:30 +05:30
Akira Ajisaka
3c932dd3d5
Revert "HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277)" 
This reverts commit 63c8e39a2ca9e768ab2b39877e596ba88b8b321a.
 2020-09-16 16:47:08 +09:00
Wanqiang Ji
63c8e39a2c
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 
(cherry picked from commit ce861836918c0c8e6f0294827e82e90edc984ec3)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64

(cherry picked from commit ffc101a27be197a63bab29cf916748289e3baa04)
 2020-09-16 16:26:36 +09:00
Jim Brennan
c83a3177dc YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 1efb54bd524d6961511712b43d3883c9e32a2b2a)
 2020-09-15 16:37:19 +00:00
Eric E Payne
1a8d9ddd14 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 17:07:09 +00:00
bibinchundatt
daff39a601 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80ad7864b332b60d5a01585fdf00848ee)
 2020-09-08 21:47:00 +00:00
Eric Badger
c3dbbd66b9 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
(cherry picked from commit 01ada576f3ecb551d99f7a61746e7563daf4ef09)
 2020-09-08 17:31:04 +00:00
Steve Loughran
5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
sguggilam
e1ac832d5d
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-27 00:06:05 -07:00
Mingliang Liu
9f94c9e60d
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit 12fb9e0600f665aca3e7ebe0be9b95ff232d520f.
 2020-08-26 11:24:03 -07:00
Uma Maheswara Rao G
0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:16:47 -07:00
S O'Donnell
0f234667dd HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bffbe9bad1c719fc3a3b734f0060df70a)
 2020-08-25 16:05:00 +01:00
sguggilam
12fb9e0600
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-25 00:08:20 -07:00
S O'Donnell
f40193298b HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b1627f4d87ddc5dcc2ec7c738ffdc576b)
 2020-08-24 13:07:22 +01:00
Chen Liang
e54128205c HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 16:28:46 -07:00
Sunil G
2f7e584c21 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 7369fc31b456e2b90893212f499a30755fab5cc9.
 2020-08-20 19:13:03 +05:30
Sunil G
7369fc31b4 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f4421c162a505ba5e5b329e4be199878a7)
 2020-08-19 12:02:40 +05:30
Chao Sun
bd3817b8cd
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fefacf2578e58cf7bf96d920f5b01dba66909ca2)
 2020-08-18 12:12:32 +09:00
Kihwal Lee
0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
 2020-08-13 19:02:03 -05:00
Akira Ajisaka
f9c251badc
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584bd253f0831d39557f54627f26cef7f7)
 2020-08-12 16:09:52 +09:00
Jonathan Hung
c18a38bd07 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 17d18a2a3a6a2014257e65dd4874c54a84bb4f9e)
 2020-08-07 17:44:26 -07:00
Ayush Saxena
4af5fc86b2 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:24:41 +05:30
Wanqiang Ji
f3ac1dde5e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86b96d6f84abd7354c3c056d3627871b)
 2020-08-06 09:37:28 +09:00
Eric Badger
647be0c0f6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 23:14:27 +00:00
Mingliang Liu
1a53bc6ecf
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:13:41 -07:00
Mingliang Liu
770735d330
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f782dfcf2deaac428a927f0de6c71def)
 2020-08-05 10:52:54 +09:00
sguggilam
31c61f4363
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:57:52 -07:00