Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,720 Commits 326 Branches 371 Tags
Commit Graph

19720 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Badger
50eba43c7f YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:01:50 +00:00
Kihwal Lee
ffe6e39c96 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d4e9a9804c777c8f845ad41e1280177)
 2020-10-16 11:41:17 -05:00
Kihwal Lee
9aefc1759b HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:18:42 -05:00
Konstantin V Shvachko
2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:52:37 -07:00
Swaroopa Kadam
0d5534430c
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-10-12 11:10:23 -07:00
Doroszlai, Attila
dd17772b53 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:43 -07:00
Jim Brennan
7bda43fbd5 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 
(cherry picked from commit e1c6804ace57ce21ba6acd1e81d73019f939e2b2)
 2020-10-08 21:13:39 +00:00
Jim Brennan
3a360b4bf1 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:33:16 +00:00
Jim Brennan
c36435c033 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8d1f546d90e92d076f32cb5681ce944)
 2020-10-06 18:38:23 +00:00
Adam Antal
b5e6ae97f4 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:53:00 +02:00
S O'Donnell
4a8b59df59 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-30 09:39:34 +01:00
Wanqiang Ji
cbc946285e
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd41885a43c7bc6e11f0c0d6e5820768aa2c)
 2020-09-29 23:08:50 +09:00
S O'Donnell
27b2c1ea7b HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 
(cherry picked from commit 5f34e3214e42098c7515f0441697980303aab643)
 2020-09-22 10:51:16 +01:00
zz
06ff4d1416
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-20 00:07:07 -07:00
hemanthboyina
43b113de69 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 
(cherry picked from commit aa582ccc2a2d7fa08cfa1a04d4cfa28c40183f14)
 2020-09-17 10:17:30 +05:30
Akira Ajisaka
3c932dd3d5
Revert "HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277)" 
This reverts commit 63c8e39a2ca9e768ab2b39877e596ba88b8b321a.
 2020-09-16 16:47:08 +09:00
Wanqiang Ji
63c8e39a2c
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 
(cherry picked from commit ce861836918c0c8e6f0294827e82e90edc984ec3)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64

(cherry picked from commit ffc101a27be197a63bab29cf916748289e3baa04)
 2020-09-16 16:26:36 +09:00
Jim Brennan
c83a3177dc YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 1efb54bd524d6961511712b43d3883c9e32a2b2a)
 2020-09-15 16:37:19 +00:00
Eric E Payne
1a8d9ddd14 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 17:07:09 +00:00
bibinchundatt
daff39a601 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80ad7864b332b60d5a01585fdf00848ee)
 2020-09-08 21:47:00 +00:00
Eric Badger
c3dbbd66b9 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
(cherry picked from commit 01ada576f3ecb551d99f7a61746e7563daf4ef09)
 2020-09-08 17:31:04 +00:00
Steve Loughran
5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
sguggilam
e1ac832d5d
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-27 00:06:05 -07:00
Mingliang Liu
9f94c9e60d
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit 12fb9e0600f665aca3e7ebe0be9b95ff232d520f.
 2020-08-26 11:24:03 -07:00
Uma Maheswara Rao G
0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:16:47 -07:00
S O'Donnell
0f234667dd HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bffbe9bad1c719fc3a3b734f0060df70a)
 2020-08-25 16:05:00 +01:00
sguggilam
12fb9e0600
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-25 00:08:20 -07:00
S O'Donnell
f40193298b HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b1627f4d87ddc5dcc2ec7c738ffdc576b)
 2020-08-24 13:07:22 +01:00
Chen Liang
e54128205c HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 16:28:46 -07:00
Sunil G
2f7e584c21 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 7369fc31b456e2b90893212f499a30755fab5cc9.
 2020-08-20 19:13:03 +05:30
Sunil G
7369fc31b4 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f4421c162a505ba5e5b329e4be199878a7)
 2020-08-19 12:02:40 +05:30
Chao Sun
bd3817b8cd
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fefacf2578e58cf7bf96d920f5b01dba66909ca2)
 2020-08-18 12:12:32 +09:00
Kihwal Lee
0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
 2020-08-13 19:02:03 -05:00
Akira Ajisaka
f9c251badc
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584bd253f0831d39557f54627f26cef7f7)
 2020-08-12 16:09:52 +09:00
Jonathan Hung
c18a38bd07 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 17d18a2a3a6a2014257e65dd4874c54a84bb4f9e)
 2020-08-07 17:44:26 -07:00
Ayush Saxena
4af5fc86b2 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:24:41 +05:30
Wanqiang Ji
f3ac1dde5e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86b96d6f84abd7354c3c056d3627871b)
 2020-08-06 09:37:28 +09:00
Eric Badger
647be0c0f6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 23:14:27 +00:00
Mingliang Liu
1a53bc6ecf
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:13:41 -07:00
Mingliang Liu
770735d330
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f782dfcf2deaac428a927f0de6c71def)
 2020-08-05 10:52:54 +09:00
sguggilam
31c61f4363
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:57:52 -07:00
Eric Badger
87816e8a51 YARN-10363. TestRMAdminCLI.testHelp is failing in branch-2.10. Contributed by 
Bilwa S T.
 2020-07-31 22:39:39 +00:00
Uma Maheswara Rao G
1768618ab9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:42:22 -07:00
Uma Maheswara Rao G
544602e3d1 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e700066394fb9f516e23537d8abb4661409cae1)
 2020-07-31 01:42:03 -07:00
Uma Maheswara Rao G
7084b273ac HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2f2ba0bd3919650ea231cedd424f77a)
 2020-07-31 01:41:45 -07:00
Uma Maheswara Rao G
ab43b7bcfb HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc9b45413febd33840b8a653e3e29440)
 2020-07-31 01:41:21 -07:00
Uma Maheswara Rao G
49a7f9ff7b HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5024aeb7fb1f8c31b8fdd0fdb88bb66)
 2020-07-31 01:40:57 -07:00
Eric E Payne
8d8366166e YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166df48a47267dbc81d124ab78267e039)
(cherry picked from commit 863689ff9a34d9712541a06757c34499b6005f12)
 2020-07-30 17:27:35 +00:00
ywheel
2dfc6bf471
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 
(cherry picked from commit cf4eb75608527a8a56ad9d13a3009ccc38b29c8e)
 2020-07-30 13:31:18 +09:00
Jonathan Hung
b0edec2918 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 
(cherry picked from commit ffb920de2a60558a9863afb31b566b8c688a8263)
 2020-07-28 13:44:40 -07:00