Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,718 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21718 Commits

Author SHA1 Message Date
maobaolong 4b1c0e4e5b HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:24:59 -07:00
Wanqiang Ji 578370d6c8
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 
(cherry picked from commit 143bdd4188)
 2020-09-29 23:08:30 +09:00
Stephen O'Donnell 71d006130e HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-09-28 12:45:28 -07:00
Eric E Payne 947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
Kihwal Lee 25a51f2f39 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 
(cherry picked from commit dfc2682213)
 2020-09-22 11:12:43 -05:00
S O'Donnell 5f34e3214e HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 2020-09-22 10:34:05 +01:00
zz 0873555f04
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:35:24 -07:00
hemanthboyina aa582ccc2a HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 10:15:18 +05:30
Akira Ajisaka 875219bc8e
Revert "HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277)" 
This reverts commit ffc101a27b.
 2020-09-16 16:46:37 +09:00
Wanqiang Ji ffc101a27b
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 
(cherry picked from commit ce86183691)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2020-09-16 16:25:34 +09:00
Jim Brennan 1efb54bd52 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-15 16:27:08 +00:00
Eric E Payne 5b14af6d09 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed17)
 2020-09-11 16:46:28 +00:00
bibinchundatt b5d24d646c YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80a)
 2020-09-08 21:05:26 +00:00
Eric Badger 01ada576f3 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d)
 2020-09-08 16:14:26 +00:00
Adam Antal 696494d663 YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 
(cherry picked from commit 34fe74da0e)
 2020-09-07 12:01:35 +02:00
Ayush Saxena 544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Kitti Nanasi 9b8eaa0572 HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 14282e311b)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi 3ba40d0ccf HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5f15a60e32)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi 0dbcb81a41 HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 951cdd7e4c)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi a78430abf4 HDFS-12946. Add a tool to check rack configuration against EC policies. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd5e7c6b72)
 2020-08-30 10:21:35 +05:30
sguggilam 970b9a283b
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:58:34 -07:00
Mingliang Liu acec431377
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit d06f0de3af.
 2020-08-26 11:23:26 -07:00
Surendra Singh Lilhore 3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a4)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G 0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran 134539f054
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210) 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!
 2020-08-25 22:47:43 +01:00
Uma Maheswara Rao G 978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:05:31 -07:00
S O'Donnell a3f12f7f20 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 15:49:54 +01:00
sguggilam d06f0de3af
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:58:36 -07:00
Joey 956c097f25 HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:06:14 +01:00
Mukund Thakur 28db29d32f HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-08-24 14:05:03 +01:00
S O'Donnell 4163b43a83 HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b16)
 2020-08-24 13:06:15 +01:00
Chen Liang 4ab14692e6 HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d)
 2020-08-20 15:26:44 -07:00
Sunil G 94723bff64 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 2a40a33dfe.
 2020-08-20 19:15:10 +05:30
Steve Loughran 42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Sunil G 2a40a33dfe YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f442)
 2020-08-19 12:00:33 +05:30
Chao Sun 5858c74a38
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fefacf2578)
 2020-08-18 12:11:57 +09:00
Akira Ajisaka 9aa78fe0d2
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584b)
 2020-08-12 16:09:35 +09:00
Gautham B A ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3)
 2020-08-11 16:47:26 +09:00
Sahil Takiar eae1387f34
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit 3b0c5016b2)
 2020-08-11 16:45:27 +09:00
Jonathan Hung 17d18a2a3a YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:52 -07:00
Ayush Saxena d6a9ed8140 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:23:02 +05:30
Wanqiang Ji ea7e41e29e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86)
 2020-08-06 09:37:03 +09:00
Eric Badger 9a1db93b1b YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925e)
 2020-08-05 19:03:48 +00:00
Mingliang Liu 41c211d762
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:01:04 -07:00
Mingliang Liu e094521298
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f7)
 2020-08-05 10:52:07 +09:00
sguggilam e484f5529c
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:47:31 -07:00
Uma Maheswara Rao G cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G 65778cdd47 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G 1369e41c65 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G 512d1d6d27 HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:38:18 -07:00