Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,384 Commits 326 Branches 371 Tags
Commit Graph

4985 Commits

Author SHA1 Message Date
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
maobaolong
4b1c0e4e5b HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:24:59 -07:00
Eric E Payne
947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
sguggilam
970b9a283b
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:58:34 -07:00
Mingliang Liu
acec431377
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit d06f0de3affbd5e8232a6fcdb9a3c396934b6a05.
 2020-08-26 11:23:26 -07:00
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran
134539f054
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210) 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!
 2020-08-25 22:47:43 +01:00
Uma Maheswara Rao G
978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:05:31 -07:00
sguggilam
d06f0de3af
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:58:36 -07:00
Joey
956c097f25 HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:06:14 +01:00
Mukund Thakur
28db29d32f HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-08-24 14:05:03 +01:00
S O'Donnell
4163b43a83 HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b1627f4d87ddc5dcc2ec7c738ffdc576b)
 2020-08-24 13:06:15 +01:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Gautham B A
ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3e956bdd01b9d24ed2108f93bb94953)
 2020-08-11 16:47:26 +09:00
Sahil Takiar
eae1387f34
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit 3b0c5016b2931e3b87153c1cf368314d229e79ff)
 2020-08-11 16:45:27 +09:00
sguggilam
e484f5529c
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:47:31 -07:00
Uma Maheswara Rao G
cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G
65778cdd47 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e700066394fb9f516e23537d8abb4661409cae1)
 2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G
1369e41c65 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2f2ba0bd3919650ea231cedd424f77a)
 2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G
655b39cc30 HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5024aeb7fb1f8c31b8fdd0fdb88bb66)
 2020-07-31 01:37:14 -07:00
Ayush Saxena
27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Chen Liang
370efc6d95 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
 2020-07-20 14:07:10 -07:00
Akira Ajisaka
0d949d375e HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
 2020-07-20 13:20:34 -07:00
Ahmed Hussein
8fd3dcc9ce HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
 2020-07-15 12:05:49 -05:00
Erik Krogen
bd37b72ea9 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:34:21 -07:00
Siyao Meng
0bd0cb1b06 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9d59cc46882df506dcea386020b1e4d)
 2020-07-09 16:31:43 -07:00
Uma Maheswara Rao G
7607f24c9e HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5584ec220b3eeaf76da85d228bcbd8b)
 2020-07-09 16:29:02 -07:00
Madhusoodan Pataki
dfb74a3eeb HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089) 
Contributed by Madhusoodan P
 2020-07-06 16:29:09 +01:00
Abhishek Das
b7d54c3dbf HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:01:31 +05:30
Virajith Jalaparti
704b273ec8 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
 2020-06-27 16:25:04 -07:00
Uma Maheswara Rao G
d030f4b2a6 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
 2020-06-27 15:56:10 -07:00
Uma Maheswara Rao G
80111fe5bb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
 2020-06-27 15:55:41 -07:00
Uma Maheswara Rao G
6c22210baa HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
 2020-06-27 15:55:11 -07:00
Uma Maheswara Rao G
4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 15:54:37 -07:00
Ayush Saxena
fc9445abbd HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c498531bf3c011f1b0ace8eddddf61f2)
 2020-06-27 15:54:13 -07:00
Abhishek Das
db5c32d2a5 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
 2020-06-27 15:53:38 -07:00
Uma Maheswara Rao G
aa8de2f43b HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def959fab6b8b7ffff66410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
 2020-06-27 15:53:07 -07:00
Uma Maheswara Rao G
852ee71354 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
(cherry picked from commit 0b5e202614f0bc20a0db6656f924fa4d2741d00c)
 2020-06-27 15:51:51 -07:00
Uma Maheswara Rao G
84ceb6d520 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
(cherry picked from commit 8e71e85af70c17f2350f794f8bc2475eb1e3acea)
 2020-06-27 15:51:28 -07:00
Abhishek Das
7cf9601987 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
(cherry picked from commit 5b248de42d2ae42710531a1514a21d60a1fca4b2)
 2020-06-27 15:50:41 -07:00
Uma Maheswara Rao G
3dcc9aed4d HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
(cherry picked from commit 544996c85702af7ae241ef2f18e2597e2b4050be)
 2020-06-27 15:49:48 -07:00
Mingliang Liu
61106467e3
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:07 -07:00
Mingliang Liu
0eee1c88ec
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:39:09 -07:00
Dhiraj
462a68a69f
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:06:32 -07:00
S O'Donnell
d245a4d7fe HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:51:45 +01:00
S O'Donnell
5b6c2b366e HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:10:50 +01:00
Akira Ajisaka
777e36448c
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:37:42 +09:00
Uma Maheswara Rao G
0d0aac1a2a HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-11 15:29:26 -07:00