Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,379 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4984 Commits

Author SHA1 Message Date
Konstantin V Shvachko 23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:42:58 -07:00
maobaolong 4b1c0e4e5b HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:24:59 -07:00
Eric E Payne 947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
sguggilam 970b9a283b
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:58:34 -07:00
Mingliang Liu acec431377
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit d06f0de3af.
 2020-08-26 11:23:26 -07:00
Surendra Singh Lilhore 3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a4)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G 0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran 134539f054
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210) 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!
 2020-08-25 22:47:43 +01:00
Uma Maheswara Rao G 978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:05:31 -07:00
sguggilam d06f0de3af
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:58:36 -07:00
Joey 956c097f25 HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:06:14 +01:00
Mukund Thakur 28db29d32f HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-08-24 14:05:03 +01:00
S O'Donnell 4163b43a83 HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b16)
 2020-08-24 13:06:15 +01:00
Steve Loughran 42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Gautham B A ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3)
 2020-08-11 16:47:26 +09:00
Sahil Takiar eae1387f34
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit 3b0c5016b2)
 2020-08-11 16:45:27 +09:00
sguggilam e484f5529c
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:47:31 -07:00
Uma Maheswara Rao G cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G 65778cdd47 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G 1369e41c65 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G 655b39cc30 HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5)
 2020-07-31 01:37:14 -07:00
Ayush Saxena 27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Chen Liang 370efc6d95 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
 2020-07-20 14:07:10 -07:00
Akira Ajisaka 0d949d375e HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65a)
 2020-07-20 13:20:34 -07:00
Ahmed Hussein 8fd3dcc9ce HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 12:05:49 -05:00
Erik Krogen bd37b72ea9 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:34:21 -07:00
Siyao Meng 0bd0cb1b06 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9)
 2020-07-09 16:31:43 -07:00
Uma Maheswara Rao G 7607f24c9e HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:29:02 -07:00
Madhusoodan Pataki dfb74a3eeb HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089) 
Contributed by Madhusoodan P
 2020-07-06 16:29:09 +01:00
Abhishek Das b7d54c3dbf HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:01:31 +05:30
Virajith Jalaparti 704b273ec8 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:25:04 -07:00
Uma Maheswara Rao G d030f4b2a6 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 15:56:10 -07:00
Uma Maheswara Rao G 80111fe5bb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 15:55:41 -07:00
Uma Maheswara Rao G 6c22210baa HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 15:55:11 -07:00
Uma Maheswara Rao G 4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 15:54:37 -07:00
Ayush Saxena fc9445abbd HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c4)
 2020-06-27 15:54:13 -07:00
Abhishek Das db5c32d2a5 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b)
 2020-06-27 15:53:38 -07:00
Uma Maheswara Rao G aa8de2f43b HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def95)
(cherry picked from commit 120ee793fc)
 2020-06-27 15:53:07 -07:00
Uma Maheswara Rao G 852ee71354 HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
(cherry picked from commit 0b5e202614)
 2020-06-27 15:51:51 -07:00
Uma Maheswara Rao G 84ceb6d520 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b)
(cherry picked from commit 8e71e85af7)
 2020-06-27 15:51:28 -07:00
Abhishek Das 7cf9601987 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec74453)
(cherry picked from commit 5b248de42d)
 2020-06-27 15:50:41 -07:00
Uma Maheswara Rao G 3dcc9aed4d HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d9)
(cherry picked from commit 544996c857)
 2020-06-27 15:49:48 -07:00
Mingliang Liu 61106467e3
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:07 -07:00
Mingliang Liu 0eee1c88ec
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:39:09 -07:00
Dhiraj 462a68a69f
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:06:32 -07:00
S O'Donnell d245a4d7fe HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e)
 2020-05-29 14:51:45 +01:00
S O'Donnell 5b6c2b366e HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:10:50 +01:00
Akira Ajisaka 777e36448c
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79e)
 2020-05-18 15:37:42 +09:00
Uma Maheswara Rao G 0d0aac1a2a HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-11 15:29:26 -07:00
Virajith Jalaparti e5e824bd6f HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-11 12:50:14 -07:00