Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,183 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5251 Commits

Author SHA1 Message Date
Steve Loughran 2029556dbb
HADOOP-17181. Handle transient stream read failures in FileSystem contract tests (#2286) 
Contributed by Steve Loughran.

* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient

The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.
 2020-09-09 11:58:26 +01:00
Bharat Viswanadham 8a3952436c
HADOOP-17245. Add OzoneFileSystem classes to core-default.xml. (#2276) 2020-09-04 18:38:48 -07:00
Uma Maheswara Rao G b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well (#2234). Contributed by Uma Maheswara Rao G. 2020-09-03 11:06:20 -07:00
He Xiaoqiao 7e671f3b65 HADOOP-17235. Erasure Coding: Remove dead code from common side. Contributed by Fei Hui. 2020-09-01 11:35:55 +08:00
Steve Loughran c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing (#2252) 
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
 2020-08-28 15:39:09 +01:00
Ayush Saxena bfd99f31fd HADOOP-17232. Erasure Coding: Typo in document. Contributed by Fei Hui. 2020-08-28 17:38:23 +05:30
sguggilam d8aaa8c338
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:45:21 -07:00
Mingliang Liu 5e52955112
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit a932796d0c.
 2020-08-26 10:41:10 -07:00
sguggilam a932796d0c
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:39:57 -07:00
Joey 960fb0aa4f
HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:01:48 +01:00
S O'Donnell 17cd8a1b16 HADOOP-17209. Erasure Coding: Native library memory leak. Contriubted by Sean Chow 2020-08-24 12:00:35 +01:00
Uma Maheswara Rao G dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 2020-08-19 09:30:41 -07:00
Steve Loughran 5092ea62ec HADOOP-13230. S3A to optionally retain directory markers. 
This adds an option to disable "empty directory" marker deletion,
so avoid throttling and other scale problems.

This feature is *not* backwards compatible.
Consult the documentation and use with care.

Contributed by Steve Loughran.

Change-Id: I69a61e7584dc36e485d5e39ff25b1e3e559a1958
 2020-08-15 12:51:08 +01:00
Siyao Meng 3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory (#2176) 2020-08-11 08:52:16 -07:00
Gautham B A 909f1e82d3
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-11 16:35:08 +09:00
Uma Maheswara Rao G 8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test
 2020-08-11 00:01:58 -07:00
sguggilam 2986058e7f
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:30:06 -07:00
Akira Ajisaka c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors (#2098) 2020-08-03 10:46:51 +09:00
belugabehr e60096c377
HADOOP-17141. Add Capability To Get Text Length (#2157) 
Contributed by David Mollitor
 2020-07-24 10:37:28 +01:00
Uma Maheswara Rao G ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 2020-07-21 23:29:10 -07:00
Masatake Iwasaki 1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Chen Liang 3833c616e0 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 2020-07-20 12:49:58 -07:00
Ayush Saxena f2033de234 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 2020-07-20 22:13:48 +05:30
Ayush Saxena 6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Mukund Thakur b21cb91c7f
HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur
 2020-07-16 18:09:59 +01:00
Ahmed Hussein 1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-07-15 11:39:06 -05:00
Erik Krogen 317fe4584a HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-14 11:22:25 -07:00
Hanisha Koneru e62d8f8412 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 2020-07-13 12:55:34 -07:00
Steve Loughran b9fa5e0182
HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:30:02 +01:00
lfengnan 84b74b335c
HDFS-15447 RBF: Add top real owners metrics for delegation tokens (#2110) 2020-07-12 00:10:12 -07:00
Uma Maheswara Rao G 3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 2020-07-11 23:50:04 -07:00
Siyao Meng 0e694b20b9
HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 2020-07-09 12:38:52 -07:00
Xiaoyu Yao f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Uma Maheswara Rao G dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 2020-07-06 18:50:03 -07:00
Madhusoodan Pataki 2f500e4635
HADOOP-17081. MetricsSystem doesn't start the sink adapters on restart (#2089) 
Contributed by Madhusoodan P
 2020-07-06 16:25:42 +01:00
Uma Maheswara Rao G 1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-07-04 00:12:10 -07:00
Abhishek Das 3b8d0f803f
HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 12:58:35 +05:30
Virajith Jalaparti bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle
 2020-06-26 13:19:16 -07:00
Uma Maheswara Rao G d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 2020-06-26 01:29:38 -07:00
lfengnan 84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch (#2047) 
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
 2020-06-23 13:12:29 -07:00
He Xiaoqiao fa14e4bc00 HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 2020-06-23 16:59:51 +08:00
Uma Maheswara Rao G 7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 2020-06-23 01:42:25 -07:00
Uma Maheswara Rao G b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 2020-06-20 00:32:02 -07:00
belugabehr 100ec8e870
HADOOP-17009: Embrace Immutability of Java Collections 2020-06-19 10:23:43 -07:00
Mehakmeet Singh 2bfb22840a
HADOOP-17020. Improve RawFileSystem Performance (#2063) 
Contributed by : Mehakmeet Singh

Co-authored-by: Rajesh Balamohan
Co-authored-by: Mehakmeet Singh
 2020-06-17 16:15:26 +01:00
Ayush Saxena c8ed33cd2a HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor. 2020-06-17 13:55:40 +05:30
Uma Maheswara Rao G 785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 2020-06-12 14:32:19 -07:00
Vinayakumar B e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Uma Maheswara Rao G 93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 2020-06-10 15:00:02 -07:00
Mingliang Liu 0c25131ca4
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:28:56 -07:00