Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,748 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5014 Commits

Author SHA1 Message Date
Jim Brennan 118d8f5aa1 HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 4bdba35b9f)
 2020-11-13 21:52:10 +00:00
Ahmed Hussein 487eaa6dea HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0f)
(cherry picked from commit 23fe3bdab3)
(cherry picked from commit ac3e10af63)
 2020-11-12 10:35:57 -08:00
Stephen Jung 9f7553bab4 HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 
(cherry picked from commit 45434c93e8)
(cherry picked from commit 0712505b59)
 2020-11-10 11:41:34 -08:00
Steve Loughran d903278607 HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 17:05:59 +05:30
hchaverr 3443d75946 HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 
(cherry picked from commit 6eacaffeea)
 2020-11-06 15:02:47 -08:00
Eric Badger 4cec4bebd9 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.

(cherry picked from commit af389d9897)
 2020-11-05 22:11:52 +00:00
Wei-Chiu Chuang d4633fcff9 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 166bee93af.
 2020-11-04 16:18:54 -08:00
Akira Ajisaka 166bee93af HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04c)
(cherry picked from commit dd1634ec3b)
(cherry picked from commit 1c7043b1ff)
 2020-11-03 11:23:19 -08:00
Ayush Saxena 66e4a8b1d7 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626)
(cherry picked from commit d2ecaf217f)
 2020-10-23 17:53:42 -07:00
Takanobu Asanuma c808a21550 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:13:59 +09:00
Konstantin V Shvachko 2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:52:37 -07:00
Steve Loughran 5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
sguggilam e1ac832d5d
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-27 00:06:05 -07:00
Mingliang Liu 9f94c9e60d
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit 12fb9e0600.
 2020-08-26 11:24:03 -07:00
Uma Maheswara Rao G 0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:16:47 -07:00
sguggilam 12fb9e0600
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-25 00:08:20 -07:00
S O'Donnell f40193298b HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b16)
 2020-08-24 13:07:22 +01:00
Akira Ajisaka f9c251badc
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584b)
 2020-08-12 16:09:52 +09:00
sguggilam 31c61f4363
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:57:52 -07:00
Uma Maheswara Rao G 1768618ab9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:42:22 -07:00
Uma Maheswara Rao G 544602e3d1 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:42:03 -07:00
Uma Maheswara Rao G 7084b273ac HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:41:45 -07:00
Uma Maheswara Rao G 49a7f9ff7b HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5)
 2020-07-31 01:40:57 -07:00
Akira Ajisaka 416edc5a7b
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 
(cherry picked from commit d5b4766158)
 2020-07-23 00:41:22 +09:00
Ayush Saxena 4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Chen Liang 1e33696bc8 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
(cherry picked from commit 370efc6d95)
 2020-07-20 15:24:07 -07:00
Akira Ajisaka 635067d03d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65a)
(cherry picked from commit 0d949d375e)
 2020-07-20 13:37:08 -07:00
Mukund Thakur cb3e378782 HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-07-17 10:40:01 +01:00
Ahmed Hussein b7b9bd32db HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 12:19:42 -05:00
Erik Krogen 0de48933c5 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:33:38 -07:00
Siyao Meng 47df2245d3 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9)
 2020-07-09 16:32:07 -07:00
Uma Maheswara Rao G 4e1abe61a2 HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:29:36 -07:00
Abhishek Das 5f84093cf8 HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:02:00 +05:30
Virajith Jalaparti c4e0098bd6 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:25:26 -07:00
Uma Maheswara Rao G 30d1d29076 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 16:09:36 -07:00
Uma Maheswara Rao G 81e62613fb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 16:09:16 -07:00
Uma Maheswara Rao G e17770dec6 HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 16:08:57 -07:00
Uma Maheswara Rao G 208701329c HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 16:08:38 -07:00
Ayush Saxena b720d7770d HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c4)
 2020-06-27 16:08:10 -07:00
Abhishek Das 0e9b6b0124 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b)
 2020-06-27 16:07:49 -07:00
Uma Maheswara Rao G 100c13967e HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def95)
(cherry picked from commit 120ee793fc)
 2020-06-27 16:07:28 -07:00
Uma Maheswara Rao G 2e1dfc152b HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
(cherry picked from commit 0b5e202614)
 2020-06-27 16:06:04 -07:00
Uma Maheswara Rao G 6ae92962d9 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b)
(cherry picked from commit 8e71e85af7)
 2020-06-27 16:05:36 -07:00
Abhishek Das 7015589f58 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec74453)
(cherry picked from commit 5b248de42d)
 2020-06-27 16:05:09 -07:00
Uma Maheswara Rao G c4857bb9c1 HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d9)
(cherry picked from commit 544996c857)
 2020-06-27 16:04:23 -07:00
Kihwal Lee 400c5accd9 HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) 
Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
(cherry picked from commit bb0b922a71)
 2020-06-08 17:25:15 -05:00
Mingliang Liu ac0e928564
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:32 -07:00
Mingliang Liu 101ce83f37
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:51:36 -07:00
Dhiraj 2c40f20deb
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:07:25 -07:00
S O'Donnell 0cd4459f4e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e)
 2020-05-29 14:53:59 +01:00