Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,818 Commits 326 Branches 371 Tags
Commit Graph

878 Commits

Author SHA1 Message Date
liubingxing
0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52ada03e099fa7dbb3efe62345ca27588)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
bitterfox
7bdb50c793 HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:43:37 +09:00
litao
ba61dd9b73 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d80db478f7e49749065c69da7e241bf0)
 2021-11-10 19:18:56 +09:00
daimin
9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:20:43 +01:00
AngersZhuuuu
1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao
cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
 2021-09-29 17:44:00 +08:00
EungsopYoo
4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
(cherry picked from commit 51a4a23e373abe9b2f6469ffcd5bc8294307f025)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein
1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Kihwal Lee
70c5e9e780 HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 
(cherry picked from commit 47002719f2aa4ff58378d528d38b0f0962a45c25)
 2021-07-16 14:24:52 -05:00
crossfire
88f3ce1127
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810beb7abb8540c051114488e69a128ea3)
(cherry picked from commit bd0b8a0bcf96fe66a93f59b74822f419cac7eb09)
 2021-07-16 11:01:03 +08:00
Wei-Chiu Chuang
11034f2488
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e5489e6080f21c308673ecc73a54ba3752)
 2021-06-17 11:38:58 +08:00
Ayush Saxena
77f61df352 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:55:06 +05:30
Takanobu Asanuma
b8fc956a4d HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682270bc6ba151875549d64d81917f425)
 2021-06-15 17:23:34 +09:00
Gautham B A
52f4f9a063
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:47:40 +09:00
Ayush Saxena
bc3c4a3268 HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:29 +05:30
Surendra Singh Lilhore
929411acce HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
 2021-02-26 10:49:48 +08:00
Jim Brennan
d74e9453d9 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 
(cherry picked from commit 62389a5a041b02efb0f310891e258fb047e4ca2a)
 2021-02-05 22:18:52 +00:00
Chao Sun
3b2a131885
HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:42:46 -08:00
Giovanni Matteo Fumarola
4af5f55623
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2020-11-10 14:44:34 +08:00
Takanobu Asanuma
0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:20 +09:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Akira Ajisaka
69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c22ccd60a8560a80e2d84129b359e630)
 2020-10-13 18:09:14 +09:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
Doroszlai, Attila
61563885f8 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:33 -07:00
Ayush Saxena
544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Ayush Saxena
27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Uma Maheswara Rao G
4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 15:54:37 -07:00
Eric Yang
e9cf454ac2
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0406380e469c76ed2e41022a6372dd7)
 2020-06-15 11:12:40 +09:00
Akira Ajisaka
dfb094d581
Revert "SPNEGO TLS verification" 
This reverts commit 9db6e96db3cd4a11075afa0c0b58476cd0e26b8a.
 2020-05-13 17:15:46 +09:00
Ayush Saxena
06f43fc628 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:43 +05:30
Eric Yang
9db6e96db3
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
 2020-05-02 21:51:41 +09:00
Chen Liang
10fe37b704 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 13:01:00 -07:00
Eric Yang
ba6b3a3848 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928bb877f26b11b5b212a100a79941f35)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
 2020-03-25 17:12:01 -07:00
Isa Hekmatizadeh
152cbc6457 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c0814ab61fce6ea8baf4b272f84c252)
 2020-03-25 16:52:52 -07:00
Surendra Singh Lilhore
e9916052b1 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:53:45 +05:30
Ayush Saxena
87532b6e3f HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:31:20 +05:30
Ayush Saxena
9acbbd885c HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a32ee172ded6c4b3b97a1e49befc4f10)
 2020-02-27 16:36:53 -08:00
Masatake Iwasaki
610805ec72 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-21 13:40:11 +09:00
Ayush Saxena
9a1ce410f6 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:09 +05:30
Kihwal Lee
f1840669cf HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java
 2020-02-11 14:50:10 -06:00
Chen Liang
9813ba2ebc [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang 2020-01-28 16:36:27 -08:00
Kihwal Lee
1bb9667137 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
 2020-01-24 09:21:09 -06:00
Masatake Iwasaki
3e17b471b6 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1fbb150ea7ff0411c41ffd3a2796208)
 2020-01-08 17:05:20 +09:00
Chandra Sanivarapu
0d94e0b1a0
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4bcfbfd76915b9ecefb2c7ea320e149)
 2019-12-25 11:12:03 +09:00
He Xiaoqiao
8ba4e13dcc HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
 2019-12-11 16:34:28 -08:00
Surendra Singh Lilhore
aa1c795dc9 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c212b26d75268ee3659aa2cadcff349)
 2019-11-06 19:37:41 +05:30