Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,529 Commits 326 Branches 371 Tags
Commit Graph

862 Commits

Author SHA1 Message Date
Surendra Singh Lilhore
929411acce HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
 2021-02-26 10:49:48 +08:00
Jim Brennan
d74e9453d9 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 
(cherry picked from commit 62389a5a041b02efb0f310891e258fb047e4ca2a)
 2021-02-05 22:18:52 +00:00
Chao Sun
3b2a131885
HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:42:46 -08:00
Giovanni Matteo Fumarola
4af5f55623
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2020-11-10 14:44:34 +08:00
Takanobu Asanuma
0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:20 +09:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Akira Ajisaka
69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c22ccd60a8560a80e2d84129b359e630)
 2020-10-13 18:09:14 +09:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
Doroszlai, Attila
61563885f8 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:33 -07:00
Ayush Saxena
544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Ayush Saxena
27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Uma Maheswara Rao G
4a9ec4d143 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 15:54:37 -07:00
Eric Yang
e9cf454ac2
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0406380e469c76ed2e41022a6372dd7)
 2020-06-15 11:12:40 +09:00
Akira Ajisaka
dfb094d581
Revert "SPNEGO TLS verification" 
This reverts commit 9db6e96db3cd4a11075afa0c0b58476cd0e26b8a.
 2020-05-13 17:15:46 +09:00
Ayush Saxena
06f43fc628 HDFS-15250. Setting dfs.client.use.datanode.hostname to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:43 +05:30
Eric Yang
9db6e96db3
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454a5f6ea84f2cf7ac0082c555e2954a7)
 2020-05-02 21:51:41 +09:00
Chen Liang
10fe37b704 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 13:01:00 -07:00
Eric Yang
ba6b3a3848 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928bb877f26b11b5b212a100a79941f35)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
 2020-03-25 17:12:01 -07:00
Isa Hekmatizadeh
152cbc6457 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c0814ab61fce6ea8baf4b272f84c252)
 2020-03-25 16:52:52 -07:00
Surendra Singh Lilhore
e9916052b1 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:53:45 +05:30
Ayush Saxena
87532b6e3f HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:31:20 +05:30
Ayush Saxena
9acbbd885c HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a32ee172ded6c4b3b97a1e49befc4f10)
 2020-02-27 16:36:53 -08:00
Masatake Iwasaki
610805ec72 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-21 13:40:11 +09:00
Ayush Saxena
9a1ce410f6 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:09 +05:30
Kihwal Lee
f1840669cf HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java
 2020-02-11 14:50:10 -06:00
Chen Liang
9813ba2ebc [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang 2020-01-28 16:36:27 -08:00
Kihwal Lee
1bb9667137 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
 2020-01-24 09:21:09 -06:00
Masatake Iwasaki
3e17b471b6 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1fbb150ea7ff0411c41ffd3a2796208)
 2020-01-08 17:05:20 +09:00
Chandra Sanivarapu
0d94e0b1a0
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4bcfbfd76915b9ecefb2c7ea320e149)
 2019-12-25 11:12:03 +09:00
He Xiaoqiao
8ba4e13dcc HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
 2019-12-11 16:34:28 -08:00
Surendra Singh Lilhore
aa1c795dc9 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c212b26d75268ee3659aa2cadcff349)
 2019-11-06 19:37:41 +05:30
Duo Zhang
a37a4bc62f HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 20:26:54 -07:00
zhaoyim
9316ca149f HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59d250788d029cb2013bb4712ef9b546)
 2019-10-25 13:10:56 -07:00
Chen Liang
a12ab911ee HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a6695f45f0d9cea5ae9aae83941d360)
 2019-10-08 12:03:04 -07:00
Surendra Singh Lilhore
a0cdebc409 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 382967be51052d59e31d8d05713645b8d3c2325b)
 2019-10-08 00:26:27 +05:30
Erik Krogen
6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
 2019-10-04 11:15:49 -07:00
Erik Krogen
69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
 2019-10-04 10:48:33 -07:00
Siyao Meng
673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:40:22 -07:00
Kitti Nanasi
cb701f4da0 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc92009fec2f0ab341f3d878e1b8864b8ea9)
 2019-10-03 16:18:23 -07:00
LeonGao91
885dd17595 HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf50ec29ffb2d8522ba2f4bc6605d8b8b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java
 2019-10-02 17:35:33 -07:00
Inigo Goiri
3510a9e5e5 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb89f4d1edbb54360622adc61ffacbde3)
 2019-10-01 16:57:20 -07:00
Chen Liang
2d7b37ea2b HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:55:10 -07:00
Anu Engineer
2e9482bd20 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14e577a7d8142bc7d49bb94014032afd)
 2019-09-18 15:23:28 +08:00
Rohith Sharma K S
7d5bb2ebb7 Preparing for 3.2.2-SNAPSHOT development. 2019-09-07 08:52:08 +05:30
Stephen O'Donnell
61fd1a74a1 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0a12cc7b4d3ef4ea951941defbbcb33)
 2019-09-02 10:06:39 -07:00
Wei-Chiu Chuang
84b2fe4b10 Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit 9c0d6e16573a055665318bce8f7d38a92c897393.
 2019-09-02 10:06:39 -07:00
Stephen O'Donnell
9c0d6e1657 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9c3dbff81fdcf243585fd3e9ae59dde)
 2019-08-29 17:38:55 -07:00