Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,622 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

645 Commits

Author SHA1 Message Date
Akira Ajisaka de4ac41528
Revert "SPNEGO TLS verification" 
This reverts commit 4b7d6667fd.
 2020-05-13 17:16:15 +09:00
Ayush Saxena 86c15b8287 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:45:13 +05:30
Eric Yang 4b7d6667fd
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:52:06 +09:00
Gabor Bota ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Eric Yang b837431a08 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

(cherry picked from commit ba6b3a3848)
 2020-03-25 17:14:18 -07:00
Isa Hekmatizadeh d64f688198 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c)
 2020-03-25 16:52:42 -07:00
Surendra Singh Lilhore a3224ff799 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:59:00 +05:30
Ayush Saxena 8566c65917 HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:31:58 +05:30
Ayush Saxena 354e63b366 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a3)
 2020-02-27 16:37:30 -08:00
Masatake Iwasaki 8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec72)
 2020-02-21 13:51:26 +09:00
Kihwal Lee c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
Ayush Saxena 83ab853733 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:30:13 +05:30
Kihwal Lee bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97b)
(cherry picked from commit f1840669cf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00
Chen Liang 5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Kihwal Lee 7dd43cec7f HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c9)
(cherry picked from commit 1bb9667137)
 2020-01-24 09:53:26 -06:00
Masatake Iwasaki c5580af1c4 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1)
 2020-01-08 17:06:49 +09:00
Chandra Sanivarapu e33651ace7
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4)
 2019-12-25 11:12:29 +09:00
He Xiaoqiao 974706c5fd HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
(cherry picked from commit 8ba4e13dcc)
 2019-12-12 02:23:22 -08:00
Surendra Singh Lilhore 4690142e86 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c)
 2019-11-06 19:48:16 +05:30
Duo Zhang 07f59c023c HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:30:34 -07:00
zhaoyim 44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
(cherry picked from commit 9316ca149f)
 2019-10-25 13:11:07 -07:00
Surendra Singh Lilhore 626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Chen Liang 01d9952f63 HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a)
 2019-10-08 12:08:17 -07:00
Vinayakumar B 7ab02a67bc HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 
(cherry picked from commit 30eceec342)
 2019-10-04 13:53:50 -07:00
Erik Krogen 9fdb849e03 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
(cherry picked from 6630c9b75d)
 2019-10-04 11:34:37 -07:00
Erik Krogen 56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa)
(cherry picked from 69b0c513a9)
 2019-10-04 10:48:17 -07:00
Siyao Meng 61dc877b00 HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b)
 2019-10-04 09:19:30 -07:00
Vinayakumar B 1a7cb7aba1 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b8)
(cherry picked from commit abe14d32d4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2019-10-03 16:59:00 -07:00
Brahma Reddy Battula ddea9d68a0 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 15d8f592b51d08a658e88c6a7a7596e0edf5793b)
(cherry picked from commit 7e10dd03b5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2019-10-03 16:16:23 -07:00
Kitti Nanasi aeb3ae1c4c HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc9200)
 2019-10-03 15:59:41 -07:00
LeonGao91 a989779f2d HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 55e40227d1d20bf88448e213937f6e3166452dc9)
 2019-10-02 17:35:25 -07:00
Inigo Goiri 2a3eb04996 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb8)
(cherry picked from commit e59ced9c60a7007551ee8a9f83ce8e266e4dbae1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2019-10-01 16:56:29 -07:00
Chen Liang 3d7730266a HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:59:08 -07:00
Anu Engineer eb5c4ce432 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14)
 2019-09-14 01:35:12 +09:00
Zhankun Tang ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Stephen O'Donnell 63b18d764a HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0)
(cherry picked from commit 2c7f49743fba587e32398c81412db9eac426cf98)
 2019-09-02 09:58:48 -07:00
Wei-Chiu Chuang 8ffaef78da Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit d42b48d4d0.
 2019-09-02 09:58:32 -07:00
Stephen O'Donnell d42b48d4d0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9)
(cherry picked from commit 9c0d6e1657)
 2019-08-29 18:26:14 -07:00
Chen Liang 0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang 3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang 3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Wei-Chiu Chuang d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde2)
(cherry picked from commit b004d305e9)
 2019-08-20 10:34:25 -07:00
Siyao Meng 224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Masatake Iwasaki e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)
 2019-08-14 09:07:00 +09:00
Chao Sun 90447baeb0 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
 2019-08-08 10:42:27 -07:00
Erik Krogen da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Erik Krogen cc22373606 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-12 12:25:19 -07:00
Erik Krogen 769b5a802b HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 35200b3671 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00