8ffc356b1e
Revert "SPNEGO TLS verification"
...
This reverts commit ba66f3b454
2020-05-13 17:14:14 +09:00
29dddb8a14
HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.
2020-05-12 15:07:51 +01:00
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. ( #2008 )
2020-05-12 09:04:03 -05:00
4c53fb9ce1
HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.
2020-05-11 16:48:34 +05:30
aab9e0b16e
HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest.
2020-05-10 11:43:30 +05:30
7fddf4855e
HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.
2020-05-06 16:55:04 +05:30
ba66f3b454
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-05-02 21:50:13 +09:00
b5b45c53a4
HDFS-15309. Remove redundant String.valueOf method on ExtendedBlockId.java. Contributed by bianqi.
2020-04-30 15:46:46 +09:00
db6252b6c3
HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore.
2020-04-29 10:58:35 +05:30
37d6582223
HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.
2020-04-17 13:32:02 +05:30
8914cf9167
Preparing for 3.4.0 development
2020-03-29 23:24:25 +05:30
f531a4a487
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.
2020-03-27 12:55:38 -07:00
d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error ( #1902 ). Contributed by zhengchenyu.
2020-03-24 23:17:22 +05:30
b74e47e31a
HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.
2020-03-22 01:44:55 +09:00
1d772dc542
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
2020-03-15 20:44:32 +05:30
74780c22eb
HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang.
2020-03-15 16:30:39 +05:30
7b637a3341
HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.
2020-03-13 19:01:23 -07:00
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
...
Contributed by Steve Loughran.
This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)
This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.
instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.
The FileSystem implementation is cut; tests updated as appropriate.
All new interfaces/classes/constants are marked as @unstable.
Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
...
Contributed by Steve Loughran.
Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.
This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.
Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
2020-03-09 14:44:28 +00:00
0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file
...
Contributed by Xiaoyu Yao.
Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails
Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.
This is a convergence of a fix for S3 meeting an issue in HDFS.
S3: a HEAD against a file can cache a 404,
-you must not do any existence checks, including deleteOnExit(),
until the file is written.
Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct.
HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.
This patch:
* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
2020-03-02 13:22:00 +00:00
97b797c314
HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.
2020-02-28 18:47:22 -08:00
ed70c115a8
HDFS-15172. Remove unnecessary deadNodeDetectInterval in DeadNodeDetector#checkDeadNodes(). Contributed by Lisheng Sun.
2020-02-21 16:36:30 -08:00
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. ( #1758 )
2020-02-21 11:56:07 +09:00
ac4b556e2d
HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena.
2020-02-19 08:20:59 +05:30
f09710bbb8
HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun
2020-02-12 20:29:35 +05:30
9b8a78d97b
HDFS-14758. Make lease hard limit configurable and reduce the default.
...
Contributed by hemanthboyina.
2020-02-11 12:40:00 -06:00
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
483397c7f7
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang
2020-01-28 15:20:36 -08:00
3e86807802
Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang."
...
This reverts commit ff8ff0f7e5
2020-01-28 15:19:47 -08:00
ff8ff0f7e5
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang.
2020-01-28 15:14:58 -08:00
f876dc228b
HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena.
2020-01-28 19:34:21 +05:30
d10f77e3c9
HDFS-15119. Allow expiration of cached locations in DFSInputStream.
...
Contributed by Ahmed Hussein.
2020-01-24 09:15:27 -06:00
92c58901d7
HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena
2020-01-23 18:18:34 +05:30
edbbc03ce7
HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. ( #1803 )
2020-01-16 23:27:50 +05:30
d7c4f8ab21
HDFS-13616. Batch listing of multiple directories ( #1725 )
2020-01-15 17:22:47 -08:00
b32757c616
HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena.
2020-01-10 08:14:56 +05:30
aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. ( #1797 )
2020-01-08 16:45:39 +09:00
59aac00283
HDFS-15066. HttpFS: Implement setErasureCodingPolicy , unsetErasureCodingPolicy , getErasureCodingPolicy. Contributed by hemanthboyina.
2020-01-07 11:10:32 +09:00
074050ca59
HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina.
2020-01-01 11:26:38 +09:00
62423910a4
HDFS-14937. [SBN read] ObserverReadProxyProvider should throw InterruptException. Contributed by xuzq.
2019-12-29 13:07:22 +05:30
df622cf4a3
HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui.
2019-12-25 11:07:25 +05:30
d8cd7098b4
HDFS-15073. Replace curator-shaded guava import with the standard one ( #1782 )
2019-12-25 10:50:01 +09:00
0e28cd8f63
HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-12 02:23:32 -08:00
c2e9783d5f
HDFS-15045. DataStreamer#createBlockOutputStream() should log exception in warn. Contributed by Ravuri Sushma Sree.
2019-12-11 08:52:55 +05:30
c8bef4d6a6
HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun.
2019-11-27 10:57:20 +08:00
9b6906fe91
HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun.
2019-11-22 10:53:55 +08:00
b3119b9ab6
HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun.
2019-11-16 11:32:41 +08:00
c36014165c
HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore.
2019-11-06 19:28:55 +05:30
30db895b59
HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() ( #1667 )
...
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-25 13:09:14 -07:00
72ae371e7a
HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko.
2019-10-08 11:56:52 -07:00
Akira Ajisaka
S O'Donnell
Takanobu Asanuma
Ayush Saxena
Eric Yang
Surendra Singh Lilhore
Brahma Reddy Battula
Chen Liang
Isa Hekmatizadeh
Wei-Chiu Chuang
Steve Loughran
Xiaoyu Yao
Inigo Goiri
Masatake Iwasaki
Kihwal Lee
Vinayakumar B
Chao Sun
Chandra Sanivarapu
He Xiaoqiao
Yiqun Lin
zhaoyim