59aac00283
HDFS-15066. HttpFS: Implement setErasureCodingPolicy , unsetErasureCodingPolicy , getErasureCodingPolicy. Contributed by hemanthboyina.
2020-01-07 11:10:32 +09:00
074050ca59
HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina.
2020-01-01 11:26:38 +09:00
62423910a4
HDFS-14937. [SBN read] ObserverReadProxyProvider should throw InterruptException. Contributed by xuzq.
2019-12-29 13:07:22 +05:30
df622cf4a3
HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui.
2019-12-25 11:07:25 +05:30
d8cd7098b4
HDFS-15073. Replace curator-shaded guava import with the standard one ( #1782 )
2019-12-25 10:50:01 +09:00
0e28cd8f63
HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-12 02:23:32 -08:00
c2e9783d5f
HDFS-15045. DataStreamer#createBlockOutputStream() should log exception in warn. Contributed by Ravuri Sushma Sree.
2019-12-11 08:52:55 +05:30
c8bef4d6a6
HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun.
2019-11-27 10:57:20 +08:00
9b6906fe91
HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun.
2019-11-22 10:53:55 +08:00
b3119b9ab6
HDFS-14648. Implement DeadNodeDetector basic model. Contributed by Lisheng Sun.
2019-11-16 11:32:41 +08:00
c36014165c
HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore.
2019-11-06 19:28:55 +05:30
30db895b59
HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() ( #1667 )
...
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-25 13:09:14 -07:00
72ae371e7a
HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko.
2019-10-08 11:56:52 -07:00
382967be51
HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore.
2019-10-08 00:14:30 +05:30
13b427fc05
HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable ( #963 ) Contributed by Sahil Takiar.
...
Reviewed-by: Siyao Meng <smeng@cloudera.com>
2019-09-27 12:25:03 -07:00
2adcc3c932
HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang.
2019-09-26 15:48:19 -07:00
1a2a352ecd
HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya.
2019-09-26 20:45:37 +05:30
e346e3638c
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
2019-09-25 12:16:41 +01:00
816d3cb908
HDFS-14837. Review of Block.java. Contributed by David Mollitor.
2019-09-24 09:54:09 -07:00
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes ( #1494 ). Contributed by Vinayakumar B.
2019-09-23 12:37:47 +05:30
3f223bebfa
HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun.
2019-09-20 11:45:01 -07:00
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 ( #1432 )
...
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
91b01a1db7
HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena.
2019-09-03 18:08:42 +05:30
915cbc91c0
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-02 09:47:04 -07:00
d207aba026
Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell."
...
This reverts commit 7bebad61d9
2019-09-02 09:43:44 -07:00
7bebad61d9
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-29 17:38:07 -07:00
29bd6f3fc3
HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun.
2019-08-28 23:58:23 +05:30
b69ac575a1
HDFS-14772. RBF: hdfs-rbf-site.xml can't be loaded automatically. Contributed by Yuxuan Wang
2019-08-27 14:32:32 +09:00
e04dcfdc57
HDFS-14583. FileStatus#toString() will throw IllegalArgumentException. Contributed by xuzq.
2019-08-22 10:22:38 -07:00
5eeb6da2d4
HDFS-14714. RBF: implement getReplicatedBlockStats interface. Contributed by Chen Zhang.
2019-08-21 09:38:17 -07:00
8aaf5e1a14
HDFS-14759. HDFS cat logs an info message.
...
Contributed by Eric Badger.
2019-08-20 20:24:19 -07:00
aa6995fde2
HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan.
2019-08-20 10:32:55 -07:00
9a1d8cfaf5
HDFS-14678. Allow triggerBlockReport to a specific namenode. ( #1252 ). Contributed by Leon Gao.
2019-08-16 08:00:51 -07:00
3c0382f1b9
HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2019-08-14 07:30:35 -07:00
da0006fe04
HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-14 08:39:40 +09:00
28a848412c
HDFS-14701. Change Log Level to warn in SlotReleaser. Contributed by Lisheng Sun.
2019-08-08 13:46:31 -07:00
e7a0b8aa83
HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira.
2019-08-02 11:48:31 -07:00
17e8cf501b
HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null ( #1192 ) Contributed by Siyao Meng.
2019-08-01 17:15:22 -07:00
3ae775d740
HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-07-30 16:01:17 -07:00
ecc8acfd24
HDFS-14673. The console log is noisy when using DNSDomainNameResolver to resolve NameNode.
2019-07-26 11:08:46 -07:00
8fb5ca3f40
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-07-11 13:23:25 -07:00
729cb3aefe
HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang.
2019-07-04 09:37:24 +08:00
041e0c0564
HDFS-14036. RBF: Add hdfs-rbf-default.xml to HdfsConfiguration by default. Contributed by Takanobu Asanuma.
2019-06-27 09:57:52 -07:00
b76b843c8b
HDFS-13371. NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X. Contributed by Sherwood Zheng.
2019-06-24 17:52:33 -07:00
38a560c6f1
Merge pull request #977 from leosunli/trunk
...
HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
2019-06-24 17:44:54 -07:00
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.
2019-06-24 11:59:07 -07:00
50aee18a84
HDFS-14268. RBF: Fix the location of the DNs in getDatanodeReport(). Contributed by Inigo Goiri.
2019-06-24 22:03:03 +05:30
f40e10b349
HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar.
2019-06-24 22:03:03 +05:30
daa1e14745
when evictableMmapped or evictable size is zero, do not throw NoSuchElementException
...
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
2019-06-24 20:17:22 +08:00
3c1a1ceea9
HDFS-14487. Missing Space in Client Error Message (Contributed by Shweta Yakkali via Daniel Templeton)
...
Change-Id: I0f8ce74a35ab24fe94fd0e57d8247bb3fa575e6f
2019-06-18 10:21:22 -07:00
62ad9885ea
HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu.
2019-06-17 20:18:53 -07:00
cc1630288e
HDFS-14556: Spelling Mistake "gloablly" ( #938 )
2019-06-16 18:32:42 -07:00
ea3b0a1844
HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
2019-06-04 08:48:05 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
d78854b928
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
...
Contributed by KWON BYUNGCHANG
2019-05-28 17:31:35 -04:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
3e5e5b028a
HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-17 19:20:28 +02:00
5847e00143
HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen.
2019-05-13 08:39:59 -07:00
1d59cc490c
HADOOP-16292. Refactor checkTrustAndSend in SaslDataTransferClient to make it cleaner. Contributed by Sherwood Zheng.
2019-05-03 10:14:17 -07:00
f1875b205e
HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena.
2019-05-03 11:22:14 +05:30
d6b7609c96
HDFS-14453. Improve Bad Sequence Number Error Message. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-02 14:25:02 -07:00
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
174b7d3126
HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
2019-04-22 13:29:56 -07:00
685cb83e4c
HDFS-14433. Remove the extra empty space in the DataStreamer logging. Contributed by Yishuang Lu. ( #747 )
2019-04-17 10:38:48 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
260d843b25
HDFS-14416. Fix TestHdfsConfigFields for field dfs.client.failover.resolver.useFQDN. Contributed by Fengnan Li.
2019-04-05 09:26:06 -07:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
49b02d4a9b
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor.
2019-03-28 11:16:01 -07:00
55fb3c32fb
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
2019-03-27 03:27:02 +09:00
55b3a718e9
HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-03-19 08:14:49 -07:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
618e009ac0
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 15:04:06 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
f940ab242d
HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena.
2019-03-05 19:26:42 +05:30
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
bad3ffd290
HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-02-22 13:58:49 -08:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
0ab7fc9200
HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-31 16:07:24 -08:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
bd2a59e508
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2018-12-24 09:39:20 -08:00
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
b8ad6c85a5
HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
71cf66e29b
HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
3bb92a1d9a
HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
2018-12-24 09:34:01 -08:00
a3aab48df0
HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Takanobu Asanuma
Ayush Saxena
Chandra Sanivarapu
He Xiaoqiao
Surendra Singh Lilhore
Yiqun Lin
zhaoyim
Chen Liang
Sahil Takiar
Steve Loughran
Inigo Goiri
Vinayakumar B
Stephen O'Donnell
Wei-Chiu Chuang
Stephen O'Donnell
Anu Engineer
LeonGao91
Siyao Meng
Masatake Iwasaki
Erik Krogen
Siyao Meng
Chao Sun
HUAN-PING SU
Weiwei Yang
Inigo Goiri
Ajay Kumar
Giovanni Matteo Fumarola
sunlisheng
Shweta Yakkali
mpicker90
huzheng
Akira Ajisaka
Eric Yang
Srinivasu Majeti
lys0716
Sahil Takiar
Yongjun Zhang
Steve Loughran
Kitti Nanasi
Dinesh Chitlangia
Konstantin V Shvachko