bad3ffd290
HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-02-22 13:58:49 -08:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
0ab7fc9200
HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-31 16:07:24 -08:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
bd2a59e508
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2018-12-24 09:39:20 -08:00
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
b8ad6c85a5
HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
71cf66e29b
HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
3bb92a1d9a
HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
2018-12-24 09:34:01 -08:00
a3aab48df0
HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
1a9ba9616a
HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
25b63e8da8
HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
aa42fb0db7
HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
f9fc01cd7f
HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
25d8e39b5c
HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
eae0a5d54a
HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
0105d7f597
HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
2018-12-24 09:33:59 -08:00
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
a109f2b32f
HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
788e7473a4
HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.
2018-12-17 11:24:57 +05:30
f858f18554
HDFS-14088. RequestHedgingProxyProvider can throw NullPointerException when failover due to no lock on currentUsedProxy. Contributed by Yuxuan Wang.
2018-12-14 11:04:36 -08:00
aa12859890
HDFS-14121. Log message about the old hosts file format is misleading
...
(Contributed by Zsolt Venczel via Daniel Templeton)
Change-Id: I7ff548f6c82e0aeb08a7a50ca7c2c827db8726bb
2018-12-14 13:54:57 +01:00
4aa0609fb0
HDFS-13985. Clearer error message for ReplicaNotFoundException
...
(Contributed by Adam Antal via Daniel Templeton)
Change-Id: I68ae7a5bedecbc204c161fc3b3ce8878fe5d493d
2018-12-13 16:21:45 +01:00
39dc7345b8
HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena.
2018-12-11 17:59:04 +05:30
f534736867
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
2018-11-30 00:18:27 +05:30
892b33e054
HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena.
2018-11-22 00:13:56 +05:30
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
887244de4a
HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen.
2018-11-06 11:18:15 -08:00
d98b881e9a
HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-11-01 17:26:00 -07:00
db7e636824
HDFS-14027. DFSStripedOutputStream should implement both hsync methods.
2018-10-29 19:06:15 -07:00
635786a511
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2018-10-23 14:53:45 -07:00
88cce32551
HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin.
2018-10-19 10:38:04 -07:00
b6fc72a025
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
2018-10-15 10:51:55 -07:00
b60ca37914
Fix potential FSImage corruption. Contributed by Daryn Sharp.
2018-10-15 15:48:26 +05:30
5ec86b445c
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
2018-10-12 09:35:52 -07:00
bf3d591f0c
HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:23:07 -07:00
3ead525c71
HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:11:10 -07:00
10185d9a77
HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-10 08:55:32 -07:00
08bb6c49a5
HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads.
...
Contributed by Xiao Chen, Hrishikesh Gadre.
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-08 20:31:57 -07:00
396ce0d9f4
HDFS-13877. HttpFS: Implement GETSNAPSHOTDIFF. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-04 15:17:47 -07:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
f6c5ef9903
HDFS-13943. [JDK10] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Akira Ajisaka.
2018-10-02 09:49:48 +09:00
28ceb34a72
HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh.
2018-09-19 03:12:20 -07:00
a1de8cbac5
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-09-14 05:22:56 +08:00
30eceec342
HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena.
2018-09-10 09:10:51 +05:30
eca1a4bfe9
HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering.
2018-09-06 14:48:00 -07:00
6e5ffb74dd
HDFS-13885. Add debug logs in dfsclient around decrypting EDEK. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-03 22:37:37 -07:00
8aa6c4f079
Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
...
This reverts commit 26c2a97c56
2018-08-30 11:44:20 -07:00
26c2a97c56
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-08-27 16:02:35 -07:00
a4121c71c2
HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
2018-08-24 18:27:30 -07:00
c67b0650ea
HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
2018-08-16 23:14:21 -07:00
39ed3a66db
HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
4402f3f855
HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.
2018-08-12 03:06:04 -07:00
3b83110d5e
HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.
2018-08-12 03:06:04 -07:00
c561cb316e
HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
68017e3349
HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
6215e35bb6
HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu
2018-08-12 03:05:57 -07:00
cd5262aba0
HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou
2018-08-12 03:05:55 -07:00
b67ae6d9d7
HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu
2018-08-12 03:05:54 -07:00
9499df7b81
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
2018-08-08 10:40:20 -07:00
849c45db18
HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
...
(Contributed by BELUGA BEHR via Daniel Templeton)
Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
2018-07-24 16:05:27 -07:00
0247cb6318
HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
2018-07-06 14:59:49 -07:00
344f324710
HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
2018-07-03 11:07:45 +02:00
1804a31515
HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-07-02 10:48:20 -07:00
fef20a446f
HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
2018-07-02 09:41:15 -07:00
5d748bd056
HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
2018-07-02 12:11:06 +02:00
6ba9974108
HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
2018-07-02 12:02:19 +02:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
fba9d7cd74
HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. Contributed by Takanobu Asanuma.
2018-06-18 10:17:07 -07:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
2e5cfe6df3
HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
2018-06-11 11:20:29 -07:00
000a678396
HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.
2018-06-08 17:57:12 -07:00
fba1c42adc
HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
2018-06-08 16:36:42 -07:00
774c1f199e
HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.
2018-06-05 21:25:08 -07:00
ba4011d64f
HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
2018-06-05 21:18:31 -07:00
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
c95b9b5c07
HDFS-13440. Log HDFS file name when client fails to connect. Contributed by Gabor Bota.
2018-05-31 15:20:59 -07:00
1361030e59
HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
2018-05-31 07:59:21 -06:00
c9b63deb53
HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
2018-05-24 09:56:23 -07:00
bc6d9d4c79
HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
2018-05-23 10:15:40 -07:00
34e8b9f9a8
HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
2018-05-23 19:10:09 +08:00
1d2640b613
HDFS-13601. Optimize ByteString conversions in PBHelper.
2018-05-22 23:55:20 -07:00
Erik Krogen
Surendra Singh Lilhore
Chen Liang
Steve Loughran
Kitti Nanasi
Akira Ajisaka
Dinesh Chitlangia
Inigo Goiri
Giovanni Matteo Fumarola
shwetayakkali@cloudera.com
Konstantin V Shvachko
Chao Sun
Plamen Jeliazkov
Zsolt Venczel
Daniel Templeton
Vinayakumar B
Brahma Reddy Battula
Jason Lowe
Pranay Singh
Xiao Chen
Hrishikesh Gadre
Weiwei Yang
Siyao Meng
Sunil G
Takanobu Asanuma
Wei-Chiu Chuang
Uma Maheswara Rao G
Rakesh Radhakrishnan
Andrew Wang
Todd Lipcon
Chris Douglas
Arpit Agarwal
Sean Mackrory
Bharat Viswanadham
Sammi Chen