7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
49b02d4a9b
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor.
2019-03-28 11:16:01 -07:00
55fb3c32fb
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
2019-03-27 03:27:02 +09:00
55b3a718e9
HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-03-19 08:14:49 -07:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
618e009ac0
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 15:04:06 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
f940ab242d
HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena.
2019-03-05 19:26:42 +05:30
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
bad3ffd290
HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-02-22 13:58:49 -08:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
0ab7fc9200
HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-31 16:07:24 -08:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
bd2a59e508
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2018-12-24 09:39:20 -08:00
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
b8ad6c85a5
HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
71cf66e29b
HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
3bb92a1d9a
HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena.
2018-12-24 09:34:01 -08:00
a3aab48df0
HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
1a9ba9616a
HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
25b63e8da8
HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
aa42fb0db7
HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
f9fc01cd7f
HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
25d8e39b5c
HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
eae0a5d54a
HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
0105d7f597
HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
2018-12-24 09:33:59 -08:00
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
a109f2b32f
HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
788e7473a4
HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.
2018-12-17 11:24:57 +05:30
f858f18554
HDFS-14088. RequestHedgingProxyProvider can throw NullPointerException when failover due to no lock on currentUsedProxy. Contributed by Yuxuan Wang.
2018-12-14 11:04:36 -08:00
aa12859890
HDFS-14121. Log message about the old hosts file format is misleading
...
(Contributed by Zsolt Venczel via Daniel Templeton)
Change-Id: I7ff548f6c82e0aeb08a7a50ca7c2c827db8726bb
2018-12-14 13:54:57 +01:00
4aa0609fb0
HDFS-13985. Clearer error message for ReplicaNotFoundException
...
(Contributed by Adam Antal via Daniel Templeton)
Change-Id: I68ae7a5bedecbc204c161fc3b3ce8878fe5d493d
2018-12-13 16:21:45 +01:00
39dc7345b8
HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena.
2018-12-11 17:59:04 +05:30
f534736867
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
2018-11-30 00:18:27 +05:30
892b33e054
HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena.
2018-11-22 00:13:56 +05:30
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
887244de4a
HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen.
2018-11-06 11:18:15 -08:00
d98b881e9a
HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-11-01 17:26:00 -07:00
db7e636824
HDFS-14027. DFSStripedOutputStream should implement both hsync methods.
2018-10-29 19:06:15 -07:00
635786a511
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2018-10-23 14:53:45 -07:00
88cce32551
HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin.
2018-10-19 10:38:04 -07:00
b6fc72a025
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
2018-10-15 10:51:55 -07:00
b60ca37914
Fix potential FSImage corruption. Contributed by Daryn Sharp.
2018-10-15 15:48:26 +05:30
5ec86b445c
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
2018-10-12 09:35:52 -07:00
bf3d591f0c
HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:23:07 -07:00
3ead525c71
HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:11:10 -07:00
10185d9a77
HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-10 08:55:32 -07:00
08bb6c49a5
HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads.
...
Contributed by Xiao Chen, Hrishikesh Gadre.
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-08 20:31:57 -07:00
396ce0d9f4
HDFS-13877. HttpFS: Implement GETSNAPSHOTDIFF. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-04 15:17:47 -07:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
f6c5ef9903
HDFS-13943. [JDK10] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Akira Ajisaka.
2018-10-02 09:49:48 +09:00
28ceb34a72
HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh.
2018-09-19 03:12:20 -07:00
a1de8cbac5
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-09-14 05:22:56 +08:00
30eceec342
HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena.
2018-09-10 09:10:51 +05:30
eca1a4bfe9
HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering.
2018-09-06 14:48:00 -07:00
6e5ffb74dd
HDFS-13885. Add debug logs in dfsclient around decrypting EDEK. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-03 22:37:37 -07:00
8aa6c4f079
Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
...
This reverts commit 26c2a97c56
2018-08-30 11:44:20 -07:00
26c2a97c56
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-08-27 16:02:35 -07:00
a4121c71c2
HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
2018-08-24 18:27:30 -07:00
c67b0650ea
HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
2018-08-16 23:14:21 -07:00
39ed3a66db
HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
4402f3f855
HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.
2018-08-12 03:06:04 -07:00
3b83110d5e
HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.
2018-08-12 03:06:04 -07:00
c561cb316e
HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
68017e3349
HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
6215e35bb6
HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu
2018-08-12 03:05:57 -07:00
cd5262aba0
HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou
2018-08-12 03:05:55 -07:00
b67ae6d9d7
HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu
2018-08-12 03:05:54 -07:00
9499df7b81
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
2018-08-08 10:40:20 -07:00
849c45db18
HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
...
(Contributed by BELUGA BEHR via Daniel Templeton)
Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
2018-07-24 16:05:27 -07:00
0247cb6318
HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
2018-07-06 14:59:49 -07:00
344f324710
HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
2018-07-03 11:07:45 +02:00
1804a31515
HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-07-02 10:48:20 -07:00
fef20a446f
HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
2018-07-02 09:41:15 -07:00
5d748bd056
HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
2018-07-02 12:11:06 +02:00
6ba9974108
HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
2018-07-02 12:02:19 +02:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
fba9d7cd74
HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. Contributed by Takanobu Asanuma.
2018-06-18 10:17:07 -07:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
2e5cfe6df3
HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
2018-06-11 11:20:29 -07:00
000a678396
HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.
2018-06-08 17:57:12 -07:00
fba1c42adc
HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
2018-06-08 16:36:42 -07:00
774c1f199e
HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.
2018-06-05 21:25:08 -07:00
ba4011d64f
HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
2018-06-05 21:18:31 -07:00
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
c95b9b5c07
HDFS-13440. Log HDFS file name when client fails to connect. Contributed by Gabor Bota.
2018-05-31 15:20:59 -07:00
1361030e59
HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
2018-05-31 07:59:21 -06:00
c9b63deb53
HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
2018-05-24 09:56:23 -07:00
bc6d9d4c79
HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
2018-05-23 10:15:40 -07:00
34e8b9f9a8
HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
2018-05-23 19:10:09 +08:00
1d2640b613
HDFS-13601. Optimize ByteString conversions in PBHelper.
2018-05-22 23:55:20 -07:00
0fc988e6a3
HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar.
2018-05-16 11:28:39 -07:00
960940e0e0
HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure.
2018-05-14 09:28:39 -07:00
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-27 19:21:37 +09:00
a8e428b2dc
HDFS-13468. Add erasure coding metrics into ReadStatistics. (Contributed by Lei (Eddy) Xu)
2018-04-26 13:54:40 -07:00
6f655aed3b
HADOOP-15404. Remove multibyte characters in DataNodeUsageReportUtil
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-26 13:54:40 -07:00
0ec88ea42b
HDFS-13301. Ozone: Remove containerPort, ratisPort and ozoneRestPort from DatanodeID and DatanodeIDProto. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
4e61bc431e
HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
...
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
5e7164c614
HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
...
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
9a914126a7
HDFS-13008. Ozone: Add DN container open/close state to container report. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
b4a3cf1476
HDFS-13070. Ozone: SCM: Support for container replica reconciliation - 1. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
4c10a849e8
HDFS-13078. Ozone: Update Ratis on Ozone to 0.1.1-alpha-8fd74ed-SNAPSHOT.
...
To fix large chunk reads (>4M) from Datanodes. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
9272e1021d
HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
377b31ffa1
HDFS-13022. Block Storage: Kubernetes dynamic persistent volume provisioner. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
ee5495456e
HDFS-13116. Ozone: Refactor Pipeline to have transport and container specific information. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
3cf07b43bc
HDFS-13127. Fix TestContainerStateManager and TestOzoneConfigurationFields. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
443425a5d9
HDFS-12522. Ozone: Remove the Priority Queues used in the Container State Manager. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
6b03fca37d
HDFS-12868. Ozone: Service Discovery API. Contributed by Nanda Kumar.
2018-04-26 05:36:04 -07:00
fb09d75379
HDFS-13018. Block Storage: make the iscsi target addres configurable for discovery. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
ba4d5a52a8
HDFS-12986. Ozone: Update ozone to latest ratis snapshot build (0.1.1-alpha-0f7169d-SNAPSHOT). Contributed by Lokesh Jain
2018-04-26 05:36:04 -07:00
94c0346f35
HDFS-13024. Ozone: ContainerStateMachine should synchronize operations between createContainer and writeChunk. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
28b87af51b
HDFS-13013. Fix closeContainer API with the right container state change. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
98d62e55c4
Revert "HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee."
...
This reverts commit 6ce5ec6761
2018-04-26 05:36:04 -07:00
98eeabbdb3
HDFS-12794. Ozone: Parallelize ChunkOutputSream Writes to container. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
98973cd0c4
HDFS-12966. Ozone: owner name should be set properly when the container allocation happens. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
3965f1ec99
HDFS-12853. Ozone: Optimize chunk writes for Ratis by avoiding double writes. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
5766792cad
HDFS-12870. Ozone: Service Discovery: REST endpoint in KSM for getServiceList. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
5f10c2d8ce
HDFS-12980. Ozone: SCM: Restructuring container state transition and event. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
87b7774488
HDFS-12958. Ozone: remove setAllocatedBytes method in ContainerInfo. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
ce19f09bd0
HDFS-12908. Ozone: write chunk call fails because of Metrics registry exception. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
4b9f66a19e
HDFS-12869. Ozone: Service Discovery: RPC endpoint in KSM for getServiceList. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
966853894f
HDFS-12795. Ozone: SCM: Support for Container LifeCycleState PENDING_CLOSE and LifeCycleEvent FULL_CONTAINER. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
26e270b908
HDFS-12751. Ozone: SCM: update container allocated size to container db for all the open containers in ContainerStateManager#close. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
fd09c2ce5b
HDFS-12890 . Ozone: XceiverClient should have upper bound on async requests. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
2f3068bc72
HDFS-12698. Ozone: Use time units in the Ozone configuration values. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
8ff98e2af3
HDFS-12000. Ozone: Container : Add key versioning support-1. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
c0c87dea9b
HDFS-12626. Ozone : delete open key entries that will no longer be closed. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
74362cb503
HDFS-12901. Ozone: SCM: Expose StorageContainerManager#getScmId through container protocol. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
7bba3943f0
HDFS-12876. Ozone: moving NodeType from OzoneConsts to Ozone.proto. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
8023ba50b5
HDFS-12807. Ozone: Expose RockDB stats via JMX for Ozone metadata stores. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
86802a461c
HDFS-12851. Ozone: Upgrade to latest ratis build. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
f9c11d952c
HDFS-12740. SCM should support a RPC to share the cluster Id with KSM and DataNodes. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
4460ac9c57
HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
740a06cdd7
HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
e395244e95
HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
cec96b296f
HDFS-12549. Ozone: OzoneClient: Support for REST protocol. Contributed by Nanda Kumar.
2018-04-26 05:36:04 -07:00
9734f505ea
HDFS-12739. Add Support for SCM --init command. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
709d56fdc9
HDFS-12474. Ozone: SCM: Handling container report with key count and container usage. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
2ec24a7542
HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar.
2018-04-26 05:36:04 -07:00
7ebe79e879
HDFS-12720. Ozone: Ratis options are not passed from KSM Client protobuf helper correctly. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
76d34bca62
HDFS-12521. Ozone: SCM should read all Container info into memory when booting up. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
6291ca1e32
HDFS-12663. Ozone: OzoneClient: Remove protobuf classes exposed to clients through OzoneBucket. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
1b56a73a4f
HDFS-12680. Ozone: SCM: Lease support for container creation. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
a467017068
HDFS-12411. Ozone: Add container usage information to DN container report. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
9bb516118d
HDFS-12490. Ozone: OzoneClient: Add creation/modification time information in OzoneVolume/OzoneBucket/OzoneKey. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
c85d3b2d25
HDFS-12593. Ozone: update Ratis to the latest snapshot. Contributed by Tsz Wo Nicholas Sze.
2018-04-26 05:36:04 -07:00
d19b4c8763
HDFS-12583. Ozone: Fix swallow exceptions which makes hard to debug failures. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
e3b51d9074
HDFS-12598. Ozone: Fix 3 node ratis replication in Ozone. Contributed by Mukul Kumar Singh
2018-04-26 05:36:04 -07:00
3504af9411
HDFS-12537. Ozone: Reduce key creation overhead in Corona. Contributed by Lokesh Jain.
2018-04-26 05:36:04 -07:00
9630621be7
HDFS-12610. Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
c3ef381011
HDFS-12543. Ozone : allow create key without specifying size. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
c17521b1bd
HDFS-12387. Ozone: Support Ratis as a first class replication mechanism. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
0b3270a8de
HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar.
2018-04-26 05:36:04 -07:00
6a754caca0
HDFS-12568. Ozone: Cleanup the ozone-default.xml. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
05246e2b32
HDFS-12060. Ozone: OzoneClient: Add list calls in RpcClient. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
7151a9b604
HDFS-12488. Ozone: OzoneRestClient timeout is not configurable. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
9530153f33
HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
6b10723ec3
HDFS-12525. Ozone: OzoneClient: Verify bucket/volume name in create calls. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
a1a3ba6529
HDFS-12454. Ozone : the sample ozone-site.xml in OzoneGettingStarted does not work. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
2804435386
HDFS-12017. Ozone: Container: Move IPC port to 98xx range. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
9ff136bb02
HDFS-12210. Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
dddded0016
Revert "Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh." to fix commit message.
...
This reverts commit 087c69ba24
2018-04-26 05:36:04 -07:00
8052374e79
Block Storage: volume creation times out while creating 3TB volume because of too many containers. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
fd1564b87e
HDFS-12506. Ozone: ListBucket is too slow. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
bf6f0cd831
HDFS-11563. Ozone: enforce DependencyConvergence uniqueVersions.
2018-04-26 05:36:04 -07:00
27ec22e20d
HDFS-12517. Ozone: mvn package is failing with out skipshade. Contributed by Bharat Viswanadham.
2018-04-26 05:36:04 -07:00
3b290fe3bf
HDFS-12466. Ozone: KSM: Make ozone.ksm.address as mandatory property for client. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
c329d3b4b4
HDFS-12329. Ozone: Ratis: Readonly calls in XceiverClientRatis should use sendReadOnly. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
77d4b18700
HDFS-12385. Ozone: OzoneClient: Refactoring OzoneClient API. Contributed by Nadakumar.
2018-04-26 05:36:04 -07:00
fe4139bdc3
HDFS-11612. Ozone: Cleanup Checkstyle issues. Contributed by Shashikant Banerjee.
2018-04-26 05:36:04 -07:00
b127ecd1ba
Revert "HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin." as it uses class not part of JRE.
...
This reverts commit c29aff44cf
2018-04-26 05:36:04 -07:00
f66a522695
HDFS-12268. Ozone: Add metrics for pending storage container requests. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
1087ce4cd2
HDFS-11676. Ozone: SCM CLI: Implement close container command. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
743be0d7c0
HDFS-12235. Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions. Contributed by Weiwei Yang and Yuanbo Liu.
2018-04-26 05:36:04 -07:00
91a8f134b7
HDFS-11834. Ozone: Fix TestArchive#testArchive. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
7b3e7eb565
HDFS-12354. Ozone: Shuffle container list for datanode BlockDeletingService. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
8654698d74
HDFS-12365. Ozone: ListVolume displays incorrect createdOn time when the volume was created by OzoneRpcClient. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
4d3b8d332d
HDFS-12283. Ozone: DeleteKey-5: Implement SCM DeletedBlockLog. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
02f8c2224f
HDFS-11888. Ozone: SCM: use state machine for open containers allocated for key/blocks. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
9e733215a8
HDFS-12327. Ozone: support setting timeout in background service. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
bd33038bf4
HDFS-12159. Ozone: SCM: Add create replication pipeline RPC. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
795ba1b3b7
HDFS-12313. Ozone: SCM: move container/pipeline StateMachine to the right package. Contributed by Xiaoyu Yao.
2018-04-26 05:36:04 -07:00
3a661b7f82
HDFS-12238. Ozone: Add valid trace ID check in sendCommandAsync. Contributed by Ajay Kumar.
2018-04-26 05:36:04 -07:00
f9bce29dcc
HDFS-12290. Block Storage: Change dfs.cblock.jscsi.server.address default bind address to 0.0.0.0. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
25586607a1
HDFS-12230. Ozone: KSM: Add creation time field in bucket info. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
c9ea431267
HDFS-12281. Ozone: Ozone-default.xml has 3 properties that do not match the default Config value. Contributed by Ajay Yadav.
2018-04-26 05:36:04 -07:00
639b4fb8a9
HDFS-12196. Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
bb9a4793d3
HDFS-12259. Ozone: OzoneClient: Refactor move ozone client from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
7290324fc8
HDFS-12267. Ozone: XceiverClientManager should not close the connection if client holds the reference. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
bb5f1c4228
HDFS-12231. Ozone: KSM: Add creation time field in volume info. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
89d8d20c73
HDFS-12246. Ozone: potential thread leaks. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
9cf40547ce
HDFS-11580. Ozone: Support asynchronus client API for SCM and containers. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
070f7e2e59
HDFS-11920. Ozone : add key partition. Contributed by Chen Liang.
2018-04-26 05:36:04 -07:00
bc413a69cc
HDFS-12195. Ozone: DeleteKey-1: KSM replies delete key request asynchronously. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
a4b3160efb
HDFS-12145. Ozone: OzoneFileSystem: Ozone & KSM should support / delimited key names. Contributed by Mukul Kumar Singh.
...
(cherry picked from commit fe0a957e92d506330ba005abba18f665eb5ac1cc)
2018-04-26 05:36:04 -07:00
8cf0c864c4
HDFS-12170. Ozone: OzoneFileSystem: KSM should maintain key creation time and modification time. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
0cc166c053
HDFS-12115. Ozone: SCM: Add queryNode RPC Call. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
f1e2592bf8
HDFS-12127. Ozone: Ozone shell: Add more testing for key shell commands. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
3d430a6076
HDFS-11493. Ozone: SCM: Add the ability to handle container reports. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
25c1b296e6
HDFS-11679. Ozone: SCM CLI: Implement list container command. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
b7b8511bae
HDFS-11773. Ozone: KSM : add listVolumes. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
0a8c903d3f
HDFS-12050. Ozone: StorageHandler: Implementation of close for releasing resources during shutdown. Contributed by Nandakumar.
2018-04-26 05:36:04 -07:00
fea1ab6192
HDFS-12007. Ozone: Enable HttpServer2 for SCM and KSM. Contributed by Elek, Marton.
2018-04-26 05:36:04 -07:00
77e3bb783b
HDFS-11970. Ozone: TestXceiverClientManager.testFreeByEviction fails occasionally. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
3b0996e2a1
HDFS-11585. Ozone: Support force update a container. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
e73d285567
HDFS-11782. Ozone: KSM: Add listKey. Contributed by Yiqun Lin.
2018-04-26 05:36:04 -07:00
938728744c
HDFS-11845. Ozone: Output error when DN handshakes with SCM. Contributed by Weiwei Yang
2018-04-26 05:36:04 -07:00
fa26dea89d
HDFS-11185. Ozone: remove disabled tests. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
0c37e05b10
HDFS-11779. Ozone: KSM: add listBuckets. Contributed by Weiwei Yang.
2018-04-26 05:36:04 -07:00
bacd1188f1
HDFS-11880. Ozone: KSM: Remove protobuf formats from KSM wrappers. Contributed by Nandakumar Vadivelu.
2018-04-26 05:36:04 -07:00
2007e85d5b
HDFS-11777. Ozone: KSM: add deleteBucket. Contributed by Nandakumar Vadivelu.
2018-04-26 05:36:04 -07:00
d6dd557b24
HDFS-11887. Shared XceiverClient should be closed if there is no open clients to avoid resource leak. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
bf8c168a80
HDFS-11781. Ozone: KSM: Add deleteKey. Contributed by Yuanbo Liu.
2018-04-26 05:36:04 -07:00
42eaecae55
HDFS-11771. Ozone: KSM: Add checkVolumeAccess. Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
a8c0976fb4
HDFS-11894. Ozone: Cleanup imports. Contributed by Weiwei Yang.
2018-04-26 05:35:59 -07:00
6b9915fcbd
HDFS-11774. Ozone: KSM: add deleteVolume. Contributed by Mukul Kumar Singh.
2018-04-26 05:34:18 -07:00
236c410881
HDFS-11776. Ozone: KSM: add SetBucketProperty. Contributed by Nandakumar Vadivelu.
2018-04-26 05:34:18 -07:00
72b228a9e6
HDFS-11853. Ozone: KSM: Add getKey. Contributed by Chen Liang.
2018-04-26 05:34:18 -07:00
b71efcf1b0
HDFS-11865. Ozone: Do not initialize Ratis cluster during datanode startup.
2018-04-26 05:34:18 -07:00
e3c8f6a24d
HDFS-11780. Ozone: KSM: Add putKey. Contributed by Chen Liang.
2018-04-26 05:34:18 -07:00
84294de9a2
HDFS-11778. Ozone: KSM: add getBucketInfo. Contributed by Nandakumar Vadivelu.
2018-04-26 05:34:18 -07:00
47c4867765
HDFS-11859. Ozone: SCM: Separate BlockLocationProtocol from ContainerLocationProtocol. Contributed by Xiaoyu Yao.
2018-04-26 05:34:18 -07:00
37642c12df
HDFS-11770. Ozone: KSM: Add setVolumeProperty. Contributed by Mukul Kumar Singh.
2018-04-26 05:34:18 -07:00
a0f51ee3d7
HDFS-11850. Ozone: Stack Overflow in XceiverClientManager because of race condition in accessing openClient. Contributed by Mukul Kumar Singh.
2018-04-26 05:34:18 -07:00
877e751c84
HDFS-11843. Ozone: XceiverClientRatis should implement XceiverClientSpi.connect().
2018-04-26 05:34:18 -07:00
4fb9064523
HDFS-11835. Block Storage: Overwrite of blocks fails. Contributed by Mukul Kumar Singh.
2018-04-26 05:34:18 -07:00
ca70300eea
HDFS-11775. Ozone: KSM: add createBucket. Contributed by Nandakumar Vadivelu.
2018-04-26 05:34:18 -07:00
c9e6d4378d
HDFS-11769. Ozone: KSM: Add createVolume API. Contributed by Mukul Kumar Singh.
2018-04-26 05:34:18 -07:00
9c9be9f7f7
HDFS-11693. Ozone:Add archive support to containers. Contributed by Anu Engineer.
2018-04-26 05:34:18 -07:00
b592df6413
HDFS-11811. Ozone: SCM: Support Delete Block. Contributed by Xiaoyu Yao.
2018-04-26 05:34:18 -07:00
Inigo Goiri
Giovanni Matteo Fumarola
Takanobu Asanuma
Erik Krogen
Sahil Takiar
Wei-Chiu Chuang
Vinayakumar B
Yongjun Zhang
Surendra Singh Lilhore
Chen Liang
Steve Loughran
Kitti Nanasi
Akira Ajisaka
Dinesh Chitlangia
shwetayakkali@cloudera.com
Konstantin V Shvachko
Chao Sun
Plamen Jeliazkov
Zsolt Venczel
Daniel Templeton
Brahma Reddy Battula
Jason Lowe
Pranay Singh
Xiao Chen
Hrishikesh Gadre
Weiwei Yang
Siyao Meng
Sunil G
Uma Maheswara Rao G
Rakesh Radhakrishnan
Andrew Wang
Todd Lipcon
Chris Douglas
Arpit Agarwal
Sean Mackrory
Bharat Viswanadham
Sammi Chen
Ewan Higgs
Lei Xu
Takanobu Asanuma
Nanda kumar
Anu Engineer
Mukul Kumar Singh
Xiaoyu Yao
Tsz-Wo Nicholas Sze
Yiqun Lin