Inigo Goiri
5740eea081
HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao.
2019-06-11 11:48:35 -07:00
Anu Engineer
101d5b5f86
HDFS-14234. Limit WebHDFS to specifc user, host, directory triples.
...
Contributed by Clay B.
2019-06-10 17:55:16 -07:00
He Xiaoqiao
bd46bdf9f9
HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 17:21:52 -07:00
Wei-Chiu Chuang
0d160a0ba8
HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed by Wei-Chiu Chuang, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-10 14:13:53 -07:00
Hanisha Koneru
7217494f40
HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 13:45:37 -07:00
Shweta Yakkali
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
Inigo Goiri
e1dfc060f8
HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena.
2019-06-06 11:59:53 -07:00
Inigo Goiri
944adc61b1
HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao.
2019-06-06 10:20:28 -07:00
Eric Yang
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
Sammi Chen
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
Kihwal Lee
9122b9b649
HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee
2019-05-30 17:28:45 -05:00
He Xiaoqiao
33c62f8f4e
HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:30:35 -07:00
Shweta Yakkali
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
Ayush Saxena
c1caab40f2
HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-29 20:56:38 -07:00
Akira Ajisaka
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
Eric Yang
d78854b928
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
...
Contributed by KWON BYUNGCHANG
2019-05-28 17:31:35 -04:00
Akira Ajisaka
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
Rakesh Radhakrishnan
37900c5639
HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.
2019-05-26 14:30:11 +05:30
Erik Krogen
55e0c134f0
HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen.
2019-05-24 13:12:59 -07:00
Konstantin V Shvachko
c31b7b8d52
HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko.
2019-05-24 12:34:29 -07:00
Inigo Goiri
c9393dd17f
Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong."
...
This reverts commit 1cb2eb0df3
.
2019-05-24 10:23:47 -07:00
Christopher Gregorian
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
Siyao Meng
9dff6eff81
HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-22 13:22:26 +09:00
Inigo Goiri
1cb2eb0df3
HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong.
2019-05-20 17:22:58 -07:00
Virajith Jalaparti
77170e70d1
HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal
2019-05-15 12:15:47 -07:00
Inigo Goiri
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
Erik Krogen
5847e00143
HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen.
2019-05-13 08:39:59 -07:00
Rakesh Radhakrishnan
9b0aace1e6
HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.
2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore
69b903bbd8
HDFS-14372. NPE while DN is shutting down. Contributed by lujie.
2019-05-05 16:33:52 +05:30
Surendra Singh Lilhore
e424392a62
HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi.
2019-05-05 13:38:09 +05:30
Giovanni Matteo Fumarola
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
Inigo Goiri
865c328930
HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota.
2019-05-02 10:09:21 -07:00
Siyao Meng
7cb46f035a
HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-01 16:48:44 -07:00
Sahil Takiar
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
Arpit Agarwal
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq.
2019-04-29 14:49:35 -07:00
Sean Mackrory
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
Erik Krogen
174b7d3126
HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
2019-04-22 13:29:56 -07:00
Inigo Goiri
5321235fe8
HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena.
2019-04-22 13:23:42 -07:00
Puleya7
13907d8479
HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-04-17 21:39:06 +09:00
Inigo Goiri
be6c8014e6
HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena.
2019-04-16 10:34:31 -07:00
Sahil Takiar
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
Chen Liang
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
Adam Antal
586826fe99
HDFS-12245. Fix INodeId javadoc
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-11 10:01:47 +09:00
Shweta
e9b859f749
HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-05 16:09:00 -07:00
Akira Ajisaka
67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji.
2019-04-05 12:29:11 +09:00
Stephen O'Donnell
d2637cb176
HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-04 04:07:18 -07:00
Inigo Goiri
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
Gabor Bota
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
Wei-Chiu Chuang
be488b6070
HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.
2019-04-03 11:00:59 -07:00
Lokesh Jain
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
Gabor Bota
53a86e2b8e
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
...
This closes #665
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-01 13:49:37 +09:00
Uma Maheswara Rao G
35ff31dd94
HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He.
2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan
f3f51284d5
HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R
2019-03-29 00:18:15 +05:30
Inigo Goiri
15d38b1bf9
HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor.
2019-03-28 03:37:33 -07:00
Takanobu Asanuma
55fb3c32fb
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
2019-03-27 03:27:02 +09:00
Stephen O'Donnell
3f6d6d2811
HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p)
...
(Contributed by Stephen O'Donnell via Daniel Templeton)
Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
2019-03-25 16:27:04 -07:00
Erik Krogen
55b3a718e9
HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-03-19 08:14:49 -07:00
Inigo Goiri
ff06ef0631
HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun.
2019-03-15 11:06:42 -07:00
Uma Maheswara Rao G
ba50a36a3e
HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He.
2019-03-14 22:21:08 -07:00
Erik Krogen
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
Stephen O'Donnell
34b14061b3
HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-12 10:17:25 -07:00
Erik Krogen
1bc282e0b3
HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
2019-03-07 08:41:23 -08:00
Wei-Chiu Chuang
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
.
2019-03-06 15:02:18 -08:00
Sahil Takiar
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
Inigo Goiri
945b504c25
HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker.
2019-03-05 17:39:52 -08:00
Vinayakumar B
f940ab242d
HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena.
2019-03-05 19:26:42 +05:30
Inigo Goiri
4b7313e640
HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker.
2019-03-04 23:03:32 -08:00
Wei-Chiu Chuang
387dbe587a
HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star.
2019-03-04 10:43:44 -08:00
Yiqun Lin
18ea0c1493
HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang.
2019-03-04 17:34:24 +08:00
Wei-Chiu Chuang
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng."
...
This reverts commit 5c10630ad8
.
2019-03-04 16:59:20 +09:00
Erik Krogen
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
Inigo Goiri
80b77deb42
HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao.
2019-03-01 09:18:51 -08:00
Brahma Reddy Battula
7a0db2f92b
HDFS-13997. Secondary NN Web UI displays nothing, and the console log shows moment is not defined. Contributed by Ayush Saxena
2019-03-01 12:24:52 +05:30
Inigo Goiri
05e4ddeee6
HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena.
2019-02-28 10:03:05 -08:00
Erik Krogen
0feba4396f
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao.
2019-02-28 08:48:10 -08:00
Konstantin V Shvachko
a6ab37192a
HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko.
2019-02-25 14:35:02 -08:00
Vinayakumar B
f4ae00c530
HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena."
2019-02-26 00:36:14 +05:30
Yongjun Zhang
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
Surendra Singh Lilhore
7d3b567194
HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi.
2019-02-23 01:08:15 +05:30
Surendra Singh Lilhore
92b53c40f0
HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie.
2019-02-21 20:36:34 +05:30
Shweta Yakkali
1bea785020
HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 14:28:37 -08:00
Inigo Goiri
f5b4e0f971
HDFS-14302. Refactor NameNodeWebHdfsMethods#generateDelegationToken() to allow better extensibility. Contributed by CR Hota.
2019-02-20 13:55:13 -08:00
Sahil Takiar
a30059bb61
HDFS-14267. Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 11:38:37 -08:00
Surendra Singh Lilhore
41e18feda3
HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar.
2019-02-20 16:56:10 +05:30
Bharat Viswanadham
51950f149e
Logging stale datanode information. Contributed by Karthik Palanisamy.
2019-02-19 17:01:07 -08:00
Kitti Nanasi
14282e311b
HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-19 12:04:56 -08:00
Inigo Goiri
7587f97127
HDFS-14296. Prefer ArrayList over LinkedList in VolumeScanner. Contributed by BELUGA BEHR.
2019-02-18 17:55:43 -08:00
Inigo Goiri
1de25d134f
HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR.
2019-02-18 15:30:38 -08:00
Inigo Goiri
235e3da90a
HDFS-14287. DataXceiverServer May Double-Close PeerServer. Contributed by BELUGA BEHR.
2019-02-18 11:00:04 -08:00
Akira Ajisaka
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
Inigo Goiri
dde0ab55aa
HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR.
2019-02-15 16:32:27 -08:00
Anu Engineer
75e15cc0c4
HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects.
...
Contributed by Elek, Marton.
2019-02-14 23:33:25 -08:00
Erik Krogen
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
Surendra Singh Lilhore
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
Shweta Yakkali
080a421911
HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali.
2019-02-14 08:05:06 -08:00
Chen Liang
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
Surendra Singh Lilhore
29b411d5f0
HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore
2019-02-13 23:41:27 +05:30
Surendra Singh Lilhore
35d4f32b32
HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena.
2019-02-13 23:06:46 +05:30
Vinayakumar B
7806403842
HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena.
2019-02-12 21:57:57 +05:30
Siyao Meng
5c10630ad8
HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-11 11:02:32 -08:00
Inigo Goiri
0ceb1b70f3
HDFS-14260. Replace synchronized method in BlockReceiver with atomic value. Contributed by BELUGA BEHR.
2019-02-11 10:09:44 -08:00
Yiqun Lin
177131793a
HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li.
2019-02-08 20:51:30 +08:00
Surendra Singh Lilhore
4be87353e3
HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-07 16:51:59 -08:00
Steve Loughran
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
.
2019-02-07 21:57:22 +00:00
Siyao Meng
6aa63452b3
HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-06 12:19:21 -08:00
Erik Krogen
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
Kitti Nanasi
49ddd8a6ed
HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-05 15:14:19 -08:00
Wangda Tan
308f3168fa
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
2019-02-05 14:03:18 -08:00
Da Zhou
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
Steve Loughran
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
Kitti Nanasi
5f15a60e32
HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-04 14:52:07 -08:00
Inigo Goiri
0e79a86582
HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar.
2019-02-04 11:59:48 -08:00
tiwalter
9aa3dc872c
HDFS-14158. Checkpointer ignores configured time period > 5 minutes
...
This closes #449
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-02-04 13:51:07 +09:00
Akira Ajisaka
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
Wei-Chiu Chuang
d1714c20e9
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1d523279da
.
2019-01-29 15:43:09 -08:00
Masatake Iwasaki
dc5af4c6b7
HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki.
2019-01-26 05:34:32 +09:00
Pranay Singh
1d523279da
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 09:02:57 -08:00
Dinesh Chitlangia
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
shwetayakkali@cloudera.com
f3e642d92b
HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
2019-01-23 15:57:39 -08:00
Kitti Nanasi
951cdd7e4c
HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-23 14:40:57 -08:00
Surendra Singh Lilhore
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
Takanobu Asanuma
7d6792e5d2
HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena.
2019-01-23 10:03:06 +09:00
Arpit Agarwal
6f0756fc0e
HDFS-14222. Make ThrottledAsyncChecker constructor public. Contributed by Arpit Agarwal.
2019-01-21 20:45:09 -08:00
Arpit Agarwal
1ff658b2ef
HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal.
2019-01-21 20:44:37 -08:00
Weiwei Yang
de34fc148c
HDFS-14207. ZKFC should catch exception when ha configuration missing. Contributed by Fei Hui.
2019-01-22 12:01:08 +08:00
Sunil G
e9962240f0
Make 3.2.0 aware to other branches - jdiff
2019-01-21 22:41:26 +05:30
Inigo Goiri
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
Giovanni Matteo Fumarola
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
Surendra Singh Lilhore
9aeaaa0479
HDFS-14198. Upload and Create button doesn't get enabled after getting reset. Contributed by Ayush Saxena.
2019-01-11 14:36:55 +05:30
shwetayakkali@cloudera.com
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
Giovanni Matteo Fumarola
1a08302897
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena.
2019-01-08 10:38:06 -08:00
Surendra Singh Lilhore
f660e5eaa3
HDFS-14163. Debug Admin Command Should Support Generic Options. Contributed by Ayush Saxena.
2019-01-03 18:58:03 +05:30
Surendra Singh Lilhore
21fe77e3ab
HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena.
2019-01-03 18:36:53 +05:30
Akira Ajisaka
f30473f2ef
HDFS-14183. [SPS] Remove the -w parameter from the -satisfystoragepolicy usage. Contributed by Ayush Saxena.
2018-12-31 03:16:26 +09:00
Wei-Chiu Chuang
e9a005dfcf
HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang.
2018-12-28 10:34:18 -08:00
Chao Sun
c546b12e45
HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
Chao Sun
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Chao Sun
c3efc299fe
HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Erik Krogen
ef3e1929dd
HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Erik Krogen
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
Erik Krogen
2904aa3808
HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Chao Sun
cbc2f932eb
HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Chen Liang
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
Chao Sun
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
Konstantin V Shvachko
b6f20c36c2
HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
1c4265d7bc
HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
b5b9b77707
HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
Chen Liang
a65bb97f5d
HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
Erik Krogen
b74a7dbf88
HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
Erik Krogen
25b63e8da8
HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
28820fb6b1
HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov.
2018-12-24 09:34:00 -08:00
Chen Liang
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Erik Krogen
aa42fb0db7
HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
Konstantin V Shvachko
25d8e39b5c
HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
Chen Liang
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Chao Sun
4c9baba066
HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
Erik Krogen
eae0a5d54a
HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
Erik Krogen
1e22f2bfbb
HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
Chao Sun
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
Konstantin V Shvachko
0105d7f597
HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
2018-12-24 09:33:59 -08:00
Erik Krogen
00e99c6594
HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
Erik Krogen
151c8ddbe4
HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
Erik Krogen
c81ac2ff02
HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Erik Krogen
091ad974cd
HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
Erik Krogen
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Plamen Jeliazkov
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
Chao Sun
3a78e5ffd8
HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
Surendra Singh Lilhore
686fcd4db3
HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar.
2018-12-24 11:02:36 +05:30
Surendra Singh Lilhore
d944d5ec46
HDFS-14165. In NameNode UI under DataNode tab ,the Capacity column is Non-Aligned. Contributed by Shubham Dewan.
2018-12-23 15:06:12 +05:30
Yiqun Lin
feb2664ac4
HDFS-13946. Log longest FSN write/read lock held stack trace.
2018-12-22 23:09:59 +08:00
Giovanni Matteo Fumarola
00d5e631b5
HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota.
2018-12-14 11:10:54 -08:00
Giovanni Matteo Fumarola
cb3382f667
HDFS-14145. TestBlockStorageMovementAttemptedItems#testNoBlockMovementAttemptFinishedReportAdded fails sporadically in Trunk. Contributed by Ayush Saxena.
2018-12-14 10:59:02 -08:00
Akira Ajisaka
ca379e1c43
HDFS-13970. Use MultiMap for CacheManager Directives to simplify the code. Contributed by BELUGA BEHR.
2018-12-14 14:36:26 +09:00
Giovanni Matteo Fumarola
561b61cfb1
HDFS-14144. TestPred fails in Trunk. Contributed by Ayush Saxena.
2018-12-12 11:16:16 -08:00
Giovanni Matteo Fumarola
60af851e59
HDFS-14137. TestMaintenanceState fails with ArrayIndexOutOfBound Exception. Contributed by Ayush Saxena.
2018-12-11 11:22:32 -08:00
Vinayakumar B
39dc7345b8
HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena.
2018-12-11 17:59:04 +05:30
Sean Mackrory
80e59e7876
HDFS-14101. Fixing underflow error in test. Contributed by Zsolt Venczel.
2018-12-10 14:03:08 -07:00
Inigo Goiri
8fc0d04517
HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-12-07 18:30:36 -08:00
Vinayakumar B
c03024a530
HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena.
2018-12-06 18:20:28 +05:30
Giovanni Matteo Fumarola
69489ff2d1
HDFS-14119. Improve GreedyPlanner Parameter Logging. Contributed by Beluga Behr.
2018-12-03 12:55:52 -08:00
Adam Antal
fb10803dfa
HDFS-13818. Extend OIV to detect FSImage corruption. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-12-03 10:34:39 -08:00
Kitti Nanasi
dd5e7c6b72
HDFS-12946. Add a tool to check rack configuration against EC policies. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-12-03 10:01:09 -08:00
Surendra Singh Lilhore
042c8ef593
HDFS-14075. Terminate the namenode when failed to start log segment. Contributed by Ayush Saxena.
2018-12-02 12:31:08 +05:30
Giovanni Matteo Fumarola
99e201dfe2
HDFS-14105. Replace TreeSet in NamenodeFsck with HashSet. Contributed by Beluga Behr.
2018-11-30 11:07:11 -08:00
Giovanni Matteo Fumarola
b09cfad432
HDFS-14106. Refactor NamenodeFsck#copyBlock. Contributed by Beluga Behr.
2018-11-30 10:47:59 -08:00
Yiqun Lin
0e36e935d9
HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng.
2018-11-30 11:31:34 +08:00
Tsz Wo Nicholas Sze
0081b02e35
HDFS-14112. Avoid recursive call to external authorizer for getContentSummary.
2018-11-29 13:55:21 -08:00
Brahma Reddy Battula
f534736867
HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
2018-11-30 00:18:27 +05:30
Ewan Higgs
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
Giovanni Matteo Fumarola
300f560fcc
HDFS-14102. Performance improvement in BlockPlacementPolicyDefault. Contributed by Beluga Behr.
2018-11-28 11:33:22 -08:00
Giovanni Matteo Fumarola
4ca3a6b21a
HDFS-14108. Performance improvement in BlockManager Data Structures. Contributed by Beluga Behr.
2018-11-28 11:25:47 -08:00
Vinayakumar B
b098281454
HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B.
2018-11-26 18:22:32 +05:30
Brahma Reddy Battula
892b33e054
HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena.
2018-11-22 00:13:56 +05:30
Akira Ajisaka
1a00b4e325
Add 2.9.2 release notes and changes documents.
2018-11-20 14:25:18 +09:00
Vinayakumar B
68d5dfdc78
HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena.
2018-11-16 18:05:10 +05:30
Vinayakumar B
8571507efa
HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena.
2018-11-16 17:26:36 +05:30
Arpit Agarwal
1e15c7e855
HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel.
2018-11-15 12:42:31 -08:00
Inigo Goiri
460a94a10f
HDFS-14045. Use different metrics in DataNode to better measure latency of heartbeat/blockReports/incrementalBlockReports of Active/Standby NN. Contributed by Jiandan Yang.
2018-11-15 10:58:57 -08:00
Inigo Goiri
993c2140cc
HDFS-14054. TestLeaseRecovery2: testHardLeaseRecoveryAfterNameNodeRestart2 and testHardLeaseRecoveryWithRenameAfterNameNodeRestart are flaky. Contributed by Zsolt Venczel.
2018-11-15 09:29:14 -08:00
Xiao Chen
9da6054ca4
Revert "HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel."
...
This reverts commit 7dc79a8b5b
.
2018-11-13 12:44:25 -08:00
Brahma Reddy Battula
e7b63baca1
HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better extensibility. Contributed by CR Hota
2018-11-13 12:45:44 +05:30
Arpit Agarwal
b6d4e19f34
HDFS-14065. Failed Storage Locations shows nothing in the Datanode Volume Failures. Contributed by Ayush Saxena.
2018-11-12 15:31:42 -08:00
Kitti Nanasi
8d99648c20
HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-11-08 10:01:19 -08:00
Inigo Goiri
887244de4a
HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen.
2018-11-06 11:18:15 -08:00
Inigo Goiri
f0ce072934
HDFS-14051. Refactor NameNodeHttpServer#initWebHdfs to specify local keytab. Contributed by CR Hota.
2018-11-06 10:05:58 -08:00
Akira Ajisaka
6430c9890a
HADOOP-15904. [JDK 11] Javadoc build failed due to bad use of '>' in hadoop-hdfs.
2018-11-06 18:42:38 +09:00
Xiao Chen
ffc9c50e07
HDFS-14053. Provide ability for NN to re-replicate based on topology changes. Contributed by Hrishikesh Gadre.
2018-11-05 21:38:39 -08:00
Inigo Goiri
f3296501e0
HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak.
2018-11-05 16:48:37 -08:00
Giovanni Matteo Fumarola
f3f5e7ad00
HDFS-14042. Fix NPE when PROVIDED storage is missing. Contributed by Virajith Jalaparti.
2018-11-05 11:02:31 -08:00
Siyao Meng
8fe85af63b
HDFS-13996. Make HttpFS' ACLs RegEx configurable. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-11-01 17:36:50 -07:00
Pranay Singh
d98b881e9a
HDFS-14008. NN should log snapshotdiff report. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-11-01 17:26:00 -07:00
Akira Ajisaka
fac9f91b29
HDFS-13942. [JDK10] Fix javadoc errors in hadoop-hdfs module. Contributed by Dinesh Chitlangia.
2018-10-31 14:43:58 +09:00
Xiao Chen
db7e636824
HDFS-14027. DFSStripedOutputStream should implement both hsync methods.
2018-10-29 19:06:15 -07:00
Wei-Chiu Chuang
63e7134df6
Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao."
...
This reverts commit fb2b72e6fc
.
2018-10-28 17:04:59 -07:00
Anu Engineer
4f10d7e23f
HDFS-14028. HDFS OIV temporary dir deletes folder.
...
Contributed by Adam Antal.
2018-10-26 16:41:23 -07:00
Inigo Goiri
1851d06eb3
HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena.
2018-10-26 09:29:12 -07:00
Arpit Agarwal
9b899f1ebd
HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal.
2018-10-25 15:14:13 -07:00
Wei-Chiu Chuang
fb2b72e6fc
HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao.
2018-10-25 09:33:31 -07:00
Ajay Kumar
97bd49fc36
HDFS-14026. Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional. Contributed by Arpit Agarwal.
2018-10-24 21:55:42 -07:00
Inigo Goiri
936fc3f3c2
HDFS-14025. TestPendingReconstruction.testPendingAndInvalidate fails. Contributed by Ayush Saxena.
2018-10-24 16:18:43 -07:00
Inigo Goiri
c1874046e2
HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen.
2018-10-24 09:20:38 -07:00
Chen Liang
635786a511
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2018-10-23 14:53:45 -07:00
Inigo Goiri
efdfe679d6
HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena.
2018-10-23 14:28:37 -07:00
Arpit Agarwal
c58811c77d
HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar.
2018-10-22 14:44:28 -07:00
Inigo Goiri
f069d38c8d
HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B.
2018-10-19 17:16:04 -07:00
Inigo Goiri
88cce32551
HDFS-9872. HDFS bytes-default configurations should accept multiple size units. Contributed by Yiqun Lin.
2018-10-19 10:38:04 -07:00
Inigo Goiri
8b64fbab1a
HDFS-13994. Improve DataNode BlockSender waitForMinLength. Contributed by BELUGA BEHR.
2018-10-19 10:36:03 -07:00
Inigo Goiri
be1cffb085
HDFS-14002. TestLayoutVersion#testNameNodeFeatureMinimumCompatibleLayoutVersions fails. Contributed by Takanobu Asanuma.
2018-10-18 15:25:53 -07:00
Inigo Goiri
0e56c883cd
HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin.
2018-10-18 10:53:30 -07:00
Xiao Chen
533138718c
HDFS-13662. TestBlockReaderLocal#testStatisticsForErasureCodingRead is flaky
2018-10-16 19:33:55 -07:00
Inigo Goiri
f90c64e624
HDFS-13993. TestDataNodeVolumeFailure#testTolerateVolumeFailuresAfterAddingMoreVolumes is flaky. Contributed by Ayush Saxena.
2018-10-16 11:22:57 -07:00
Xiao Chen
b6fc72a025
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
2018-10-15 10:51:55 -07:00
Vinayakumar B
b60ca37914
Fix potential FSImage corruption. Contributed by Daryn Sharp.
2018-10-15 15:48:26 +05:30
Vinayakumar B
6e0e6daaf3
HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena.
2018-10-12 17:34:10 +05:30
Vinayakumar B
de44e1064f
HDFS-13156. HDFS Block Placement Policy - Client Local Rack. Contributed by Ayush Saxena.
2018-10-12 17:27:23 +05:30
Shweta
c0118105b7
HDFS-13973. getErasureCodingPolicy should log path in audit event. Contributed by Shweta.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-11 20:39:41 -07:00
Inigo Goiri
3532aa3886
HDFS-13968. BlockReceiver Array-Based Queue. Contributed by BELUGA BEHR.
2018-10-11 13:49:39 -07:00
Steve Loughran
2addebb94f
HDFS-13951. HDFS DelegationTokenFetcher can't print non-HDFS tokens in a tokenfile.
...
Contributed by Steve Loughran.
2018-10-11 20:03:56 +01:00
Hrishikesh Gadre
bf3d591f0c
HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:23:07 -07:00
Weiwei Yang
3ead525c71
HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-10 10:11:10 -07:00
Kitti Nanasi
10185d9a77
HDFS-13882. Set a maximum delay for retrying locateFollowingBlock. Contributed by Kitti Nanasi.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-10 08:55:32 -07:00
Nanda kumar
b39b802422
HDFS-13949. Correct the description of dfs.datanode.disk.check.timeout in hdfs-default.xml. Contributed by Toshihiro Suzuki.
2018-10-10 12:29:07 +05:30
Akira Ajisaka
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2018-10-10 10:16:57 +09:00
Inigo Goiri
605622c87b
HDFS-11396. TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out. Contributed by Ayush Saxena.
2018-10-09 17:03:21 -07:00
Hrishikesh Gadre
08bb6c49a5
HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads.
...
Contributed by Xiao Chen, Hrishikesh Gadre.
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-08 20:31:57 -07:00
Yiqun Lin
1043795f7f
HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore.
2018-10-09 10:33:13 +08:00
Inigo Goiri
73c660b43f
HDFS-13958. Miscellaneous Improvements for FsVolumeSpi. Contributed by BELUGA BEHR.
2018-10-05 09:32:19 -07:00
Yiqun Lin
619e490333
HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap.
2018-10-05 09:55:08 +08:00
Adam Antal
cc2babc1f7
HDFS-13950. ACL documentation update to indicate that ACL entries are capped by 32. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-04 15:22:44 -07:00
Inigo Goiri
1dc0adfac0
HDFS-13947. Review of DirectoryScanner Class. Contributed by BELUGA BEHR.
2018-10-03 11:19:57 -07:00
Sunil G
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
Yiqun Lin
5689355783
HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore.
2018-10-02 09:43:14 +08:00
Takanobu Asanuma
e2113500df
HDFS-8196. Post enabled Erasure Coding Policies on NameNode UI. Contributed by Kitti Nanasi and Kai Sasaki.
2018-09-29 00:53:35 +09:00
Akira Ajisaka
f9c0221623
HDFS-13932. [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions. Contributed by Andrew Purtell.
2018-09-27 11:44:42 +09:00
Brahma Reddy Battula
e0ff8e2c10
HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula
2018-09-26 23:44:16 +05:30
Inigo Goiri
6275b42870
HDFS-13927. Addendum: Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena.
2018-09-26 09:50:09 -07:00
Inigo Goiri
4758b4b6da
HDFS-13927. Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena.
2018-09-21 15:32:28 -07:00
Anu Engineer
6fc293fece
HDFS-13892. Disk Balancer: Make execute command documentation better.
...
Contributed by Ranith Sardar.
2018-09-19 20:48:41 -07:00
Arpit Agarwal
2eb597b151
HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena.
2018-09-19 13:48:27 -07:00
Chen Liang
a30b4f9e71
HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen.
2018-09-19 13:22:37 -07:00
Inigo Goiri
090272d7de
HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang.
2018-09-19 13:00:30 -07:00
Inigo Goiri
56e0d635e0
HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena.
2018-09-19 09:30:25 -07:00
Wei-Chiu Chuang
28ceb34a72
HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh.
2018-09-19 03:12:20 -07:00
Shweta
27978bcb66
HDFS-13833. Improve BlockPlacementPolicyDefault's consider load logic. Contributed by Shweta.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-18 20:23:50 -07:00
Mingliang Liu
6ff509c32a
HDFS-11719. Arrays.fill() wrong index in BlockSender.readChecksum() exception handling. Contributed by Tao Zhang
2018-09-18 10:25:28 -07:00
Inigo Goiri
d1541932db
HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng.
2018-09-17 14:42:03 -07:00
Anu Engineer
fdf5a3fd63
HDFS-13919. Documentation: Improper formatting in Disk Balancer for Settings.
...
Contributed by Ayush Saxena.
2018-09-17 10:08:23 -07:00
Wei-Chiu Chuang
a1de8cbac5
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-09-14 05:22:56 +08:00
Inigo Goiri
e1b242a984
HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang.
2018-09-13 09:16:58 -07:00
Daniel Templeton
78bd3b1db9
HDFS-13846. Safe blocks counter is not decremented correctly if the block is striped
...
(Contributed by Kitti Nanasi via Daniel Templeton)
Change-Id: Id41747a67dc946fdf0dbde90643bb1ea7e9e0f70
2018-09-12 11:18:55 -07:00
Brahma Reddy Battula
b2432d254c
HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang.
2018-09-12 06:30:57 +05:30
Xiao Chen
8e54da1511
HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp.
2018-09-10 22:34:23 -07:00
Yiqun Lin
0da49642fc
HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu.
2018-09-10 13:57:36 +08:00
Vinayakumar B
eef3bafae8
HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena.
2018-09-10 09:22:59 +05:30
Vinayakumar B
30eceec342
HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena.
2018-09-10 09:10:51 +05:30
Hrishikesh Gadre
335a8139f5
HDFS-13820. Add an ability to disable CacheReplicationMonitor. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-07 14:59:06 -07:00
Sean Mackrory
410dd3faa5
HDFS-13744. OIV tool should better handle control characters present in file or directory names. Contributed by Zsolt Venczel.
2018-09-07 12:39:33 -06:00
Giovanni Matteo Fumarola
eca1a4bfe9
HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering.
2018-09-06 14:48:00 -07:00
Giovanni Matteo Fumarola
9af96d4ed4
HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak.
2018-09-05 10:50:25 -07:00
Hrishikesh Gadre
6ccb809c2d
HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre.
...
Signed-off-by: Xiao Chen <xiao@apache.org>
2018-09-04 21:53:42 -07:00
Vinayakumar B
3801436e49
HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena.
2018-09-03 14:37:57 +05:30
Wei-Chiu Chuang
8aa6c4f079
Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
...
This reverts commit 26c2a97c56
.
2018-08-30 11:44:20 -07:00
Vinayakumar B
c36d69a7b3
HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B.
...
(cherry picked from commit 65977e5d81
)
2018-08-30 22:15:51 +05:30
Yiqun Lin
582cb10ec7
HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui.
2018-08-30 11:21:13 +08:00
Zsolt Venczel
3e18b957eb
HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel.
2018-08-28 15:13:43 -07:00
Shweta
33f42efc94
HDFS-13837. Enable debug log for LeaseRenewer in TestDistributedFileSystem. Contributed by Shweta.
2018-08-28 13:56:32 -07:00
Wei-Chiu Chuang
26c2a97c56
HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
2018-08-27 16:02:35 -07:00
Yiqun Lin
b9b964d253
HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu.
2018-08-27 14:55:46 +08:00
Surendra Singh Lilhore
96c4575d73
HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore.
2018-08-24 08:14:57 +05:30
Vinayakumar B
770d9d9bb6
HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena
2018-08-21 09:33:19 +05:30
Giovanni Matteo Fumarola
79c97f6a0b
HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He.
2018-08-17 14:40:00 -07:00
Márton Elek
77b015000a
HADOOP-8807. Update README and website to reflect HADOOP-8662. Contributed by Andras Bokor.
2018-08-17 11:10:29 +02:00
Xiao Chen
c67b0650ea
HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
2018-08-16 23:14:21 -07:00
Wei-Chiu Chuang
1290e3c647
HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang.
2018-08-16 16:29:38 -07:00
Daniel Templeton
8512e1a91b
HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
...
(Contributed by Siyao Meng via Daniel Templeton)
Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
2018-08-16 15:00:45 -07:00
Yiqun Lin
6df606f1b4
HDFS-13829. Remove redundant condition judgement in DirectoryScanner#scan. Contributed by liaoyuxiangqin.
2018-08-16 18:44:18 +08:00
Xiao Chen
7dc79a8b5b
HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel.
2018-08-15 13:53:47 -07:00
Xiao Chen
ef02f9b233
HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin.
2018-08-15 09:23:05 -07:00
Akira Ajisaka
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
Daniel Templeton
4a5006b1d0
HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky
...
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
2018-08-14 17:22:15 -07:00
Wei-Chiu Chuang
363bd16e31
HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal.
2018-08-14 17:19:00 -07:00
Xiao Chen
cede33997f
HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi.
2018-08-14 11:57:22 -07:00
Wei-Chiu Chuang
61a9b4f58b
HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan.
2018-08-14 11:51:27 -07:00
Wei-Chiu Chuang
4023eeba05
HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh.
2018-08-13 17:36:13 -07:00
Wei-Chiu Chuang
23854443ef
HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng.
2018-08-13 16:12:37 -07:00
Arpit Agarwal
f760a544a7
HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar.
2018-08-13 11:35:19 -07:00
drankye
475bff6e8e
HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao.
2018-08-13 17:32:56 +08:00
Uma Maheswara Rao Gangumalla
3ac07b720b
HDFS-13808: [SPS]: Remove unwanted FSNamesystem #isFileOpenedForWrite() and #getFileInfo() function. Contributed by Rakesh R.
2018-08-12 03:06:07 -07:00
Uma Maheswara Rao G
39ed3a66db
HDFS-13076: [SPS]: Cleanup work for HDFS-10285 merge. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
dfcb331ba3
HDFS-13076: [SPS]: Addendum. Resolve conflicts after rebasing branch to trunk. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
db3f227d8a
HDFS-13076: [SPS]: Resolve conflicts after rebasing HDFS-10285 branch to trunk. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
Uma Maheswara Rao G
66e8f9b315
HDFS-13381 : [SPS]: Use DFSUtilClient#makePathFromFileId() to prepare satisfier file path. Contributed by Rakesh R.
2018-08-12 03:06:06 -07:00
Rakesh Radhakrishnan
2acc50b826
HDFS-13165: [SPS]: Collects successfully moved block details via IBR. Contributed by Rakesh R.
2018-08-12 03:06:05 -07:00
Surendra Singh Lilhore
75ccc1396b
HDFS-13166: [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls. Contributed by Rakesh R.
2018-08-12 03:06:05 -07:00
Rakesh Radhakrishnan
8467ec24fb
HDFS-13110: [SPS]: Reduce the number of APIs in NamenodeProtocol used by external satisfier. Contributed by Rakesh R.
2018-08-12 03:06:05 -07:00
Uma Maheswara Rao G
4402f3f855
HDFS-13097: [SPS]: Fix the branch review comments(Part1). Contributed by Surendra Singh.
2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
d3de4fb2a0
HDFS-13077. [SPS]: Fix review comments of external storage policy satisfier. Contributed by Rakesh R.
2018-08-12 03:06:04 -07:00
Rakesh Radhakrishnan
5845c36c16
HDFS-13050: [SPS]: Create start/stop script to start external SPS process. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:04 -07:00
Surendra Singh Lilhore
99594b48b8
HDFS-13075. [SPS]: Provide External Context implementation. Contributed by Uma Maheswara Rao G.
2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
3b83110d5e
HDFS-13057: [SPS]: Revisit configurations to make SPS service modes internal/external/none. Contributed by Rakesh R.
2018-08-12 03:06:04 -07:00
Uma Maheswara Rao G
b0cb8d9bb4
HDFS-13033: [SPS]: Implement a mechanism to do file block movements for external SPS. Contributed by Rakesh R.
2018-08-12 03:06:03 -07:00
Rakesh Radhakrishnan
3159b39cf8
HDFS-13025. [SPS]: Implement a mechanism to scan the files for external SPS. Contributed by Uma Maheswara Rao G.
2018-08-12 03:06:03 -07:00
Rakesh Radhakrishnan
8d4f74e733
HDFS-12911. [SPS]: Modularize the SPS code and expose necessary interfaces for external/internal implementations. Contributed by Uma Maheswara Rao G
2018-08-12 03:06:03 -07:00
Surendra Singh Lilhore
05d4daf6ba
HDFS-12982 : [SPS]: Reduce the locking and cleanup the Namesystem access. Contributed by Rakesh R.
2018-08-12 03:06:02 -07:00
Uma Maheswara Rao G
78420719eb
HDFS-12955: [SPS]: Move SPS classes to a separate package. Contributed by Rakesh R.
2018-08-12 03:06:02 -07:00
Surendra Singh Lilhore
c561cb316e
HDFS-12106: [SPS]: Improve storage policy satisfier configurations. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
9b83f94f35
HDFS-12790: [SPS]: Rebasing HDFS-10285 branch after HDFS-10467, HDFS-12599 and HDFS-11968 commits. Contributed by Rakesh R.
2018-08-12 03:06:02 -07:00
Rakesh Radhakrishnan
68017e3349
HDFS-12310: [SPS]: Provide an option to track the status of in progress requests. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:02 -07:00
Surendra Singh Lilhore
5780f0624d
HDFS-12556: [SPS] : Block movement analysis should be done in read lock.
2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
00eceed233
HDFS-12570: [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements. Contributed by Rakesh R.
2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
bfd3f8bd8a
HDFS-12291: [SPS]: Provide a mechanism to recursively iterate and satisfy storage policy of all the files under the given dir. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
7ea24fc06c
HDFS-12225: [SPS]: Optimize extended attributes for tracking SPS movements. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
0e820f16af
HDFS-12214: [SPS]: Fix review comments of StoragePolicySatisfier feature. Contributed by Rakesh R.
2018-08-12 03:06:01 -07:00
Uma Maheswara Rao G
3b601f2c0e
HDFS-12152: [SPS]: Re-arrange StoragePolicySatisfyWorker stopping sequence to improve thread cleanup time. Contributed by Rakesh R.
2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
4bcf61c696
HDFS-12141: [SPS]: Fix checkstyle warnings. Contributed by Rakesh R.
2018-08-12 03:06:00 -07:00
Rakesh Radhakrishnan
9e82e5a86e
HDFS-12146. [SPS]: Fix TestStoragePolicySatisfierWithStripedFile#testSPSWhenFileHasLowRedundancyBlocks. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:00 -07:00
Rakesh Radhakrishnan
68af4e199a
HDFS-11874. [SPS]: Document the SPS feature. Contributed by Uma Maheswara Rao G
2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
5eb24ef7e7
HDFS-11264: [SPS]: Double checks to ensure that SPS/Mover are not running together. Contributed by Rakesh R.
2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
0b360b16ab
HDFS-11965: [SPS]: Should give chance to satisfy the low redundant blocks before removing the xattr. Contributed by Surendra Singh Lilhore.
2018-08-12 03:06:00 -07:00
Uma Maheswara Rao G
00cf207192
HDFS-11670: [SPS]: Add CLI command for satisfy storage policy operations. Contributed by Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
46f7523eb8
HDFS-11966. [SPS] Correct the log in BlockStorageMovementAttemptedItems#blockStorageMovementResultCheck. Contributed by Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
20f9c62336
HDFS-11726. [SPS]: StoragePolicySatisfier should not select same storage type as source and destination in same datanode. Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
d638a7dc03
HDFS-11762. [SPS]: Empty files should be ignored in StoragePolicySatisfier. Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Uma Maheswara Rao G
e53f89ccc3
HDFS-11883: [SPS] : Handle NPE in BlockStorageMovementTracker when dropSPSWork() called. Contributed by Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Uma Maheswara Rao G
5ce332dc9a
HDFS-11695: [SPS]: Namenode failed to start while loading SPS xAttrs from the edits log. Contributed by Surendra Singh Lilhore.
2018-08-12 03:05:59 -07:00
Rakesh Radhakrishnan
6fe6c549e8
HDFS-11572. [SPS]: SPS should clean Xattrs when no blocks required to satisfy for a file. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
695a402fca
HDFS-11334: [SPS]: NN switch and rescheduling movements can lead to have more than one coordinator for same file blocks. Contributed by Rakesh R.
2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
11a08a7e8f
HDFS-11338: [SPS]: Fix timeout issue in unit tests caused by longger NN down time. Contributed by Wei Zhou and Rakesh R
2018-08-12 03:05:58 -07:00
Uma Maheswara Rao G
c00be44463
HDFS-11336: [SPS]: Remove xAttrs when movements done or SPS disabled. Contributed by Yuanbo Liu.
2018-08-12 03:05:58 -07:00
Rakesh Radhakrishnan
9b15f5418d
HDFS-11239: [SPS]: Check Mover file ID lease also to determine whether Mover is running. Contributed by Wei Zhou
2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
e34331c31d
HDFS-11243. [SPS]: Add a protocol command from NN to DN for dropping the SPS work and queues. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
f8fc96a66e
HDFS-11309. [SPS]: chooseTargetTypeInSameNode should pass accurate block size to chooseStorage4Block while choosing target. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:57 -07:00
Rakesh Radhakrishnan
681d2804c9
HDFS-11186. [SPS]: Daemon thread of SPS should start only in Active NN. Contributed by Wei Zhou
2018-08-12 03:05:57 -07:00
Uma Maheswara Rao G
6215e35bb6
HDFS-11150: [SPS]: Provide persistence when satisfying storage policy. Contributed by Yuanbo Liu
2018-08-12 03:05:57 -07:00
Uma Maheswara Rao G
df2b551e79
HDFS-11293: [SPS]: Local DN should be given preference as source node, when target available in same node. Contributed by Yuanbo Liu and Uma Maheswara Rao G
2018-08-12 03:05:56 -07:00
Rakesh Radhakrishnan
422f870607
HDFS-11289. [SPS]: Make SPS movement monitor timeouts configurable. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:56 -07:00
Uma Maheswara Rao G
81ed364b63
HDFS-11193 : [SPS]: Erasure coded files should be considered for satisfying storage policy. Contributed by Rakesh R
2018-08-12 03:05:56 -07:00
Uma Maheswara Rao G
b7bed9f00a
HDFS-11248: [SPS]: Handle partial block location movements. Contributed by Rakesh R
2018-08-12 03:05:55 -07:00
Uma Maheswara Rao G
d81611fe55
HDFS-11032: [SPS]: Handling of block movement failure at the coordinator datanode. Contributed by Rakesh R
2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
5179d99b7e
HDFS-11123. [SPS] Make storage policy satisfier daemon work on/off dynamically. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
cd5262aba0
HDFS-10885. [SPS]: Mover tool should not be allowed to run when Storage Policy Satisfier is on. Contributed by Wei Zhou
2018-08-12 03:05:55 -07:00
Rakesh Radhakrishnan
b07291e176
HDFS-11151. [SPS]: StoragePolicySatisfier should gracefully handle when there is no target node with the required storage type. Contributed by Rakesh R
2018-08-12 03:05:54 -07:00
Rakesh Radhakrishnan
b67ae6d9d7
HDFS-10802. [SPS]: Add satisfyStoragePolicy API in HdfsAdmin. Contributed by Yuanbo Liu
2018-08-12 03:05:54 -07:00
Uma Maheswara Rao G
19b5aee3e4
HDFS-11068: [SPS]: Provide unique trackID to track the block movement sends to coordinator. Contributed by Rakesh R
2018-08-12 03:05:54 -07:00
Rakesh Radhakrishnan
047526b4c2
HDFS-11029. [SPS]:Provide retry mechanism for the blocks which were failed while moving its storage at DNs. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:53 -07:00
Rakesh Radhakrishnan
0f2d1ddc2c
HDFS-10954. [SPS]: Provide mechanism to send blocks movement result back to NN from coordinator DN. Contributed by Rakesh R
2018-08-12 03:05:53 -07:00
Uma Maheswara Rao G
24add8c2f8
HDFS-10884: [SPS]: Add block movement tracker to track the completion of block movement future tasks at DN. Contributed by Rakesh R
2018-08-12 03:05:53 -07:00
Rakesh Radhakrishnan
e2a15d18bb
HDFS-10801. [SPS]: Protocol buffer changes for sending storage movement commands from NN to DN. Contributed by Rakesh R
2018-08-12 03:05:53 -07:00
Uma Maheswara Rao G
1438da4944
HDFS-10800: [SPS]: Daemon thread in Namenode to find blocks placed in other storage than what the policy specifies. Contributed by Uma Maheswara Rao G
2018-08-12 03:05:53 -07:00
Kai Zheng
5692887395
HDFS-10794. [SPS]: Provide storage policy satisfy worker at DN for co-ordinating the block storage movement work. Contributed by Rakesh R
2018-08-12 03:05:52 -07:00
Chao Sun
a2a8c48699
Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun.
2018-08-10 15:59:39 -07:00
Virajith Jalaparti
15241c6349
HDFS-13795. Fix potential NPE in InMemoryLevelDBAliasMapServer.
2018-08-10 09:38:40 -07:00
Chen Liang
5326a7906d
HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun.
2018-08-09 10:11:47 -07:00
Wangda Tan
49c687608b
Make 3.1.1 awared by other branches
2018-08-08 13:02:12 -07:00
Xiao Chen
9499df7b81
HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi.
2018-08-08 10:40:20 -07:00
Márton Elek
36c0d742d4
HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr.
2018-08-08 17:31:55 +02:00
Ewan Higgs
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
Vinayakumar B
4203bc738c
HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena
2018-08-08 12:42:20 +05:30
Xiao Chen
6677717c68
HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell.
2018-08-07 22:05:17 -07:00
Vinayakumar B
2b0f977241
HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena.
2018-08-08 07:53:17 +05:30
Xiao Chen
0f8cb127cd
HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization
...
between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
2018-08-07 16:13:41 -07:00
Virajith Jalaparti
b1a59b1644
HDFS-13796. Allow verbosity of InMemoryLevelDBAliasMapServer to be configurable.
2018-08-07 10:15:28 -07:00
Takanobu Asanuma
2b18bb4f37
HDFS-11610. sun.net.spi.nameservice.NameService has moved to a new location. Contributed by Akira Ajisaka.
2018-08-03 23:38:05 +09:00
Brahma Reddy Battula
3108d27edd
HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani
2018-07-30 15:50:04 +05:30
Arpit Agarwal
1c40bc2836
HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain.
2018-07-27 10:14:01 -07:00
Anu Engineer
64e739e344
HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception.
...
Contributed by Gabor Bota.
2018-07-27 06:11:56 -07:00
Xiao Chen
be150a17b1
HDFS-13622. mkdir should print the parent directory in the error message when parent directories do not exist. Contributed by Shweta.
2018-07-26 10:24:32 -07:00
Wei-Chiu Chuang
e95c5e9f62
HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin.
2018-07-25 11:04:18 -07:00
Xiao Chen
26864471c2
HDFS-13761. Add toString Method to AclFeature Class. Contributed by Shweta.
2018-07-24 21:42:47 -07:00
Daniel Templeton
849c45db18
HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
...
(Contributed by BELUGA BEHR via Daniel Templeton)
Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
2018-07-24 16:05:27 -07:00
Konstantin V Shvachko
0a1e922f3d
Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal.
2018-07-16 18:24:18 -07:00
Wei-Chiu Chuang
88b2794244
HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng.
2018-07-16 10:51:23 -07:00
Akira Ajisaka
88625f5cd9
HADOOP-15531. Use commons-text instead of commons-lang in some classes to fix deprecation warnings. Contributed by Takanobu Asanuma.
2018-07-13 11:42:12 -04:00
Xiao Chen
87eeb26e72
HDFS-13663. Should throw exception when incorrect block size is set. Contributed by Shweta.
2018-07-12 20:24:11 -07:00
Xiao Chen
b37074be5a
HDFS-12837. Intermittent failure in TestReencryptionWithKMS.
2018-07-11 21:03:19 -07:00
Akira Ajisaka
418cc7f3ae
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota.
2018-07-11 14:46:43 -04:00
Wei-Chiu Chuang
162228e8db
HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng.
2018-07-11 10:02:08 -07:00
Bharat Viswanadham
c275a9a6a0
Merge trunk into HDDS-48
2018-07-09 12:13:03 -07:00
Andrew Wang
eecb5baaaa
HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi.
2018-07-09 15:17:21 +02:00
Xiao Chen
936e0df0d3
HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer.
2018-07-06 21:00:32 -07:00
Wei-Chiu Chuang
0247cb6318
HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
2018-07-06 14:59:49 -07:00
Bharat Viswanadham
59a3038bc3
HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham
2018-07-02 21:43:18 -07:00
Inigo Goiri
7296b644f7
HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.
2018-07-02 16:36:25 -07:00
Inigo Goiri
1804a31515
HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-07-02 10:48:20 -07:00
Andrew Wang
5d748bd056
HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
2018-07-02 12:11:06 +02:00
Andrew Wang
6ba9974108
HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
2018-07-02 12:02:19 +02:00
Inigo Goiri
73746c5da7
HDFS-13707. [PROVIDED Storage] Fix failing integration tests in ITestProvidedImplementation. Contributed by Virajith Jalaparti.
2018-06-29 09:56:13 -07:00
Bharat Viswanadham
c104525168
Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48
2018-06-28 14:28:23 -07:00
Akira Ajisaka
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
Yiqun Lin
30728aced4
HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.
2018-06-22 10:50:54 +08:00
Xiao Chen
32f867a6a9
HDFS-13682. Cannot create encryption zone after KMS auth token expires.
2018-06-20 15:58:26 -07:00
Akira Ajisaka
fba9d7cd74
HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. Contributed by Takanobu Asanuma.
2018-06-18 10:17:07 -07:00
Chris Douglas
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
Inigo Goiri
8762e9cf10
HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.
2018-06-15 16:49:06 -07:00
Inigo Goiri
d31a3ce767
HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.
2018-06-15 15:59:19 -07:00
Wei-Chiu Chuang
c966a3837a
HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.
2018-06-15 13:36:46 -07:00
Inigo Goiri
eebeb6033f
HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.
2018-06-15 13:07:54 -07:00
Inigo Goiri
43d994e4a6
HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.
2018-06-15 10:33:28 -07:00
Mukul Kumar Singh
3e37a9a70b
HDFS-13679. Fix Typo in javadoc for ScanInfoPerBlockPool#addAll. Contributed by Shashikant Banerjee.
2018-06-15 14:58:20 +05:30
Inigo Goiri
5d7449d2b8
HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.
2018-06-14 13:43:14 -07:00
Inigo Goiri
8d4926f38b
HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.
2018-06-14 09:58:50 -07:00
Arpit Agarwal
418cff4820
Merge remote-tracking branch 'apache-commit/trunk' into HDDS-48
2018-06-14 09:38:20 -07:00
Yiqun Lin
8e7548d33b
HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.
2018-06-13 20:05:55 +08:00
Jitendra Pandey
f4c7c91123
HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan.
2018-06-13 00:36:02 -07:00
Yongjun Zhang
7c3dc39083
Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release.
...
(cherry picked from commit 5a82f10e32
)
2018-06-11 15:13:18 -07:00
Inigo Goiri
2e5cfe6df3
HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
2018-06-11 11:20:29 -07:00
Bharat Viswanadham
ccfb816d39
HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar
2018-06-09 16:43:03 -07:00
Arpit Agarwal
000a678396
HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal.
2018-06-08 17:57:12 -07:00
Xiao Chen
cf4108313d
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail.
2018-06-08 15:14:11 -07:00
Wei-Chiu Chuang
e39b113db0
HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.
2018-06-07 08:30:06 -07:00
Brahma Reddy Battula
9654dd1f47
HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.
2018-06-07 10:16:02 +05:30
Hanisha Koneru
772c95395b
HDDS-140. Add DU usage to VolumeInfo.
2018-06-06 16:21:21 -07:00
Akira Ajisaka
0e3c31579d
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.
2018-06-05 17:53:24 +09:00
Inigo Goiri
8d31ddcfeb
HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.
2018-06-04 18:28:11 -07:00
Inigo Goiri
16316b6011
HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.
2018-06-04 18:21:38 -07:00
Inigo Goiri
a28848104f
HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.
2018-06-04 18:12:43 -07:00
Rushabh Shah
e2289c8d14
HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah
2018-06-04 09:19:03 -05:00
Wei-Chiu Chuang
bccdfeee0a
HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.
2018-06-04 07:03:05 -07:00
Xiao Chen
9efb4b7db0
HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.
2018-06-03 22:33:34 -07:00
Inigo Goiri
8f83b9abf9
HDFS-13648. Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
2018-06-02 20:19:59 -07:00
Inigo Goiri
e308ac88d3
HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.
2018-06-02 20:12:35 -07:00
Chao Sun
ff013d2c95
HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.
2018-05-31 17:37:44 -07:00
Wei-Chiu Chuang
3f4a29813b
HDFS-13646. DFSAdmin doesn't display specialized help for triggerBlockReport. Contributed by Takanobu Asanuma.
2018-05-31 15:26:08 -07:00
Konstantin V Shvachko
ebe5853a45
HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko.
2018-05-31 14:56:32 -07:00
Inigo Goiri
1f324e9661
HDFS-13631. TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.
2018-05-31 10:12:38 -07:00
Sean Mackrory
1361030e59
HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
2018-05-31 07:59:21 -06:00
Inigo Goiri
47c31ff16b
HDFS-13629. Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.
2018-05-30 10:22:04 -07:00
Inigo Goiri
8197b9b560
HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.
2018-05-30 10:13:52 -07:00