645d67bc4f
HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu.
2019-06-21 18:27:03 -07:00
1524e2e6c5
HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi.
2019-06-21 18:17:36 -07:00
272b96d243
HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 20:23:05 -07:00
98d2065643
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 20:17:24 -07:00
5962a518bd
HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 09:40:29 -07:00
71ecd2e411
HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
2019-06-19 10:27:53 -07:00
f5ecc0bc08
HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-19 10:06:40 -07:00
d3ac516665
Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti."
...
This reverts commit a1c3868c4f
2019-06-19 06:27:02 -07:00
48e564f7e2
HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar.
2019-06-18 22:28:21 -07:00
a1c3868c4f
HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-18 22:17:49 -07:00
f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya.
2019-06-18 21:58:37 -07:00
81ec909411
HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-18 12:01:26 -07:00
3ab77d9bc9
HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.
2019-06-18 09:58:29 -07:00
335c1c9938
HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti.
2019-06-18 00:35:48 -07:00
54cdde38c7
HDFS-12314. Typo in the TestDataNodeHotSwapVolumes.testAddOneNewVolume(). Contributed by Oleg Danilov.
2019-06-17 23:11:25 -07:00
d41310a15d
HDFS-12315. Use Path instead of String to check closedFiles set. Contributed by Oleg Danilov.
2019-06-17 23:06:02 -07:00
e3172221ea
HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 21:48:16 -07:00
6822193ee6
HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-17 16:20:30 -07:00
7314185c4a
Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
...
This reverts commit ae4143a529
2019-06-17 16:18:48 -07:00
a50c35bb81
Revert "HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
...
This reverts commit cdc5de6448
2019-06-17 16:18:37 -07:00
f9a7b442fd
HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai.
2019-06-17 15:18:17 -07:00
8370a0ae16
HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-15 06:47:10 -07:00
b24efa11ea
HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri.
2019-06-14 13:37:23 -07:00
cdc5de6448
HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-14 13:01:39 -07:00
ae4143a529
HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-14 10:42:54 -07:00
9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method
...
Closes #967
2019-06-14 16:09:37 +02:00
4f455290b1
HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell.
2019-06-13 18:26:53 -07:00
50de0874d0
HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-12 10:30:23 -07:00
4ea6c2f457
HADOOP-16354. Enable AuthFilter as default for WebHDFS.
...
Contributed by Prabhu Joseph
2019-06-11 18:41:08 -04:00
5740eea081
HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao.
2019-06-11 11:48:35 -07:00
101d5b5f86
HDFS-14234. Limit WebHDFS to specifc user, host, directory triples.
...
Contributed by Clay B.
2019-06-10 17:55:16 -07:00
bd46bdf9f9
HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 17:21:52 -07:00
0d160a0ba8
HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed by Wei-Chiu Chuang, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-10 14:13:53 -07:00
7217494f40
HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 13:45:37 -07:00
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
e1dfc060f8
HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena.
2019-06-06 11:59:53 -07:00
944adc61b1
HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao.
2019-06-06 10:20:28 -07:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
9122b9b649
HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee
2019-05-30 17:28:45 -05:00
33c62f8f4e
HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:30:35 -07:00
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
c1caab40f2
HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-29 20:56:38 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
d78854b928
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
...
Contributed by KWON BYUNGCHANG
2019-05-28 17:31:35 -04:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
37900c5639
HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.
2019-05-26 14:30:11 +05:30
55e0c134f0
HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen.
2019-05-24 13:12:59 -07:00
c31b7b8d52
HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko.
2019-05-24 12:34:29 -07:00
c9393dd17f
Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong."
...
This reverts commit 1cb2eb0df3
2019-05-24 10:23:47 -07:00
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
9dff6eff81
HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-22 13:22:26 +09:00
1cb2eb0df3
HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong.
2019-05-20 17:22:58 -07:00
77170e70d1
HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal
2019-05-15 12:15:47 -07:00
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
5847e00143
HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen.
2019-05-13 08:39:59 -07:00
9b0aace1e6
HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.
2019-05-08 17:20:21 +05:30
69b903bbd8
HDFS-14372. NPE while DN is shutting down. Contributed by lujie.
2019-05-05 16:33:52 +05:30
e424392a62
HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi.
2019-05-05 13:38:09 +05:30
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
865c328930
HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota.
2019-05-02 10:09:21 -07:00
7cb46f035a
HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-01 16:48:44 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq.
2019-04-29 14:49:35 -07:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
174b7d3126
HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
2019-04-22 13:29:56 -07:00
5321235fe8
HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena.
2019-04-22 13:23:42 -07:00
13907d8479
HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-04-17 21:39:06 +09:00
be6c8014e6
HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena.
2019-04-16 10:34:31 -07:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
586826fe99
HDFS-12245. Fix INodeId javadoc
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-11 10:01:47 +09:00
e9b859f749
HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-05 16:09:00 -07:00
67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji.
2019-04-05 12:29:11 +09:00
d2637cb176
HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-04 04:07:18 -07:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
be488b6070
HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.
2019-04-03 11:00:59 -07:00
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
53a86e2b8e
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
...
This closes #665
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-01 13:49:37 +09:00
35ff31dd94
HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He.
2019-03-30 23:33:25 -07:00
f3f51284d5
HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R
2019-03-29 00:18:15 +05:30
15d38b1bf9
HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor.
2019-03-28 03:37:33 -07:00
55fb3c32fb
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
2019-03-27 03:27:02 +09:00
3f6d6d2811
HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p)
...
(Contributed by Stephen O'Donnell via Daniel Templeton)
Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
2019-03-25 16:27:04 -07:00
55b3a718e9
HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-03-19 08:14:49 -07:00
ff06ef0631
HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun.
2019-03-15 11:06:42 -07:00
ba50a36a3e
HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He.
2019-03-14 22:21:08 -07:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
34b14061b3
HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-12 10:17:25 -07:00
1bc282e0b3
HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
2019-03-07 08:41:23 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
945b504c25
HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker.
2019-03-05 17:39:52 -08:00
f940ab242d
HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena.
2019-03-05 19:26:42 +05:30
4b7313e640
HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker.
2019-03-04 23:03:32 -08:00
387dbe587a
HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star.
2019-03-04 10:43:44 -08:00
18ea0c1493
HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang.
2019-03-04 17:34:24 +08:00
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng."
...
This reverts commit 5c10630ad8
2019-03-04 16:59:20 +09:00
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
80b77deb42
HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao.
2019-03-01 09:18:51 -08:00
7a0db2f92b
HDFS-13997. Secondary NN Web UI displays nothing, and the console log shows moment is not defined. Contributed by Ayush Saxena
2019-03-01 12:24:52 +05:30
05e4ddeee6
HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena.
2019-02-28 10:03:05 -08:00
0feba4396f
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao.
2019-02-28 08:48:10 -08:00
a6ab37192a
HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko.
2019-02-25 14:35:02 -08:00
f4ae00c530
HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena."
2019-02-26 00:36:14 +05:30
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
7d3b567194
HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi.
2019-02-23 01:08:15 +05:30
92b53c40f0
HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie.
2019-02-21 20:36:34 +05:30
1bea785020
HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 14:28:37 -08:00
f5b4e0f971
HDFS-14302. Refactor NameNodeWebHdfsMethods#generateDelegationToken() to allow better extensibility. Contributed by CR Hota.
2019-02-20 13:55:13 -08:00
a30059bb61
HDFS-14267. Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 11:38:37 -08:00
41e18feda3
HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar.
2019-02-20 16:56:10 +05:30
51950f149e
Logging stale datanode information. Contributed by Karthik Palanisamy.
2019-02-19 17:01:07 -08:00
14282e311b
HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-19 12:04:56 -08:00
7587f97127
HDFS-14296. Prefer ArrayList over LinkedList in VolumeScanner. Contributed by BELUGA BEHR.
2019-02-18 17:55:43 -08:00
1de25d134f
HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR.
2019-02-18 15:30:38 -08:00
235e3da90a
HDFS-14287. DataXceiverServer May Double-Close PeerServer. Contributed by BELUGA BEHR.
2019-02-18 11:00:04 -08:00
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
dde0ab55aa
HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR.
2019-02-15 16:32:27 -08:00
75e15cc0c4
HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects.
...
Contributed by Elek, Marton.
2019-02-14 23:33:25 -08:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
080a421911
HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali.
2019-02-14 08:05:06 -08:00
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
29b411d5f0
HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore
2019-02-13 23:41:27 +05:30
35d4f32b32
HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena.
2019-02-13 23:06:46 +05:30
7806403842
HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena.
2019-02-12 21:57:57 +05:30
5c10630ad8
HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-11 11:02:32 -08:00
0ceb1b70f3
HDFS-14260. Replace synchronized method in BlockReceiver with atomic value. Contributed by BELUGA BEHR.
2019-02-11 10:09:44 -08:00
177131793a
HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li.
2019-02-08 20:51:30 +08:00
4be87353e3
HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-07 16:51:59 -08:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
6aa63452b3
HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-06 12:19:21 -08:00
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
49ddd8a6ed
HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-05 15:14:19 -08:00
308f3168fa
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
2019-02-05 14:03:18 -08:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
5f15a60e32
HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-04 14:52:07 -08:00
0e79a86582
HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar.
2019-02-04 11:59:48 -08:00
9aa3dc872c
HDFS-14158. Checkpointer ignores configured time period > 5 minutes
...
This closes #449
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-02-04 13:51:07 +09:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
d1714c20e9
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1d523279da
2019-01-29 15:43:09 -08:00
dc5af4c6b7
HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki.
2019-01-26 05:34:32 +09:00
1d523279da
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 09:02:57 -08:00
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
f3e642d92b
HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
2019-01-23 15:57:39 -08:00
951cdd7e4c
HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-23 14:40:57 -08:00
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
7d6792e5d2
HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena.
2019-01-23 10:03:06 +09:00
6f0756fc0e
HDFS-14222. Make ThrottledAsyncChecker constructor public. Contributed by Arpit Agarwal.
2019-01-21 20:45:09 -08:00
1ff658b2ef
HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal.
2019-01-21 20:44:37 -08:00
de34fc148c
HDFS-14207. ZKFC should catch exception when ha configuration missing. Contributed by Fei Hui.
2019-01-22 12:01:08 +08:00
e9962240f0
Make 3.2.0 aware to other branches - jdiff
2019-01-21 22:41:26 +05:30
f048512bb8
HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena.
2019-01-16 10:14:22 -08:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
9aeaaa0479
HDFS-14198. Upload and Create button doesn't get enabled after getting reset. Contributed by Ayush Saxena.
2019-01-11 14:36:55 +05:30
4ab5260b7e
HDFS-14132. Add BlockLocation.isStriped() to determine if block is replicated or Striped
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0ed8996a0bae2ad2c7d3513143195533f7191af8
2019-01-08 17:04:57 -08:00
1a08302897
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena.
2019-01-08 10:38:06 -08:00
f660e5eaa3
HDFS-14163. Debug Admin Command Should Support Generic Options. Contributed by Ayush Saxena.
2019-01-03 18:58:03 +05:30
21fe77e3ab
HDFS-14184. [SPS] Add support for URI based path in satisfystoragepolicy command. Contributed by Ayush Saxena.
2019-01-03 18:36:53 +05:30
f30473f2ef
HDFS-14183. [SPS] Remove the -w parameter from the -satisfystoragepolicy usage. Contributed by Ayush Saxena.
2018-12-31 03:16:26 +09:00
e9a005dfcf
HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang.
2018-12-28 10:34:18 -08:00
c546b12e45
HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
c9d7737431
HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko.
2018-12-24 09:34:01 -08:00
fa8550337d
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
c3efc299fe
HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
ef3e1929dd
HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
b73fb70f97
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2018-12-24 09:34:01 -08:00
2904aa3808
HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
cbc2f932eb
HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
652b257478
HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang.
2018-12-24 09:34:01 -08:00
8c49135078
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2018-12-24 09:34:01 -08:00
b6f20c36c2
HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng.
2018-12-24 09:34:00 -08:00
1c4265d7bc
HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
b5b9b77707
HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
a65bb97f5d
HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
b74a7dbf88
HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
25b63e8da8
HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun.
2018-12-24 09:34:00 -08:00
28820fb6b1
HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov.
2018-12-24 09:34:00 -08:00
d2e85b0b6d
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
aa42fb0db7
HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
25d8e39b5c
HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko.
2018-12-24 09:34:00 -08:00
66ec3a67e8
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
4c9baba066
HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
eae0a5d54a
HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
2018-12-24 09:34:00 -08:00
1e22f2bfbb
HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen.
2018-12-24 09:34:00 -08:00
64b7cf59bd
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
0105d7f597
HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko.
2018-12-24 09:33:59 -08:00
00e99c6594
HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
151c8ddbe4
HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
c81ac2ff02
HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen.
2018-12-24 09:33:59 -08:00
e880660a20
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
091ad974cd
HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
9bf0696c73
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
c9d73437e8
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2018-12-24 09:33:59 -08:00
3a78e5ffd8
HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
686fcd4db3
HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar.
2018-12-24 11:02:36 +05:30
d944d5ec46
HDFS-14165. In NameNode UI under DataNode tab ,the Capacity column is Non-Aligned. Contributed by Shubham Dewan.
2018-12-23 15:06:12 +05:30
Wei-Chiu Chuang
Lokesh Jain
Takanobu Asanuma
Surendra Singh Lilhore
David Mollitor
Virajith Jalaparti
Bharat Viswanadham
Elek, Marton
Inigo Goiri
Oleg Danilov
Weiwei Yang
Santosh Marella
Adam Antal
Giovanni Matteo Fumarola
dineshchitlangia
Stephen O'Donnell
Eric Yang
Anu Engineer
He Xiaoqiao
Hanisha Koneru
Shweta Yakkali
Sammi Chen
Kihwal Lee
Ayush Saxena
Akira Ajisaka
Rakesh Radhakrishnan
Erik Krogen
Konstantin V Shvachko
Christopher Gregorian
Siyao Meng
Sahil Takiar
Arpit Agarwal
Sean Mackrory
Puleya7
Sahil Takiar
Chen Liang
Gabor Bota
Uma Maheswara Rao G
Sahil Takiar
Vinayakumar B
Yiqun Lin
Brahma Reddy Battula
Yongjun Zhang
Kitti Nanasi
Steve Loughran
Wangda Tan
Da Zhou
tiwalter
Masatake Iwasaki
Pranay Singh
Dinesh Chitlangia
Sunil G
Chao Sun
Plamen Jeliazkov