40150da1e1
HDFS-14922. Prevent snapshot modification time got change on startup. Contributed by hemanthboyina.
2019-11-12 10:53:54 -08:00
77934bc07b
HDFS-14975. Add CR for SetECPolicyCommand usage. Contributed by Fei Hui.
2019-11-11 11:05:34 +05:30
6663d6a5c2
HDFS-14928. UI: unifying the WebUI across different components. Contributed by Xieming Li
2019-11-11 14:04:57 +09:00
320008bb7c
HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina.
2019-11-09 22:14:35 +05:30
247584eb63
HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan.
2019-11-07 22:49:01 +05:30
dd900259c4
HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko.
2019-11-06 09:56:19 -08:00
9d0d580031
HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen.
2019-11-06 09:30:04 -08:00
9e287054a8
HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun.
2019-11-06 22:02:03 +05:30
dcf55838ae
HDFS-14880. Correct the sequence of statistics & exit message in balencer. Contributed by Renukaprasad C.
2019-11-06 21:45:03 +05:30
c36014165c
HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore.
2019-11-06 19:28:55 +05:30
bfb8f28cc9
HDFS-14775. Add Timestamp for longest FSN write/read lock held log. Contributed by Chen Zhang.
2019-11-05 09:19:52 -08:00
b643a1cbe8
HDFS-14938. Add check if excludedNodes contain scope in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Lisheng Sun.
2019-11-05 09:23:20 +05:30
2ffec347eb
HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui.
2019-11-05 01:37:15 +05:30
eb73ba6ed5
HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao.
2019-11-04 09:40:42 -08:00
de6b8b0c0b
HDFS-14925. Rename operation should check nest snapshot ( #1670 )
...
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.
Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
2019-11-01 16:37:53 -07:00
7d7acb004a
HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong.
2019-11-01 22:50:10 +05:30
02009c3bb7
HDFS-14768. EC : Busy DN replica should be consider in live replica check. Contributed by guojh.
2019-11-01 22:15:05 +05:30
9d25ae7669
HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui.
2019-10-31 23:49:37 +05:30
d9fbedc4ae
HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun.
2019-10-31 23:39:27 +05:30
e6137d0e50
Make upstream aware of 2.10.0 release
2019-10-30 20:58:32 -07:00
fa4904cdca
HDFS-14935. Refactor DFSNetworkTopology#isNodeInScope. Contributed by Lisheng Sun.
2019-10-29 12:30:24 +05:30
30ed24a421
HDFS-14730. Removed unused configuration dfs.web.authentication.filter.
...
Contributed by Chen Zhang
2019-10-28 19:27:32 -04:00
9ef6ed9c1c
HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger
2019-10-28 16:37:04 +00:00
30db895b59
HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() ( #1667 )
...
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-25 13:09:14 -07:00
862526530a
HDFS-14933. Fixing a typo in documentation of Observer NameNode. Contributed by Xieming Li
2019-10-25 15:23:05 +09:00
0db0f1e399
HDFS-14917. Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html". Contributed by Xieming Li
2019-10-25 10:27:38 +09:00
a1b4eebcc9
HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. ( #1672 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2019-10-24 13:09:52 -07:00
ac6b6a6a85
HADOOP-16614. Add aarch64 support for dependent leveldbjni.
...
Contributed by liusheng
close #1546
2019-10-24 11:45:57 -04:00
ee699dc26c
HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena.
2019-10-24 12:14:09 +05:30
a901405ad8
HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-22 13:31:15 -07:00
6020505943
HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena.
2019-10-22 18:44:22 +05:30
19f35cfd57
HDFS-14918. Remove useless getRedundancyThread from BlockManagerTestUtil. Contributed by Fei Hui.
2019-10-22 18:34:02 +05:30
72003b19bf
HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun.
2019-10-21 17:31:43 -07:00
c4e27ef773
HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He.
2019-10-21 16:33:15 -05:00
74c2329fc3
HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena.
2019-10-21 18:35:12 +05:30
447f46d962
HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui.
2019-10-20 06:10:59 +05:30
3d41f33018
HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.
...
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-18 08:25:02 -07:00
54dc6b7d72
HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore.
2019-10-17 22:28:47 +05:30
5527d79adb
HDFS-14810. Review FSNameSystem editlog sync. Contributed by Xiaoqiao He.
2019-10-17 21:56:30 +05:30
336abbd873
HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina.
2019-10-14 23:31:43 +05:30
fabd41fa48
HDFS-14856. Fetch file ACLs while mounting external store. ( #1478 )
2019-10-14 09:44:56 -07:00
5f4641a120
HDFS-14238. A log in NNThroughputBenchmark should change log level to INFO instead of ERROR. Contributed by Shen Yinjie.
2019-10-12 21:06:23 +05:30
d76e2655ac
HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore.
2019-10-09 23:20:28 +05:30
72ae371e7a
HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko.
2019-10-08 11:56:52 -07:00
91320b4461
HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti.
2019-10-08 14:14:14 +05:30
382967be51
HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore.
2019-10-08 00:14:30 +05:30
aa24add8f0
HDFS-14890. Fixed namenode and journalnode startup on Windows.
...
Contributed by Siddharth Wagle
2019-10-04 13:13:10 -04:00
b23bdaf085
HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina
2019-10-04 16:47:06 +09:00
c99a12167f
HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-03 22:13:50 -07:00
844b766da5
HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. ( #1573 )"
2019-10-03 21:32:48 -07:00
a3fe404244
HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C.
2019-10-03 22:34:55 +05:30
1303255aee
HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang.
2019-10-02 16:38:20 -07:00
3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li.
2019-10-02 11:03:16 +09:00
6ef6594c7e
HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. ( #1370 )
...
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.
Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37
* Fix checkstyle
Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
2019-10-01 21:16:46 +05:30
b3275ab1f2
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko.
2019-09-30 16:48:10 -07:00
760b523e58
Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao."
...
This reverts commit 0feba4396f
2019-09-29 13:11:31 -07:00
2200871607
HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun.
2019-09-28 22:03:53 +05:30
c4c8d5fd0e
HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
2019-09-28 21:44:44 +05:30
0d5d0b914a
Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao."
...
This reverts commit ce58c05f1d
2019-09-28 21:32:15 +05:30
13b427fc05
HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable ( #963 ) Contributed by Sahil Takiar.
...
Reviewed-by: Siyao Meng <smeng@cloudera.com>
2019-09-27 12:25:03 -07:00
ce58c05f1d
HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
2019-09-27 18:58:19 +05:30
2870668cfa
Make upstream aware of 3.1.3 release.
2019-09-27 16:44:07 +08:00
a3f68933d7
HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui.
2019-09-26 19:54:07 +05:30
66400c1cbb
HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena.
2019-09-25 01:31:15 +05:30
eb96a3093e
HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena.
2019-09-25 01:16:30 +05:30
afa1006a53
HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor.
2019-09-24 09:58:42 -07:00
c30e495557
HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar.
2019-09-23 21:22:50 +05:30
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes ( #1494 ). Contributed by Vinayakumar B.
2019-09-23 12:37:47 +05:30
4c0a7a9e13
Make upstream aware of 3.2.1 release.
2019-09-23 06:20:54 +00:00
659c88801d
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.
2019-09-22 22:02:54 +05:30
3f223bebfa
HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun.
2019-09-20 11:45:01 -07:00
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 ( #1432 )
...
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
4ed0aefe9f
HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang.
2019-09-18 16:47:46 -07:00
5dd859a8a0
HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-09-18 09:34:01 -07:00
f580a87079
HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun.
2019-09-17 14:55:34 -07:00
e04b8a46c3
HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina.
2019-09-14 10:39:20 +05:30
06ad6540b3
HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-13 10:36:45 -07:00
4852a90e4b
HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina.
2019-09-13 00:17:13 +05:30
2ff2a7f612
HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina.
2019-09-12 19:43:18 +05:30
d1c303a497
HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming.
2019-09-12 19:11:50 +05:30
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor.
2019-09-12 12:41:57 +09:00
c92a3e94d8
HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun.
2019-09-06 10:22:28 -07:00
d98c54816d
HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green.
2019-09-06 18:12:42 +05:30
fa7f03fc56
HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C.
2019-09-06 17:55:30 +05:30
494d75eb2b
HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang.
2019-09-05 21:15:17 -07:00
72d8b92ba5
HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun.
2019-09-05 11:44:02 -07:00
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun.
2019-09-03 09:29:58 -07:00
91b01a1db7
HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena.
2019-09-03 18:08:42 +05:30
5ff76cb8bc
HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina.
2019-09-03 12:37:09 +05:30
915cbc91c0
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-02 09:47:04 -07:00
d207aba026
Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell."
...
This reverts commit 7bebad61d9
2019-09-02 09:43:44 -07:00
c3abfcefdd
HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena.
2019-09-02 07:24:04 +05:30
96f7dc1992
HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui.
2019-08-30 14:31:05 +05:30
e220dac15c
HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B.
2019-08-30 08:40:13 +05:30
915271245b
Make DataNodePeerMetrics#minOutlierDetectionSamples configurable ( #1314 ). Contributed by Lisheng Sun.
...
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
2019-08-29 18:35:26 -07:00
7bebad61d9
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-29 17:38:07 -07:00
fcb7884bfc
HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-29 16:56:14 -07:00
3b22fcd377
HDFS-14256. Review Logging of NameNode Class. Contributed by David Mollitor.
2019-08-29 14:21:09 -07:00
ffca734c62
HDFS-14104. Review getImageTxIdToRetain. Contributed by David Mollitor.
2019-08-29 14:18:35 -07:00
f600fbb6c4
HDFS-11246. FSNameSystem#logAuditEvent should be called outside the read or write locks. Contributed by He Xiaoqiao, Kuhu Shukla.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Kuhu Shukla <kshukla@apache.org>
2019-08-29 10:10:52 -07:00
29bd6f3fc3
HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun.
2019-08-28 23:58:23 +05:30
6e37d65b03
HDFS-14760. Log INFO mode if snapshot usage and actual usage differ. Contributed by CR Hota.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 16:20:25 -07:00
dde9399b37
HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 15:26:59 -07:00
ce8eb1283a
Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee."
...
This reverts commit 62445021d5
2019-08-27 23:23:44 +05:30
8ab7020e64
HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns
2019-08-27 10:30:31 -07:00
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle.
2019-08-26 15:57:55 -07:00
ebef99dcf4
HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui.
2019-08-23 12:18:43 -07:00
d699022fce
HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen.
2019-08-23 11:33:21 -07:00
bd7baea5a5
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui.
2019-08-23 15:28:27 +09:00
abc8fde4ca
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui.
2019-08-23 14:44:34 +09:00
b67812ea21
HDFS-14617. Improve fsimage load time by writing sub-sections to the fsimage index ( #1028 ). Contributed by Stephen O'Donnell.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-08-22 17:09:57 -07:00
93daf69f90
HDFS-14675. Increase Balancer Defaults Further. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Gabor Bota <gabota@apache.org>
2019-08-22 15:21:33 -07:00
e04dcfdc57
HDFS-14583. FileStatus#toString() will throw IllegalArgumentException. Contributed by xuzq.
2019-08-22 10:22:38 -07:00
76790a1e67
HDFS-14358. Provide LiveNode and DeadNode filter in DataNode UI. Contributed by hemanthboyina.
2019-08-22 12:27:45 +05:30
0f598aed13
HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang.
2019-08-22 02:25:06 +05:30
57f7370174
Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow."
...
This reverts commit b58a35f374
2019-08-21 08:59:49 -07:00
3a145e2918
HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-20 15:55:26 -07:00
269b543367
HADOOP-16523. Minor spell mistake in comment : Line number 101 ( #388 )
2019-08-20 13:36:23 -07:00
4cb22cd867
HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai.
2019-08-20 10:37:11 -07:00
bd92462321
HDFS-14729. Upgrade Bootstrap and jQuery versions used in HDFS UIs. Contributed by Vivek Ratnavel Subramanian. This closes #1297
2019-08-20 18:42:19 +05:30
62445021d5
HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee.
2019-08-20 14:37:01 +05:30
360a96f342
HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang.
2019-08-19 13:08:55 -07:00
abae6ff2a2
HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su.
...
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-19 11:14:51 -07:00
b8db5b9a98
HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore.
...
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-18 18:13:35 -07:00
b58a35f374
HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow.
2019-08-18 17:46:00 -07:00
d873ddd656
HDFS-14127. Add a description about the observer read configuration. Contributed by xiangheng.
2019-08-17 04:39:22 -07:00
528378784f
HDFS-12012. Fix spelling mistakes in BPServiceActor.java. Contributed by chencan.
2019-08-17 04:37:05 -07:00
a38b9e137e
HDFS-14456:HAState#prepareToEnterState neednt a lock ( #770 ) Contributed by hunshenshi.
2019-08-16 14:53:06 -07:00
9a1d8cfaf5
HDFS-14678. Allow triggerBlockReport to a specific namenode. ( #1252 ). Contributed by Leon Gao.
2019-08-16 08:00:51 -07:00
34681643e9
HDFS-14719. Correct the safemode threshold value in BlockManagerSafeMode. Contributed by hemanthboyina.
2019-08-15 12:12:18 +05:30
0a85af959c
HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee.
2019-08-15 10:16:25 +05:30
3c0382f1b9
HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2019-08-14 07:30:35 -07:00
c13ec7ab66
HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng.
2019-08-13 17:15:26 -07:00
633b7c1cfe
HDFS-14625. Make DefaultAuditLogger class in FSnamesystem to Abstract. Contributed by hemanthboyina.
2019-08-13 16:50:49 -07:00
da0006fe04
HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-14 08:39:40 +09:00
e9b6b81de4
HDFS-13505. Turn on HDFS ACLs by default. Contributed by Siyao Meng.
2019-08-13 19:17:10 +05:30
454420e4f2
HDFS-14708. TestLargeBlockReport#testBlockReportSucceedsWithLargerLengthLimit fails in trunk. Contributed by Lisheng Sun.
2019-08-13 08:27:20 +05:30
c92b49876a
HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory ( #1274 ) Contributed by Siyao Meng.
2019-08-12 17:26:20 -07:00
8a77a224c7
HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-09 18:40:28 -07:00
e02ffed1b1
HDFS-12125. Document the missing EC removePolicy command ( #1258 ) Contributed by Wenxin He, Siyao Meng.
...
Co-authored-by: Wenxin He <wenxin.he@gmail.com>
2019-08-09 18:00:22 -07:00
ce3c5a3e3b
HDFS-14623. In NameNode Web UI, for Head the file (first 32K) old data is showing. Contributed by hemanthboyina.
2019-08-09 15:41:37 -07:00
865021b8c9
HDFS-14195. OIV: print out storage policy id in oiv Delimited output. Contributed by Wang, Xinglong.
2019-08-09 15:37:29 -07:00
f6fa865d6f
HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He.
2019-08-09 14:07:54 +05:30
6ad9a11494
HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao.
2019-08-08 13:50:30 -07:00
2265872c2d
HDFS-14705. Remove unused configuration dfs.min.replication. Contributed by CR Hota.
2019-08-08 13:48:29 -07:00
23f91f68b8
HDFS-14662. Document the usage of the new Balancer "asService" parameter. Contributed by Chen Zhang.
2019-08-08 13:45:29 -07:00
b0799148cf
HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-08 13:36:39 -07:00
11f750e6a7
HDFS-14616. Add the warn log when the volume available space isn't enough. Contributed by liying.
2019-08-08 00:49:58 +05:30
de64253892
HDFS-14608. DataNode#DataTransfer should be named. Contributed by Inigo Goiri.
2019-08-08 00:37:56 +05:30
827dbb11e2
HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen.
2019-08-07 09:25:58 -07:00
a5bb1e8ee8
HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
2019-08-07 10:18:11 +08:00
954ff36360
HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-06 15:30:50 -07:00
35e0a01d7b
HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-06 07:57:46 -07:00
11272159bb
HDFS-14679. Failed to add erasure code policies with example template. Contributed by Yuan Zhou.
2019-08-06 08:42:15 +05:30
8f40856f76
HDFS-14669. TestDirectoryScanner#testDirectoryScannerInFederatedCluster fails intermittently in trunk. Contributed by qiang Liu.
2019-08-03 12:15:41 +05:30
e503db5f44
HDFS-12826. Document Saying the RPC port, But it's required IPC port in HDFS Federation Document. Contributed by usharani.
2019-08-03 12:05:46 +05:30
e7a0b8aa83
HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira.
2019-08-02 11:48:31 -07:00
b964b81f85
HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq.
2019-08-02 10:54:32 -07:00
c2d00c8450
HDFS-13131. Modifying testcase testEnableAndDisableErasureCodingPolicy. Contributed by chencan.
2019-08-01 19:11:43 -07:00
99bf1dc9eb
HDFS-14683. WebHDFS: Add erasureCodingPolicy field to GETCONTENTSUMMARY response ( #1189 ) Contributed by Siyao Meng.
2019-08-01 17:14:07 -07:00
32607dbd98
HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun.
2019-08-01 15:48:36 -07:00
99f88c30cb
HDFS-14419. Avoid repeated calls to the listOpenFiles function. Contributed by HuangTao.
2019-07-30 16:50:06 -07:00
3ae775d740
HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-07-30 16:01:17 -07:00
1f26cc8705
HDFS-13783. Add an option to the Balancer to make it run as a long-running service. Contributed by Chen Zhang.
2019-07-30 15:42:55 -07:00
7849bdcf70
HDFS-14449. Expose total number of DT in JMX for Namenode. Contributed by Fengnan Li.
2019-07-30 13:45:27 -07:00
8deced0073
HDFS-14677. TestDataNodeHotSwapVolumes#testAddVolumesConcurrently fails intermittently in trunk. Contributed by Chen Zhang.
2019-07-30 22:17:39 +05:30
8053085388
HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai.
2019-07-29 14:31:34 -07:00
02bd02b5af
HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun.
2019-07-28 08:41:42 +05:30
ce99cc31e9
HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
2019-07-26 08:31:00 +05:30
6b8107ad97
HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-07-26 09:35:23 +09:00
62deab17a3
HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li.
2019-07-25 06:51:07 +05:30
377f95bbe8
HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun.
2019-07-23 08:37:55 +05:30
5e6cc6fe8a
HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang.
2019-07-17 14:37:16 -07:00
af0665c7f5
HDFS-13647. Fix the description of storageType option for space quota. Contributed by Takanobu Asanuma.
2019-07-17 23:22:32 +05:30
f77d54c243
HDFS-14642. processMisReplicatedBlocks does not return correct processed count. Contributed by Stephen O'Donnell.
2019-07-16 08:14:27 +05:30
30a8f840f1
HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He.
2019-07-15 13:18:23 +05:30
e98adb00b7
HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He.
2019-07-15 13:02:37 +05:30
f9fab9f22a
HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee.
2019-07-12 15:42:06 +05:30
00dd843a1a
HDFS-14466. Add a regression test for HDFS-14323.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-12 10:04:55 +09:00
8fb5ca3f40
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-07-11 13:23:25 -07:00
5747f6cff5
HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell.
2019-07-11 08:55:44 -07:00
eccc9a40de
HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao.
2019-07-10 11:11:52 -07:00
cdb20adfcc
HDDS-1611. Evaluate ACL on volume bucket key and prefix to authorize access. Contributed by Ajay Kumar. ( #973 )
2019-07-10 11:03:58 -07:00
993dc8726b
HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
2019-07-10 10:53:34 +08:00
4632708148
HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun.
2019-07-08 14:46:25 -07:00
ec851e4db2
HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL.
2019-07-08 15:23:46 +08:00
9c90729486
HDFS-13694. Making md5 computing being in parallel with image loading.
2019-07-05 10:17:37 -07:00
1c254a8ec7
HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina.
2019-07-04 23:20:42 +09:00
729cb3aefe
HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang.
2019-07-04 09:37:24 +08:00
d8bac50e12
HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. ( #1015 )
2019-07-01 13:54:49 -07:00
d203045c30
HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward.
2019-06-28 16:06:43 -07:00
4a212242d9
HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao.
2019-06-27 10:00:05 -07:00
041e7a7dee
HDFS-14598. Findbugs warning caused by HDFS-12487.
...
Contributed by He Xiaoqiao.
2019-06-24 19:07:22 -07:00
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.
2019-06-24 11:59:07 -07:00
dd8c2b92df
HDFS-13853. RBF: RouterAdmin update cmd is overwriting the entry not updating the existing. Contributed by Ayush Saxena.
2019-06-24 22:03:04 +05:30
6c42d40504
HDFS-14316. RBF: Support unavailable subclusters for mount points with multiple destinations. Contributed by Inigo Goiri.
2019-06-24 22:03:04 +05:30
2a2d5eb441
HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri.
2019-06-24 22:03:03 +05:30
8b8ff5ccbc
HDFS-14249. RBF: Tooling to identify the subcluster location of a file. Contributed by Inigo Goiri.
2019-06-24 22:03:03 +05:30
f4e2bfce58
HDFS-13856. RBF: RouterAdmin should support dfsrouteradmin -refreshRouterArgs command. Contributed by yanghuafeng.
2019-06-24 22:03:02 +05:30
8f6f9d9c83
HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad.
2019-06-24 22:03:02 +05:30
645d67bc4f
HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu.
2019-06-21 18:27:03 -07:00
1524e2e6c5
HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi.
2019-06-21 18:17:36 -07:00
272b96d243
HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 20:23:05 -07:00
98d2065643
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 20:17:24 -07:00
5962a518bd
HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 09:40:29 -07:00
71ecd2e411
HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu.
2019-06-19 10:27:53 -07:00
f5ecc0bc08
HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-19 10:06:40 -07:00
d3ac516665
Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti."
...
This reverts commit a1c3868c4f
2019-06-19 06:27:02 -07:00
48e564f7e2
HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar.
2019-06-18 22:28:21 -07:00
a1c3868c4f
HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-18 22:17:49 -07:00
f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya.
2019-06-18 21:58:37 -07:00
81ec909411
HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-18 12:01:26 -07:00
3ab77d9bc9
HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.
2019-06-18 09:58:29 -07:00
335c1c9938
HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti.
2019-06-18 00:35:48 -07:00
54cdde38c7
HDFS-12314. Typo in the TestDataNodeHotSwapVolumes.testAddOneNewVolume(). Contributed by Oleg Danilov.
2019-06-17 23:11:25 -07:00
d41310a15d
HDFS-12315. Use Path instead of String to check closedFiles set. Contributed by Oleg Danilov.
2019-06-17 23:06:02 -07:00
e3172221ea
HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 21:48:16 -07:00
6822193ee6
HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-17 16:20:30 -07:00
7314185c4a
Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
...
This reverts commit ae4143a529
2019-06-17 16:18:48 -07:00
a50c35bb81
Revert "HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
...
This reverts commit cdc5de6448
2019-06-17 16:18:37 -07:00
f9a7b442fd
HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai.
2019-06-17 15:18:17 -07:00
8370a0ae16
HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-15 06:47:10 -07:00
b24efa11ea
HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri.
2019-06-14 13:37:23 -07:00
cdc5de6448
HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-14 13:01:39 -07:00
ae4143a529
HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-06-14 10:42:54 -07:00
9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method
...
Closes #967
2019-06-14 16:09:37 +02:00
4f455290b1
HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell.
2019-06-13 18:26:53 -07:00
50de0874d0
HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-12 10:30:23 -07:00
4ea6c2f457
HADOOP-16354. Enable AuthFilter as default for WebHDFS.
...
Contributed by Prabhu Joseph
2019-06-11 18:41:08 -04:00
5740eea081
HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao.
2019-06-11 11:48:35 -07:00
101d5b5f86
HDFS-14234. Limit WebHDFS to specifc user, host, directory triples.
...
Contributed by Clay B.
2019-06-10 17:55:16 -07:00
bd46bdf9f9
HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 17:21:52 -07:00
0d160a0ba8
HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed by Wei-Chiu Chuang, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-10 14:13:53 -07:00
7217494f40
HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 13:45:37 -07:00
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
e1dfc060f8
HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena.
2019-06-06 11:59:53 -07:00
944adc61b1
HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao.
2019-06-06 10:20:28 -07:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
9122b9b649
HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee
2019-05-30 17:28:45 -05:00
33c62f8f4e
HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:30:35 -07:00
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
c1caab40f2
HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-29 20:56:38 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
d78854b928
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
...
Contributed by KWON BYUNGCHANG
2019-05-28 17:31:35 -04:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
37900c5639
HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He.
2019-05-26 14:30:11 +05:30
55e0c134f0
HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen.
2019-05-24 13:12:59 -07:00
c31b7b8d52
HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko.
2019-05-24 12:34:29 -07:00
c9393dd17f
Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong."
...
This reverts commit 1cb2eb0df3
2019-05-24 10:23:47 -07:00
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
9dff6eff81
HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-22 13:22:26 +09:00
1cb2eb0df3
HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong.
2019-05-20 17:22:58 -07:00
77170e70d1
HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal
2019-05-15 12:15:47 -07:00
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
5847e00143
HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen.
2019-05-13 08:39:59 -07:00
9b0aace1e6
HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He.
2019-05-08 17:20:21 +05:30
69b903bbd8
HDFS-14372. NPE while DN is shutting down. Contributed by lujie.
2019-05-05 16:33:52 +05:30
e424392a62
HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi.
2019-05-05 13:38:09 +05:30
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
865c328930
HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota.
2019-05-02 10:09:21 -07:00
7cb46f035a
HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-01 16:48:44 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq.
2019-04-29 14:49:35 -07:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
174b7d3126
HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen.
2019-04-22 13:29:56 -07:00
5321235fe8
HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena.
2019-04-22 13:23:42 -07:00
13907d8479
HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-04-17 21:39:06 +09:00
be6c8014e6
HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena.
2019-04-16 10:34:31 -07:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
586826fe99
HDFS-12245. Fix INodeId javadoc
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-11 10:01:47 +09:00
e9b859f749
HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-05 16:09:00 -07:00
67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji.
2019-04-05 12:29:11 +09:00
d2637cb176
HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-04 04:07:18 -07:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
be488b6070
HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang.
2019-04-03 11:00:59 -07:00
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
53a86e2b8e
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project
...
This closes #665
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-01 13:49:37 +09:00
35ff31dd94
HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He.
2019-03-30 23:33:25 -07:00
f3f51284d5
HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R
2019-03-29 00:18:15 +05:30
15d38b1bf9
HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor.
2019-03-28 03:37:33 -07:00
55fb3c32fb
HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
2019-03-27 03:27:02 +09:00
3f6d6d2811
HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p)
...
(Contributed by Stephen O'Donnell via Daniel Templeton)
Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
2019-03-25 16:27:04 -07:00
55b3a718e9
HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen.
2019-03-19 08:14:49 -07:00
ff06ef0631
HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun.
2019-03-15 11:06:42 -07:00
ba50a36a3e
HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He.
2019-03-14 22:21:08 -07:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
34b14061b3
HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-12 10:17:25 -07:00
1bc282e0b3
HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
2019-03-07 08:41:23 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
945b504c25
HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker.
2019-03-05 17:39:52 -08:00
f940ab242d
HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena.
2019-03-05 19:26:42 +05:30
4b7313e640
HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker.
2019-03-04 23:03:32 -08:00
387dbe587a
HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star.
2019-03-04 10:43:44 -08:00
18ea0c1493
HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang.
2019-03-04 17:34:24 +08:00
6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng."
...
This reverts commit 5c10630ad8
2019-03-04 16:59:20 +09:00
fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen.
2019-03-04 16:52:04 +09:00
80b77deb42
HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao.
2019-03-01 09:18:51 -08:00
7a0db2f92b
HDFS-13997. Secondary NN Web UI displays nothing, and the console log shows moment is not defined. Contributed by Ayush Saxena
2019-03-01 12:24:52 +05:30
05e4ddeee6
HDFS-14324. Fix TestDataNodeVolumeFailure. Contributed by Ayush Saxena.
2019-02-28 10:03:05 -08:00
0feba4396f
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao.
2019-02-28 08:48:10 -08:00
a6ab37192a
HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by xiangheng and Konstantin Shvachko.
2019-02-25 14:35:02 -08:00
f4ae00c530
HDFS-7133. Support clearing namespace quota on '/'. Contributed by Ayush Saxena."
2019-02-26 00:36:14 +05:30
f7a27cdee4
HDFS-14118. Support using DNS to resolve nameservices to IP addresses. Contributed by Fengnan Li.
2019-02-23 09:35:36 -08:00
7d3b567194
HDFS-14298. Improve log messages of ECTopologyVerifier. Contributed by Kitti Nanasi.
2019-02-23 01:08:15 +05:30
92b53c40f0
HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie.
2019-02-21 20:36:34 +05:30
1bea785020
HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 14:28:37 -08:00
f5b4e0f971
HDFS-14302. Refactor NameNodeWebHdfsMethods#generateDelegationToken() to allow better extensibility. Contributed by CR Hota.
2019-02-20 13:55:13 -08:00
a30059bb61
HDFS-14267. Add test_libhdfs_ops to libhdfs tests, mark libhdfs_read/write.c as examples. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-20 11:38:37 -08:00
41e18feda3
HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar.
2019-02-20 16:56:10 +05:30
51950f149e
Logging stale datanode information. Contributed by Karthik Palanisamy.
2019-02-19 17:01:07 -08:00
14282e311b
HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-19 12:04:56 -08:00
7587f97127
HDFS-14296. Prefer ArrayList over LinkedList in VolumeScanner. Contributed by BELUGA BEHR.
2019-02-18 17:55:43 -08:00
1de25d134f
HDFS-9596. Remove Shuffle Method From DFSUtil. Contributed by BELUGA BEHR.
2019-02-18 15:30:38 -08:00
235e3da90a
HDFS-14287. DataXceiverServer May Double-Close PeerServer. Contributed by BELUGA BEHR.
2019-02-18 11:00:04 -08:00
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
dde0ab55aa
HDFS-14258. Introduce Java Concurrent Package To DataXceiverServer Class. Contributed by BELUGA BEHR.
2019-02-15 16:32:27 -08:00
75e15cc0c4
HDDS-1103.Fix rat/findbug/checkstyle errors in ozone/hdds projects.
...
Contributed by Elek, Marton.
2019-02-14 23:33:25 -08:00
64f28f9efa
HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen.
2019-02-14 11:22:04 -08:00
0d7a5ac5f5
HDFS-13209. DistributedFileSystem.create should allow an option to provide StoragePolicy. Contributed by Ayush Saxena.
2019-02-14 22:13:14 +05:30
080a421911
HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali.
2019-02-14 08:05:06 -08:00
024c87291c
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-02-13 12:40:31 -08:00
29b411d5f0
HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore
2019-02-13 23:41:27 +05:30
35d4f32b32
HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena.
2019-02-13 23:06:46 +05:30
7806403842
HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena.
2019-02-12 21:57:57 +05:30
5c10630ad8
HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-11 11:02:32 -08:00
0ceb1b70f3
HDFS-14260. Replace synchronized method in BlockReceiver with atomic value. Contributed by BELUGA BEHR.
2019-02-11 10:09:44 -08:00
177131793a
HDFS-14172. Avoid NPE when SectionName#fromString returns null. Contributed by Xiang Li.
2019-02-08 20:51:30 +08:00
4be87353e3
HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-07 16:51:59 -08:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
6aa63452b3
HDFS-14242. OIV WebImageViewer: NPE when param op is not specified. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-06 12:19:21 -08:00
d3de8e162b
HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun.
2019-02-06 09:54:47 -08:00
49ddd8a6ed
HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-05 15:14:19 -08:00
308f3168fa
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
2019-02-05 14:03:18 -08:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
5f15a60e32
HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-02-04 14:52:07 -08:00
0e79a86582
HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar.
2019-02-04 11:59:48 -08:00
9aa3dc872c
HDFS-14158. Checkpointer ignores configured time period > 5 minutes
...
This closes #449
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-02-04 13:51:07 +09:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
d1714c20e9
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1d523279da
2019-01-29 15:43:09 -08:00
dc5af4c6b7
HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki.
2019-01-26 05:34:32 +09:00
1d523279da
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 09:02:57 -08:00
45caeee6cf
HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-25 07:37:01 -08:00
f3e642d92b
HDFS-14185. Cleanup method calls to static Assert methods in TestAddStripedBlocks
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I0d533f575a405ed3affd05994a4208bde7d9cbe9
2019-01-23 15:57:39 -08:00
951cdd7e4c
HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-01-23 14:40:57 -08:00
221e308cb5
HDFS-14153. [SPS] : Add Support for Storage Policy Satisfier in WEBHDFS. Contributed by Ayush Saxena.
2019-01-23 17:00:37 +05:30
Inigo Goiri
Ayush Saxena
Takanobu Asanuma
Surendra Singh Lilhore
Chen Liang
Wei-Chiu Chuang
Zhao Junwang
Jonathan Hung
Eric Yang
Eric Badger
zhaoyim
Vinayakumar B
Mukul Kumar Singh
Kihwal Lee
Yuming Wang
Ashvin
Stephen O'Donnell
Virajith Jalaparti
Akira Ajisaka
Konstantin V Shvachko
Sahil Takiar
Zhankun Tang
Rohith Sharma K S
Rakesh Radhakrishnan
Erik Krogen
Xiaoyu Yao
leosunli
Stephen O'Donnell
Istvan Fajth
He Xiaoqiao
CR Hota
Shashikant Banerjee
Jonathan Hung
Arpit Agarwal
Sandeep Kumar
Sunil G
HUAN-PING SU
hunshenshi
LeonGao91
Siyao Meng
Masatake Iwasaki
Siyao Meng
Yiqun Lin
Lokesh Jain
Chao Sun
Ajay Yadav
Weiwei Yang
paulward24
Anu Engineer
Ajay Kumar
Giovanni Matteo Fumarola
David Mollitor
Bharat Viswanadham
Elek, Marton
Oleg Danilov
Santosh Marella
Adam Antal
dineshchitlangia
Hanisha Koneru
Shweta Yakkali
Sammi Chen
Christopher Gregorian
Sean Mackrory
Puleya7
Sahil Takiar
Gabor Bota
Uma Maheswara Rao G
Sahil Takiar
Brahma Reddy Battula
Yongjun Zhang
Kitti Nanasi
Steve Loughran
Wangda Tan
Da Zhou
tiwalter
Pranay Singh
Dinesh Chitlangia