ff6a492d66
HDFS-14916. RBF: line breaks are missing from the output of 'hdfs dfsrouteradmin -ls'. Contributed by Xieming Li.
2019-10-21 18:20:00 +05:30
447f46d962
HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui.
2019-10-20 06:10:59 +05:30
e6f95eb0f7
HDFS-14887. RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable. Contributed by hemanthboyina.
2019-10-19 13:24:58 -07:00
3d41f33018
HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.
...
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-18 08:25:02 -07:00
54dc6b7d72
HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore.
2019-10-17 22:28:47 +05:30
5527d79adb
HDFS-14810. Review FSNameSystem editlog sync. Contributed by Xiaoqiao He.
2019-10-17 21:56:30 +05:30
375224edeb
HDFS-14739. RBF: LS command for mount point shows wrong owner and permission information. Contributed by Jinglun.
2019-10-16 19:26:01 +05:30
336abbd873
HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina.
2019-10-14 23:31:43 +05:30
fabd41fa48
HDFS-14856. Fetch file ACLs while mounting external store. ( #1478 )
2019-10-14 09:44:56 -07:00
5f4641a120
HDFS-14238. A log in NNThroughputBenchmark should change log level to INFO instead of ERROR. Contributed by Shen Yinjie.
2019-10-12 21:06:23 +05:30
6e5cd5273f
HDFS-14899. Use Relative URLS in Hadoop HDFS RBF. Contributed by David Mollitor.
2019-10-12 13:26:47 +05:30
eeb58a07e2
HDFS-14898. Use Relative URLS in Hadoop HDFS HTTP FS. Contributed by David Mollitor.
2019-10-10 01:03:33 +05:30
d76e2655ac
HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore.
2019-10-09 23:20:28 +05:30
72ae371e7a
HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko.
2019-10-08 11:56:52 -07:00
91320b4461
HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti.
2019-10-08 14:14:14 +05:30
761594549e
HDFS-14814. RBF: RouterQuotaUpdateService supports inherited rule. Contributed by Jinglun.
2019-10-08 14:01:44 +05:30
382967be51
HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore.
2019-10-08 00:14:30 +05:30
aa24add8f0
HDFS-14890. Fixed namenode and journalnode startup on Windows.
...
Contributed by Siddharth Wagle
2019-10-04 13:13:10 -04:00
b23bdaf085
HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina
2019-10-04 16:47:06 +09:00
c99a12167f
HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-03 22:13:50 -07:00
844b766da5
HDFS-14889. Ability to check if a block has a replica on provided storage. Contributed by Ashvin Agrawal. ( #1573 )"
2019-10-03 21:32:48 -07:00
a3fe404244
HDFS-14881. Safemode 'forceExit' option, doesn’t shown in help message. Contributed by Renukaprasad C.
2019-10-03 22:34:55 +05:30
5a7483ca5c
HDFS-14888. RBF: Enable Parallel Test Profile for builds. Contributed by Ayush Saxena.
2019-10-03 19:21:27 +05:30
1303255aee
HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang.
2019-10-02 16:38:20 -07:00
3df733c250
HDFS-14885. UI: Fix a typo on WebUI of DataNode. Contributed by Xieming Li.
2019-10-02 11:03:16 +09:00
6ef6594c7e
HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. ( #1370 )
...
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.
Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37
* Fix checkstyle
Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
2019-10-01 21:16:46 +05:30
b3275ab1f2
HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko.
2019-09-30 16:48:10 -07:00
760b523e58
Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao."
...
This reverts commit 0feba4396f
2019-09-29 13:11:31 -07:00
d8313b2274
HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun.
2019-09-28 17:20:44 -07:00
2200871607
HDFS-14876. Remove unused imports from TestBlockMissingException.java and TestClose.java. Contributed by Lisheng Sun.
2019-09-28 22:03:53 +05:30
c4c8d5fd0e
HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
2019-09-28 21:44:44 +05:30
0d5d0b914a
Revert "HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao."
...
This reverts commit ce58c05f1d
2019-09-28 21:32:15 +05:30
13b427fc05
HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable ( #963 ) Contributed by Sahil Takiar.
...
Reviewed-by: Siyao Meng <smeng@cloudera.com>
2019-09-27 12:25:03 -07:00
ce58c05f1d
HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao.
2019-09-27 18:58:19 +05:30
2870668cfa
Make upstream aware of 3.1.3 release.
2019-09-27 16:44:07 +08:00
2adcc3c932
HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang.
2019-09-26 15:48:19 -07:00
b1e55cfb55
HDFS-14461. RBF: Fix intermittently failing kerberos related unit test. Contributed by Xiaoqiao He.
2019-09-26 15:26:23 -07:00
1a2a352ecd
HDFS-11934. Add assertion to TestDefaultNameNodePort#testGetAddressFromConf. Contributed by Nikhil Navadiya.
2019-09-26 20:45:37 +05:30
a3f68933d7
HDFS-14873. Fix dfsadmin doc for triggerBlockReport. Contributed by Fei Hui.
2019-09-26 19:54:07 +05:30
bdaaa3bbf2
HDFS-14832. RBF: Add Icon for ReadOnly False. Contributed by hemanthboyina
2019-09-25 23:48:49 +09:00
e346e3638c
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
2019-09-25 12:16:41 +01:00
3f89084ac7
HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-09-25 10:19:30 +09:00
66400c1cbb
HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena.
2019-09-25 01:31:15 +05:30
eb96a3093e
HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena.
2019-09-25 01:16:30 +05:30
afa1006a53
HDFS-14843. Double Synchronization in BlockReportLeaseManager. Contributed by David Mollitor.
2019-09-24 09:58:42 -07:00
816d3cb908
HDFS-14837. Review of Block.java. Contributed by David Mollitor.
2019-09-24 09:54:09 -07:00
43203b466d
HDFS-14868. RBF: Fix typo in TestRouterQuota. Contributed by Jinglun.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-24 09:38:58 -07:00
c30e495557
HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar.
2019-09-23 21:22:50 +05:30
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes ( #1494 ). Contributed by Vinayakumar B.
2019-09-23 12:37:47 +05:30
4c0a7a9e13
Make upstream aware of 3.2.1 release.
2019-09-23 06:20:54 +00:00
659c88801d
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.
2019-09-22 22:02:54 +05:30
aa938662f9
HDFS-14833. RBF: Router Update Doesn't Sync Quota. Contributed by Ayush Saxena.
2019-09-21 09:53:13 +05:30
3f223bebfa
HDFS-14844. Make buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream configurable. Contributed by Lisheng Sun.
2019-09-20 11:45:01 -07:00
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 ( #1432 )
...
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
a79f286c6f
HDFS-14609. RBF: Security should use common AuthenticationFilter. Contributed by Chen Zhang.
2019-09-19 11:00:10 -07:00
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM.
...
Contributed by Malcolm Taylor.
Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
2019-09-19 16:00:05 +01:00
4ed0aefe9f
HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang.
2019-09-18 16:47:46 -07:00
5dd859a8a0
HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2019-09-18 09:34:01 -07:00
3cf6e4272f
HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs
...
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-09-17 15:51:39 -07:00
f580a87079
HDFS-14795. Add Throttler for writing block. Contributed by Lisheng Sun.
2019-09-17 14:55:34 -07:00
e04b8a46c3
HDFS-14799. Do Not Call Map containsKey In Conjunction with get. Contributed by hemanthboyina.
2019-09-14 10:39:20 +05:30
06ad6540b3
HDFS-14821. Make HDFS-14617 (fsimage sub-sections) off by default. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-13 10:36:45 -07:00
4852a90e4b
HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina.
2019-09-13 00:17:13 +05:30
2ff2a7f612
HDFS-14798. Synchronize invalidateBlocks in DatanodeDescriptor. Contributed by hemanthboyina.
2019-09-12 19:43:18 +05:30
d1c303a497
HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming.
2019-09-12 19:11:50 +05:30
68612a0410
HDFS-14840. Use Java Conccurent Instead of Synchronization in BlockPoolTokenSecretManager. Contributed by David Mollitor.
2019-09-12 12:41:57 +09:00
c255333e20
HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li
2019-09-11 16:54:08 +09:00
524b553a5f
HDFS-14835. RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager. ( #1414 )
2019-09-11 10:32:07 +09:00
c92a3e94d8
HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun.
2019-09-06 10:22:28 -07:00
d98c54816d
HDFS-13913. LazyPersistFileScrubber.run() should log meaningful warn message. Contributed by Daniel Green.
2019-09-06 18:12:42 +05:30
fa7f03fc56
HDFS-14826. dfs.ha.zkfc.port property duplicated in hdfs-default.xml. Contributed by Renukaprasad C.
2019-09-06 17:55:30 +05:30
494d75eb2b
HDFS-14784. Add more methods to WebHdfsTestUtil to support tests outside of package. Contributed by Chen Zhang.
2019-09-05 21:15:17 -07:00
72d8b92ba5
HDFS-12904. Add DataTransferThrottler to the Datanode transfers. Contributed by Lisheng Sun.
2019-09-05 11:44:02 -07:00
2f70b52a5b
HDFS-14812. RBF: MountTableRefresherService should load cache when refresh. Contributed by xuzq.
2019-09-05 08:29:56 +05:30
05704754a0
HDFS-14777. RBF: Set ReadOnly is failing for mount Table but actually readonly succeed to set. Contributed by Ranith Sardar.
2019-09-04 15:09:09 +05:30
3c117163a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun.
2019-09-03 09:29:58 -07:00
91b01a1db7
HDFS-14807. SetTimes updates all negative values apart from -1. Contributed by Ayush Saxena.
2019-09-03 18:08:42 +05:30
5ff76cb8bc
HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina.
2019-09-03 12:37:09 +05:30
915cbc91c0
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-02 09:47:04 -07:00
d207aba026
Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell."
...
This reverts commit 7bebad61d9
2019-09-02 09:43:44 -07:00
040f6e93bb
HDFS-14654. RBF: TestRouterRpc#testNamenodeMetrics is flaky. Contributed by Chen Zhang.
2019-09-02 13:01:52 +05:30
c3abfcefdd
HDFS-13843. RBF: Add optional parameter -d for detailed listing of mount points. Contributed by Ayush Saxena.
2019-09-02 07:24:04 +05:30
18d74fe41c
HDFS-14711. RBF: RBFMetrics throws NullPointerException if stateStore disabled. Contributed by Chen Zhang.
2019-09-01 12:53:10 +05:30
96f7dc1992
HDFS-14796. Define LOG instead of BlockManager.LOG in ErasureCodingWork/ReplicationWork. Contributed by Fei Hui.
2019-08-30 14:31:05 +05:30
e220dac15c
HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B.
2019-08-30 08:40:13 +05:30
915271245b
Make DataNodePeerMetrics#minOutlierDetectionSamples configurable ( #1314 ). Contributed by Lisheng Sun.
...
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
2019-08-29 18:35:26 -07:00
7bebad61d9
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-29 17:38:07 -07:00
fcb7884bfc
HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-29 16:56:14 -07:00
3b22fcd377
HDFS-14256. Review Logging of NameNode Class. Contributed by David Mollitor.
2019-08-29 14:21:09 -07:00
ffca734c62
HDFS-14104. Review getImageTxIdToRetain. Contributed by David Mollitor.
2019-08-29 14:18:35 -07:00
f600fbb6c4
HDFS-11246. FSNameSystem#logAuditEvent should be called outside the read or write locks. Contributed by He Xiaoqiao, Kuhu Shukla.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Kuhu Shukla <kshukla@apache.org>
2019-08-29 10:10:52 -07:00
8e779a151e
HDFS-14721. RBF: ProxyOpComplete is not accurate in FederationRPCPerformanceMonitor. Contributed by xuzq.
2019-08-29 20:08:38 +05:30
29bd6f3fc3
HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun.
2019-08-28 23:58:23 +05:30
48cb583906
HDFS-14710. RBF: Improve some RPC performance by using previous block. Contributed by xuzq.
2019-08-28 10:48:00 -07:00
6e37d65b03
HDFS-14760. Log INFO mode if snapshot usage and actual usage differ. Contributed by CR Hota.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 16:20:25 -07:00
dde9399b37
HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 15:26:59 -07:00
ce8eb1283a
Revert "HDDS-1610. applyTransaction failure should not be lost on restart. Contributed by Shashikant Banerjee."
...
This reverts commit 62445021d5
2019-08-27 23:23:44 +05:30
8ab7020e64
HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns
2019-08-27 10:30:31 -07:00
b69ac575a1
HDFS-14772. RBF: hdfs-rbf-site.xml can't be loaded automatically. Contributed by Yuxuan Wang
2019-08-27 14:32:32 +09:00
07e3cf952e
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle.
2019-08-26 15:57:55 -07:00
Ayush Saxena
Inigo Goiri
Yuming Wang
Surendra Singh Lilhore
Ashvin
Chen Liang
Eric Yang
Takanobu Asanuma
Stephen O'Donnell
Virajith Jalaparti
Akira Ajisaka
Wei-Chiu Chuang
Konstantin V Shvachko
Sahil Takiar
Zhankun Tang
Steve Loughran
Prabhu Joseph
Vinayakumar B
Rohith Sharma K S
Rakesh Radhakrishnan
Malcolm Taylor
Sahil Takiar
Erik Krogen
Xiaoyu Yao
leosunli
Stephen O'Donnell
Istvan Fajth
He Xiaoqiao
CR Hota
Shashikant Banerjee
Jonathan Hung
Arpit Agarwal