41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. ( #2233 )
2020-08-27 14:54:52 +05:30
2ffe00fc46
HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-08-26 23:05:32 -07:00
ca8e7a7725
HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina.
2020-08-27 12:10:39 +09:00
6e618b6a7e
HDFS-15536. RBF: Clear Quota in Router was not consistent.
2020-08-26 13:03:08 +05:30
6a49bf9bff
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
2020-08-25 15:18:36 +01:00
a7830423c5
HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.
2020-08-24 21:16:00 +08:00
83fd15b412
HDFS-14504. Rename with Snapshots does not honor quota limit.
2020-08-23 16:10:47 +05:30
f734455e5d
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
2020-08-20 15:25:10 -07:00
4813a37023
HDFS-15535. RBF: Fix Namespace path to snapshot path resolution for snapshot API. Contributed by Ayush Saxena.
2020-08-20 23:54:35 +05:30
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. ( #2229 ). Contributed by Uma Maheswara Rao G.
2020-08-19 09:30:41 -07:00
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS ( #2220 )
2020-08-18 19:28:19 +09:00
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. ( #2172 )
...
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.
* Addressed review comments.
* Rebased to latest trunk and added snapshotTrashRoot config check.
* Addressed review comments.
* Removede empty line added in SnapshotManager.Java.
* Addressed whitespace issues.
2020-08-17 12:56:13 +05:30
e3d1966f58
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
2020-08-15 15:07:58 +05:30
b93dd7c281
HDFS-15519. Check inaccessible INodes in FsImageValidation. ( #2224 )
2020-08-14 10:10:01 -07:00
15a76e8d65
HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. ( #2219 )
2020-08-14 20:46:48 +05:30
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots ( #2212 )
2020-08-13 22:36:15 +05:30
e592ec5f8b
HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. ( #2217 )
...
* HDFS-15518. Fixed String operationName = ListSnapshot.
* HDFS-15518. Changed ListSnapshot to operationName.
Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>
2020-08-13 00:09:38 +05:30
10716040a8
HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang
2020-08-12 09:02:47 +01:00
11cec9ab94
HDFS-15523. Fix findbugs warnings from HDFS-15520. ( #2218 )
2020-08-11 21:41:28 -07:00
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory ( #2176 )
2020-08-11 08:52:16 -07:00
909f1e82d3
HADOOP-17196. Fix C/C++ standard warnings ( #2208 )
...
* Passing C/C++ standard flags -std is
not cross-compiler friendly as not all
compilers support all values.
* Thus, we need to make use of the
appropriate flags provided by CMake in
order to specify the C/C++ standards.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-08-11 16:35:08 +09:00
8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false ( #2205 )
...
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false
* Used LambdaTestUtils#intercept in test
2020-08-11 00:01:58 -07:00
592127bdc2
HDFS-15520 Use visitor pattern to visit namespace tree ( #2203 )
2020-08-11 12:30:30 +05:30
32895f4f7e
HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.
2020-08-11 13:49:56 +09:00
7938ebfb9d
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
2020-08-11 11:14:58 +09:00
9062814bab
HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
2020-08-11 10:58:56 +09:00
c2a17659d1
HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.
2020-08-08 14:33:53 +05:30
975b6024dd
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:19:17 +05:30
1d5ccc790b
HDFS-15512. Remove smallBufferSize in DFSClient. ( #2191 )
2020-08-06 22:47:12 +05:30
58def7cecb
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion ( #2188 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-08-04 20:48:45 -07:00
e072d33327
HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable ( #2175 )
2020-08-04 14:10:29 +05:30
82f3ffcd64
HDFS-15503. File and directory permissions are not able to be modified from WebUI.
2020-08-03 23:21:01 +05:30
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors ( #2098 )
2020-08-03 10:46:51 +09:00
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. ( #2181 )
2020-08-02 21:59:56 +05:30
528a799a78
HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.
2020-08-02 12:02:37 +05:30
e756fe3590
HDFS-14950. fix missing libhdfspp lib in dist-package ( #1947 )
...
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
2020-07-31 16:49:49 +09:00
05b3337a46
HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots ( #2165 )
2020-07-30 23:06:51 +05:30
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. ( #2166 )
2020-07-29 21:33:25 +05:30
026dce5334
HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. ( #2135 )
2020-07-27 09:55:11 -07:00
2d12496643
HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr ( #2163 )
2020-07-22 16:16:27 -07:00
ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. ( #2160 ). Contributed by Uma Maheswara Rao G.
2020-07-21 23:29:10 -07:00
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 )
2020-07-22 13:40:20 +09:00
8b7695bb26
HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.
2020-07-21 09:00:07 -07:00
d57462f2da
HDFS-15479. Ordered snapshot deletion: make it a configurable feature ( #2156 )
2020-07-21 11:36:24 +05:30
d9441f95c3
HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.
2020-07-21 11:13:05 +05:30
3833c616e0
HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.
2020-07-20 12:49:58 -07:00
6cbd8854ee
HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.
2020-07-20 21:49:05 +05:30
2cec50cf16
HDFS-15463. Add a tool to validate FsImage. ( #2140 )
2020-07-20 11:44:30 +05:30
4101b0c0ed
HDFS-15476 Make AsyncStream executor private ( #2151 )
2020-07-19 23:26:54 +05:30
8a9a674ef1
HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu.
2020-07-18 15:36:55 +05:30
6bcb24d269
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-18 14:33:43 +05:30
1f71c4ae71
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 11:39:06 -05:00
98fcffe93f
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 09:53:18 -05:00
85d4718ed7
HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.
2020-07-15 10:32:25 +05:30
cce5a6f609
HDFS-15385 Upgrade boost library to 1.72 ( #2051 )
...
* Removed the asio-1.10.2 that was
checked in under libhdfspp/third_party
directory.
2020-07-15 13:39:48 +09:00
bdce75d737
HDFS-15371. Nonstandard characters exist in NameNode.java ( #2032 )
...
Contributed by zhaoyim
2020-07-14 14:42:12 +01:00
b9fa5e0182
HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext.
...
Contributed by Steve Loughran.
Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
2020-07-13 13:30:02 +01:00
b97fea65e7
HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-13 14:12:48 +08:00
84b74b335c
HDFS-15447 RBF: Add top real owners metrics for delegation tokens ( #2110 )
2020-07-12 00:10:12 -07:00
3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links ( #2132 ). Contributed by Uma Maheswara Rao G.
2020-07-11 23:50:04 -07:00
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. ( #2085 )
2020-07-09 11:33:37 -07:00
4f26454a7d
HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang.
2020-07-08 00:02:39 +08:00
f77bbc2123
HDFS-15312. Apply umask when creating directory by WebHDFS ( #2096 )
2020-07-06 19:01:46 -07:00
dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.
2020-07-06 18:50:03 -07:00
e820baa6e6
HDFS-15417. RBF: Get the datanode report from cache for federation WebHDFS operations ( #2080 )
2020-07-06 16:17:09 -07:00
834372f404
HDFS-15451. Do not discard non-initial block report for provided storage. ( #2119 ). Contributed by Shanyu Zhao.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-07-06 23:43:34 +08:00
55a2ae80dc
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.
2020-07-04 13:45:49 -07:00
f86f15cf20
HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.
2020-07-04 13:00:43 +05:30
1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-07-04 00:12:10 -07:00
9ac498e300
HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.
2020-07-01 12:38:07 +08:00
2a67e2b1a0
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-30 07:10:17 -07:00
c71ce7ac33
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
2020-06-28 16:02:47 +09:00
8db38c98a6
HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.
2020-06-27 19:13:01 +05:30
bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable ( #2100 )
...
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable
* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests
* Address Uma's comments on PR#2100
* Sort lists in test to match without concern to order
* Address comments, fix checkstyle and fix failing tests
* Fix checkstyle
2020-06-26 13:19:16 -07:00
d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-06-26 01:29:38 -07:00
84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch ( #2047 )
...
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
2020-06-23 13:12:29 -07:00
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
2020-06-23 01:42:25 -07:00
17ffcab5f6
HDFS-14546. Document block placement policies. Contributed by Amithsha.
2020-06-21 17:27:28 +05:30
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
2020-06-20 00:32:02 -07:00
d50e93ce7b
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-18 06:45:15 -07:00
edf716a5c3
Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell."
...
This reverts commit 730a39d138
2020-06-18 06:45:15 -07:00
123777823e
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
2020-06-18 12:26:22 +01:00
9cbd76cc77
HDFS-15346. FedBalance tool implementation. Contributed by Jinglun.
2020-06-18 13:33:25 +08:00
c8ed33cd2a
HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.
2020-06-17 13:55:40 +05:30
730a39d138
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-16 15:59:13 -07:00
81d8a887b0
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-06-15 10:56:16 +09:00
f41a144077
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
2020-06-15 09:15:53 +09:00
719b53a79d
HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.
2020-06-13 09:35:05 -07:00
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
2020-06-12 14:32:19 -07:00
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:16:33 +05:30
93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.
2020-06-10 15:00:02 -07:00
b735a77717
HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.
2020-06-10 12:06:16 +05:30
635e6a16d0
HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi.
2020-06-10 11:50:37 +05:30
8525874561
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:53:33 +05:30
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
2020-06-06 08:11:57 -07:00
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
2020-06-05 22:19:38 -07:00
76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
2020-06-05 10:58:21 -07:00
2326123705
HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.
2020-06-05 19:15:05 +05:30
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
2020-06-05 11:11:49 +01:00
e8cb2ae409
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.
2020-06-03 12:47:15 +05:30
Tsz-Wo Nicholas Sze
Stephen O'Donnell
Takanobu Asanuma
hemanthboyina
S O'Donnell
He Xiaoqiao
Chen Liang
Ayush Saxena
Uma Maheswara Rao G
Siyao Meng
bshashikant
Vivek Ratnavel Subramanian
Aryan Gupta
Gautham B A
Akira Ajisaka
Mingliang Liu
Yuan
touchida
Masatake Iwasaki
Inigo Goiri
Suraj Naik
Ahmed Hussein
zhaoyim
Steve Loughran
lfengnan
Xiaoyu Yao
Ye Ni
Shanyu Zhao
Virajith Jalaparti
Wei-Chiu Chuang
Yiqun Lin
Eric Yang
Vinayakumar B
Toshihiro Suzuki