Commit Graph

9031 Commits

Author SHA1 Message Date
Ayush Saxena ae13a5ccbe HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang. 2020-05-31 17:10:59 +05:30
Wanqiang Ji d9838f2d42
HADOOP-17055. Remove residual code of Ozone (#2039) 2020-05-29 16:49:18 +09:00
Ayush Saxena a838d871a7 HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He. 2020-05-28 20:52:58 +05:30
Ayush Saxena 9b38be43c6 HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 2020-05-28 13:00:04 +05:30
Inigo Goiri 2148a8fe64 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 2020-05-27 11:06:13 -07:00
Ayush Saxena 6c9f75cf16 HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena. 2020-05-26 16:26:57 +05:30
Ayush Saxena f43a152b97 HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun. 2020-05-24 18:03:41 +05:30
Ayush Saxena f4901d0778 HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun. 2020-05-24 17:59:16 +05:30
Inigo Goiri 6e04b00df1 HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun. 2020-05-23 09:58:19 -07:00
Ayush Saxena e0ae232f66 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 2020-05-23 18:38:32 +05:30
Ayush Saxena f99fcb26ab HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena. 2020-05-23 18:29:31 +05:30
Takanobu Asanuma 4d22d1c58f HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 2020-05-23 17:28:38 +09:00
Uma Maheswara Rao G 4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 2020-05-21 21:34:58 -07:00
leosunli be374faf42
HDFS-13639. SlotReleaser is not fast enough (#1885) 2020-05-21 13:21:17 -07:00
Yiqun Lin 1983eea62d HDFS-15340. RBF: Implement BalanceProcedureScheduler basic framework. Contributed by Jinglun. 2020-05-20 10:39:40 +08:00
Abhishek Das ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 2020-05-18 22:27:12 -07:00
Chen Liang 7bb902bc0d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 10:58:52 -07:00
Ayush Saxena c84e6beada HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena. 2020-05-18 22:24:34 +05:30
Wei-Chiu Chuang 2abcf7762a HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 2020-05-18 09:23:09 -07:00
Wei-Chiu Chuang 4525292d41 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)"
This reverts commit 86e6aa8eec.
2020-05-18 09:22:05 -07:00
Wei-Chiu Chuang 50caba1a92 HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 2020-05-18 08:40:38 -07:00
He Xiaoqiao a3f44dacc1 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-05-18 07:09:43 -07:00
pustota2009 86e6aa8eec
HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
2020-05-18 07:04:04 -07:00
Ayush Saxena a3809d2023 HDFS-15082. RBF: Check each component length of destination path when add/update mount entry. Contributed by Xiaoqiao He. 2020-05-17 19:45:34 +05:30
Ayush Saxena 6e416a83d1 HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 2020-05-17 03:06:27 +05:30
Ayush Saxena 178336f8a8 HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He. 2020-05-16 16:35:06 +05:30
Uma Maheswara Rao G ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 2020-05-14 17:29:35 -07:00
Surendra Singh Lilhore 1958cb7c2b YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:09:34 +05:30
Ayush Saxena 0918433b4d YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:36:20 +05:30
Xiaoyu Yao 3cacf1ce56
HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. (#2004) 2020-05-13 11:47:19 -07:00
Surendra Singh Lilhore 743c2e9071 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 2020-05-13 15:01:07 +05:30
Akira Ajisaka 8ffc356b1e
Revert "SPNEGO TLS verification"
This reverts commit ba66f3b454.
2020-05-13 17:14:14 +09:00
Ayush Saxena 936bf09c37 HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 2020-05-12 21:54:54 +05:30
Elixir Kook a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007)
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
Xiaoyu Yao 047d8879e7
HDFS-15345. RouterPermissionChecker#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. 2020-05-12 08:31:04 -07:00
S O'Donnell 29dddb8a14 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:07:51 +01:00
Takanobu Asanuma 928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. (#2008) 2020-05-12 09:04:03 -05:00
Ayush Saxena 8dad38c0be HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:34:26 +05:30
Ayush Saxena 0fe49036e5 HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu. 2020-05-12 13:11:31 +05:30
Ayush Saxena 4c53fb9ce1 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:48:34 +05:30
Ayush Saxena aab9e0b16e HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:43:30 +05:30
Ayush Saxena cb64e993c2 HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi. 2020-05-10 00:11:46 +05:30
Ayush Saxena d59de27c01 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:53:49 +05:30
Inigo Goiri 130f89e068 HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 2020-05-06 13:18:32 -07:00
Ayush Saxena 7fddf4855e HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 16:55:04 +05:30
Wei-Chiu Chuang adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998)
Reviewed-by: Arpit Agarwal <arp@apache.org>
2020-05-05 16:04:54 -07:00
Uma Maheswara Rao G 9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 2020-05-04 17:55:40 -07:00
Babneet Singh 1996351b0b
HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951)
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
2020-05-04 13:08:53 -07:00
Konstantin V Shvachko ebb878bab9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 2020-05-04 10:29:50 -07:00
Ayush Saxena 8dace8ff3a HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi. 2020-05-03 23:59:17 +05:30
Ayush Saxena d6fc482a54 Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:09:14 +05:30
Mingliang Liu 44de193bec
HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li 2020-05-02 20:56:29 -07:00
Akira Ajisaka f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-02 13:02:27 -07:00
Eric Yang ba66f3b454
SPNEGO TLS verification
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-05-02 21:50:13 +09:00
Shashikant Banerjee 82343790ee HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-05-01 12:16:02 +05:30
Mingliang Liu 263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2020-04-30 14:15:28 -07:00
Akira Ajisaka b5b45c53a4
HDFS-15309. Remove redundant String.valueOf method on ExtendedBlockId.java. Contributed by bianqi. 2020-04-30 15:46:46 +09:00
Inigo Goiri 31b2f687ef HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina. 2020-04-29 13:41:36 -07:00
S O'Donnell 9ca6298a9a HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 2020-04-29 15:56:06 +01:00
Surendra Singh Lilhore db6252b6c3 HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 2020-04-29 10:58:35 +05:30
Aryan Gupta 816042e62b
HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 2020-04-29 09:19:59 +05:30
Akira Ajisaka 5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 2020-04-28 06:17:14 +09:00
Toshihiro Suzuki 62c26b91fd
HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 (#1979) 2020-04-27 17:20:47 +09:00
Mingliang Liu 74d4a18735
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:43:02 -07:00
Dhiraj 453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964)
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:04:32 -07:00
Wei-Chiu Chuang 453771f170 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 2020-04-24 11:53:24 -07:00
Mingliang Liu 5958af4dbc
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 13:31:02 -07:00
Akira Ajisaka e069a06137
HDFS-14742. RBF: TestRouterFaultTolerant tests are flaky 2020-04-20 08:41:18 -07:00
Takanobu Asanuma 1fdfaebd98 HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina. 2020-04-20 23:09:33 +09:00
Toshihiro Suzuki 1824aee9da
HDFS-15217 Add more information to longest write/read lock held log 2020-04-18 13:52:07 -07:00
Surendra Singh Lilhore 8e6227441a HDFS-15218. RBF: MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by Surendra Singh Lilhore. 2020-04-18 20:07:21 +05:30
Ayush Saxena 37d6582223 HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 2020-04-17 13:32:02 +05:30
Stephen O'Donnell 3481895f8a HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-04-16 20:18:25 -07:00
Ayush Saxena cc5c1da7c1 HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang. 2020-04-15 23:01:58 +05:30
Ayush Saxena 8d49229c37 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-04-12 12:05:50 +05:30
Ayush Saxena 275c478330 HDFS-15261. RBF: Add Block Related Metrics. Contributed by Ayush Saxena. 2020-04-09 23:44:36 +05:30
Wei-Chiu Chuang 061afcdf30
HDFS-15269. NameNode should check the authorization API version only … (#1945)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-04-09 09:19:35 -07:00
Ayush Saxena 1189af4746 HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena. 2020-04-08 15:55:14 +05:30
Toshihiro Suzuki c12ddbd1de
HDFS-15249 ThrottledAsyncChecker is not thread-safe. (#1922) 2020-04-07 14:38:09 +09:00
Inigo Goiri 0b855b9f35 HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun. 2020-04-06 13:05:33 -07:00
Akira Ajisaka 1695d8d59c
HDFS-15258. RBF: Mark Router FSCK unstable. (#1934) 2020-04-04 00:13:41 +09:00
Takanobu Asanuma 736659e0e1 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 2020-04-02 19:21:02 +09:00
Ayush Saxena 4a3eb10972 HDFS-15051. RBF: Impose directory level permissions for Mount entries. Contributed by Xiaoqiao He. 2020-04-02 11:06:01 +05:30
He Xiaoqiao d3b5951572 HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-04-01 16:36:46 -07:00
Inigo Goiri 80b877a72f HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 2020-03-30 12:29:21 -07:00
Ayush Saxena 960c9ebaea HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 2020-03-30 02:40:25 +05:30
Brahma Reddy Battula 8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
Inigo Goiri f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena 696a663cd0 HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Chen Liang f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00
Ayush Saxena cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Inigo Goiri 679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
Wei-Chiu Chuang 0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909) 2020-03-25 16:03:26 -07:00
Inigo Goiri cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Arpit Agarwal a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle.
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
2020-03-25 10:28:30 -07:00
Toshihiro Suzuki d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Isa Hekmatizadeh d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 2020-03-24 23:17:22 +05:30
Ayush Saxena ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Karthik Palanisamy 5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
He Xiaoqiao 5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 22:41:31 -07:00
Akira Ajisaka f59f6891c8
HDFS-15232. Fix libhdfspp test failures with GCC 7. (#1906) 2020-03-24 11:36:14 +09:00
He Xiaoqiao af64ce2f4a HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 12:43:51 -07:00
Ayush Saxena 62c8015683 HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena. 2020-03-22 11:44:18 +05:30
Takanobu Asanuma b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
Ayush Saxena f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Ayush Saxena 654db35fa2 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-03-18 21:00:45 +05:30
Wei-Chiu Chuang 096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
2020-03-18 13:27:13 +00:00
Surendra Singh Lilhore 1d772dc542 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:44:32 +05:30
Ayush Saxena 74780c22eb HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:30:39 +05:30
Ayush Saxena 8a7c54995a HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina. 2020-03-15 16:16:27 +05:30
Wei-Chiu Chuang 7b637a3341 HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun. 2020-03-13 19:01:23 -07:00
He Xiaoqiao e9955bb8ff HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-03-13 18:55:06 -07:00
Wei-Chiu Chuang 4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... (#1829)
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
2020-03-13 11:29:44 -07:00
Ayush Saxena fb1d7635ae HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang. 2020-03-13 21:10:03 +05:30
S O'Donnell 25497799cb HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:24:19 +00:00
Wei-Chiu Chuang 20903f72b4 HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun. 2020-03-12 19:23:12 -07:00
Inigo Goiri 0695f7a538 HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang. 2020-03-12 17:22:11 -07:00
Surendra Singh Lilhore f736408a83 HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree. 2020-03-12 18:59:17 +05:30
Steve Loughran c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
Contributed by Steve Loughran.

This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)

This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.

instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.

The FileSystem implementation is cut; tests updated as appropriate.

All new interfaces/classes/constants are marked as @unstable.

Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
Steve Loughran d4d4c37810
HADOOP-14630 Contract Tests to verify create, mkdirs and rename under a file is forbidden
Contributed by Steve Loughran.

Not all stores do complete validation here; in particular the S3A
Connector does not: checking up the entire directory tree to see if a path matches
is a file significantly slows things down.

This check does take place in S3A mkdirs(), which walks backwards up the list of
parent paths until it finds a directory (success) or a file (failure).
In practice production applications invariably create destination directories
before writing 1+ file into them -restricting check purely to the mkdirs()
call deliver significant speed up while implicitly including the checks.

Change-Id: I2c9df748e92b5655232e7d888d896f1868806eb0
2020-03-09 14:44:28 +00:00
Inigo Goiri 4062217189 HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena. 2020-03-06 10:14:53 -08:00
Inigo Goiri 3afd4cbe89 HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina. 2020-03-04 10:13:23 -08:00
Inigo Goiri 1d4d0fcbe1 HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein. 2020-03-04 10:02:54 -08:00
Akshay Nehe bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. (#1871) 2020-03-04 11:07:31 +09:00
Xiaoyu Yao 0dd8956f2e
HADOOP-16885. Encryption zone file copy failure leaks a temp file
Contributed by Xiaoyu Yao.

Contains HDFS-14892. Close the output stream if createWrappedOutputStream() fails

Copying file through the FsShell command into an HDFS encryption zone where
the caller lacks permissions is leaks a temp ._COPYING file
and potentially a wrapped stream unclosed.

This is a convergence of a fix for S3 meeting an issue in HDFS.

S3: a HEAD against a file can cache a 404, 
 -you must not do any existence checks, including deleteOnExit(),
  until the file is written. 

Hence: HADOOP-16490, only register files for deletion the create worked
and the upload is not direct. 

HDFS-14892. HDFS doesn't close wrapped streams when IOEs are raised on
create() failures. Which means that an entry is retained on the NN.
-you need to register a file with deleteOnExit() even if the file wasn't
created.

This patch:

* Moves the deleteOnExit to ensure the created file get deleted cleanly.
* Fixes HDFS to close the wrapped stream on failures.
2020-03-02 13:22:00 +00:00
Ayush Saxena 1a636da041 HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun. 2020-02-29 09:55:29 +05:30
Inigo Goiri 97b797c314 HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun. 2020-02-28 18:47:22 -08:00
Takanobu Asanuma 9a3a28018a HDFS-15190. HttpFS: Add Support for Storage Policy Satisfier. Contributed by hemanthboyina. 2020-02-29 10:30:26 +09:00
Surendra Singh Lilhore fccfb02672 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:33:06 +05:30
Ayush Saxena 429da635ec HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong. 2020-02-28 00:31:55 +05:30
Ayush Saxena cd2c6b1aac HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest. 2020-02-27 22:19:35 +05:30
Istvan Fajth 57aa048516
HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 2020-02-27 07:48:14 -08:00
Surendra Singh Lilhore 2059f255d3 HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena. 2020-02-27 14:08:42 +05:30
Ayush Saxena 033a3d7ff9 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 2020-02-27 04:02:26 +05:30
Ayush Saxena 209630472a HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun. 2020-02-27 02:22:24 +05:30
Kihwal Lee 352a4ec16d HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein. 2020-02-26 09:33:29 -06:00
Stephen O'Donnell 900430b990 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-25 13:28:50 -08:00
Wei-Chiu Chuang 1c5d2f1fdc HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 2020-02-24 13:39:16 -08:00
Ayush Saxena 7f8685f476 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-02-24 20:38:04 +05:30
Ayush Saxena 93b8f453b9 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:01 +05:30
Ayush Saxena b5698e0c33 HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun. 2020-02-24 00:07:18 +05:30
Ayush Saxena 9eb7a8bdf8 HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi. 2020-02-23 14:25:39 +05:30
Ayush Saxena ba9025c7cd HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun. 2020-02-22 14:27:26 +05:30
Konstantin V Shvachko ecbcb058b8 HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 2020-02-21 17:53:37 -08:00
Inigo Goiri ed70c115a8 HDFS-15172. Remove unnecessary deadNodeDetectInterval in DeadNodeDetector#checkDeadNodes(). Contributed by Lisheng Sun. 2020-02-21 16:36:30 -08:00
Konstantin V Shvachko 6f84269bcd HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 2020-02-21 10:51:14 -08:00
Masatake Iwasaki 2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 2020-02-21 11:56:07 +09:00
Inigo Goiri ec7507162c HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham. 2020-02-19 11:33:58 -08:00
Ayush Saxena ac4b556e2d HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena. 2020-02-19 08:20:59 +05:30
Ayush Saxena 84f7638840 HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri. 2020-02-18 00:43:33 +05:30
xiaoxiaopan118 439d935e1d
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 2020-02-17 15:55:10 +09:00
Surendra Singh Lilhore 810783d443 HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree. 2020-02-16 12:15:03 +05:30
Ayush Saxena c75756fe13 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:07:24 +05:30
Surendra Singh Lilhore a98352ced1 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 16:57:41 +05:30
Akira Ajisaka 0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router (#1832)
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
2020-02-13 10:06:07 +09:00
Ayush Saxena f09710bbb8 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:29:35 +05:30
Ayush Saxena 3df0adaaea HDFS-15127. RBF: Do not allow writes when a subcluster is unavailable for HASH_ALL mount points. Contributed by Inigo Goiri 2020-02-12 19:41:04 +05:30
Kihwal Lee 9b8a78d97b HDFS-14758. Make lease hard limit configurable and reduce the default.
Contributed by hemanthboyina.
2020-02-11 12:40:00 -06:00
Stephen O'Donnell d7c136b9ed HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-11 08:00:15 -08:00
Ayush Saxena 6191d4b4a0 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 2020-02-09 23:32:22 +05:30
Ayush Saxena d23317b102 HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-08 10:33:57 +05:30
Vinayakumar B 7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Chen Liang ce7b8b5634 HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 2020-02-04 12:12:35 -08:00
Kihwal Lee 10a60fbe20 HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 2020-02-04 12:22:35 -06:00
Stephen O'Donnell 1e3a0b0d93 HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2020-01-31 16:13:02 -08:00
Kihwal Lee 799d4c1cf4 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails
intermittently. Contributed by Ahmed Hussein.
2020-01-29 11:00:27 -06:00
Chen Liang 483397c7f7 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-28 15:20:36 -08:00
Chen Liang 3e86807802 Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang."
This reverts commit ff8ff0f7e5.
2020-01-28 15:19:47 -08:00
Chen Liang ff8ff0f7e5 [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang. 2020-01-28 15:14:58 -08:00
Inigo Goiri 1839c467f6 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 2020-01-28 10:10:35 -08:00
Inigo Goiri 061421fc6d HDFS-15145. HttpFS: getAclStatus() returns permission as null. Contributed by hemanthboyina. 2020-01-28 10:04:38 -08:00
Ayush Saxena 87c198468b HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:09:38 +05:30
Ayush Saxena f876dc228b HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena. 2020-01-28 19:34:21 +05:30
Ayush Saxena 6d008c0d39 HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest 2020-01-25 11:47:29 +05:30
Kihwal Lee d10f77e3c9 HDFS-15119. Allow expiration of cached locations in DFSInputStream.
Contributed by Ahmed Hussein.
2020-01-24 09:15:27 -06:00
Ayush Saxena 92c58901d7 HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena 2020-01-23 18:18:34 +05:30
Kihwal Lee bd03053ea2 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 2020-01-22 09:14:38 -06:00
Inigo Goiri 8cfc3673dc HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 2020-01-21 13:29:20 -08:00
Inigo Goiri b657822b98 HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein. 2020-01-21 13:22:53 -08:00
Akira Ajisaka f6d20daf40
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 2020-01-21 18:03:24 +09:00
Akira Ajisaka 1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 2020-01-21 10:58:32 +09:00
Inigo Goiri 263413e838 HDFS-15112. RBF: Do not return FileNotFoundException when a subcluster is unavailable. 2020-01-16 10:39:14 -08:00
Vinayakumar B edbbc03ce7
HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. (#1803) 2020-01-16 23:27:50 +05:30
Chao Sun d7c4f8ab21 HDFS-13616. Batch listing of multiple directories (#1725) 2020-01-15 17:22:47 -08:00
Wei-Chiu Chuang 6b86a5110e HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 2020-01-13 10:48:53 -08:00
Vinayakumar B 52b360a928
HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. (#1801) 2020-01-13 10:40:29 +05:30
Ayush Saxena cebce0a348 HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:29:11 +05:30
Ayush Saxena 7b62409ace HDFS-15108. RBF: MembershipNamenodeResolver should invalidate cache incase of active namenode update. Contributed by Ayush Saxena. 2020-01-11 10:21:31 +05:30
Chen Liang 26a969ec73 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 2020-01-10 17:06:59 -08:00
Kihwal Lee 5fb901ac40 HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 2020-01-10 12:43:21 -06:00
Takanobu Asanuma 9da294a140 HDFS-15110. HttpFS: post requests are not supported for path "/". Contributed by hemanthboyina. 2020-01-10 17:53:19 +09:00
Akira Ajisaka 0315ef8448 HDFS-15100. RBF: Print stacktrace when DFSRouter fails to fetch/parse JMX output from NameNode. (#1800) 2020-01-10 13:16:57 +09:00
Ayush Saxena b32757c616 HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena. 2020-01-10 08:14:56 +05:30
Takanobu Asanuma 782c0556fb HDFS-15102. HttpFS: put requests are not supported for path "/". Contributed by hemanthboyina. 2020-01-10 09:52:13 +09:00
Surendra Singh Lilhore bf45f3b80a HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:04:05 +05:30
Ayush Saxena 8fe01db34a HDFS-15094. RBF: Reuse ugi string in ConnectionPoolID. Contributed by Ayush Saxena. 2020-01-09 09:02:38 +05:30
Ayush Saxena fd30f4c52b HDFS-15096. RBF: GetServerDefaults Should be Cached At Router. Contributed by Ayush Saxena. 2020-01-09 08:26:51 +05:30
Rakesh Radhakrishnan 7030722e5d HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 2020-01-08 14:25:17 +05:30
Masatake Iwasaki aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 2020-01-08 16:45:39 +09:00
Masatake Iwasaki a43c177f1d HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 2020-01-08 11:28:34 +09:00
Takanobu Asanuma 59aac00283 HDFS-15066. HttpFS: Implement setErasureCodingPolicy , unsetErasureCodingPolicy , getErasureCodingPolicy. Contributed by hemanthboyina. 2020-01-07 11:10:32 +09:00
Takanobu Asanuma 4a76ab777f HDFS-15090. RBF: MountPoint Listing Should Return Flag Values Of Destination. Contributed by Ayush Saxena. 2020-01-06 18:09:59 +09:00
luhuachao 77ae7b9ce2 HDFS-15089. RBF: SmallFix for RBFMetrics in doc (#1786) 2020-01-06 12:31:13 +09:00
Ayush Saxena f8644fbe9f HDFS-15091. Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock. Contributed by Ayush Saxena. 2020-01-04 19:02:59 +05:30
Masatake Iwasaki 037ec8cfb1 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2020-01-04 01:55:36 +09:00
Rakesh Radhakrishnan d79cce20ab HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 2020-01-02 11:44:00 +05:30
Takanobu Asanuma 074050ca59 HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina. 2020-01-01 11:26:38 +09:00
Ayush Saxena 62423910a4 HDFS-14937. [SBN read] ObserverReadProxyProvider should throw InterruptException. Contributed by xuzq. 2019-12-29 13:07:22 +05:30
Surendra Singh Lilhore ee51eadda0 HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:15:54 +05:30
Takanobu Asanuma dc32f583af HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 2019-12-28 21:32:15 +09:00
Ayush Saxena 8730a7bf60 HDFS-15003. RBF: Make Router support storage type quota. Contributed by Jinglun. 2019-12-27 09:22:58 +05:30
Ayush Saxena 80f91d14ab HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 2019-12-27 09:12:44 +05:30
Ayush Saxena 300505c562 HDFS-15054. Delete Snapshot not updating new modification time. Contributed by hemanthboyina. 2019-12-25 12:42:50 +05:30
Ayush Saxena df622cf4a3 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 2019-12-25 11:07:25 +05:30
Chandra Sanivarapu d8cd7098b4 HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 2019-12-25 10:50:01 +09:00
Konstantin V Shvachko b98ac2a3af HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 2019-12-24 12:08:34 -08:00
Inigo Goiri 52d7b745c6 HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 2019-12-19 09:42:17 -08:00
Inigo Goiri b86895485d HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 2019-12-19 09:34:43 -08:00
Shashikant Banerjee fdd96e46d1 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 2019-12-18 22:50:46 +05:30
Inigo Goiri 24080666e5 HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun. 2019-12-16 18:41:45 -08:00
Masatake Iwasaki dc6cf17b34 HDFS-15048. Fix findbug in DirectoryScanner. 2019-12-16 19:00:57 +09:00
Ayush Saxena 7a87007545 HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina. 2019-12-15 00:33:32 +05:30
Ayush Saxena 72aee114f8 HDFS-15053. RBF: Add permission check for safemode operation. Contributed by Xiaoqiao He. 2019-12-14 17:57:23 +05:30
Inigo Goiri 7fe924b1c0 HDFS-15016. RBF: getDatanodeReport() should return the latest update. Contributed by Inigo Goiri. 2019-12-13 10:51:14 -08:00
Chen Liang 65c4660bcd HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 2019-12-12 10:22:05 -08:00
Masatake Iwasaki bdd00f10b4
HDFS-15047. Document the new decommission monitor (HDFS-14854). (#1755) 2019-12-13 00:35:41 +09:00
He Xiaoqiao 0e28cd8f63 HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-12 02:23:32 -08:00
Takanobu Asanuma 93bb368094 HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li 2019-12-12 10:56:19 +09:00
Erik Krogen c174d50b30 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 2019-12-11 09:39:37 -08:00
Surendra Singh Lilhore c2e9783d5f HDFS-15045. DataStreamer#createBlockOutputStream() should log exception in warn. Contributed by Ravuri Sushma Sree. 2019-12-11 08:52:55 +05:30
Stephen O'Donnell c93cb6790e HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell.
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-10 17:16:43 -08:00
Takanobu Asanuma c4733377d0
HDFS-15040. RBF: Secured Router should not run when SecretManager is not running. (#1745) 2019-12-10 16:03:32 +09:00
Akira Ajisaka 9f09852051
HDFS-15043. RBF: The detail of the Exception is not shown in ZKDelegationTokenSecretManagerImpl (#1748) 2019-12-10 16:00:20 +09:00
Masatake Iwasaki 4dffd81bb7
HDFS-14522. Allow compact property description in xml in httpfs. (#1737) 2019-12-10 10:30:57 +09:00
Masatake Iwasaki 11cd5b6e39 HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-12-07 08:46:45 +09:00
Wei-Chiu Chuang ecd461f940 HDFS-14751. Synchronize on diffs in DirectoryScanner. Contributed by Lisheng Sun. 2019-12-06 15:10:40 -08:00
Wei-Chiu Chuang 313b76f8e9 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-12-06 15:10:40 -08:00
Ayush Saxena 705b172b95 HDFS-14998. [SBN read] Update Observer Namenode doc for ZKFC after HDFS-14130. Contributed by Fei Hui. 2019-12-06 23:04:09 +05:30
Ayush Saxena 83a14559e5 HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui. 2019-12-05 18:52:01 +05:30
Yiqun Lin 47fdae7904 HDFS-13811. RBF: Race condition between router admin quota update and periodic quota update service. Contributed by Jinglun. 2019-12-04 18:19:11 +08:00
Ayush Saxena 0c217feed8 HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun. 2019-12-03 09:18:09 +05:30
Takanobu Asanuma 4ede8bce28 HDFS-9695. HTTPFS - CHECKACCESS operation missing. Contributed by hemanthboyina 2019-12-03 09:38:36 +09:00
Ayush Saxena 6b2d6d4aaf HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina. 2019-11-29 23:55:37 +05:30
Surendra Singh Lilhore 44f7b9159d HDFS-15013. Reduce NameNode overview tab response time. Contributed by HuangTao. 2019-11-29 00:42:31 +05:30
Surendra Singh Lilhore 0384687811 HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-28 23:49:35 +05:30
Ayush Saxena 46166bd8d1 HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena. 2019-11-28 18:28:20 +05:30
Yiqun Lin c3659f8f94 HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun. 2019-11-28 14:41:49 +08:00
Yiqun Lin 2b452b4e60 HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 2019-11-28 10:43:35 +08:00
Yiqun Lin c8bef4d6a6 HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun. 2019-11-27 10:57:20 +08:00