Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,884 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8464 Commits

Author SHA1 Message Date
Inigo Goiri 4a212242d9 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 2019-06-27 10:00:05 -07:00
Inigo Goiri 041e0c0564 HDFS-14036. RBF: Add hdfs-rbf-default.xml to HdfsConfiguration by default. Contributed by Takanobu Asanuma. 2019-06-27 09:57:52 -07:00
Anu Engineer 041e7a7dee HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.
 2019-06-24 19:07:22 -07:00
Inigo Goiri b76b843c8b HDFS-13371. NPE for FsServerDefaults.getKeyProviderUri() for clientProtocol communication between 2.7 and 3.X. Contributed by Sherwood Zheng. 2019-06-24 17:52:33 -07:00
Inigo Goiri 38a560c6f1
Merge pull request #977 from leosunli/trunk 
HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
 2019-06-24 17:44:54 -07:00
Ajay Kumar 95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Inigo Goiri 719d57bf46 HDFS-14545. RBF: Router should support GetUserMappingsProtocol. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Akira Ajisaka 8627f6532b HDFS-14550. RBF: Failed to get statistics from NameNodes before 2.9.0. Contributed by He Xiaoqiao. 2019-06-24 22:03:04 +05:30
Ayush Saxena ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena ade8d3b60e HDFS-14508. RBF: Clean-up and refactor UI components. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena 6915d7e13c HDFS-13480. RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena ddbe08db33 HDFS-13955. RBF: Support secure Namenode in NamenodeHeartbeatService. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Inigo Goiri b062dd462d HDFS-14475. RBF: Expose router security enabled status on the UI. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena b6fff8c81e HDFS-13787. RBF: Add Snapshot related ClientProtocol APIs. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena d240eec136 HDFS-14516. RBF: Create hdfs-rbf-site.xml for RBF specific properties. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena 9b197c2893 HDFS-13909. RBF: Add Cache pools and directives related ClientProtocol APIs. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena ffbb6b6557 HDFS-13255. RBF: Fail when try to remove mount point paths. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena 8e4267650f HDFS-14440. RBF: Optimize the file write process in case of multiple destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula 2636a54ffd HDFS-13995. RBF: Security documentation. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 393f15176d HDFS-14447. RBF: Router should support RefreshUserMappingsProtocol. Contributed by Shen Yinjie. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 3deb5d345f HDFS-14490. RBF: Remove unnecessary quota checks. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena 62fa53a01d HDFS-14210. RBF: ACL commands should work over all the destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 32841178ba HDFS-14426. RBF: Add delegation token total count as one of the federation metrics. Contributed by Fengnan Li. 2019-06-24 22:03:04 +05:30
Ayush Saxena 203664e6b2 HDFS-14454. RBF: getContentSummary() should allow non-existing folders. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena b522b52bb1 HDFS-14457. RBF: Add order text SPACE in CLI command 'hdfs dfsrouteradmin'. Contributed by luhuachao. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula 021a43b1a4 HDFS-13972. RBF: Support for Delegation Token (WebHDFS). Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena de7da9b69e HDFS-14422. RBF: Router shouldn't allow READ operations in safe mode. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena 0f9b8d7a75 HDFS-14369. RBF: Fix trailing / for webhdfs. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena dd8c2b92df HDFS-13853. RBF: RouterAdmin update cmd is overwriting the entry not updating the existing. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena 6c42d40504 HDFS-14316. RBF: Support unavailable subclusters for mount points with multiple destinations. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena 0dbd87874a HDFS-14388. RBF: Prevent loading metric system when disabled. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Ayush Saxena 9a9fbbe145 HDFS-14351. RBF: Optimize configuration item resolving for monitor namenode. Contributed by He Xiaoqiao and Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola f539e2a4ee HDFS-14343. RBF: Fix renaming folders spread across multiple subclusters. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 2a2d5eb441 HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri fcabc8f0e4 HDFS-14335. RBF: Fix heartbeat typos in the Router. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri 6cdf8db55c HDFS-14331. RBF: IOE While Removing Mount Entry. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri 1ce25e702b HDFS-14329. RBF: Add maintenance nodes to federation metrics. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri 7bbe01a196 HDFS-14259. RBF: Fix safemode message for Router. Contributed by Ranith Sadar. 2019-06-24 22:03:03 +05:30
Inigo Goiri 1c7ab59be3 HDFS-14322. RBF: Security manager should not load if security is disabled. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula c4b1fa91fa HDFS-14052. RBF: Use Router keytab for WebHDFS. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri de719b08b5 HDFS-14307. RBF: Update tests to use internal Whitebox instead of Mockito. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 8b8ff5ccbc HDFS-14249. RBF: Tooling to identify the subcluster location of a file. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 50aee18a84 HDFS-14268. RBF: Fix the location of the DNs in getDatanodeReport(). Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri e2a3c4494b HDFS-14226. RBF: Setting attributes should set on all subclusters' directories. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula 75f8b6ccfa HDFS-13358. RBF: Support for Delegation Token (RPC). Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri 7e63e37dc5 HDFS-14230. RBF: Throw RetriableException instead of IOException when no namenodes available. Contributed by Fei Hui. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 912b90f91e HDFS-14252. RBF : Exceptions are exposing the actual sub cluster path. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore 9c4e55685b HDFS-14225. RBF : MiniRouterDFSCluster should configure the failover proxy provider for namespace. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Inigo Goiri 9eed3a49df HDFS-14215. RBF: Remove dependency on availability of default namespace. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula acdf911c01 HDFS-14224. RBF: NPE in getContentSummary() for getEcPolicy() in case of multiple destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula 8b9b58b58a HDFS-14223. RBF: Add configuration documents for using multiple sub-clusters. Contributed by Takanobu Asanuma. 2019-06-24 22:03:03 +05:30
Yiqun Lin 020f83f51f HDFS-14209. RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable. Contributed by Shubham Dewan. 2019-06-24 22:03:03 +05:30
Inigo Goiri 235406d904 HDFS-14156. RBF: rollEdit() command fails with Router. Contributed by Shubham Dewan. 2019-06-24 22:03:03 +05:30
Vinayakumar B c012b09fb6 HDFS-14193. RBF: Inconsistency with the Default Namespace. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore 7b61cbf672 HDFS-14129. addendum to HDFS-14129. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore f40e10b349 HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Yiqun Lin 221f24cbdc HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri f4e2bfce58 HDFS-13856. RBF: RouterAdmin should support dfsrouteradmin -refreshRouterArgs command. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Surendra Singh Lilhore b8bcbd0ed2 HDFS-14191. RBF: Remove hard coded router status from FederationMetrics. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Yiqun Lin 4244653e43 HDFS-14150. RBF: Quotas of the sub-cluster should be removed when removing the mount point. Contributed by Takanobu Asanuma. 2019-06-24 22:03:02 +05:30
Inigo Goiri f3cbf0eb9a HDFS-14161. RBF: Throw StandbyException instead of IOException so that client can retry when can not get connection. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Inigo Goiri 1dc01e59af HDFS-14167. RBF: Add stale nodes to federation metrics. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Yiqun Lin 8f6f9d9c83 HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad. 2019-06-24 22:03:02 +05:30
Takanobu Asanuma 3d97142dff HDFS-14151. RBF: Make the read-only column of Mount Table clearly understandable. 2019-06-24 22:03:02 +05:30
Yiqun Lin bbe859177d HDFS-13869. RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Takanobu Asanuma 01b4126b4e HDFS-14152. RBF: Fix a typo in RouterAdmin usage. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Yiqun Lin 94a8dec168 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Yiqun Lin 0ca7142c11 Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui." 
This reverts commit 7c0d6f65fde12ead91ed7c706521ad1d3dc995f8.
 2019-06-24 22:03:02 +05:30
Yiqun Lin 6aa7aabff8 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Surendra Singh Lilhore b320caecb3 HDFS-14085. RBF: LS command for root shows wrong owner and permission information. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula 19088e1b49 HDFS-14089. RBF: Failed to specify server's Kerberos pricipal name in NamenodeHeartbeatService. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula f2355c7063 HDFS-13776. RBF: Add Storage policies related ClientProtocol APIs. Contributed by Dibyendu Karmakar. 2019-06-24 22:03:02 +05:30
Yiqun Lin f4bd1114ff HDFS-14082. RBF: Add option to fail operations when a subcluster is unavailable. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri fa55eacd35 HDFS-13834. RBF: Connection creator thread should catch Throwable. Contributed by CR Hota. 2019-06-24 22:03:02 +05:30
Inigo Goiri 04caaba488 HDFS-13852. RBF: The DN_REPORT_TIME_OUT and DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula ebfd2d8a4e HDFS-12284. addendum to HDFS-12284. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula 6f2c871b05 HDFS-12284. RBF: Support for Kerberos authentication. Contributed by Sherwood Zheng and Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri 7b0bc492ed HDFS-14024. RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService. Contributed by CR Hota. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula c5065bf20b HDFS-13845. RBF: The default MountTableResolver should fail resolving multi-destination paths. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Yiqun Lin b3fee1d2bf HDFS-14011. RBF: Add more information to HdfsFileStatus for a mount point. Contributed by Akira Ajisaka. 2019-06-24 22:03:02 +05:30
Vinayakumar B 41c94a636b HDFS-13906. RBF: Add multiple paths for dfsrouteradmin 'rm' and 'clrquota' commands. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Wei-Chiu Chuang 05145404d5 HDFS-14339. Inconsistent log level practices in RpcProgramNfs3.java. Contributed by Anuhan Torgonshar. 2019-06-24 08:30:48 -07:00
sunlisheng daa1e14745 when evictableMmapped or evictable size is zero, do not throw NoSuchElementException 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-06-24 20:17:22 +08:00
Wei-Chiu Chuang 645d67bc4f HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 2019-06-21 18:27:03 -07:00
Wei-Chiu Chuang 1524e2e6c5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 2019-06-21 18:17:36 -07:00
Lokesh Jain 272b96d243 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:23:05 -07:00
Takanobu Asanuma 98d2065643 HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:17:24 -07:00
Surendra Singh Lilhore 5962a518bd HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 09:40:29 -07:00
Wei-Chiu Chuang 71ecd2e411 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-19 10:27:53 -07:00
David Mollitor f5ecc0bc08 HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-19 10:06:40 -07:00
Wei-Chiu Chuang d3ac516665 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit a1c3868c4f.
 2019-06-19 06:27:02 -07:00
Wei-Chiu Chuang 48e564f7e2 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-18 22:28:21 -07:00
Virajith Jalaparti a1c3868c4f HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 22:17:49 -07:00
Bharat Viswanadham f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya. 2019-06-18 21:58:37 -07:00
Elek, Marton 81ec909411 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 12:01:26 -07:00
Shweta Yakkali 3c1a1ceea9 HDFS-14487. Missing Space in Client Error Message (Contributed by Shweta Yakkali via Daniel Templeton) 
Change-Id: I0f8ce74a35ab24fe94fd0e57d8247bb3fa575e6f
 2019-06-18 10:21:22 -07:00
Inigo Goiri 3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Wei-Chiu Chuang 335c1c9938 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 2019-06-18 00:35:48 -07:00
Oleg Danilov 54cdde38c7 HDFS-12314. Typo in the TestDataNodeHotSwapVolumes.testAddOneNewVolume(). Contributed by Oleg Danilov. 2019-06-17 23:11:25 -07:00
Oleg Danilov d41310a15d HDFS-12315. Use Path instead of String to check closedFiles set. Contributed by Oleg Danilov. 2019-06-17 23:06:02 -07:00
Wei-Chiu Chuang 098c325a78 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 2019-06-17 22:45:52 -07:00
Weiwei Yang e3172221ea HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 21:48:16 -07:00
Wei-Chiu Chuang 62ad9885ea HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 2019-06-17 20:18:53 -07:00
Akira Ajisaka a95e87d827 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 17:03:25 -07:00
Santosh Marella 6822193ee6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:20:30 -07:00
Wei-Chiu Chuang 7314185c4a Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit ae4143a529.
 2019-06-17 16:18:48 -07:00
Wei-Chiu Chuang a50c35bb81 Revert "HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit cdc5de6448.
 2019-06-17 16:18:37 -07:00
Wei-Chiu Chuang f9a7b442fd HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-17 15:18:17 -07:00
mpicker90 cc1630288e HDFS-14556: Spelling Mistake "gloablly" (#938) 2019-06-16 18:32:42 -07:00
Adam Antal 8370a0ae16 HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-15 06:47:10 -07:00
Giovanni Matteo Fumarola b24efa11ea HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri. 2019-06-14 13:37:23 -07:00
Santosh Marella cdc5de6448 HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 13:01:39 -07:00
Santosh Marella ae4143a529 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 10:42:54 -07:00
dineshchitlangia 9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method 
Closes #967
 2019-06-14 16:09:37 +02:00
Inigo Goiri 4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Stephen O'Donnell 50de0874d0 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-12 10:30:23 -07:00
Eric Yang 4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Inigo Goiri 5740eea081 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-06-11 11:48:35 -07:00
Anu Engineer 101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
He Xiaoqiao bd46bdf9f9 HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 17:21:52 -07:00
Wei-Chiu Chuang 0d160a0ba8 HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed by Wei-Chiu Chuang, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-10 14:13:53 -07:00
Hanisha Koneru 7217494f40 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 13:45:37 -07:00
Shweta Yakkali d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Inigo Goiri e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri 944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang 294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
huzheng ea3b0a1844 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 2019-06-04 08:48:05 -07:00
Takanobu Asanuma 35f1014b3e HDFS-13654. Use a random secret when a secret file doesn't exist in HttpFS. This should be default. 2019-05-31 10:29:24 +09:00
Kihwal Lee 9122b9b649 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 2019-05-30 17:28:45 -05:00
He Xiaoqiao 33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali 6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Akira Ajisaka afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Eric Yang d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Akira Ajisaka 9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Rakesh Radhakrishnan 37900c5639 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 2019-05-26 14:30:11 +05:30
Erik Krogen 55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Konstantin V Shvachko c31b7b8d52 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:34:29 -07:00
Inigo Goiri c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Siyao Meng 9dff6eff81 HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-22 13:22:26 +09:00
Inigo Goiri 1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Srinivasu Majeti 3e5e5b028a HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-17 19:20:28 +02:00
Virajith Jalaparti 77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Sahil Takiar e2dfdaee7c HDFS-14482: Crash when using libhdfs with bad classpath 2019-05-14 11:04:06 -07:00
Inigo Goiri 389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Erik Krogen 5847e00143 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 2019-05-13 08:39:59 -07:00
Rakesh Radhakrishnan 9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore 69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Surendra Singh Lilhore e424392a62 HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi. 2019-05-05 13:38:09 +05:30
Chen Liang 1d59cc490c HADOOP-16292. Refactor checkTrustAndSend in SaslDataTransferClient to make it cleaner. Contributed by Sherwood Zheng. 2019-05-03 10:14:17 -07:00
Vinayakumar B f1875b205e HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena. 2019-05-03 11:22:14 +05:30
Shweta d6b7609c96 HDFS-14453. Improve Bad Sequence Number Error Message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-02 14:25:02 -07:00
Giovanni Matteo Fumarola 7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Inigo Goiri 865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Siyao Meng 7cb46f035a HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-01 16:48:44 -07:00
Sahil Takiar 4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00
Arpit Agarwal 4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 2019-04-29 14:49:35 -07:00
Sean Mackrory a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Erik Krogen 174b7d3126 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-04-22 13:29:56 -07:00
Inigo Goiri 5321235fe8 HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena. 2019-04-22 13:23:42 -07:00
lys0716 685cb83e4c HDFS-14433. Remove the extra empty space in the DataStreamer logging. Contributed by Yishuang Lu. (#747) 2019-04-17 10:38:48 -07:00
Puleya7 13907d8479 HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-04-17 21:39:06 +09:00
Inigo Goiri be6c8014e6 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 2019-04-16 10:34:31 -07:00
Sahil Takiar 2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer. 
Author:    Sahil Takiar <stakiar@cloudera.com>
 2019-04-12 18:12:02 -07:00
Chen Liang 626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Adam Antal 586826fe99
HDFS-12245. Fix INodeId javadoc 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-11 10:01:47 +09:00
Shweta e9b859f749 HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-05 16:09:00 -07:00
Inigo Goiri 260d843b25 HDFS-14416. Fix TestHdfsConfigFields for field dfs.client.failover.resolver.useFQDN. Contributed by Fengnan Li. 2019-04-05 09:26:06 -07:00
Akira Ajisaka 67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 2019-04-05 12:29:11 +09:00
Stephen O'Donnell d2637cb176 HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-04 04:07:18 -07:00
Inigo Goiri 7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Gabor Bota d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Wei-Chiu Chuang be488b6070 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-03 11:00:59 -07:00
Lokesh Jain cf268114c9 HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-02 12:24:55 -07:00
Gabor Bota 53a86e2b8e
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-01 13:49:37 +09:00
Uma Maheswara Rao G 35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Giovanni Matteo Fumarola 49b02d4a9b HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2019-03-28 11:16:01 -07:00
Inigo Goiri 15d38b1bf9 HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor. 2019-03-28 03:37:33 -07:00
Sahil Takiar 18c57cf046 HDFS-14304: High lock contention on hdfsHashMutex in libhdfs 
This closes #595

Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-03-26 20:32:51 -07:00
Sahil Takiar fe29b3901b
HDFS-14348: Fix JNI exception handling issues in libhdfs 
This closes #600

Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-03-26 16:33:34 -07:00
Takanobu Asanuma 55fb3c32fb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 2019-03-27 03:27:02 +09:00
Stephen O'Donnell 3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Ajay Kumar 399563fec6 HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia. 2019-03-20 11:20:45 -07:00
Erik Krogen 55b3a718e9 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-03-19 08:14:49 -07:00
Inigo Goiri ff06ef0631 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 2019-03-15 11:06:42 -07:00
Uma Maheswara Rao G ba50a36a3e HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 2019-03-14 22:21:08 -07:00
Erik Krogen 66357574ae HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 2019-03-13 13:15:56 -07:00
Stephen O'Donnell 34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen 1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Sahil Takiar 618e009ac0 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 15:04:06 -08:00
Wei-Chiu Chuang 6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri 945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Inigo Goiri 4b7313e640 HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker. 2019-03-04 23:03:32 -08:00
Sahil Takiar cb0fa0ce3f HDFS-14321. Fix -Xcheck:jni issues in libhdfs, run ctest with -Xcheck:jni enabled. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-04 12:37:57 -08:00
Wei-Chiu Chuang 387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00