Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
22,842 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8460 Commits

Author SHA1 Message Date
Inigo Goiri 38a560c6f1
Merge pull request #977 from leosunli/trunk 
HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
 2019-06-24 17:44:54 -07:00
Ajay Kumar 95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar. 2019-06-24 11:59:07 -07:00
Inigo Goiri 719d57bf46 HDFS-14545. RBF: Router should support GetUserMappingsProtocol. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Akira Ajisaka 8627f6532b HDFS-14550. RBF: Failed to get statistics from NameNodes before 2.9.0. Contributed by He Xiaoqiao. 2019-06-24 22:03:04 +05:30
Ayush Saxena ba221dc50e HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena ade8d3b60e HDFS-14508. RBF: Clean-up and refactor UI components. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena 6915d7e13c HDFS-13480. RBF: Separate namenodeHeartbeat and routerHeartbeat to different config key. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena ddbe08db33 HDFS-13955. RBF: Support secure Namenode in NamenodeHeartbeatService. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Inigo Goiri b062dd462d HDFS-14475. RBF: Expose router security enabled status on the UI. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena b6fff8c81e HDFS-13787. RBF: Add Snapshot related ClientProtocol APIs. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena d240eec136 HDFS-14516. RBF: Create hdfs-rbf-site.xml for RBF specific properties. Contributed by Takanobu Asanuma. 2019-06-24 22:03:04 +05:30
Ayush Saxena 9b197c2893 HDFS-13909. RBF: Add Cache pools and directives related ClientProtocol APIs. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena ffbb6b6557 HDFS-13255. RBF: Fail when try to remove mount point paths. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena 8e4267650f HDFS-14440. RBF: Optimize the file write process in case of multiple destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula 2636a54ffd HDFS-13995. RBF: Security documentation. Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 393f15176d HDFS-14447. RBF: Router should support RefreshUserMappingsProtocol. Contributed by Shen Yinjie. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 3deb5d345f HDFS-14490. RBF: Remove unnecessary quota checks. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena 62fa53a01d HDFS-14210. RBF: ACL commands should work over all the destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Giovanni Matteo Fumarola 32841178ba HDFS-14426. RBF: Add delegation token total count as one of the federation metrics. Contributed by Fengnan Li. 2019-06-24 22:03:04 +05:30
Ayush Saxena 203664e6b2 HDFS-14454. RBF: getContentSummary() should allow non-existing folders. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena b522b52bb1 HDFS-14457. RBF: Add order text SPACE in CLI command 'hdfs dfsrouteradmin'. Contributed by luhuachao. 2019-06-24 22:03:04 +05:30
Brahma Reddy Battula 021a43b1a4 HDFS-13972. RBF: Support for Delegation Token (WebHDFS). Contributed by CR Hota. 2019-06-24 22:03:04 +05:30
Ayush Saxena de7da9b69e HDFS-14422. RBF: Router shouldn't allow READ operations in safe mode. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena 0f9b8d7a75 HDFS-14369. RBF: Fix trailing / for webhdfs. Contributed by Akira Ajisaka. 2019-06-24 22:03:04 +05:30
Ayush Saxena dd8c2b92df HDFS-13853. RBF: RouterAdmin update cmd is overwriting the entry not updating the existing. Contributed by Ayush Saxena. 2019-06-24 22:03:04 +05:30
Ayush Saxena 6c42d40504 HDFS-14316. RBF: Support unavailable subclusters for mount points with multiple destinations. Contributed by Inigo Goiri. 2019-06-24 22:03:04 +05:30
Ayush Saxena 0dbd87874a HDFS-14388. RBF: Prevent loading metric system when disabled. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Ayush Saxena 9a9fbbe145 HDFS-14351. RBF: Optimize configuration item resolving for monitor namenode. Contributed by He Xiaoqiao and Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola f539e2a4ee HDFS-14343. RBF: Fix renaming folders spread across multiple subclusters. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 2a2d5eb441 HDFS-14334. RBF: Use human readable format for long numbers in the Router UI. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri fcabc8f0e4 HDFS-14335. RBF: Fix heartbeat typos in the Router. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri 6cdf8db55c HDFS-14331. RBF: IOE While Removing Mount Entry. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri 1ce25e702b HDFS-14329. RBF: Add maintenance nodes to federation metrics. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Inigo Goiri 7bbe01a196 HDFS-14259. RBF: Fix safemode message for Router. Contributed by Ranith Sadar. 2019-06-24 22:03:03 +05:30
Inigo Goiri 1c7ab59be3 HDFS-14322. RBF: Security manager should not load if security is disabled. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula c4b1fa91fa HDFS-14052. RBF: Use Router keytab for WebHDFS. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri de719b08b5 HDFS-14307. RBF: Update tests to use internal Whitebox instead of Mockito. Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 8b8ff5ccbc HDFS-14249. RBF: Tooling to identify the subcluster location of a file. Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 50aee18a84 HDFS-14268. RBF: Fix the location of the DNs in getDatanodeReport(). Contributed by Inigo Goiri. 2019-06-24 22:03:03 +05:30
Inigo Goiri e2a3c4494b HDFS-14226. RBF: Setting attributes should set on all subclusters' directories. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula 75f8b6ccfa HDFS-13358. RBF: Support for Delegation Token (RPC). Contributed by CR Hota. 2019-06-24 22:03:03 +05:30
Inigo Goiri 7e63e37dc5 HDFS-14230. RBF: Throw RetriableException instead of IOException when no namenodes available. Contributed by Fei Hui. 2019-06-24 22:03:03 +05:30
Giovanni Matteo Fumarola 912b90f91e HDFS-14252. RBF : Exceptions are exposing the actual sub cluster path. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore 9c4e55685b HDFS-14225. RBF : MiniRouterDFSCluster should configure the failover proxy provider for namespace. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Inigo Goiri 9eed3a49df HDFS-14215. RBF: Remove dependency on availability of default namespace. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula acdf911c01 HDFS-14224. RBF: NPE in getContentSummary() for getEcPolicy() in case of multiple destinations. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Brahma Reddy Battula 8b9b58b58a HDFS-14223. RBF: Add configuration documents for using multiple sub-clusters. Contributed by Takanobu Asanuma. 2019-06-24 22:03:03 +05:30
Yiqun Lin 020f83f51f HDFS-14209. RBF: setQuota() through router is working for only the mount Points under the Source column in MountTable. Contributed by Shubham Dewan. 2019-06-24 22:03:03 +05:30
Inigo Goiri 235406d904 HDFS-14156. RBF: rollEdit() command fails with Router. Contributed by Shubham Dewan. 2019-06-24 22:03:03 +05:30
Vinayakumar B c012b09fb6 HDFS-14193. RBF: Inconsistency with the Default Namespace. Contributed by Ayush Saxena. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore 7b61cbf672 HDFS-14129. addendum to HDFS-14129. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Surendra Singh Lilhore f40e10b349 HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar. 2019-06-24 22:03:03 +05:30
Yiqun Lin 221f24cbdc HDFS-14206. RBF: Cleanup quota modules. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri f4e2bfce58 HDFS-13856. RBF: RouterAdmin should support dfsrouteradmin -refreshRouterArgs command. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Surendra Singh Lilhore b8bcbd0ed2 HDFS-14191. RBF: Remove hard coded router status from FederationMetrics. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Yiqun Lin 4244653e43 HDFS-14150. RBF: Quotas of the sub-cluster should be removed when removing the mount point. Contributed by Takanobu Asanuma. 2019-06-24 22:03:02 +05:30
Inigo Goiri f3cbf0eb9a HDFS-14161. RBF: Throw StandbyException instead of IOException so that client can retry when can not get connection. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Inigo Goiri 1dc01e59af HDFS-14167. RBF: Add stale nodes to federation metrics. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Yiqun Lin 8f6f9d9c83 HDFS-13443. RBF: Update mount table cache immediately after changing (add/update/remove) mount table entries. Contributed by Mohammad Arshad. 2019-06-24 22:03:02 +05:30
Takanobu Asanuma 3d97142dff HDFS-14151. RBF: Make the read-only column of Mount Table clearly understandable. 2019-06-24 22:03:02 +05:30
Yiqun Lin bbe859177d HDFS-13869. RBF: Handle NPE for NamenodeBeanMetrics#getFederationMetrics. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Takanobu Asanuma 01b4126b4e HDFS-14152. RBF: Fix a typo in RouterAdmin usage. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Yiqun Lin 94a8dec168 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Yiqun Lin 0ca7142c11 Revert "HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui." 
This reverts commit 7c0d6f65fde12ead91ed7c706521ad1d3dc995f8.
 2019-06-24 22:03:02 +05:30
Yiqun Lin 6aa7aabff8 HDFS-14114. RBF: MIN_ACTIVE_RATIO should be configurable. Contributed by Fei Hui. 2019-06-24 22:03:02 +05:30
Surendra Singh Lilhore b320caecb3 HDFS-14085. RBF: LS command for root shows wrong owner and permission information. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula 19088e1b49 HDFS-14089. RBF: Failed to specify server's Kerberos pricipal name in NamenodeHeartbeatService. Contributed by Ranith Sardar. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula f2355c7063 HDFS-13776. RBF: Add Storage policies related ClientProtocol APIs. Contributed by Dibyendu Karmakar. 2019-06-24 22:03:02 +05:30
Yiqun Lin f4bd1114ff HDFS-14082. RBF: Add option to fail operations when a subcluster is unavailable. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri fa55eacd35 HDFS-13834. RBF: Connection creator thread should catch Throwable. Contributed by CR Hota. 2019-06-24 22:03:02 +05:30
Inigo Goiri 04caaba488 HDFS-13852. RBF: The DN_REPORT_TIME_OUT and DN_REPORT_CACHE_EXPIRE should be configured in RBFConfigKeys. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula ebfd2d8a4e HDFS-12284. addendum to HDFS-12284. Contributed by Inigo Goiri. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula 6f2c871b05 HDFS-12284. RBF: Support for Kerberos authentication. Contributed by Sherwood Zheng and Inigo Goiri. 2019-06-24 22:03:02 +05:30
Inigo Goiri 7b0bc492ed HDFS-14024. RBF: ProvidedCapacityTotal json exception in NamenodeHeartbeatService. Contributed by CR Hota. 2019-06-24 22:03:02 +05:30
Brahma Reddy Battula c5065bf20b HDFS-13845. RBF: The default MountTableResolver should fail resolving multi-destination paths. Contributed by yanghuafeng. 2019-06-24 22:03:02 +05:30
Yiqun Lin b3fee1d2bf HDFS-14011. RBF: Add more information to HdfsFileStatus for a mount point. Contributed by Akira Ajisaka. 2019-06-24 22:03:02 +05:30
Vinayakumar B 41c94a636b HDFS-13906. RBF: Add multiple paths for dfsrouteradmin 'rm' and 'clrquota' commands. Contributed by Ayush Saxena. 2019-06-24 22:03:02 +05:30
Wei-Chiu Chuang 05145404d5 HDFS-14339. Inconsistent log level practices in RpcProgramNfs3.java. Contributed by Anuhan Torgonshar. 2019-06-24 08:30:48 -07:00
sunlisheng daa1e14745 when evictableMmapped or evictable size is zero, do not throw NoSuchElementException 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
 2019-06-24 20:17:22 +08:00
Wei-Chiu Chuang 645d67bc4f HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 2019-06-21 18:27:03 -07:00
Wei-Chiu Chuang 1524e2e6c5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 2019-06-21 18:17:36 -07:00
Lokesh Jain 272b96d243 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:23:05 -07:00
Takanobu Asanuma 98d2065643 HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 20:17:24 -07:00
Surendra Singh Lilhore 5962a518bd HDFS-14581. Appending to EC files crashes NameNode. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-20 09:40:29 -07:00
Wei-Chiu Chuang 71ecd2e411 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-19 10:27:53 -07:00
David Mollitor f5ecc0bc08 HDFS-14103. Review Logging of BlockPlacementPolicyDefault. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-19 10:06:40 -07:00
Wei-Chiu Chuang d3ac516665 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit a1c3868c4f.
 2019-06-19 06:27:02 -07:00
Wei-Chiu Chuang 48e564f7e2 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-18 22:28:21 -07:00
Virajith Jalaparti a1c3868c4f HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 22:17:49 -07:00
Bharat Viswanadham f9ee97de98
HDFS-14398. Update HAState.java to fix typos. Contributed by Nikhil Navadiya. 2019-06-18 21:58:37 -07:00
Elek, Marton 81ec909411 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 12:01:26 -07:00
Shweta Yakkali 3c1a1ceea9 HDFS-14487. Missing Space in Client Error Message (Contributed by Shweta Yakkali via Daniel Templeton) 
Change-Id: I0f8ce74a35ab24fe94fd0e57d8247bb3fa575e6f
 2019-06-18 10:21:22 -07:00
Inigo Goiri 3ab77d9bc9 HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao. 2019-06-18 09:58:29 -07:00
Wei-Chiu Chuang 335c1c9938 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 2019-06-18 00:35:48 -07:00
Oleg Danilov 54cdde38c7 HDFS-12314. Typo in the TestDataNodeHotSwapVolumes.testAddOneNewVolume(). Contributed by Oleg Danilov. 2019-06-17 23:11:25 -07:00
Oleg Danilov d41310a15d HDFS-12315. Use Path instead of String to check closedFiles set. Contributed by Oleg Danilov. 2019-06-17 23:06:02 -07:00
Wei-Chiu Chuang 098c325a78 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 2019-06-17 22:45:52 -07:00
Weiwei Yang e3172221ea HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 21:48:16 -07:00
Wei-Chiu Chuang 62ad9885ea HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 2019-06-17 20:18:53 -07:00
Akira Ajisaka a95e87d827 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-17 17:03:25 -07:00
Santosh Marella 6822193ee6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:20:30 -07:00
Wei-Chiu Chuang 7314185c4a Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit ae4143a529.
 2019-06-17 16:18:48 -07:00
Wei-Chiu Chuang a50c35bb81 Revert "HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit cdc5de6448.
 2019-06-17 16:18:37 -07:00
Wei-Chiu Chuang f9a7b442fd HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-17 15:18:17 -07:00
mpicker90 cc1630288e HDFS-14556: Spelling Mistake "gloablly" (#938) 2019-06-16 18:32:42 -07:00
Adam Antal 8370a0ae16 HDFS-14203. Refactor OIV Delimited output entry building mechanism. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-15 06:47:10 -07:00
Giovanni Matteo Fumarola b24efa11ea HDFS-14549. EditLogTailer shouldn't output full stack trace when interrupted. Contributed by Inigo Goiri. 2019-06-14 13:37:23 -07:00
Santosh Marella cdc5de6448 HDFS-12914. Addendum patch. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 13:01:39 -07:00
Santosh Marella ae4143a529 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 10:42:54 -07:00
dineshchitlangia 9ebbda342f
HADOOP-16372. Fix typo in DFSUtil getHttpPolicy method 
Closes #967
 2019-06-14 16:09:37 +02:00
Inigo Goiri 4f455290b1 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:26:53 -07:00
Stephen O'Donnell 50de0874d0 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-12 10:30:23 -07:00
Eric Yang 4ea6c2f457 HADOOP-16354. Enable AuthFilter as default for WebHDFS. 
Contributed by Prabhu Joseph
 2019-06-11 18:41:08 -04:00
Inigo Goiri 5740eea081 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-06-11 11:48:35 -07:00
Anu Engineer 101d5b5f86 HDFS-14234. Limit WebHDFS to specifc user, host, directory triples. 
Contributed by Clay B.
 2019-06-10 17:55:16 -07:00
He Xiaoqiao bd46bdf9f9 HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 17:21:52 -07:00
Wei-Chiu Chuang 0d160a0ba8 HDFS-10210. Remove the defunct startKdc profile from hdfs. Contributed by Wei-Chiu Chuang, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2019-06-10 14:13:53 -07:00
Hanisha Koneru 7217494f40 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 13:45:37 -07:00
Shweta Yakkali d6d95d2686 HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-10 10:23:18 -07:00
Inigo Goiri e1dfc060f8 HDFS-14486. The exception classes in some throw statements do not accurately describe why they are thrown. Contributed by Ayush Saxena. 2019-06-06 11:59:53 -07:00
Inigo Goiri 944adc61b1 HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 2019-06-06 10:20:28 -07:00
Eric Yang 294695dd57 HADOOP-16314. Make sure all web end points are covered by the same authentication filter. 
Contributed by Prabhu Joseph
 2019-06-05 18:55:13 -04:00
Sammi Chen d1aad44490 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 2019-06-05 21:33:00 +08:00
huzheng ea3b0a1844 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 2019-06-04 08:48:05 -07:00
Takanobu Asanuma 35f1014b3e HDFS-13654. Use a random secret when a secret file doesn't exist in HttpFS. This should be default. 2019-05-31 10:29:24 +09:00
Kihwal Lee 9122b9b649 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 2019-05-30 17:28:45 -05:00
He Xiaoqiao 33c62f8f4e HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:30:35 -07:00
Shweta Yakkali 6f5a36c13c HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-30 13:21:58 -07:00
Ayush Saxena c1caab40f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-29 20:56:38 -07:00
Akira Ajisaka afd844059c HADOOP-16331. Fix ASF License check in pom.xml 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:25:13 +09:00
Eric Yang d78854b928 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG
 2019-05-28 17:31:35 -04:00
Akira Ajisaka 9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Rakesh Radhakrishnan 37900c5639 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 2019-05-26 14:30:11 +05:30
Erik Krogen 55e0c134f0 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 2019-05-24 13:12:59 -07:00
Konstantin V Shvachko c31b7b8d52 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:34:29 -07:00
Inigo Goiri c9393dd17f Revert "HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong." 
This reverts commit 1cb2eb0df3.
 2019-05-24 10:23:47 -07:00
Christopher Gregorian f96a2df38d HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 2019-05-23 10:28:37 -07:00
Siyao Meng 9dff6eff81 HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-22 13:22:26 +09:00
Inigo Goiri 1cb2eb0df3 HDFS-14353. Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong. 2019-05-20 17:22:58 -07:00
Srinivasu Majeti 3e5e5b028a HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-17 19:20:28 +02:00
Virajith Jalaparti 77170e70d1 HDFS-14390. Provide kerberos support for AliasMap service used by Provided storage. Contributed by Ashvin Agrawal 2019-05-15 12:15:47 -07:00
Sahil Takiar e2dfdaee7c HDFS-14482: Crash when using libhdfs with bad classpath 2019-05-14 11:04:06 -07:00
Inigo Goiri 389e640f0c HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 2019-05-13 11:46:16 -07:00
Erik Krogen 5847e00143 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 2019-05-13 08:39:59 -07:00
Rakesh Radhakrishnan 9b0aace1e6 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-05-08 17:20:21 +05:30
Surendra Singh Lilhore 69b903bbd8 HDFS-14372. NPE while DN is shutting down. Contributed by lujie. 2019-05-05 16:33:52 +05:30
Surendra Singh Lilhore e424392a62 HDFS-14438. Fix typo in OfflineEditsVisitorFactory. Contributed by bianqi. 2019-05-05 13:38:09 +05:30
Chen Liang 1d59cc490c HADOOP-16292. Refactor checkTrustAndSend in SaslDataTransferClient to make it cleaner. Contributed by Sherwood Zheng. 2019-05-03 10:14:17 -07:00
Vinayakumar B f1875b205e HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena. 2019-05-03 11:22:14 +05:30
Shweta d6b7609c96 HDFS-14453. Improve Bad Sequence Number Error Message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-02 14:25:02 -07:00
Giovanni Matteo Fumarola 7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Inigo Goiri 865c328930 HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 2019-05-02 10:09:21 -07:00
Siyao Meng 7cb46f035a HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-05-01 16:48:44 -07:00
Sahil Takiar 4877f0aa51 HDFS-3246: pRead equivalent for direct read path (#597) 
HDFS-3246: pRead equivalent for direct read path

Contributed by Sahil Takiar
 2019-04-30 14:52:16 -07:00
Arpit Agarwal 4b4200f1f8
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 2019-04-29 14:49:35 -07:00
Sean Mackrory a703dae25e HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota. 2019-04-24 10:39:00 -06:00
Erik Krogen 174b7d3126 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-04-22 13:29:56 -07:00
Inigo Goiri 5321235fe8 HDFS-14445. TestTrySendErrorReportWhenNNThrowsIOException fails in trunk. Contributed by Ayush Saxena. 2019-04-22 13:23:42 -07:00
lys0716 685cb83e4c HDFS-14433. Remove the extra empty space in the DataStreamer logging. Contributed by Yishuang Lu. (#747) 2019-04-17 10:38:48 -07:00
Puleya7 13907d8479 HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-04-17 21:39:06 +09:00
Inigo Goiri be6c8014e6 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 2019-04-16 10:34:31 -07:00
Sahil Takiar 2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer. 
Author:    Sahil Takiar <stakiar@cloudera.com>
 2019-04-12 18:12:02 -07:00
Chen Liang 626fec652b HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-04-12 17:37:51 -07:00
Adam Antal 586826fe99
HDFS-12245. Fix INodeId javadoc 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-11 10:01:47 +09:00
Shweta e9b859f749 HDFS-14371. Improve Logging in FSNamesystem by adding parameterized logging. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-05 16:09:00 -07:00
Inigo Goiri 260d843b25 HDFS-14416. Fix TestHdfsConfigFields for field dfs.client.failover.resolver.useFQDN. Contributed by Fengnan Li. 2019-04-05 09:26:06 -07:00
Akira Ajisaka 67020f0950
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 2019-04-05 12:29:11 +09:00
Stephen O'Donnell d2637cb176 HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-04 04:07:18 -07:00
Inigo Goiri 7b5b783f66 HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li. 2019-04-03 16:11:13 -07:00
Gabor Bota d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Wei-Chiu Chuang be488b6070 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-03 11:00:59 -07:00
Lokesh Jain cf268114c9 HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-02 12:24:55 -07:00
Gabor Bota 53a86e2b8e
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-01 13:49:37 +09:00
Uma Maheswara Rao G 35ff31dd94 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-03-30 23:33:25 -07:00
Rakesh Radhakrishnan f3f51284d5 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 2019-03-29 00:18:15 +05:30
Giovanni Matteo Fumarola 49b02d4a9b HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2019-03-28 11:16:01 -07:00
Inigo Goiri 15d38b1bf9 HDFS-14295. Add Threadpool for DataTransfers. Contributed by David Mollitor. 2019-03-28 03:37:33 -07:00
Sahil Takiar 18c57cf046 HDFS-14304: High lock contention on hdfsHashMutex in libhdfs 
This closes #595

Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-03-26 20:32:51 -07:00
Sahil Takiar fe29b3901b
HDFS-14348: Fix JNI exception handling issues in libhdfs 
This closes #600

Signed-off-by: Todd Lipcon <todd@apache.org>
 2019-03-26 16:33:34 -07:00
Takanobu Asanuma 55fb3c32fb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 2019-03-27 03:27:02 +09:00
Stephen O'Donnell 3f6d6d2811 HDFS-14359. Inherited ACL permissions masked when parent directory does not exist (mkdir -p) 
(Contributed by Stephen O'Donnell via Daniel Templeton)

Change-Id: Ia83f799a8f56aa8057a967b234f184683395fa41
 2019-03-25 16:27:04 -07:00
Ajay Kumar 399563fec6 HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia. 2019-03-20 11:20:45 -07:00
Erik Krogen 55b3a718e9 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-03-19 08:14:49 -07:00
Inigo Goiri ff06ef0631 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 2019-03-15 11:06:42 -07:00
Uma Maheswara Rao G ba50a36a3e HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 2019-03-14 22:21:08 -07:00
Erik Krogen 66357574ae HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 2019-03-13 13:15:56 -07:00
Stephen O'Donnell 34b14061b3 HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-12 10:17:25 -07:00
Erik Krogen 1bc282e0b3 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 2019-03-07 08:41:23 -08:00
Sahil Takiar 618e009ac0 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 15:04:06 -08:00
Wei-Chiu Chuang 6192c1fe3b Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar." 
This reverts commit f5a4b43a49.
 2019-03-06 15:02:18 -08:00
Sahil Takiar f5a4b43a49 HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-06 14:58:45 -08:00
Inigo Goiri 945b504c25 HDFS-14326. Add CorruptFilesCount to JMX. Contributed by Danny Becker. 2019-03-05 17:39:52 -08:00
Vinayakumar B f940ab242d HDFS-7663. Erasure Coding: Append on striped file. Contributed by Ayush Saxena. 2019-03-05 19:26:42 +05:30
Inigo Goiri 4b7313e640 HDFS-14336. Fix checkstyle for NameNodeMXBean. Contributed by Danny Becker. 2019-03-04 23:03:32 -08:00
Sahil Takiar cb0fa0ce3f HDFS-14321. Fix -Xcheck:jni issues in libhdfs, run ctest with -Xcheck:jni enabled. Contributed by Sahil Takiar. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-03-04 12:37:57 -08:00
Wei-Chiu Chuang 387dbe587a HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 2019-03-04 10:43:44 -08:00
Yiqun Lin 18ea0c1493 HDFS-14182. Datanode usage histogram is clicked to show ip list. Contributed by fengchuang. 2019-03-04 17:34:24 +08:00
Wei-Chiu Chuang 6c4d566955
Revert "HDFS-14261. Kerberize JournalNodeSyncer unit test. Contributed by Siyao Meng." 
This reverts commit 5c10630ad8.
 2019-03-04 16:59:20 +09:00
Erik Krogen fc17ba172b
HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-03-04 16:52:04 +09:00
Inigo Goiri 80b77deb42 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 2019-03-01 09:18:51 -08:00