Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,975 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5070 Commits

Author SHA1 Message Date
Chao Sun 305b9a652a Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 
(cherry picked from commit a2a8c48699)
 2018-08-10 16:17:04 -07:00
Wei-Chiu Chuang 5ef8fd5119 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 
(cherry picked from commit e95c5e9f62)
(cherry picked from commit 964f3454d1)
(cherry picked from commit 0af04ad89f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 2018-07-25 11:26:19 -07:00
Yiqun Lin e6fcaccf5a HDFS-13583. RBF: Router admin clrQuota is not synchronized with nameservice. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 17a87977f2)
 2018-07-24 11:21:37 +08:00
Yiqun Lin 8a68a1590f HDFS-13743. RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver. Contributed by Takanobu Asanuma. 
(cherry picked from commit 7b25fb949b)
 2018-07-20 17:33:31 +08:00
Yiqun Lin 4c2ab22130 HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF document. Contributed by Takanobu Asanuma. 
(cherry picked from commit 1af87df242)
 2018-07-17 10:49:50 +08:00
Konstantin V Shvachko fccb3c02f0 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal. 
(cherry picked from commit 0a1e922f3d)
 2018-07-16 18:44:01 -07:00
Inigo Goiri 436fceebf9 HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed by Chao Sun. 
(cherry picked from commit 359ea4e181)
 2018-07-16 09:47:57 -07:00
Akira Ajisaka ff46db2ea6
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota. 2018-07-12 13:53:08 -04:00
Yiqun Lin fbe7192815 HDFS-13726. RBF: Fix RBF configuration links. Contributed by Takanobu Asanuma. 
(cherry picked from commit 2ae13d41dc)
 2018-07-11 22:17:43 +08:00
Andrew Wang 913417bbea HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 2018-07-09 15:17:26 +02:00
Yiqun Lin 512d20e9c5 HDFS-13710. RBF: setQuota and getQuotaUsage should check the dfs.federation.router.quota.enable. Contributed by yanghuafeng. 
(cherry picked from commit 43f7fe8aae)
 2018-07-09 15:08:39 +08:00
Yiqun Lin 1a96293c63 HDFS-13528. RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 3b637155a4)
 2018-07-04 15:07:47 +08:00
Inigo Goiri 3e177a1d62 HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 7296b644f7)
 2018-07-02 16:38:13 -07:00
Inigo Goiri 9e148aa0bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 8762e9cf10)
 2018-06-15 16:51:03 -07:00
Inigo Goiri b52316d796 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak. 
(cherry picked from commit d31a3ce767)
 2018-06-15 16:00:45 -07:00
Sean Mackrory b122961bc0 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel. 2018-06-15 16:28:36 -06:00
Inigo Goiri fe831cb5b2 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang. 2018-06-15 13:15:20 -07:00
Inigo Goiri 55b50eecc4 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit 43d994e4a6)
 2018-06-15 10:35:05 -07:00
Inigo Goiri fcd7ae2745 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak. 2018-06-14 14:12:54 -07:00
Inigo Goiri 06c87edf51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 8d4926f38b)
 2018-06-14 10:00:26 -07:00
Rohith Sharma K S 0af3bea05d YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:30:02 +05:30
Inigo Goiri 8059390798 HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 2018-06-11 13:47:08 -07:00
Chao Sun 4049fa37af HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun. 
(cherry picked from commit fba1c42adc)
 2018-06-08 16:44:36 -07:00
Yiqun Lin 1465830f7e HDFS-13644. Backport HDFS-10376 to branch-2. Contributed by Zsolt Venczel. 2018-06-07 18:10:52 +08:00
Inigo Goiri 65d6943b4f HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu. 
(cherry picked from commit a28848104f)
 2018-06-04 18:14:34 -07:00
Rushabh Shah 0caf40efe5 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah 
(cherry picked from commit e2289c8d14)
(cherry picked from commit 4ecb7aaf77)
 2018-06-04 10:07:24 -05:00
Inigo Goiri c8f183edd3 HDFS-13637. RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota. 
(cherry picked from commit e11d674049)
 2018-06-01 17:01:02 -07:00
Chao Sun 87e805ba4d HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun. 
(cherry picked from commit ff013d2c95)
 2018-05-31 17:44:39 -07:00
Inigo Goiri 6b9ef2d5b7 HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu. 2018-05-30 10:10:25 -07:00
Yiqun Lin cb962139ee HDFS-13626. Fix incorrect username when deny the setOwner operation. Contributed by Zsolt Venczel. 2018-05-30 16:55:39 +08:00
Inigo Goiri 86736cb279 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9dbf4f0166)
 2018-05-28 16:36:06 -07:00
Inigo Goiri 27091d2d16 HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu. 2018-05-25 19:48:16 -07:00
Inigo Goiri 1542484161 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
 2018-05-25 09:12:44 -07:00
Andrew Wang 097347c857 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient. 
(cherry picked from commit c9b63deb53)
 2018-05-24 09:56:54 -07:00
Inigo Goiri c5ff9553fb HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c13dea87d9)
 2018-05-23 11:37:55 -07:00
Inigo Goiri 411a2f609b HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang. 2018-05-23 09:48:35 -07:00
Andrew Wang 0f0d29a8d1 HDFS-13601. Optimize ByteString conversions in PBHelper. 2018-05-22 23:58:11 -07:00
Inigo Goiri 8f45d119a1 HDFS-13590. Backport HDFS-12378 to branch-2. Contributed by Lukas Majercak. 2018-05-18 11:07:12 -07:00
Inigo Goiri 1af41dadc2 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9775ecb235)
 2018-05-18 09:47:50 -07:00
Inigo Goiri b0e0ec1648 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu. 
(cherry picked from commit 57b893de3d)
 2018-05-18 09:40:10 -07:00
Inigo Goiri a5ec44c655 HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 53b807a6a8)
 2018-05-17 17:04:59 -07:00
Inigo Goiri 6df8fa1bc9 HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit a97a2042f2)
 2018-05-17 16:55:01 -07:00
Inigo Goiri 315ffd1ada HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 65476458fa)
 2018-05-17 14:49:37 -07:00
Inigo Goiri b87cd6354b HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 26f1e22fc9)
 2018-05-17 13:36:53 -07:00
Inigo Goiri 33dfbed8d2 HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c0ec061e28)
 2018-05-17 09:23:11 -07:00
Kihwal Lee 033b25cbe9 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0ce6290de6)
 2018-05-17 09:29:08 -05:00
Inigo Goiri d2a3bd1430 HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu. 
(cherry picked from commit 454de3b543)
 2018-05-16 19:24:36 -07:00
Inigo Goiri 9c96822096 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 55d5549087)
 2018-05-16 10:10:37 -07:00
Inigo Goiri d163f72310 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu. 2018-05-16 09:33:16 -07:00
Inigo Goiri 3d48a895a6 HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu. 
(cherry picked from commit 63480976a0)
 2018-05-15 15:36:58 -07:00
Inigo Goiri cdb80d6dcb HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 2d6195c0af)
 2018-05-15 11:22:41 -07:00
Inigo Goiri cc9203615a HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu. 
(cherry picked from commit eaca7980ad)
 2018-05-15 10:29:44 -07:00
Inigo Goiri 4ae1ed5b17 HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 92ebd466c7)
 2018-05-15 10:23:28 -07:00
Hanisha Koneru c998c126b8 HDFS-13544. Improve logging for JournalNode in federated cluster. 
(cherry picked from commit 6beb25ab7e)
 2018-05-14 10:21:59 -07:00
Sammi Chen eabc588d22 Add 2.9.1 release notes and changes documents 2018-05-14 15:36:30 +08:00
Inigo Goiri d88d9f2874 HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu. 2018-05-11 09:50:40 -07:00
Yiqun Lin 11794e5eda HDFS-13346. RBF: Fix synchronization of router quota and nameservice quota. 2018-05-11 18:56:59 +08:00
Yiqun Lin b83ff77409 Revert "HDFS-13346. RBF: Fix synchronization of router quota and nameservice quota." 
This reverts commit d4a141e7b0.
 2018-05-11 18:53:12 +08:00
Yiqun Lin d4a141e7b0 HDFS-13346. RBF: Fix synchronization of router quota and nameservice quota. 
(cherry picked from commit a922b9c82c)
 2018-05-11 15:06:45 +08:00
Chris Douglas b2acaa52d2 HDFS-13272. DataNodeHttpServer to have configurable HttpServer2 threads. Contributed by Erik Krogen 2018-05-10 14:53:39 -07:00
Inigo Goiri a756d3e7c8 HDFS-13537. TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang. 
(cherry picked from commit 914b98a713)
 2018-05-09 16:53:45 -07:00
Xiao Chen 5679920793 HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi. 
(cherry picked from commit 696a4be0da)
 2018-05-07 15:51:33 -07:00
Xiao Chen d7f0af0b65 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445." 
This reverts commit 2444d70a40.
 2018-05-07 14:11:13 -07:00
Inigo Goiri e7b5c0091d HDFS-13525. RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin. 
(cherry picked from commit a3b416f69d)
 2018-05-03 11:26:51 -07:00
Yiqun Lin 06632c0665 HDFS-13488. RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri. 
(cherry picked from commit 37269261d1)
(cherry picked from commit 5fef28d0d4)
 2018-05-02 15:02:24 +08:00
Inigo Goiri 9fd93ee533 HDFS-13503. Fix TestFsck test failures on Windows. Contributed by Xiao Liang. 2018-05-01 08:15:07 -07:00
Inigo Goiri cb3414a279 HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 2018-04-30 15:24:21 -07:00
Inigo Goiri c0c788aafc HDFS-13509. Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang. 2018-04-28 09:07:56 -07:00
Inigo Goiri c8441811fd HDFS-13508. RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S. 
(cherry picked from commit 484440602c)
 2018-04-27 16:30:07 -07:00
Inigo Goiri ae32e2140c HDFS-13434. RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari. 
(cherry picked from commit f469628bba)
 2018-04-27 15:16:46 -07:00
Yiqun Lin 87c9fff352 HDFS-13499. RBF: Show disabled name services in the UI. Contributed by Inigo Goiri. 
(cherry picked from commit 0a293de00d)
 2018-04-27 10:41:16 +08:00
Inigo Goiri 1c3d746fc0 HDFS-13326. RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li. 
(cherry picked from commit c394051a3d4d9d531f418503cb519606ae2b2e69)
 2018-04-26 13:14:08 -07:00
Wei-Chiu Chuang a5fc638235 HDFS-13492. Limit httpfs binds to certain IP addresses in branch-2. Contributed by Wei-Chiu Chuang. 2018-04-26 09:55:03 -07:00
Yiqun Lin 33ffc96085 HDFS-13484. RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 30fef0bf1e5c8c0ca073df99ad9b33cb0e4431a5)
 2018-04-25 15:25:16 +08:00
Yiqun Lin cc5416d94a HDFS-13490. RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri. 
(cherry picked from commit b06601acce)
 2018-04-24 11:29:06 +08:00
Inigo Goiri db9da432b9 HDFS-13336. Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang. 
(cherry picked from commit df92a17e02)
 2018-04-23 19:14:03 -07:00
Chris Douglas 99e82e2c2c HDFS-13408. MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang 
(cherry picked from commit f411de6a79)
(cherry picked from commit cf272c5179)
(cherry picked from commit 956ab12ede)
 2018-04-23 11:15:28 -07:00
Yiqun Lin a975250303 HDFS-13478. RBF: Disabled Nameservice store API. Contributed by Inigo Goiri. 
(cherry picked from commit b6dae26f44)
 2018-04-21 14:53:12 +08:00
Inigo Goiri a3c1735951 HDFS-13453. RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 1134af9ad1)
 2018-04-19 14:58:02 -07:00
Xiao Chen 2444d70a40 HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445. 
(cherry picked from commit 6503593711)
 2018-04-18 22:43:24 -05:00
Shashikant Banerjee 228869e5f9
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
 2018-04-19 08:38:11 +09:00
Erik Krogen 8f341c673f
HDFS-12828. OIV ReverseXML Processor fails with escaped characters 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 2d9e791a90)
 2018-04-18 14:40:45 +09:00
Inigo Goiri b4b10786c3 HDFS-13462. Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak. 2018-04-17 15:18:01 -07:00
Wei Yan 8d720daa49 HDFS-13466. RBF: Add more router-related information to the UI. 
(cherry picked from commit e4313e7e47)
 2018-04-17 15:06:48 -07:00
Yiqun Lin 88cb461c87 HDFS-13435. RBF: Improve the error loggings for printing the stack trace. 
(cherry picked from commit c4d3636c21)
 2018-04-17 11:27:30 +08:00
Yiqun Lin a7721082dc HDFS-13418. NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie. 2018-04-13 17:59:35 +08:00
Inigo Goiri 20472bdfd2 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun." 
This reverts commit 7e692425d5.
 2018-04-12 08:44:17 -07:00
Akira Ajisaka f055a53b43
HDFS-7101. Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu. 
(cherry picked from commit 113af12cfb)
 2018-04-12 17:48:45 +09:00
Akira Ajisaka be627ccfe3
HDFS-13427. Fix the section titles of transparent encryption document. 
(cherry picked from commit c7cd362afd)
 2018-04-12 10:49:07 +09:00
Inigo Goiri 940d3e6f74 HDFS-13386. RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 18de6f2042)
 2018-04-11 12:04:18 -07:00
Inigo Goiri 9af5823dbb HDFS-13428. RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR. 
(cherry picked from commit 6a881cd842fb4eab596a386b39670ea03536891a)
 2018-04-11 10:42:17 -07:00
Wei Yan c65ac673a1 HDFS-13045. RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri. 
(cherry picked from commit 8433f6256c)
 2018-04-11 09:08:11 -07:00
Yiqun Lin ea51ef44fc HDFS-13384. RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri. 
(cherry picked from commit e87be8a2a4)
 2018-04-10 15:39:46 +08:00
Yiqun Lin 9baebf4b27 HDFS-13410. RBF: Support federation with no subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit a92200f4a6)
 2018-04-10 14:47:55 +08:00
Inigo Goiri 78ec00155e HDFS-13380. RBF: mv/rm fail after the directory exceeded the quota limit. Contributed by Yiqun Lin. 
(cherry picked from commit e9b9f48dad)
 2018-04-09 10:11:38 -07:00
Inigo Goiri 7e692425d5 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit ac32b3576d)
 2018-04-09 09:18:42 -07:00
Yiqun Lin 14827da747 HDFS-13402. RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu. 
(cherry picked from commit 5700556cd6)
 2018-04-08 12:07:44 +08:00
Sean Mackrory b412958f70 HDFS-13176. WebHdfs file path gets truncated when having semicolon inside. Contributed by Zsolt Venczel. 2018-04-06 07:41:58 -06:00
Wei Yan 689a8252aa HDFS-13353. RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma. 
(cherry picked from commit 3121e8c293)
 2018-04-05 13:23:07 -07:00
Inigo Goiri 4276a858f8 HDFS-13337. Backport HDFS-4275 to branch-2.9. Contributed by Xiao Liang. 2018-04-03 13:55:44 -07:00
Inigo Goiri 254ec1e4d3 HDFS-13364. RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri. 2018-04-03 09:29:20 -07:00
Xiao Chen 251fc1fb40 HADOOP-15317. Improve NetworkTopology chooseRandom's loop. 
(cherry picked from commit 57374c4737)
 2018-04-02 22:55:03 -07:00
Inigo Goiri c6a013c9e0 HDFS-13289. RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar. 
(cherry picked from commit dc8e343201)
 2018-03-31 09:47:36 -07:00
Konstantin V Shvachko de419e2777 HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie. 
(cherry picked from commit acfd764fcc)
 2018-03-30 19:08:12 -07:00
Arpit Agarwal 7e5c8faeb7 HDFS-13314. NameNode should optionally exit if it detects FsImage corruption. Contributed by Arpit Agarwal. 2018-03-28 13:08:04 -07:00
Yiqun Lin f5aa36e194 HDFS-13347. RBF: Cache datanode reports. Contributed by Inigo Goiri. 
(cherry picked from commit a71656c1c1)
 2018-03-28 11:59:36 -07:00
Inigo Goiri 6b187d2ee0 HDFS-13352. RBF: Add xsl stylesheet for hdfs-rbf-default.xml. Contributed by Takanobu Asanuma. 
(cherry picked from commit 4cef01147f)
 2018-03-27 10:33:31 -07:00
Inigo Goiri 90609f3561 HDFS-13204. RBF: Optimize name service safe mode icon. Contributed by liuhongtong. 
(cherry picked from commit 947b282984)
 2018-03-26 09:44:34 -07:00
Yiqun Lin 622664648c HDFS-13291. RBF: Implement available space based OrderResolver. Contributed by Yiqun Lin. 
(cherry picked from commit cfc3a1c8f0)
 2018-03-26 20:03:20 +08:00
Wei Yan 69c08f3dd3 HDFS-12512. RBF: Add WebHDFS. 
(cherry picked from commit 6e31a09084)
 2018-03-23 10:39:08 -07:00
Inigo Goiri 3db8c7ff68 HDFS-12792. RBF: Test Router-based federation using HDFSContract. Contributed by Inigo Goiri. 
(cherry picked from commit e196d158a2)
 2018-03-22 13:37:06 -07:00
Kihwal Lee d2c3a56ad8 HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong. 
(cherry picked from commit 66461ed98b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
 2018-03-22 12:31:52 -05:00
Inigo Goiri 06bebcc7d2 HDFS-13318. RBF: Fix FindBugs in hadoop-hdfs-rbf. Contributed by Ekanth S. 
(cherry picked from commit 3d0ee8b804)
 2018-03-22 09:23:25 -07:00
Konstantin V Shvachko 52752840f8 HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan. 
(cherry picked from commit 8d898ab25f)
 2018-03-21 17:21:28 -07:00
Inigo Goiri 8bbd560805 HDFS-13250. Addendum to fix branch-2 compilation. RBF: Router to manage requests across multiple subclusters. Contributed by Inigo Goiri. 2018-03-21 11:38:17 -07:00
Yiqun Lin 0b6d5a0731 HDFS-13250. RBF: Router to manage requests across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit 2caba999bb)
 2018-03-21 11:42:21 +08:00
Yiqun Lin fb77ac6243 HDFS-13307. RBF: Improve the use of setQuota command. Contributed by liuhongtong. 
(cherry picked from commit 69fe4407eb)
 2018-03-21 10:55:43 +08:00
weiy 4aa34324b2 HDFS-13215. RBF: Move Router to its own module. 2018-03-19 23:23:16 -07:00
Inigo Goiri f278494a5f HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 2018-03-16 14:59:01 -07:00
Inigo Goiri 5e917a1501 Revert "HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak." 
This reverts commit 972c322301.
 2018-03-16 14:26:09 -07:00
Inigo Goiri 972c322301 HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 
(cherry picked from commit 08ff1586d5)
 2018-03-16 10:31:02 -07:00
Inigo Goiri 4615c0bba2 HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit e71bc00a47)
 2018-03-16 09:15:23 -07:00
Inigo Goiri 3a5f50fa6e Revert "HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri." 
This reverts commit ccf07fdafd.
 2018-03-16 09:12:30 -07:00
Inigo Goiri b1b10007a4 HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-03-15 11:30:52 -07:00
Inigo Goiri ccf07fdafd HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit e71bc00a47)
 2018-03-15 10:51:04 -07:00
Yiqun Lin 3fad968624 HDFS-13261. Fix incorrect null value check. Contributed by Jianfei Jiang. 2018-03-15 11:12:25 +08:00
Inigo Goiri 299b40ff8b HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router. Contributed by Wei Yan. 
(cherry picked from commit 41fc7f80be)
 2018-03-14 13:16:32 -07:00
Yiqun Lin b3d56cb835 HDFS-12773. RBF: Improve State Store FS implementation. Contributed by Inigo Goiri. 
(cherry picked from commit 76be6cbf6c)
 2018-03-14 11:25:08 +08:00
Chris Douglas 340cd5f1b1 HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
(cherry picked from commit a03c8ea61f)
 2018-03-12 20:08:46 -07:00
Yiqun Lin 38c21ec651 HDFS-13226. RBF: Throw the exception if mount table entry validated failed. Contributed by maobaolong. 
(cherry picked from commit 19292bc264)
 2018-03-13 11:06:53 +08:00
Yiqun Lin 5edbd1a258 HDFS-13253. RBF: Quota management incorrect parent-child relationship judgement. Contributed by Yiqun Lin. 
(cherry picked from commit 7fab787de7)
 2018-03-13 10:33:12 +08:00
Akira Ajisaka b0c80f1c81
HDFS-12156. TestFSImage fails without -Pnative 
(cherry picked from commit 319defafc1)
 2018-03-13 11:29:44 +09:00
Yiqun Lin 2ca6cf3170 HDFS-10803. TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space available. Contributed by Yiqun Lin. 
(cherry picked from commit 4afd50b106)
 2018-03-13 10:18:23 +08:00
Inigo Goiri 7fded36e4c HDFS-13241. RBF: TestRouterSafemode failed if the port 8888 is in use. Contributed by maobaolong. 
(cherry picked from commit 91c82c90f0)
 2018-03-12 17:30:45 -07:00
Inigo Goiri 737dca1959 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin. 2018-03-12 12:54:42 -07:00
Anu Engineer 88c749a903 HDFS-10618. TestPendingReconstruction#testPendingAndInvalidate is flaky due to race condition. Contributed by Eric Badger. 2018-03-12 12:14:21 -07:00
Inigo Goiri 457895d50f Revert "HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin." 
This reverts commit ab0f127084.
 2018-03-12 11:44:06 -07:00
Inigo Goiri ab0f127084 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin. 
(cherry picked from commit ac627f561f)
 2018-03-12 10:04:51 -07:00
Yiqun Lin 3a34924896 HDFS-13240. RBF: Update some inaccurate document descriptions. Contributed by Yiqun Lin. 
(cherry picked from commit 4743d4a2c7)
 2018-03-10 11:33:44 +08:00
Inigo Goiri 0dadcf9ea3 HDFS-13232. RBF: ConnectionPool should return first usable connection. Contributed by Ekanth S. 
(cherry picked from commit 8133cd5305)
 2018-03-09 18:28:59 -08:00
Inigo Goiri 4b6b07690e HDFS-13212. RBF: Fix router location cache issue. Contributed by Weiwei Wu. 
(cherry picked from commit afe1a3ccd5)
 2018-03-09 17:21:10 -08:00
Yiqun Lin aa748c6bca HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan. 
(cherry picked from commit 122805b43a)
 2018-03-09 15:51:56 +08:00
Kihwal Lee 0aa52d4085 HDFS-12614. FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured. Contributed by Manoj Govindassamy. 2018-03-08 14:22:17 -06:00
Inigo Goiri 95f34ce462 HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun. 
(cherry picked from commit 0c2b969e01)
(cherry picked from commit 44abf77d9b)
 2018-03-08 10:24:41 -08:00
Yiqun Lin a5d9691b54 HDFS-13214. RBF: Complete document of Router configuration. Contributed by Yiqun Lin. 
(cherry picked from commit 58ea2d7a65)
 2018-03-07 15:51:42 +08:00
Xiaoyu Yao 313fdb9968 HDFS-13109. Support fully qualified hdfs path in EZ commands. Contributed by Hanisha Koneru. 
(cherry picked from commit edf9445708)
 2018-03-06 20:01:51 -08:00
Xiaoyu Yao 4f3c4e4514 HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao. 
(cherry picked from commit 8faf0b50d4)
 2018-03-06 18:49:13 -08:00
Xiao Chen 0882725c88 HDFS-13040. Kerberized inotify client fails despite kinit properly. Contributed by Istvan Fajth, Wei-Chiu Chuang, Xiao Chen. 2018-03-05 09:44:42 -08:00
Yiqun Lin 08cd911b69 HDFS-13194. CachePool permissions incorrectly checked. Contributed by Jianfei Jiang. 
(cherry picked from commit a9c14b1119)
 2018-02-28 10:25:49 +08:00
Inigo Goiri 2f18785f39 HDFS-13199. RBF: Fix the hdfs router page missing label icon issue. Contributed by maobaolong. 
(cherry picked from commit d86f301d46)
 2018-02-27 17:42:28 -08:00
Yiqun Lin 1b0aa0e87a HDFS-13184. RBF: Improve the unit test TestRouterRPCClientRetries. Contributed by Yiqun Lin. 
(cherry picked from commit 1e85a995d1)
 2018-02-27 10:54:37 +08:00
Wei Yan 54803ebe49 HDFS-13187. RBF: Fix Routers information shown in the web UI. Contributed by Inigo Goiri and Wei Yan. 
(cherry picked from commit 6ce9f79cc9)
 2018-02-26 15:32:27 -08:00
Kihwal Lee 4b43f2aa56 HDFS-12070. Failed block recovery leaves files open indefinitely and at risk for data loss. Contributed by Kihwal Lee. 
(cherry picked from commit 451265a83d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java
 2018-02-26 10:58:07 -06:00
Arpit Agarwal 79af42f095 HDFS-12781. After Datanode down, In Namenode UI Datanode tab is throwing warning message.. Contributed by Brahma Reddy Battula. 2018-02-24 15:12:36 -08:00
Xiao Chen 80f7165374 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException. 
(cherry picked from commit 51088d3233)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
 2018-02-23 14:00:33 -08:00
Gabor Bota 696e8e0299
HADOOP-15236. Fix typo in RequestHedgingProxyProvider and RequestHedgingRMFailoverProxyProvider 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c36b4aa31c)
(cherry picked from commit b8942cb3bf)
 2018-02-23 13:58:08 +09:00
Xiao Chen 49ed7d7fc9 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Gabor Bota, Wei-Chiu Chuang. 2018-02-19 19:15:02 -08:00
Yiqun Lin a278ad6b91 HDFS-13119. RBF: Manage unavailable clusters. Contributed by Yiqun Lin. 2018-02-20 09:58:28 +08:00
Kihwal Lee fe044e69c3 HDFS-13112. Token expiration edits may cause log corruption or deadlock. Contributed by Daryn Sharp. 
(cherry picked from commit 47473952e5)
 2018-02-15 15:38:49 -06:00
Kihwal Lee b725fd6924 xattr api cleanup 
(cherry picked from commit da59acd8ca)
 2018-02-15 11:28:45 -06:00
Brahma Reddy Battula 24af0bc144 HDFS-8693. Addendum patch to execute the command using UGI. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 35c17351ca)
 2018-02-12 22:25:44 +05:30
Arpit Agarwal 82530c966a HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:02:42 -08:00
Yiqun Lin d49074f0a2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin. 
(cherry picked from commit 543f3abbee)
 2018-02-09 14:00:55 +08:00
Brahma Reddy Battula 9873eb63a7 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 2018-02-08 18:33:11 +05:30
Yongjun Zhang b7f7fb003e HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang. 
(cherry picked from commit f491f717e9)

Conflicts:

	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
(cherry picked from commit 60d82ac61f)
 2018-02-07 15:35:41 -08:00
Yongjun Zhang 6dfea05049 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang. 
(cherry picked from commit 4e9a59ce16)
 2018-02-02 23:33:10 -08:00
Yiqun Lin 7cd3770fee HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 
(cherry picked from commit b0627c891b)
 2018-02-02 17:38:32 +08:00
Inigo Goiri 7b8cc048cc HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 6ca7204ceb)
 2018-02-01 10:38:39 -08:00
Yiqun Lin e5cac88c82 HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 
(cherry picked from commit dbb9dded33)
 2018-01-31 09:14:31 -08:00
Weiwei Yang 0c2c4c20cb HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 
(cherry picked from commit 2e7331ca26)
 2018-01-30 21:40:07 -08:00
Inigo Goiri fffa7dc377 HDFS-13083. RBF: Fix doc error setting up client. Contributed by tartarus. 
(cherry picked from commit 5206b2c7ca)
 2018-01-30 18:28:21 -08:00
Kihwal Lee eda786ea12 HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah. 2018-01-29 17:52:04 -06:00
Xiao Chen 0922ea0aaf HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi. 
(cherry picked from commit b63dcd583f)
(cherry picked from commit 75be15cf02)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
 2018-01-28 22:30:14 -08:00
Arpit Agarwal 7be5a46a41 HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:36 -08:00
Wei Yan cefe4b7bb6 HDFS-13049. RBF: Inconsistent Router OPTS config in branch-2 and branch-3. Contributed by Wei Yan. 2018-01-25 12:58:16 -08:00
Yiqun Lin 9ce24d2d50 HDFS-13042. RBF: Heartbeat Router State. Contributed by Inigo Goiri. 
(cherry picked from commit 7721fff744)
 2018-01-25 15:54:20 +08:00
Jianfei Jiang f4656e2150
HDFS-13063. Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit eb2dd08688)
 2018-01-25 15:32:13 +09:00
Inigo Goiri 1534936ac1 HDFS-12772. RBF: Federation Router State State Store internal API. Contributed by Inigo Goiri. 
(cherry picked from commit 95743c672e)
 2018-01-23 19:17:22 -08:00
Inigo Goiri c6139332f9 HDFS-13028. RBF: Fix spurious TestRouterRpc#testProxyGetStats. Contributed by Inigo Goiri. 
(cherry picked from commit 130f8bce00)
 2018-01-19 09:52:29 -08:00
Yiqun Lin 65507053b4 HDFS-12973. RBF: Document global quota supporting in federation. Contributed by Yiqun Lin. 
(cherry picked from commit 9e4f52d323)
 2018-01-19 14:21:07 +08:00
Vinayakumar B b3eeea0b6e HDFS-9049. Make Datanode Netty reverse proxy port to be configurable. Contributed by Vinayakumar B. 
(cherry picked from commit 09efdfe9e1)
 2018-01-17 14:22:29 +05:30
Brahma Reddy Battula 8e7ce0eb4c HDFS-8693. refreshNamenodes does not support adding a new standby to a running DN. Contributed by Ajith S. 
(cherry picked from commit 880b9d24ff)
 2018-01-16 16:29:13 +05:30
Yiqun Lin e5612f0389 HDFS-12972. RBF: Display mount table quota info in Web UI and admin command. Contributed by Yiqun Lin. 
(cherry picked from commit 9afb8025d6)
 2018-01-13 13:20:07 +08:00
Wei-Chiu Chuang 2273499aef HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. Contributed by Vinayakumar B. 2018-01-12 10:01:57 -08:00
Yiqun Lin 66d58d2b9a HDFS-12934. RBF: Federation supports global quota. Contributed by Yiqun Lin. 2018-01-10 19:48:47 +08:00
Inigo Goiri 215a3d800a HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by Inigo Goiri. 
(cherry picked from commit d9006d8a4e)
 2018-01-09 19:01:10 -08:00
Manoj Govindassamy eadd3cecf9 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 2018-01-08 16:21:21 -08:00
Wei Yan 223bb935e2 HDFS-12945. Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods. Contributed by Wei Yan. 
(cherry picked from commit 2ee0d64ace)
 2018-01-08 15:23:57 -08:00
Inigo Goiri f7fedac6cd HDFS-12988. RBF: Mount table entries not properly updated in the local cache. Contributed by Inigo Goiri. 
(cherry picked from commit 83b513ac6d)
 2018-01-05 09:13:52 -08:00
Xiao Chen 50b233d356 HDFS-9023. When NN is not able to identify DN for replication, reason behind it can be logged. 2017-12-28 11:55:24 -08:00
Yiqun Lin df3212aad5 HDFS-12937. RBF: Add more unit tests for router admin commands. Contributed by Yiqun Lin. 
(cherry picked from commit e040c97b77)
 2017-12-19 15:35:40 +08:00
Tsz-Wo Nicholas Sze c4e58ea823 HDFS-12347. TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently. Contributed by Bharat Viswanadham 2017-12-19 10:04:00 +08:00
Erik Krogen da4e2f38e1 HDFS-12818. Support multiple storages in DataNodeCluster / SimulatedFSDataset. Contributed by Erik Krogen. 
(cherry picked from commit 94576b17fb)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
 2017-12-18 12:23:20 -08:00
Jason Lowe 9ae8d1a8de HDFS-12881. Output streams closed with IOUtils suppressing write errors. Contributed by Ajay Kumar 2017-12-15 14:47:54 -06:00
Yiqun Lin 24c4d2a7a2 HDFS-12895. RBF: Add ACL support for mount table. Contributed by Yiqun Lin. 
(cherry picked from commit ee028bfdf1)
 2017-12-15 14:13:54 +08:00
Kihwal Lee 08a50da95f HDFS-12907. Allow read-only access to reserved raw for non-superusers. Contributed by Rushabh S Shah. 2017-12-14 15:03:47 -06:00
Yiqun Lin 22bc62058b HDFS-12883. RBF: Document Router and State Store metrics. Contributed by Yiqun Lin. 
(cherry picked from commit 91c96bdf8e)
 2017-12-14 16:03:34 +08:00
Inigo Goiri 67c2826941 HDFS-12875. RBF: Complete logic for -readonly option of dfsrouteradmin add command. Contributed by Inigo Goiri. 
(cherry picked from commit 5cd1056ad7)
 2017-12-11 15:17:17 -08:00
Xiao Chen 31436a12d5 HADOOP-15056. Fix TestUnbuffer#testUnbufferException failure. Contributed by Jack Bearden. 
(cherry picked from commit 19e0894209)
(cherry picked from commit 1d1401ad38)
 2017-12-07 21:20:33 -08:00
Xiao Chen c9d48352c2 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-12-07 21:20:33 -08:00