Commit Graph

7071 Commits

Author SHA1 Message Date
Xiao Chen 5d16301ad9 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi.
(cherry picked from commit cede33997f)
2018-08-14 11:59:05 -07:00
Wei-Chiu Chuang 6f2a46578f HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan.
(cherry picked from commit 61a9b4f58b)
(cherry picked from commit 7a115e8de4)
2018-08-14 11:52:55 -07:00
Wei-Chiu Chuang d502abe947 HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh.
(cherry picked from commit 4023eeba05)
(cherry picked from commit 74d4fd975a)
2018-08-13 17:37:36 -07:00
Wei-Chiu Chuang 4b606eb9fd HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng.
(cherry picked from commit 23854443ef)
(cherry picked from commit b38649c59a)
2018-08-13 16:14:01 -07:00
Vinayakumar B eecfbaff3b HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena.
(cherry picked from commit 2b0f977241)
2018-08-13 13:58:37 -07:00
Chen Liang 2d985b6c81 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:47:17 -07:00
Chao Sun 07054c5994 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun.
(cherry picked from commit a2a8c48699)
2018-08-10 16:12:05 -07:00
Wangda Tan 8b4392769c Make 3.1.1 awared by other branches
(cherry picked from commit 49c687608b)
(cherry picked from commit 781620a44d)
2018-08-08 13:07:09 -07:00
Xiao Chen 8e5081569f HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell.
(cherry picked from commit 6677717c68)
2018-08-07 22:05:59 -07:00
Brahma Reddy Battula abc32c863e HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani
(cherry picked from commit 3108d27edd)
2018-07-30 16:08:49 +05:30
Arpit Agarwal eb40d4fab4 HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:11 -07:00
Anu Engineer faaf72dcb3 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception.
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
2018-07-27 09:35:02 -07:00
Wei-Chiu Chuang 0af04ad89f HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin.
(cherry picked from commit e95c5e9f62)
(cherry picked from commit 964f3454d1)
2018-07-25 11:05:36 -07:00
Daniel Templeton 3210b3d8aa HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
(cherry picked from commit 00c476abd8)
2018-07-24 16:13:30 -07:00
Yiqun Lin 097ae7fdc8 HDFS-13743. RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver. Contributed by Takanobu Asanuma.
(cherry picked from commit 7b25fb949b)
2018-07-20 17:32:27 +08:00
Yiqun Lin 34afb9f946 HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF document. Contributed by Takanobu Asanuma.
(cherry picked from commit 1af87df242)
2018-07-17 10:48:36 +08:00
Konstantin V Shvachko 475c282939 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal.
(cherry picked from commit 0a1e922f3d)
2018-07-16 18:33:36 -07:00
Wei-Chiu Chuang 0d2b983dc2 HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng.
(cherry picked from commit 88b2794244)
(cherry picked from commit ac9155d6cb)
2018-07-16 10:53:30 -07:00
Inigo Goiri 3d47ef1d31 HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed by Chao Sun.
(cherry picked from commit 359ea4e181)
2018-07-16 09:47:22 -07:00
Xiao Chen 883df537d6 HDFS-12837. Intermittent failure in TestReencryptionWithKMS.
(cherry picked from commit b37074be5a)
2018-07-11 21:03:53 -07:00
Akira Ajisaka 4d5473c89d
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota.
(cherry picked from commit 418cc7f3ae)
2018-07-11 14:48:12 -04:00
Wei-Chiu Chuang 74aa4be92d HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng.
(cherry picked from commit 162228e8db)
(cherry picked from commit f9fa3cb157)
2018-07-11 10:04:38 -07:00
Yiqun Lin 241f7bb7ad HDFS-13726. RBF: Fix RBF configuration links. Contributed by Takanobu Asanuma.
(cherry picked from commit 2ae13d41dc)
2018-07-11 22:19:32 +08:00
Andrew Wang 0e57e81b0c HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi.
(cherry picked from commit eecb5baaaa)
2018-07-09 15:17:48 +02:00
Wei-Chiu Chuang 68290606ea HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
(cherry picked from commit 0247cb6318)
(cherry picked from commit f5f4d0b7e7)
2018-07-06 15:01:35 -07:00
Andrew Wang 896b51578f HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
(cherry picked from commit 344f324710)
2018-07-03 11:08:05 +02:00
Inigo Goiri 98f2444a7d HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 7296b644f7)
2018-07-02 16:37:36 -07:00
Wei-Chiu Chuang 563d56be62 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
(cherry picked from commit fef20a446f)
(cherry picked from commit 2ce892fcf5)
2018-07-02 09:44:16 -07:00
Andrew Wang d34260f122 HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
(cherry picked from commit 5d748bd056)
2018-07-02 12:11:14 +02:00
Andrew Wang 7cd769b31c HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
(cherry picked from commit 6ba9974108)
2018-07-02 12:02:27 +02:00
Yiqun Lin db1f37a711 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.
(cherry picked from commit 30728aced4)
2018-06-22 10:55:09 +08:00
Xiao Chen aca66f6155 HDFS-13682. Cannot create encryption zone after KMS auth token expires.
(cherry picked from commit 32f867a6a9)
2018-06-20 15:59:45 -07:00
Surendra Singh Lilhore 5667582bc4 HDFS-12857. StoragePolicyAdmin should support schema based path. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 30941d99c9)
2018-06-19 10:09:05 -07:00
Inigo Goiri 11d3db2dba HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.
(cherry picked from commit 8762e9cf10)
2018-06-15 16:50:20 -07:00
Inigo Goiri 69abd1fc5c HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.
(cherry picked from commit d31a3ce767)
2018-06-15 16:00:16 -07:00
Wei-Chiu Chuang 73badce597 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.
(cherry picked from commit c966a3837a)
(cherry picked from commit 975d4b3d60)
2018-06-15 13:38:11 -07:00
Inigo Goiri fe1926a50f HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit eebeb6033f)
2018-06-15 13:08:47 -07:00
Inigo Goiri 94e0b9839f HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit 43d994e4a6)
2018-06-15 10:34:33 -07:00
Inigo Goiri a3c0c03899 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 8d4926f38b)
2018-06-14 09:59:50 -07:00
Yiqun Lin b94b071b10 HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.
(cherry picked from commit 8e7548d33b)
2018-06-13 20:24:12 +08:00
Rohith Sharma K S 96e7b7e8ae YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri.
(cherry picked from commit 2df73dace0)
2018-06-12 17:25:22 +05:30
Yongjun Zhang 8202c334fe Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release.
(cherry picked from commit 5a82f10e32)
2018-06-11 14:28:44 -07:00
Inigo Goiri 6eea6d28de HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
2018-06-11 11:18:04 -07:00
Bharat Viswanadham 7c144e4ae4 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar
(cherry picked from commit ccfb816d39)
2018-06-09 16:44:47 -07:00
Arpit Agarwal ea05171ceb HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:58:02 -07:00
Chao Sun d62b0f84f1 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
(cherry picked from commit fba1c42adc)
2018-06-08 16:42:18 -07:00
Brahma Reddy Battula a45e9812d6 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 9654dd1f47)
(cherry picked from commit 630f43faf7)
2018-06-07 09:18:20 -07:00
Wei-Chiu Chuang 8e423a7779 HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.
(cherry picked from commit e39b113db0)
(cherry picked from commit b2e919dc1a)
2018-06-07 08:32:53 -07:00
Xiao Chen 3843abe482 HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
(cherry picked from commit ba4011d64f)
2018-06-05 21:19:19 -07:00
Akira Ajisaka cc735146dc
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.
(cherry picked from commit 0e3c31579d)
2018-06-05 17:55:33 +09:00
Inigo Goiri e794ff2dcb HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 8d31ddcfeb)
2018-06-04 18:29:12 -07:00
Inigo Goiri 076dc95c76 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 16316b6011)
2018-06-04 18:22:35 -07:00
Inigo Goiri 2e5f849ea5 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.
(cherry picked from commit a28848104f)
2018-06-04 18:13:50 -07:00
Rushabh Shah 4ecb7aaf77 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah
(cherry picked from commit e2289c8d14)
2018-06-04 09:58:38 -05:00
Wei-Chiu Chuang 63fc8f82cb HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.
(cherry picked from commit bccdfeee0a)
(cherry picked from commit 302c36ffdc)
2018-06-04 07:04:32 -07:00
Xiao Chen 3ee5fd16f4 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.
(cherry picked from commit 9efb4b7db0)
2018-06-03 22:37:36 -07:00
Inigo Goiri a274d21ea3 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit e308ac88d3)
2018-06-02 20:13:55 -07:00
Inigo Goiri d40c49257a HDFS-13637. RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota.
(cherry picked from commit e11d674049)
2018-06-01 17:00:12 -07:00
Chao Sun c6becec392 HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.
(cherry picked from commit ff013d2c95)
2018-05-31 17:42:41 -07:00
Konstantin V Shvachko 3d7e345823 HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 15:00:58 -07:00
Inigo Goiri 33c4f60b06 HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
2018-05-30 10:07:16 -07:00
Yongjun Zhang f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Inigo Goiri 128cfa101a HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9dbf4f0166)
2018-05-28 16:35:32 -07:00
Inigo Goiri 1c1553670d HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.
(cherry picked from commit 8605a38514)
2018-05-25 19:46:16 -07:00
Inigo Goiri ee31d9d0dc HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 13d2528907)
2018-05-25 13:33:30 -07:00
Inigo Goiri 3b0da097a4 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
2018-05-25 09:11:34 -07:00
Andrew Wang 731abae557 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
(cherry picked from commit c9b63deb53)
2018-05-24 09:56:40 -07:00
Inigo Goiri 506f61e186 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.
(cherry picked from commit cddbbe5f69)
2018-05-23 12:13:06 -07:00
Inigo Goiri c5f29d62eb HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit c13dea87d9)
2018-05-23 11:37:27 -07:00
Inigo Goiri 8d8ef081a2 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit c0c9b7a8ef)
2018-05-23 09:47:32 -07:00
Sammi Chen 6db710b9d8 HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
(cherry picked from commit 34e8b9f9a8)
(cherry picked from commit 7d71b3a1cc)
2018-05-23 19:20:32 +08:00
Andrew Wang 5b3a6e3678 HDFS-13601. Optimize ByteString conversions in PBHelper.
(cherry picked from commit 1d2640b613)
(cherry picked from commit d47c0fc39a)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
2018-05-22 23:57:05 -07:00
Tsz-Wo Nicholas Sze 2acd8e4ecc HDFS-12813. RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1. Contributed by Mukul Kumar Singh 2018-05-19 11:12:13 -07:00
Inigo Goiri 97aa781d3d HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9775ecb235)
2018-05-18 09:47:22 -07:00
Inigo Goiri d923e05798 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.
(cherry picked from commit 57b893de3d)
2018-05-18 09:38:48 -07:00
Inigo Goiri cdc16b3a2f HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.
(cherry picked from commit 53b807a6a8)
2018-05-17 17:04:23 -07:00
Inigo Goiri ccd7d50faf HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit a97a2042f2)
2018-05-17 16:54:17 -07:00
Inigo Goiri 6b2d86debe HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 65476458fa)
2018-05-17 14:49:03 -07:00
Inigo Goiri d3457b38e6 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 26f1e22fc9)
2018-05-17 13:36:16 -07:00
Inigo Goiri fb0a258b3c HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.
(cherry picked from commit c0ec061e28)
2018-05-17 09:22:48 -07:00
Kihwal Lee 0943112d85 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee.
(cherry picked from commit 0ce6290de6)
2018-05-17 09:28:17 -05:00
Inigo Goiri fcf0b6a8dd HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu.
(cherry picked from commit 454de3b543)
2018-05-16 19:24:00 -07:00
Inigo Goiri 16e89a2771 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 55d5549087)
2018-05-16 10:10:08 -07:00
Inigo Goiri 0a2cdd2730 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 6b67161ca3)
2018-05-16 09:32:39 -07:00
Inigo Goiri eadfb55195 HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu.
(cherry picked from commit 63480976a0)
2018-05-15 15:36:30 -07:00
Inigo Goiri 1b4c39fd2f HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 2d6195c0af)
2018-05-15 11:22:12 -07:00
Inigo Goiri d1b17f997c HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu.
(cherry picked from commit eaca7980ad)
2018-05-15 10:29:04 -07:00
Inigo Goiri 9a722150c6 HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 92ebd466c7)
2018-05-15 10:22:52 -07:00
Hanisha Koneru b00c0a08a3 HDFS-13544. Improve logging for JournalNode in federated cluster.
(cherry picked from commit 6beb25ab7e)
2018-05-14 10:16:06 -07:00
Xiao Chen 220e38ec02 HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure.
(cherry picked from commit 960940e0e0)
2018-05-14 09:29:22 -07:00
Sammi Chen a700d05f72 Add 2.9.1 release notes and changes documents
(cherry picked from commit f3f544b004)
2018-05-14 15:18:19 +08:00
Inigo Goiri 3b57a236ed HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu.
(cherry picked from commit d50c4d71dc)
2018-05-11 09:49:21 -07:00
Inigo Goiri d72a4b6303 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun."
This reverts commit 1049e180a7.
2018-05-10 09:24:45 -07:00
Inigo Goiri e466b4e2a7 HDFS-13537. TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang.
(cherry picked from commit 914b98a713)
2018-05-09 16:53:09 -07:00
Inigo Goiri 1049e180a7 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.
(cherry picked from commit 63803e7051)
2018-05-09 13:56:58 -07:00
Yongjun Zhang 21bf02023c Revert "HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem"
This reverts commit 218a21a0f0.
2018-05-08 22:22:06 -07:00
Xiao Chen fe66ed7da5 HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi.
(cherry picked from commit 696a4be0da)
2018-05-07 15:39:33 -07:00
Xiao Chen 22384580b2 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445."
This reverts commit 9f19c3b03e.
2018-05-07 13:56:26 -07:00
Inigo Goiri 21442dcaa3 HDFS-13525. RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin.
(cherry picked from commit a3b416f69d)
2018-05-03 11:26:00 -07:00
Yiqun Lin 221db515aa HDFS-13488. RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri.
(cherry picked from commit 37269261d1)
(cherry picked from commit 5fef28d0d4)
2018-05-02 15:06:53 +08:00
Inigo Goiri d5aa4b9a9a HDFS-13503. Fix TestFsck test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit 9e2cfb2d3f)
2018-05-01 08:14:14 -07:00
Inigo Goiri 0e31e014f3 HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 2018-04-30 15:22:30 -07:00
Inigo Goiri af9bc982ce HDFS-13509. Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit eb7fe1d588)
2018-04-28 09:06:58 -07:00
Inigo Goiri d83254312b HDFS-13508. RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S.
(cherry picked from commit 484440602c)
2018-04-27 16:29:23 -07:00
Inigo Goiri 71e52c936d HDFS-13434. RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari.
(cherry picked from commit f469628bba)
2018-04-27 15:16:03 -07:00
Yiqun Lin 1c65529dc3 HDFS-13499. RBF: Show disabled name services in the UI. Contributed by Inigo Goiri.
(cherry picked from commit 0a293de00d)
2018-04-27 10:40:13 +08:00
Inigo Goiri a71d184f4f HDFS-13326. RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li.
(cherry picked from commit c394051a3d4d9d531f418503cb519606ae2b2e69)
2018-04-26 13:13:02 -07:00
Yiqun Lin caffd538af HDFS-13484. RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri.
(cherry picked from commit 30fef0bf1e5c8c0ca073df99ad9b33cb0e4431a5)
2018-04-25 15:39:16 +08:00
Yiqun Lin 14cfcf6b71 HDFS-13490. RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri.
(cherry picked from commit b06601acce)
2018-04-24 11:28:06 +08:00
Inigo Goiri 7bb2709c4a HDFS-13336. Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang.
(cherry picked from commit df92a17e02)
2018-04-23 19:13:25 -07:00
Lei Xu 463fcfb50a Update CHANGES, RELEASENOTES, and jdiff for 3.0.2 release.
(cherry picked from commit f6ecb76d0b)

Conflicts:
	hadoop-project-dist/pom.xml
2018-04-23 14:17:29 -07:00
Chris Douglas 956ab12ede HDFS-13408. MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang
(cherry picked from commit f411de6a79)
(cherry picked from commit cf272c5179)
2018-04-23 11:15:15 -07:00
Arpit Agarwal e8f62357c0 HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal. 2018-04-23 10:09:16 -07:00
Yiqun Lin 45b11fe359 HDFS-13478. RBF: Disabled Nameservice store API. Contributed by Inigo Goiri.
(cherry picked from commit b6dae26f44)
2018-04-21 14:43:49 +08:00
Sangjin Lee 6becb3de0b
HDFS-10183. Prevent race condition during class initialization. Contributed by Pavel Avgustinov.
(cherry picked from commit f40969a141)
2018-04-20 21:10:39 -07:00
Inigo Goiri f8330d969d HDFS-13453. RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar.
(cherry picked from commit 1134af9ad1)
2018-04-19 14:57:33 -07:00
Xiao Chen 9f19c3b03e HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445.
(cherry picked from commit 6503593711)
2018-04-18 22:35:30 -05:00
Shashikant Banerjee 2c4b1c2b15
HADOOP-15396. Some java source files are executable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
2018-04-19 08:36:59 +09:00
Erik Krogen 1f8fd96eaf
HDFS-12828. OIV ReverseXML Processor fails with escaped characters
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 2d9e791a90)
2018-04-18 14:40:24 +09:00
Wei Yan 0124c7ffd6 HDFS-13466. RBF: Add more router-related information to the UI.
(cherry picked from commit e4313e7e47)
2018-04-17 15:04:05 -07:00
Inigo Goiri 282bbd56e8 HDFS-13462. Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak.
(cherry picked from commit c9b33514b8)
2018-04-17 14:29:20 -07:00
Yiqun Lin 24bdad925e HDFS-13435. RBF: Improve the error loggings for printing the stack trace.
(cherry picked from commit c4d3636c21)
2018-04-17 11:26:13 +08:00
Wei-Chiu Chuang dc01e323ed HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota.
(cherry picked from commit e66e287efe)
(cherry picked from commit ca8bb322be)
2018-04-13 12:20:01 -07:00
Akira Ajisaka 64105868e9
HDFS-13436. Fix javadoc of package-info.java
(cherry picked from commit 9b0b9f2af2)
2018-04-13 13:25:19 +09:00
Lei Xu 3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Akira Ajisaka 94daa4f3b3
HDFS-13427. Fix the section titles of transparent encryption document.
(cherry picked from commit c7cd362afd)
2018-04-12 10:46:40 +09:00
Inigo Goiri 5fe2b97dd0 HDFS-13386. RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar.
(cherry picked from commit 18de6f2042)
2018-04-11 12:03:57 -07:00
Inigo Goiri 670abc4121 HDFS-13428. RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR.
(cherry picked from commit 6a881cd842fb4eab596a386b39670ea03536891a)
2018-04-11 10:41:40 -07:00
Wei Yan 8433f6256c HDFS-13045. RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri. 2018-04-11 08:51:47 -07:00
Rakesh Radhakrishnan afbdd8fdcf HDFS-13328. Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore.
(cherry picked from commit f89594f0b8)
2018-04-10 23:43:28 +05:30
Inigo Goiri fe7a70e586 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun."
This reverts commit 877f963a05.
2018-04-10 10:08:34 -07:00
Weiwei Yang 1a0ce93696 HDFS-13420. License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka.
(cherry picked from commit 6729047a8b)
2018-04-10 22:16:25 +08:00
Yiqun Lin d4e438ff63 HDFS-13384. RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri.
(cherry picked from commit e87be8a2a4)
2018-04-10 15:38:18 +08:00
Yiqun Lin 0a216c3285 HDFS-13410. RBF: Support federation with no subclusters. Contributed by Inigo Goiri.
(cherry picked from commit a92200f4a6)
2018-04-10 14:34:25 +08:00
Inigo Goiri 877f963a05 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.
(cherry picked from commit ac32b3576d)
2018-04-09 09:18:03 -07:00
Yiqun Lin c3d8eda9bd HDFS-13402. RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu.
(cherry picked from commit 5700556cd6)
2018-04-08 12:05:32 +08:00
Manoj Govindassamy 218a21a0f0 HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem
(cherry picked from commit 133d7ca76e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
2018-04-06 15:19:01 -07:00
Arpit Agarwal 492a597942 HDFS-11968. ViewFS: StoragePolicies commands fail with HDFS federation. Contributed by Mukul Kumar Singh.
(cherry picked from commit b91305119b)
2018-04-06 15:19:01 -07:00
Wei Yan 9638c3fe67 HDFS-13353. RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma.
(cherry picked from commit 3121e8c293)
2018-04-05 13:19:05 -07:00
Lei Xu 9fe337096d HDFS-13350. Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu).
(cherry picked from commit 1cbf23df145af01692b8aaa438642b64e330cd05)
2018-04-05 09:54:15 -07:00
Yiqun Lin f2d89d7bf9 HDFS-13364. Addendum to fix branch-3.0 compilation. 2018-04-05 09:52:50 +08:00
Yiqun Lin 794ecb0284 HDFS-13364. RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri.
(cherry picked from commit 2be64eb201)
2018-04-03 15:26:02 +08:00
Xiao Chen ce37f586f5 HADOOP-15317. Improve NetworkTopology chooseRandom's loop.
(cherry picked from commit 57374c4737)
2018-04-02 22:49:03 -07:00
Inigo Goiri 8266696aa3 HDFS-13289. RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar.
(cherry picked from commit dc8e343201)
2018-03-31 09:47:06 -07:00
Konstantin V Shvachko 4ba5efa96f HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie.
(cherry picked from commit acfd764fcc)
2018-03-30 19:05:01 -07:00
Yiqun Lin 7483acf484
HDFS-13349. Unresolved merge conflict in ViewFs.md
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-30 23:28:59 +09:00
Xiao Chen cd74a281a7 HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge.
(cherry picked from commit 2c6cfad5a3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
2018-03-29 16:07:09 -07:00
Arpit Agarwal 198f2bc35e HDFS-13314. NameNode should optionally exit if it detects FsImage corruption. Contributed by Arpit Agarwal. 2018-03-28 12:49:09 -07:00
Yiqun Lin c903efeb51 HDFS-13347. RBF: Cache datanode reports. Contributed by Inigo Goiri.
(cherry picked from commit a71656c1c1)
2018-03-28 11:05:58 +08:00
Inigo Goiri 7cc06217de HDFS-13352. RBF: Add xsl stylesheet for hdfs-rbf-default.xml. Contributed by Takanobu Asanuma.
(cherry picked from commit 4cef01147f)
2018-03-27 10:32:56 -07:00
Inigo Goiri b8e82a4f66 HDFS-13204. RBF: Optimize name service safe mode icon. Contributed by liuhongtong.
(cherry picked from commit 947b282984)
2018-03-26 09:43:59 -07:00
Yiqun Lin fdcd139c55 HDFS-13291. RBF: Implement available space based OrderResolver. Contributed by Yiqun Lin.
(cherry picked from commit cfc3a1c8f0)
2018-03-26 18:37:40 +08:00
Lei Xu beb98bb419 Update 3.0.1 jdiff file and jdiff stable api version 2018-03-23 11:51:40 -07:00
Wei Yan f4d6fee96b HDFS-12512. RBF: Add WebHDFS.
(cherry picked from commit 6e31a09084)
2018-03-23 08:55:26 -07:00
Arpit Agarwal 26b08e002f
HDFS-12781. After Datanode down, In Namenode UI Datanode tab is throwing warning message.. Contributed by Brahma Reddy Battula.
(cherry picked from commit 46c93453e9)
2018-03-23 18:12:48 +09:00
Inigo Goiri 50b6dc9f3e HDFS-12792. RBF: Test Router-based federation using HDFSContract. Contributed by Inigo Goiri.
(cherry picked from commit e196d158a2)
2018-03-22 13:35:24 -07:00
Kihwal Lee 7212c98ba5 HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong.
(cherry picked from commit 66461ed98b)
2018-03-22 12:29:21 -05:00
Inigo Goiri 471f758673 HDFS-13318. RBF: Fix FindBugs in hadoop-hdfs-rbf. Contributed by Ekanth S.
(cherry picked from commit 3d0ee8b804)
2018-03-22 09:22:46 -07:00
Konstantin V Shvachko 987d90aa4c HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan.
(cherry picked from commit 8d898ab25f)
2018-03-21 17:17:41 -07:00
Yiqun Lin e65db99799 HDFS-13250. RBF: Router to manage requests across multiple subclusters. Contributed by Inigo Goiri.
(cherry picked from commit 2caba999bb)
2018-03-21 11:41:14 +08:00
weiy 62a819d5ee HDFS-13215. RBF: Move Router to its own module. 2018-03-19 23:15:53 -07:00
Inigo Goiri c6b1125a24 HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak.
(cherry picked from commit 08ff1586d5)
2018-03-16 10:30:34 -07:00
Inigo Goiri ba9a656ffa HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri.
(cherry picked from commit e71bc00a47)
2018-03-15 10:35:20 -07:00
Inigo Goiri 6b8a60b62a HDFS-12723. TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently. Contributed by Ajay Kumar.
(cherry picked from commit 6de135169e)
2018-03-15 10:15:46 -07:00
Inigo Goiri cfa7cc2aa4 HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router. Contributed by Wei Yan.
(cherry picked from commit 41fc7f80be)
2018-03-14 13:15:51 -07:00
Sammi Chen e97234f274 HDFS-11600. Refactor TestDFSStripedOutputStreamWithFailure test classes. Contributed by Sammi Chen.
(cherry picked from commit ad1b988a82)
2018-03-14 14:51:51 +08:00
Yiqun Lin a105069729 HDFS-12773. RBF: Improve State Store FS implementation. Contributed by Inigo Goiri.
(cherry picked from commit 76be6cbf6c)
2018-03-14 11:31:35 +08:00
Xiao Chen fc6f3e2a6f HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma.
(cherry picked from commit 8211a3d469)
2018-03-13 11:18:22 -07:00
Xiao Chen f23ef858ff Revert "HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma."
This reverts commit 752e116f41.
2018-03-13 10:39:50 -07:00
Xiao Chen c0aa249a1d HDFS-12587. Use Parameterized tests in TestBlockInfoStriped and TestLowRedundancyBlockQueues to test all EC policies. Contributed by Takanobu Asanuma.
(cherry picked from commit 3b8dbc2cb7)
2018-03-13 10:14:39 -07:00
Xiao Chen 8f4bfb3ed5 HDFS-13239. Fix non-empty dir warning message when setting default EC policy. Contributed by Bharat Viswanadham.
(cherry picked from commit d6931c30c5)
2018-03-13 10:08:09 -07:00
Xiao Chen 752e116f41 HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma.
(cherry picked from commit 84c1095586)
2018-03-13 09:58:50 -07:00
Xiaoyu Yao 2a93ba690c HDFS-12455. WebHDFS - Adding "snapshot enabled" status to ListStatus query result. Contributed by Ajay Kumar.
(cherry picked from commit 107c177782)
2018-03-12 20:23:52 -07:00
Yiqun Lin 840c83bf7a HDFS-13226. RBF: Throw the exception if mount table entry validated failed. Contributed by maobaolong. 2018-03-13 11:13:15 +08:00
Chris Douglas a03c8ea61f HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
2018-03-12 20:06:22 -07:00
Yiqun Lin 430cdfefc0 HDFS-13253. RBF: Quota management incorrect parent-child relationship judgement. Contributed by Yiqun Lin. 2018-03-13 10:41:44 +08:00
Akira Ajisaka 89105a623b
HDFS-12156. TestFSImage fails without -Pnative
(cherry picked from commit 319defafc1)
2018-03-13 11:27:46 +09:00
Inigo Goiri 177dd6580d HDFS-13241. RBF: TestRouterSafemode failed if the port 8888 is in use. Contributed by maobaolong.
(cherry picked from commit 91c82c90f0)
2018-03-12 17:30:00 -07:00
Chris Douglas 2210593dad HADOOP-14742. Document multi-URI replication Inode for ViewFS. Contributed by Gera Shegalov
(cherry picked from commit ddb67ca707)
(cherry picked from commit c9364b3bce)
2018-03-12 13:55:10 -07:00
Inigo Goiri 29cf29a853 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin.
(cherry picked from commit ac627f561f)
2018-03-12 10:05:43 -07:00
Yiqun Lin 2e607f2103 HDFS-13240. RBF: Update some inaccurate document descriptions. Contributed by Yiqun Lin.
(cherry picked from commit 4743d4a2c7)
2018-03-10 11:32:12 +08:00
Inigo Goiri 465c3f0e24 HDFS-13232. RBF: ConnectionPool should return first usable connection. Contributed by Ekanth S.
(cherry picked from commit 8133cd5305)
2018-03-09 18:27:17 -08:00
Inigo Goiri c453fcb23f HDFS-13212. RBF: Fix router location cache issue. Contributed by Weiwei Wu.
(cherry picked from commit afe1a3ccd5)
2018-03-09 17:20:29 -08:00
Hanisha Koneru bd2ebf031c HDFS-13244. Add stack, conf, metrics links to utilities dropdown in NN webUI. Contributed by Bharat Viswanadham.
(cherry picked from commit 4eeff62f69)
2018-03-09 15:49:09 -08:00
Lokesh Jain 80aabd72c2
HDFS-13190. Document WebHDFS support for snapshot diff
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Xiaoyu Yao <xyao@apache.org>
(cherry picked from commit 7b0dc31020)
2018-03-09 15:07:34 -08:00
Hanisha Koneru dec53bb9e1 HDFS-11394. Support for getting erasure coding policy through WebHDFS#FileStatus.
(cherry picked from commit 9a082fbe6e)
2018-03-09 13:39:49 -08:00
Yiqun Lin f879504fe1 HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan. 2018-03-09 15:48:36 +08:00
Inigo Goiri 68cff8c235 HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e01)
2018-03-08 09:33:24 -08:00
Hanisha Koneru 0b1e9665df HDFS-13225. StripeReader#checkMissingBlocks() 's IOException info is incomplete. Contributed by lufei.
(cherry picked from commit 19ae4429f9)
2018-03-07 14:22:01 -08:00
Yiqun Lin 253ef33f80 HDFS-13214. RBF: Complete document of Router configuration. Contributed by Yiqun Lin. 2018-03-07 16:07:05 +08:00
Xiaoyu Yao b74df5523c HDFS-13109. Support fully qualified hdfs path in EZ commands. Contributed by Hanisha Koneru.
(cherry picked from commit edf9445708)
2018-03-06 18:30:10 -08:00
Xiao Chen 283d48a4f7 HDFS-13170. Port webhdfs unmaskedpermission parameter to HTTPFS. Contributed by Stephen O'Donnell.
(cherry picked from commit 92c15e19272a8bb59f8704c44d4cd713d3a58bcb)
2018-03-06 09:58:09 -08:00
Xiao Chen d28db463c2 HDFS-13040. Kerberized inotify client fails despite kinit properly. Contributed by Istvan Fajth, Wei-Chiu Chuang, Xiao Chen.
(cherry picked from commit c75105f07b)
2018-03-02 16:47:36 -08:00
Xiaoyu Yao 908ca63c03 HDFS-13081. Datanode#checkSecureConfig should allow SASL and privileged HTTP. Contributed by Ajay Kumar.
(cherry picked from commit f20e10b2dd)
2018-02-28 09:48:12 -08:00
Xiaoyu Yao ce356dcbf6 HDFS-13114. CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path. Contributed by Hanisha Koneru.
(cherry picked from commit 90fbe5c6834a6685e65542fa69e396fad2678927)
2018-02-28 08:44:14 -08:00
Inigo Goiri 0eff245d3d HDFS-13199. RBF: Fix the hdfs router page missing label icon issue. Contributed by maobaolong.
(cherry picked from commit d86f301d46)
2018-02-27 17:41:42 -08:00
Yiqun Lin 09017aee90 HDFS-13184. RBF: Improve the unit test TestRouterRPCClientRetries. Contributed by Yiqun Lin.
(cherry picked from commit 1e85a995d1)
2018-02-27 10:52:49 +08:00
Chao Sun 26395aef67 HDFS-13145. SBN crash when transition to ANN with in-progress edit tailing enabled. Contributed by Chao Sun.
(cherry picked from commit ae290a4bb4)
2018-02-26 16:18:51 -08:00
Wei Yan 1fb87df87e HDFS-13187. RBF: Fix Routers information shown in the web UI. Contributed by Inigo Goiri and Wei Yan.
(cherry picked from commit 6ce9f79cc9)
2018-02-26 15:29:11 -08:00
Kihwal Lee 1087b9af8c HDFS-12070. Failed block recovery leaves files open indefinitely and at risk for data loss. Contributed by Kihwal Lee.
(cherry picked from commit 451265a83d)
2018-02-26 10:30:50 -06:00