Commit Graph

7149 Commits

Author SHA1 Message Date
Chen Liang e05b38f132 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:48:12 -07:00
Arpit Agarwal 4e197a4b46 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:36:15 -07:00
Chao Sun baca64cc68 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun.
(cherry picked from commit a2a8c48699)
2018-08-10 16:07:02 -07:00
Xiao Chen 7d1e2e1fbb HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer. 2018-08-10 13:53:34 -07:00
Wangda Tan 781620a44d Make 3.1.1 awared by other branches
(cherry picked from commit 49c687608b)
2018-08-08 13:04:05 -07:00
Xiao Chen bf03b25f4b HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell.
(cherry picked from commit 6677717c68)
2018-08-07 22:05:51 -07:00
Wangda Tan 5583711419 Preparing for 3.1.2 release
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
2018-07-31 13:08:55 -07:00
Brahma Reddy Battula 6f6d9ee9bf HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani
(cherry picked from commit 3108d27edd)
2018-07-30 15:57:58 +05:30
Arpit Agarwal 0eb187f72b HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:20 -07:00
Anu Engineer 2ebd553072 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception.
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
2018-07-27 09:43:44 -07:00
Wei-Chiu Chuang 964f3454d1 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin.
(cherry picked from commit e95c5e9f62)
2018-07-25 11:05:18 -07:00
Daniel Templeton 00c476abd8 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
2018-07-24 16:12:43 -07:00
Yiqun Lin 5aca0588ea HDFS-13583. RBF: Router admin clrQuota is not synchronized with nameservice. Contributed by Dibyendu Karmakar.
(cherry picked from commit 17a87977f2)
2018-07-24 11:19:21 +08:00
Yiqun Lin 823d576a66 HDFS-13743. RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver. Contributed by Takanobu Asanuma.
(cherry picked from commit 7b25fb949b)
2018-07-20 17:31:13 +08:00
Yiqun Lin 228508a0ee HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF document. Contributed by Takanobu Asanuma.
(cherry picked from commit 1af87df242)
2018-07-17 10:47:07 +08:00
Konstantin V Shvachko 53c7d82d53 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal.
(cherry picked from commit 0a1e922f3d)
2018-07-16 18:29:43 -07:00
Wei-Chiu Chuang ac9155d6cb HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng.
(cherry picked from commit 88b2794244)
2018-07-16 10:52:56 -07:00
Inigo Goiri 4898edf4f7 HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed by Chao Sun.
(cherry picked from commit 359ea4e181)
2018-07-16 09:47:00 -07:00
Xiao Chen d5d987550a HDFS-12837. Intermittent failure in TestReencryptionWithKMS.
(cherry picked from commit b37074be5a)
2018-07-11 21:03:45 -07:00
Akira Ajisaka fe256a98ff
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota.
(cherry picked from commit 418cc7f3ae)
2018-07-11 14:47:21 -04:00
Wei-Chiu Chuang f9fa3cb157 HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng.
(cherry picked from commit 162228e8db)
2018-07-11 10:04:07 -07:00
Yiqun Lin 99e98bf19a HDFS-13726. RBF: Fix RBF configuration links. Contributed by Takanobu Asanuma.
(cherry picked from commit 2ae13d41dc)
2018-07-11 22:15:07 +08:00
Andrew Wang 242b5acdb3 HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi.
(cherry picked from commit eecb5baaaa)
2018-07-09 15:17:34 +02:00
Yiqun Lin e994c4f8aa HDFS-13710. RBF: setQuota and getQuotaUsage should check the dfs.federation.router.quota.enable. Contributed by yanghuafeng.
(cherry picked from commit 43f7fe8aae)
2018-07-09 15:07:47 +08:00
Wei-Chiu Chuang f5f4d0b7e7 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
(cherry picked from commit 0247cb6318)
2018-07-06 15:01:10 -07:00
Yiqun Lin 48addcdbbe HDFS-13528. RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries. Contributed by Dibyendu Karmakar.
(cherry picked from commit 3b637155a4)
2018-07-04 15:30:17 +08:00
Andrew Wang 3b266abdad HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
(cherry picked from commit 344f324710)
2018-07-03 11:07:57 +02:00
Bharat Viswanadham 0d7b811ee6 HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham
(cherry picked from commit 59a3038bc3)
2018-07-02 21:48:09 -07:00
Inigo Goiri 177ccbc43f HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 7296b644f7)
2018-07-02 16:37:12 -07:00
Wei-Chiu Chuang 2ce892fcf5 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
(cherry picked from commit fef20a446f)
2018-07-02 09:43:44 -07:00
Andrew Wang 2d2639beaa HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
(cherry picked from commit 5d748bd056)
2018-07-02 12:11:11 +02:00
Andrew Wang 0268b954e0 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
(cherry picked from commit 6ba9974108)
2018-07-02 12:02:48 +02:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Yiqun Lin 6b5b58a495 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.
(cherry picked from commit 30728aced4)
2018-06-22 10:53:33 +08:00
Xiao Chen e8655b8c72 HDFS-13682. Cannot create encryption zone after KMS auth token expires.
(cherry picked from commit 32f867a6a9)
2018-06-20 15:59:23 -07:00
Inigo Goiri 88e8a3e5bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.
(cherry picked from commit 8762e9cf10)
2018-06-15 16:49:51 -07:00
Inigo Goiri 7c06be3912 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.
(cherry picked from commit d31a3ce767)
2018-06-15 15:59:49 -07:00
Wei-Chiu Chuang 975d4b3d60 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.
(cherry picked from commit c966a3837a)
2018-06-15 13:37:19 -07:00
Inigo Goiri 271ea7a3d9 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit eebeb6033f)
2018-06-15 13:08:24 -07:00
Inigo Goiri 4add4c1be0 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit 43d994e4a6)
2018-06-15 10:34:08 -07:00
Inigo Goiri fd7cd596b7 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.
(cherry picked from commit 5d7449d2b8)
2018-06-14 13:43:51 -07:00
Inigo Goiri e45541be51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 8d4926f38b)
2018-06-14 09:59:28 -07:00
Yiqun Lin 5c1248b2da HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.
(cherry picked from commit 8e7548d33b)
2018-06-13 20:10:46 +08:00
Xiao Chen 3b646a9836 HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.
(cherry picked from commit 774c1f199e)
2018-06-12 11:17:54 -07:00
Rohith Sharma K S af0309d793 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri.
(cherry picked from commit 2df73dace0)
2018-06-12 17:23:22 +05:30
Yongjun Zhang 425fe4e217 Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release.
(cherry picked from commit 5a82f10e32)
2018-06-11 14:31:46 -07:00
Inigo Goiri f0150f024f HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
2018-06-11 11:17:37 -07:00
Bharat Viswanadham c0d46a84a6 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar
(cherry picked from commit ccfb816d39)
2018-06-09 16:43:58 -07:00
Arpit Agarwal 33b6a68c81 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:30 -07:00
Chao Sun d827ffe245 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
(cherry picked from commit fba1c42adc)
2018-06-08 16:40:13 -07:00
Wei-Chiu Chuang b2e919dc1a HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.
(cherry picked from commit e39b113db0)
2018-06-07 08:31:30 -07:00
Brahma Reddy Battula 630f43faf7 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 9654dd1f47)
2018-06-07 10:21:00 +05:30
Xiao Chen f1c91e9fda HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
(cherry picked from commit ba4011d64f)
2018-06-05 21:19:11 -07:00
Akira Ajisaka f358a77610
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.
(cherry picked from commit 0e3c31579d)
2018-06-05 17:55:01 +09:00
Inigo Goiri ec5ef0bb20 HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 8d31ddcfeb)
2018-06-04 18:28:40 -07:00
Inigo Goiri 19b37e7080 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 16316b6011)
2018-06-04 18:22:09 -07:00
Inigo Goiri 76546d23e2 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.
(cherry picked from commit a28848104f)
2018-06-04 18:13:14 -07:00
Rushabh Shah 6646b16450 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah
(cherry picked from commit e2289c8d14)
2018-06-04 09:21:14 -05:00
Wei-Chiu Chuang 302c36ffdc HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.
(cherry picked from commit bccdfeee0a)
2018-06-04 07:03:35 -07:00
Xiao Chen fde45994a3 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.
(cherry picked from commit 9efb4b7db0)
2018-06-03 22:37:27 -07:00
Inigo Goiri 90060d52f6 HDFS-13648. Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit 8f83b9abf9)
2018-06-02 20:20:26 -07:00
Inigo Goiri 8a9c2d7b97 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit e308ac88d3)
2018-06-02 20:13:21 -07:00
Inigo Goiri fdacc8a088 HDFS-13637. RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota.
(cherry picked from commit e11d674049)
2018-06-01 16:59:35 -07:00
Sean Mackrory 58b5b3aa75 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
(cherry picked from commit 1361030e59)
2018-05-31 22:23:19 -06:00
Chao Sun 2bbdf362cf HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.
(cherry picked from commit ff013d2c95)
2018-05-31 17:39:46 -07:00
Konstantin V Shvachko 442dd87dcd HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 14:58:06 -07:00
Inigo Goiri ad11e40557 HDFS-13631. TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.
(cherry picked from commit 1f324e9661)
2018-05-31 10:13:14 -07:00
Inigo Goiri 5c09dadd73 HDFS-13629. Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.
(cherry picked from commit 47c31ff16b)
2018-05-30 10:23:00 -07:00
Inigo Goiri 6dc40bab4e HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
2018-05-30 10:02:50 -07:00
Inigo Goiri bc80290f28 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9dbf4f0166)
2018-05-28 16:34:31 -07:00
Inigo Goiri cc696f80d3 HDFS-13627. TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 91d7c74e6a)
2018-05-28 10:27:23 -07:00
Takanobu Asanuma 499a90f152
HDFS-13628. Update Archival Storage doc for Provided Storage
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 04757e5864)
2018-05-28 19:06:58 +09:00
Inigo Goiri c7d77a3bfe HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.
(cherry picked from commit 8605a38514)
2018-05-25 19:45:48 -07:00
Inigo Goiri 71ab85adce HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 13d2528907)
2018-05-25 13:33:05 -07:00
Inigo Goiri 2d3d7a67f2 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
2018-05-25 09:11:07 -07:00
Andrew Wang 31d597f0c7 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
(cherry picked from commit c9b63deb53)
2018-05-24 09:56:37 -07:00
Inigo Goiri f57e91a348 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.
(cherry picked from commit cddbbe5f69)
2018-05-23 12:12:40 -07:00
Inigo Goiri a138496379 HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit c13dea87d9)
2018-05-23 11:36:34 -07:00
Bharat Viswanadham d0d3d5b2ad HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
(cherry picked from commit bc6d9d4c79)
2018-05-23 10:23:45 -07:00
Inigo Goiri 0c268a0755 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit c0c9b7a8ef)
2018-05-23 09:47:10 -07:00
Sammi Chen 7d71b3a1cc HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
(cherry picked from commit 34e8b9f9a8)
2018-05-23 19:12:04 +08:00
Andrew Wang d47c0fc39a HDFS-13601. Optimize ByteString conversions in PBHelper.
(cherry picked from commit 1d2640b613)
2018-05-22 23:55:35 -07:00
Inigo Goiri 8a86269c51 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9775ecb235)
2018-05-18 09:46:29 -07:00
Inigo Goiri da5eff8325 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.
(cherry picked from commit 57b893de3d)
2018-05-18 09:38:23 -07:00
Inigo Goiri df54cdbaea HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.
(cherry picked from commit 53b807a6a8)
2018-05-17 17:03:56 -07:00
Inigo Goiri f67865ed7e HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit a97a2042f2)
2018-05-17 16:53:53 -07:00
Inigo Goiri 8c463a2a9c HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 65476458fa)
2018-05-17 14:48:40 -07:00
Inigo Goiri e5ff9f3922 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 26f1e22fc9)
2018-05-17 13:35:40 -07:00
Inigo Goiri b0c89690ed HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.
(cherry picked from commit c0ec061e28)
2018-05-17 09:22:29 -07:00
Kihwal Lee 4f1ec6a515 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee.
(cherry picked from commit 0ce6290de6)
2018-05-17 09:26:13 -05:00
Inigo Goiri 1d1806f862 HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu.
(cherry picked from commit 454de3b543)
2018-05-16 19:23:33 -07:00
Arpit Agarwal 45dd200a2c HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar. 2018-05-16 11:28:51 -07:00
Inigo Goiri e12ddd9394 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 55d5549087)
2018-05-16 10:09:18 -07:00
Inigo Goiri 5310542fb4 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 6b67161ca3)
2018-05-16 09:32:15 -07:00
Inigo Goiri 899a7ec2fb HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu.
(cherry picked from commit 63480976a0)
2018-05-15 15:36:03 -07:00
Inigo Goiri 34fbc1c2eb HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 2d6195c0af)
2018-05-15 11:21:44 -07:00
Inigo Goiri c168329cc4 HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu.
(cherry picked from commit eaca7980ad)
2018-05-15 10:28:01 -07:00
Inigo Goiri 583b9b418a HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 92ebd466c7)
2018-05-15 10:22:19 -07:00
Hanisha Koneru 4f55941390 HDFS-13544. Improve logging for JournalNode in federated cluster.
(cherry picked from commit 6beb25ab7e)
2018-05-14 10:14:38 -07:00
Xiao Chen 6e731eb20d HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure.
(cherry picked from commit 960940e0e0)
2018-05-14 09:29:14 -07:00