Commit Graph

6970 Commits

Author SHA1 Message Date
Hrishikesh Gadre fde3b5ac22 HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre.
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d)
2018-09-04 21:54:43 -07:00
Vinayakumar B 9cf35d99b9 HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena.
(cherry picked from commit 3801436e49)
2018-09-03 14:39:11 +05:30
Wei-Chiu Chuang 9f0cd2ade2 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
This reverts commit 41732e7b28.
2018-08-30 11:58:42 -07:00
Vinayakumar B 65977e5d81 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 2018-08-30 22:14:47 +05:30
Yiqun Lin 8a3be0d5a4 HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui.
(cherry picked from commit 582cb10ec7)
2018-08-30 11:22:57 +08:00
Zsolt Venczel 5cbb9b1ca9 HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel.
(cherry picked from commit 3e18b957eb)
2018-08-28 15:14:13 -07:00
Yiqun Lin 2cbc3c7d43 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu.
(cherry picked from commit b9b964d253)
2018-08-28 09:36:02 -07:00
Wei-Chiu Chuang 41732e7b28 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
(cherry picked from commit 26c2a97c56)
2018-08-27 16:03:52 -07:00
Vinayakumar B 8df2eb8119 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:34:57 +05:30
Wei-Chiu Chuang 865650052b HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-20 14:57:40 -07:00
Xiao Chen 0424715207 HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-20 14:57:07 -07:00
Xiao Chen 81fbfe500e HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin. 2018-08-20 14:56:47 -07:00
Daniel Templeton c15853f87a HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
2018-08-20 14:54:58 -07:00
drankye 975d60685e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-20 14:53:32 -07:00
Xiao Chen c0ac0a5337 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-20 14:52:28 -07:00
Márton Elek 366517ba99 HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr. 2018-08-20 14:52:04 -07:00
Vinayakumar B a517ee4f5b HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena 2018-08-20 14:51:53 -07:00
Xiao Chen 754069906b HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization
between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
2018-08-20 14:51:07 -07:00
Wei-Chiu Chuang 78fb14ba49 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:24 -07:00
Xiao Chen 06f0d5e257 HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
(cherry picked from commit c67b0650ea)
2018-08-16 23:15:48 -07:00
Daniel Templeton fbedf89377 HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
(cherry picked from commit 8512e1a91b)
2018-08-16 15:01:34 -07:00
Xiao Chen 83d0d82a69 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi.
(cherry picked from commit cede33997f)
2018-08-14 11:57:50 -07:00
Wei-Chiu Chuang 7a115e8de4 HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan.
(cherry picked from commit 61a9b4f58b)
2018-08-14 11:52:24 -07:00
Wei-Chiu Chuang 74d4fd975a HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh.
(cherry picked from commit 4023eeba05)
2018-08-13 17:37:12 -07:00
Wei-Chiu Chuang b38649c59a HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng.
(cherry picked from commit 23854443ef)
2018-08-13 16:13:36 -07:00
Vinayakumar B b3f29e3504 HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena.
(cherry picked from commit 2b0f977241)
2018-08-13 13:58:27 -07:00
Chen Liang e05b38f132 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:48:12 -07:00
Arpit Agarwal 4e197a4b46 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:36:15 -07:00
Chao Sun baca64cc68 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun.
(cherry picked from commit a2a8c48699)
2018-08-10 16:07:02 -07:00
Xiao Chen 7d1e2e1fbb HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer. 2018-08-10 13:53:34 -07:00
Wangda Tan 781620a44d Make 3.1.1 awared by other branches
(cherry picked from commit 49c687608b)
2018-08-08 13:04:05 -07:00
Xiao Chen bf03b25f4b HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell.
(cherry picked from commit 6677717c68)
2018-08-07 22:05:51 -07:00
Wangda Tan 5583711419 Preparing for 3.1.2 release
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
2018-07-31 13:08:55 -07:00
Brahma Reddy Battula 6f6d9ee9bf HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani
(cherry picked from commit 3108d27edd)
2018-07-30 15:57:58 +05:30
Arpit Agarwal 0eb187f72b HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:20 -07:00
Anu Engineer 2ebd553072 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception.
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
2018-07-27 09:43:44 -07:00
Wei-Chiu Chuang 964f3454d1 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin.
(cherry picked from commit e95c5e9f62)
2018-07-25 11:05:18 -07:00
Daniel Templeton 00c476abd8 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
2018-07-24 16:12:43 -07:00
Konstantin V Shvachko 53c7d82d53 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal.
(cherry picked from commit 0a1e922f3d)
2018-07-16 18:29:43 -07:00
Wei-Chiu Chuang ac9155d6cb HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng.
(cherry picked from commit 88b2794244)
2018-07-16 10:52:56 -07:00
Xiao Chen d5d987550a HDFS-12837. Intermittent failure in TestReencryptionWithKMS.
(cherry picked from commit b37074be5a)
2018-07-11 21:03:45 -07:00
Akira Ajisaka fe256a98ff
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota.
(cherry picked from commit 418cc7f3ae)
2018-07-11 14:47:21 -04:00
Wei-Chiu Chuang f9fa3cb157 HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng.
(cherry picked from commit 162228e8db)
2018-07-11 10:04:07 -07:00
Andrew Wang 242b5acdb3 HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi.
(cherry picked from commit eecb5baaaa)
2018-07-09 15:17:34 +02:00
Wei-Chiu Chuang f5f4d0b7e7 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
(cherry picked from commit 0247cb6318)
2018-07-06 15:01:10 -07:00
Bharat Viswanadham 0d7b811ee6 HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham
(cherry picked from commit 59a3038bc3)
2018-07-02 21:48:09 -07:00
Inigo Goiri 177ccbc43f HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 7296b644f7)
2018-07-02 16:37:12 -07:00
Andrew Wang 2d2639beaa HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
(cherry picked from commit 5d748bd056)
2018-07-02 12:11:11 +02:00
Andrew Wang 0268b954e0 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
(cherry picked from commit 6ba9974108)
2018-07-02 12:02:48 +02:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Yiqun Lin 6b5b58a495 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham.
(cherry picked from commit 30728aced4)
2018-06-22 10:53:33 +08:00
Xiao Chen e8655b8c72 HDFS-13682. Cannot create encryption zone after KMS auth token expires.
(cherry picked from commit 32f867a6a9)
2018-06-20 15:59:23 -07:00
Inigo Goiri 88e8a3e5bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang.
(cherry picked from commit 8762e9cf10)
2018-06-15 16:49:51 -07:00
Inigo Goiri 7c06be3912 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak.
(cherry picked from commit d31a3ce767)
2018-06-15 15:59:49 -07:00
Wei-Chiu Chuang 975d4b3d60 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth.
(cherry picked from commit c966a3837a)
2018-06-15 13:37:19 -07:00
Inigo Goiri 271ea7a3d9 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit eebeb6033f)
2018-06-15 13:08:24 -07:00
Inigo Goiri 4add4c1be0 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang.
(cherry picked from commit 43d994e4a6)
2018-06-15 10:34:08 -07:00
Inigo Goiri fd7cd596b7 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak.
(cherry picked from commit 5d7449d2b8)
2018-06-14 13:43:51 -07:00
Inigo Goiri e45541be51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak.
(cherry picked from commit 8d4926f38b)
2018-06-14 09:59:28 -07:00
Yiqun Lin 5c1248b2da HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun.
(cherry picked from commit 8e7548d33b)
2018-06-13 20:10:46 +08:00
Yongjun Zhang 425fe4e217 Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release.
(cherry picked from commit 5a82f10e32)
2018-06-11 14:31:46 -07:00
Inigo Goiri f0150f024f HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
2018-06-11 11:17:37 -07:00
Bharat Viswanadham c0d46a84a6 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar
(cherry picked from commit ccfb816d39)
2018-06-09 16:43:58 -07:00
Arpit Agarwal 33b6a68c81 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:30 -07:00
Wei-Chiu Chuang b2e919dc1a HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang.
(cherry picked from commit e39b113db0)
2018-06-07 08:31:30 -07:00
Brahma Reddy Battula 630f43faf7 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 9654dd1f47)
2018-06-07 10:21:00 +05:30
Akira Ajisaka f358a77610
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia.
(cherry picked from commit 0e3c31579d)
2018-06-05 17:55:01 +09:00
Inigo Goiri ec5ef0bb20 HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 8d31ddcfeb)
2018-06-04 18:28:40 -07:00
Inigo Goiri 19b37e7080 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu.
(cherry picked from commit 16316b6011)
2018-06-04 18:22:09 -07:00
Inigo Goiri 76546d23e2 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu.
(cherry picked from commit a28848104f)
2018-06-04 18:13:14 -07:00
Rushabh Shah 6646b16450 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah
(cherry picked from commit e2289c8d14)
2018-06-04 09:21:14 -05:00
Wei-Chiu Chuang 302c36ffdc HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel.
(cherry picked from commit bccdfeee0a)
2018-06-04 07:03:35 -07:00
Xiao Chen fde45994a3 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel.
(cherry picked from commit 9efb4b7db0)
2018-06-03 22:37:27 -07:00
Inigo Goiri 90060d52f6 HDFS-13648. Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit 8f83b9abf9)
2018-06-02 20:20:26 -07:00
Inigo Goiri 8a9c2d7b97 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit e308ac88d3)
2018-06-02 20:13:21 -07:00
Sean Mackrory 58b5b3aa75 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
(cherry picked from commit 1361030e59)
2018-05-31 22:23:19 -06:00
Chao Sun 2bbdf362cf HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun.
(cherry picked from commit ff013d2c95)
2018-05-31 17:39:46 -07:00
Konstantin V Shvachko 442dd87dcd HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 14:58:06 -07:00
Inigo Goiri ad11e40557 HDFS-13631. TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu.
(cherry picked from commit 1f324e9661)
2018-05-31 10:13:14 -07:00
Inigo Goiri 5c09dadd73 HDFS-13629. Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu.
(cherry picked from commit 47c31ff16b)
2018-05-30 10:23:00 -07:00
Inigo Goiri 6dc40bab4e HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu.
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
2018-05-30 10:02:50 -07:00
Inigo Goiri bc80290f28 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9dbf4f0166)
2018-05-28 16:34:31 -07:00
Inigo Goiri cc696f80d3 HDFS-13627. TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 91d7c74e6a)
2018-05-28 10:27:23 -07:00
Takanobu Asanuma 499a90f152
HDFS-13628. Update Archival Storage doc for Provided Storage
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 04757e5864)
2018-05-28 19:06:58 +09:00
Inigo Goiri c7d77a3bfe HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu.
(cherry picked from commit 8605a38514)
2018-05-25 19:45:48 -07:00
Inigo Goiri 71ab85adce HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 13d2528907)
2018-05-25 13:33:05 -07:00
Inigo Goiri 2d3d7a67f2 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
2018-05-25 09:11:07 -07:00
Inigo Goiri f57e91a348 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen.
(cherry picked from commit cddbbe5f69)
2018-05-23 12:12:40 -07:00
Inigo Goiri a138496379 HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit c13dea87d9)
2018-05-23 11:36:34 -07:00
Bharat Viswanadham d0d3d5b2ad HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
(cherry picked from commit bc6d9d4c79)
2018-05-23 10:23:45 -07:00
Inigo Goiri 0c268a0755 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang.
(cherry picked from commit c0c9b7a8ef)
2018-05-23 09:47:10 -07:00
Sammi Chen 7d71b3a1cc HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
(cherry picked from commit 34e8b9f9a8)
2018-05-23 19:12:04 +08:00
Andrew Wang d47c0fc39a HDFS-13601. Optimize ByteString conversions in PBHelper.
(cherry picked from commit 1d2640b613)
2018-05-22 23:55:35 -07:00
Inigo Goiri 8a86269c51 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu.
(cherry picked from commit 9775ecb235)
2018-05-18 09:46:29 -07:00
Inigo Goiri da5eff8325 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu.
(cherry picked from commit 57b893de3d)
2018-05-18 09:38:23 -07:00
Inigo Goiri df54cdbaea HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.
(cherry picked from commit 53b807a6a8)
2018-05-17 17:03:56 -07:00
Inigo Goiri f67865ed7e HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit a97a2042f2)
2018-05-17 16:53:53 -07:00
Inigo Goiri 8c463a2a9c HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 65476458fa)
2018-05-17 14:48:40 -07:00
Inigo Goiri e5ff9f3922 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu.
(cherry picked from commit 26f1e22fc9)
2018-05-17 13:35:40 -07:00
Inigo Goiri b0c89690ed HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu.
(cherry picked from commit c0ec061e28)
2018-05-17 09:22:29 -07:00