Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,226 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7095 Commits

Author SHA1 Message Date
Arpit Agarwal 8b4f9b3e2c HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd)
 2018-10-25 15:14:48 -07:00
Wei-Chiu Chuang ef9f8ca13d HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
(cherry picked from commit fd02c501c0)
 2018-10-25 09:37:31 -07:00
Inigo Goiri 4b1f2ecd4c HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd)
 2018-10-24 16:14:14 -07:00
Xiao Chen 96cedb87b9 Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
 2018-10-24 16:10:25 -07:00
Inigo Goiri 2e43966b56 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2)
 2018-10-24 09:38:40 -07:00
Arpit Agarwal 8548864ccb HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d)
 2018-10-22 14:44:58 -07:00
Xiao Chen d1749fbaae HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a025)
 2018-10-15 11:00:30 -07:00
Vinayakumar B bed2f407a9 HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 6e0e6daaf3)
 2018-10-12 17:36:03 +05:30
Hrishikesh Gadre 323b76bccf HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-09 19:54:34 -07:00
Yiqun Lin 665036c5f7 HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 1043795f7f)
 2018-10-09 10:36:45 +08:00
Yiqun Lin dd70adf318 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. 
(cherry picked from commit 619e490333)
 2018-10-05 10:08:28 +08:00
Yiqun Lin 65af98b58a HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 5689355783)
 2018-10-02 09:46:23 +08:00
Brahma Reddy Battula 02f8b5da47 HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10)
 2018-09-26 23:51:59 +05:30
Chen Liang b4069343bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:28:35 -07:00
Arpit Agarwal 58f339e491 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b151)
 2018-09-19 13:49:06 -07:00
Inigo Goiri 2f7222a27b HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:16 -07:00
Inigo Goiri e1af3c9bb7 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0)
 2018-09-19 09:31:18 -07:00
Wei-Chiu Chuang 7442b7941d HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a72)
 2018-09-19 03:12:55 -07:00
Mingliang Liu bff2230591 HDFS-11719. Arrays.fill() wrong index in BlockSender.readChecksum() exception handling. Contributed by Tao Zhang 2018-09-18 10:40:16 -07:00
Inigo Goiri 508fd55c79 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:38:08 -07:00
Wei-Chiu Chuang 829399a9f3 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5)
 2018-09-14 05:24:24 +08:00
Inigo Goiri 2f7d226589 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a984)
 2018-09-13 09:17:37 -07:00
Brahma Reddy Battula 74c00641f9 HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 
(cherry picked from commit b2432d254c)
 2018-09-12 06:34:35 +05:30
Xiao Chen 2dd27c999b HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 
(cherry picked from commit 8e54da1511)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2018-09-10 22:44:48 -07:00
Vinayakumar B b8eaeca7e6 HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena. 
(cherry picked from commit eef3bafae8)
 2018-09-10 09:24:08 +05:30
Hrishikesh Gadre fde3b5ac22 HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d)
 2018-09-04 21:54:43 -07:00
Vinayakumar B 9cf35d99b9 HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena. 
(cherry picked from commit 3801436e49)
 2018-09-03 14:39:11 +05:30
Wei-Chiu Chuang 9f0cd2ade2 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 41732e7b28.
 2018-08-30 11:58:42 -07:00
Vinayakumar B 65977e5d81 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 2018-08-30 22:14:47 +05:30
Yiqun Lin 8a3be0d5a4 HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 
(cherry picked from commit 582cb10ec7)
 2018-08-30 11:22:57 +08:00
Zsolt Venczel 5cbb9b1ca9 HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 
(cherry picked from commit 3e18b957eb)
 2018-08-28 15:14:13 -07:00
Yiqun Lin 2cbc3c7d43 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 
(cherry picked from commit b9b964d253)
 2018-08-28 09:36:02 -07:00
Wei-Chiu Chuang 41732e7b28 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit 26c2a97c56)
 2018-08-27 16:03:52 -07:00
Vinayakumar B 8df2eb8119 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:34:57 +05:30
Wei-Chiu Chuang 865650052b HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-20 14:57:40 -07:00
Xiao Chen 0424715207 HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-20 14:57:07 -07:00
Xiao Chen 81fbfe500e HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin. 2018-08-20 14:56:47 -07:00
Daniel Templeton c15853f87a HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky 
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
 2018-08-20 14:54:58 -07:00
drankye 975d60685e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-20 14:53:32 -07:00
Xiao Chen c0ac0a5337 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-20 14:52:28 -07:00
Márton Elek 366517ba99 HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr. 2018-08-20 14:52:04 -07:00
Vinayakumar B a517ee4f5b HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena 2018-08-20 14:51:53 -07:00
Xiao Chen 754069906b HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization 
between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
 2018-08-20 14:51:07 -07:00
Wei-Chiu Chuang 78fb14ba49 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:24 -07:00
Xiao Chen 06f0d5e257 HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 
(cherry picked from commit c67b0650ea)
 2018-08-16 23:15:48 -07:00
Daniel Templeton fbedf89377 HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
(cherry picked from commit 8512e1a91b)
 2018-08-16 15:01:34 -07:00
Xiao Chen 83d0d82a69 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 
(cherry picked from commit cede33997f)
 2018-08-14 11:57:50 -07:00
Wei-Chiu Chuang 7a115e8de4 HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 
(cherry picked from commit 61a9b4f58b)
 2018-08-14 11:52:24 -07:00
Wei-Chiu Chuang 74d4fd975a HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 
(cherry picked from commit 4023eeba05)
 2018-08-13 17:37:12 -07:00
Wei-Chiu Chuang b38649c59a HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 
(cherry picked from commit 23854443ef)
 2018-08-13 16:13:36 -07:00
Vinayakumar B b3f29e3504 HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena. 
(cherry picked from commit 2b0f977241)
 2018-08-13 13:58:27 -07:00
Chen Liang e05b38f132 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:48:12 -07:00
Arpit Agarwal 4e197a4b46 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:36:15 -07:00
Chao Sun baca64cc68 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 
(cherry picked from commit a2a8c48699)
 2018-08-10 16:07:02 -07:00
Xiao Chen 7d1e2e1fbb HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer. 2018-08-10 13:53:34 -07:00
Wangda Tan 781620a44d Make 3.1.1 awared by other branches 
(cherry picked from commit 49c687608b)
 2018-08-08 13:04:05 -07:00
Xiao Chen bf03b25f4b HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell. 
(cherry picked from commit 6677717c68)
 2018-08-07 22:05:51 -07:00
Wangda Tan 5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Brahma Reddy Battula 6f6d9ee9bf HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani 
(cherry picked from commit 3108d27edd)
 2018-07-30 15:57:58 +05:30
Arpit Agarwal 0eb187f72b HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:20 -07:00
Anu Engineer 2ebd553072 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception. 
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
 2018-07-27 09:43:44 -07:00
Wei-Chiu Chuang 964f3454d1 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 
(cherry picked from commit e95c5e9f62)
 2018-07-25 11:05:18 -07:00
Daniel Templeton 00c476abd8 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica 
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
 2018-07-24 16:12:43 -07:00
Konstantin V Shvachko 53c7d82d53 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal. 
(cherry picked from commit 0a1e922f3d)
 2018-07-16 18:29:43 -07:00
Wei-Chiu Chuang ac9155d6cb HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng. 
(cherry picked from commit 88b2794244)
 2018-07-16 10:52:56 -07:00
Xiao Chen d5d987550a HDFS-12837. Intermittent failure in TestReencryptionWithKMS. 
(cherry picked from commit b37074be5a)
 2018-07-11 21:03:45 -07:00
Akira Ajisaka fe256a98ff
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota. 
(cherry picked from commit 418cc7f3ae)
 2018-07-11 14:47:21 -04:00
Wei-Chiu Chuang f9fa3cb157 HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng. 
(cherry picked from commit 162228e8db)
 2018-07-11 10:04:07 -07:00
Andrew Wang 242b5acdb3 HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 
(cherry picked from commit eecb5baaaa)
 2018-07-09 15:17:34 +02:00
Wei-Chiu Chuang f5f4d0b7e7 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel. 
(cherry picked from commit 0247cb6318)
 2018-07-06 15:01:10 -07:00
Bharat Viswanadham 0d7b811ee6 HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham 
(cherry picked from commit 59a3038bc3)
 2018-07-02 21:48:09 -07:00
Inigo Goiri 177ccbc43f HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 7296b644f7)
 2018-07-02 16:37:12 -07:00
Andrew Wang 2d2639beaa HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon. 
(cherry picked from commit 5d748bd056)
 2018-07-02 12:11:11 +02:00
Andrew Wang 0268b954e0 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon. 
(cherry picked from commit 6ba9974108)
 2018-07-02 12:02:48 +02:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Yiqun Lin 6b5b58a495 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham. 
(cherry picked from commit 30728aced4)
 2018-06-22 10:53:33 +08:00
Xiao Chen e8655b8c72 HDFS-13682. Cannot create encryption zone after KMS auth token expires. 
(cherry picked from commit 32f867a6a9)
 2018-06-20 15:59:23 -07:00
Inigo Goiri 88e8a3e5bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 8762e9cf10)
 2018-06-15 16:49:51 -07:00
Inigo Goiri 7c06be3912 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak. 
(cherry picked from commit d31a3ce767)
 2018-06-15 15:59:49 -07:00
Wei-Chiu Chuang 975d4b3d60 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth. 
(cherry picked from commit c966a3837a)
 2018-06-15 13:37:19 -07:00
Inigo Goiri 271ea7a3d9 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit eebeb6033f)
 2018-06-15 13:08:24 -07:00
Inigo Goiri 4add4c1be0 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit 43d994e4a6)
 2018-06-15 10:34:08 -07:00
Inigo Goiri fd7cd596b7 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak. 
(cherry picked from commit 5d7449d2b8)
 2018-06-14 13:43:51 -07:00
Inigo Goiri e45541be51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 8d4926f38b)
 2018-06-14 09:59:28 -07:00
Yiqun Lin 5c1248b2da HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun. 
(cherry picked from commit 8e7548d33b)
 2018-06-13 20:10:46 +08:00
Yongjun Zhang 425fe4e217 Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release. 
(cherry picked from commit 5a82f10e32)
 2018-06-11 14:31:46 -07:00
Inigo Goiri f0150f024f HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
 2018-06-11 11:17:37 -07:00
Bharat Viswanadham c0d46a84a6 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar 
(cherry picked from commit ccfb816d39)
 2018-06-09 16:43:58 -07:00
Arpit Agarwal 33b6a68c81 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:30 -07:00
Wei-Chiu Chuang b2e919dc1a HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e39b113db0)
 2018-06-07 08:31:30 -07:00
Brahma Reddy Battula 630f43faf7 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9654dd1f47)
 2018-06-07 10:21:00 +05:30
Akira Ajisaka f358a77610
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia. 
(cherry picked from commit 0e3c31579d)
 2018-06-05 17:55:01 +09:00
Inigo Goiri ec5ef0bb20 HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 8d31ddcfeb)
 2018-06-04 18:28:40 -07:00
Inigo Goiri 19b37e7080 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 16316b6011)
 2018-06-04 18:22:09 -07:00
Inigo Goiri 76546d23e2 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu. 
(cherry picked from commit a28848104f)
 2018-06-04 18:13:14 -07:00
Rushabh Shah 6646b16450 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah 
(cherry picked from commit e2289c8d14)
 2018-06-04 09:21:14 -05:00
Wei-Chiu Chuang 302c36ffdc HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel. 
(cherry picked from commit bccdfeee0a)
 2018-06-04 07:03:35 -07:00
Xiao Chen fde45994a3 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel. 
(cherry picked from commit 9efb4b7db0)
 2018-06-03 22:37:27 -07:00
Inigo Goiri 90060d52f6 HDFS-13648. Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 8f83b9abf9)
 2018-06-02 20:20:26 -07:00
Inigo Goiri 8a9c2d7b97 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit e308ac88d3)
 2018-06-02 20:13:21 -07:00
Sean Mackrory 58b5b3aa75 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel. 
(cherry picked from commit 1361030e59)
 2018-05-31 22:23:19 -06:00
Chao Sun 2bbdf362cf HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun. 
(cherry picked from commit ff013d2c95)
 2018-05-31 17:39:46 -07:00
Konstantin V Shvachko 442dd87dcd HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 14:58:06 -07:00
Inigo Goiri ad11e40557 HDFS-13631. TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu. 
(cherry picked from commit 1f324e9661)
 2018-05-31 10:13:14 -07:00
Inigo Goiri 5c09dadd73 HDFS-13629. Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu. 
(cherry picked from commit 47c31ff16b)
 2018-05-30 10:23:00 -07:00
Inigo Goiri 6dc40bab4e HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu. 
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
 2018-05-30 10:02:50 -07:00
Inigo Goiri bc80290f28 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9dbf4f0166)
 2018-05-28 16:34:31 -07:00
Inigo Goiri cc696f80d3 HDFS-13627. TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 91d7c74e6a)
 2018-05-28 10:27:23 -07:00
Takanobu Asanuma 499a90f152
HDFS-13628. Update Archival Storage doc for Provided Storage 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 04757e5864)
 2018-05-28 19:06:58 +09:00
Inigo Goiri c7d77a3bfe HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu. 
(cherry picked from commit 8605a38514)
 2018-05-25 19:45:48 -07:00
Inigo Goiri 71ab85adce HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 13d2528907)
 2018-05-25 13:33:05 -07:00
Inigo Goiri 2d3d7a67f2 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
 2018-05-25 09:11:07 -07:00
Inigo Goiri f57e91a348 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen. 
(cherry picked from commit cddbbe5f69)
 2018-05-23 12:12:40 -07:00
Inigo Goiri a138496379 HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c13dea87d9)
 2018-05-23 11:36:34 -07:00
Bharat Viswanadham d0d3d5b2ad HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru 
(cherry picked from commit bc6d9d4c79)
 2018-05-23 10:23:45 -07:00
Inigo Goiri 0c268a0755 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit c0c9b7a8ef)
 2018-05-23 09:47:10 -07:00
Sammi Chen 7d71b3a1cc HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen. 
(cherry picked from commit 34e8b9f9a8)
 2018-05-23 19:12:04 +08:00
Andrew Wang d47c0fc39a HDFS-13601. Optimize ByteString conversions in PBHelper. 
(cherry picked from commit 1d2640b613)
 2018-05-22 23:55:35 -07:00
Inigo Goiri 8a86269c51 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9775ecb235)
 2018-05-18 09:46:29 -07:00
Inigo Goiri da5eff8325 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu. 
(cherry picked from commit 57b893de3d)
 2018-05-18 09:38:23 -07:00
Inigo Goiri df54cdbaea HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 53b807a6a8)
 2018-05-17 17:03:56 -07:00
Inigo Goiri f67865ed7e HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit a97a2042f2)
 2018-05-17 16:53:53 -07:00
Inigo Goiri 8c463a2a9c HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 65476458fa)
 2018-05-17 14:48:40 -07:00
Inigo Goiri e5ff9f3922 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 26f1e22fc9)
 2018-05-17 13:35:40 -07:00
Inigo Goiri b0c89690ed HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c0ec061e28)
 2018-05-17 09:22:29 -07:00
Kihwal Lee 4f1ec6a515 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0ce6290de6)
 2018-05-17 09:26:13 -05:00
Inigo Goiri 1d1806f862 HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu. 
(cherry picked from commit 454de3b543)
 2018-05-16 19:23:33 -07:00
Arpit Agarwal 45dd200a2c HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar. 2018-05-16 11:28:51 -07:00
Inigo Goiri e12ddd9394 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 55d5549087)
 2018-05-16 10:09:18 -07:00
Inigo Goiri 5310542fb4 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 6b67161ca3)
 2018-05-16 09:32:15 -07:00
Inigo Goiri 899a7ec2fb HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu. 
(cherry picked from commit 63480976a0)
 2018-05-15 15:36:03 -07:00
Inigo Goiri 34fbc1c2eb HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 2d6195c0af)
 2018-05-15 11:21:44 -07:00
Inigo Goiri c168329cc4 HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu. 
(cherry picked from commit eaca7980ad)
 2018-05-15 10:28:01 -07:00
Inigo Goiri 583b9b418a HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 92ebd466c7)
 2018-05-15 10:22:19 -07:00
Hanisha Koneru 4f55941390 HDFS-13544. Improve logging for JournalNode in federated cluster. 
(cherry picked from commit 6beb25ab7e)
 2018-05-14 10:14:38 -07:00
Xiao Chen 6e731eb20d HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure. 
(cherry picked from commit 960940e0e0)
 2018-05-14 09:29:14 -07:00
Sammi Chen e5281d9f8b Add 2.9.1 release notes and changes documents 2018-05-14 15:26:00 +08:00
Inigo Goiri e332e011ae HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu. 
(cherry picked from commit d50c4d71dc)
 2018-05-11 09:48:45 -07:00
Xiao Chen 6f3bbf364c HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi. 
(cherry picked from commit 696a4be0da)
 2018-05-07 15:39:24 -07:00
Xiao Chen d762932d79 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445." 
This reverts commit 128b09ee43.
 2018-05-07 13:47:52 -07:00
Inigo Goiri 4aadfb3c92 HDFS-13503. Fix TestFsck test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 9e2cfb2d3f)
 2018-05-01 08:13:30 -07:00
Inigo Goiri 4c13e7e3a0 HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 
(cherry picked from commit fc074a359c)
 2018-04-30 13:31:03 -07:00
Inigo Goiri ce62991648 HDFS-13509. Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit eb7fe1d588)
 2018-04-28 09:05:59 -07:00
Inigo Goiri 921a3e1678 HDFS-13326. RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li. 
(cherry picked from commit c394051a3d4d9d531f418503cb519606ae2b2e69)
 2018-04-26 13:01:16 -07:00
Yiqun Lin 999ea44b9f HDFS-13484. RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 30fef0bf1e5c8c0ca073df99ad9b33cb0e4431a5)
 2018-04-25 15:23:54 +08:00
Inigo Goiri 5b2427cd73 HDFS-13336. Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang. 
(cherry picked from commit df92a17e02)
 2018-04-23 19:12:53 -07:00
Bharat Viswanadham caf4d5000e HDFS-13356. Balancer:Set default value of minBlockSize to 10mb. Contributed by Bharat Viswanadham 
(cherry picked from commit 9b5375e0c1)
 2018-04-23 16:10:28 -07:00
Chris Douglas cf272c5179 HDFS-13408. MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang 
(cherry picked from commit f411de6a79)
 2018-04-23 11:13:52 -07:00
Arpit Agarwal 1f486a0649 HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal. 2018-04-23 10:09:02 -07:00
Sangjin Lee 9f648f6e59 HDFS-10183. Prevent race condition during class initialization. Contributed by Pavel Avgustinov. 
(cherry picked from commit f40969a141)
 2018-04-20 20:36:31 -07:00
Xiao Chen 128b09ee43 HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445. 
(cherry picked from commit 6503593711)
 2018-04-18 22:29:59 -05:00
Bharat Viswanadham 63c1b9d50c HDFS-13464. Fix javadoc in FsVolumeList#handleVolumeFailures. Contributed by Shashikant Banerjee 
(cherry picked from commit bf7694d020)
 2018-04-18 08:24:52 -07:00
Mukul Kumar Singh a07ba66ee9 HDFS-13426. Fix javadoc in FsDatasetAsyncDiskService#removeVolume. Contributed by Shashikant Banerjee. 2018-04-18 16:30:40 +05:30
Mukul Kumar Singh d99c8aa898 HDFS-13463. Fix javadoc in FsDatasetImpl#checkAndUpdate. Contributed by Shashikant Banerjee. 2018-04-18 16:30:24 +05:30
Erik Krogen fb350e8112
HDFS-12828. OIV ReverseXML Processor fails with escaped characters 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 2d9e791a90)
 2018-04-18 14:40:01 +09:00
Inigo Goiri 4be0fa28da HDFS-13462. Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak. 
(cherry picked from commit c9b33514b8)
 2018-04-17 14:20:23 -07:00
Bharat Viswanadham fc11555caf HDFS-13129. Add a test for DfsAdmin refreshSuperUserGroupsConfiguration. Contributed by Mukul Kumar Singh 
(cherry picked from commit a68b04370f)
 2018-04-17 12:41:54 -07:00
Arpit Agarwal 43a2af5569 HDFS-13439. Add test case for read block operation when it is moved. Contributed by Ajay Kumar. 2018-04-16 14:18:41 -07:00
Xiao Chen 805e33b62c HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-16 11:58:10 -07:00
Wei-Chiu Chuang ca8bb322be HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota. 
(cherry picked from commit e66e287efe)
 2018-04-13 12:18:49 -07:00
Xiaoyu Yao 867135af28 HDFS-11043. TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun. 2018-04-13 10:14:47 -07:00
Yongjun Zhang 56acbe476b HDFS-13315. Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang. 2018-04-13 10:14:24 -07:00
Bharat Viswanadham 8df8cad019 HDFS-13438. Fix javadoc in FsVolumeList#removeVolume. Contributed by Shashikant Banerjee. 
(cherry picked from commit fa8b88ab2b)
 2018-04-13 09:05:24 -07:00
Yiqun Lin 16763d8229 HDFS-13418. NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie. 
(cherry picked from commit 0725953efe)
 2018-04-13 17:57:15 +08:00
Akira Ajisaka 5531c31f16
HDFS-13436. Fix javadoc of package-info.java 
(cherry picked from commit 9b0b9f2af2)
 2018-04-13 13:24:42 +09:00
Akira Ajisaka 5f4762d950
HDFS-7101. Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu. 
(cherry picked from commit 113af12cfb)
 2018-04-12 17:48:14 +09:00
Akira Ajisaka 2d82a21b84
HDFS-13427. Fix the section titles of transparent encryption document. 
(cherry picked from commit c7cd362afd)
 2018-04-12 10:46:14 +09:00
Rakesh Radhakrishnan 3414bf6dbb HDFS-13328. Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit f89594f0b8)
 2018-04-10 23:40:26 +05:30
Weiwei Yang b104de3f64 HDFS-13420. License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka. 2018-04-10 22:13:08 +08:00
Wangda Tan 25bf1d3ac6 Updated branch-3.1 to track 3.1.1 
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
 2018-04-09 10:58:50 -07:00
Wangda Tan a336561e2b Added CHANGES/RELEASES/Jdiff for 3.1.0 release 
Change-Id: Ied5067a996151c04d15cad46c46ac98b60c37b39
(cherry picked from commit 2d96570452a72569befdf9cfe9b90c9fa2e0e261)
 2018-04-05 15:52:02 -07:00
Lei Xu 2543a9dd77 HDFS-13350. Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu). 
(cherry picked from commit 1cbf23df145af01692b8aaa438642b64e330cd05)
 2018-04-05 10:01:05 -07:00
Xiao Chen dcb2176c5d HADOOP-15317. Improve NetworkTopology chooseRandom's loop. 
(cherry picked from commit 57374c4737)
 2018-04-02 22:48:54 -07:00
Konstantin V Shvachko 4ce45a8ec7 HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie. 
(cherry picked from commit acfd764fcc)
 2018-03-30 18:57:18 -07:00
Xiao Chen 2960592c6f HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge. 
(cherry picked from commit 2c6cfad5a3)
 2018-03-29 16:07:01 -07:00
Arpit Agarwal d54e19a913 HDFS-13314. NameNode should optionally exit if it detects FsImage corruption. Contributed by Arpit Agarwal. 2018-03-28 12:49:14 -07:00
Lei Xu 4859cd7cc9 Update 3.0.1 jdiff file and jdiff stable api version 2018-03-23 11:53:30 -07:00
weiy d17aa8d052 HDFS-12512. RBF: Add WebHDFS. 
(cherry picked from commit 6e31a09084)
 2018-03-23 08:40:04 -07:00
Kihwal Lee c245ecedda HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong. 
(cherry picked from commit 66461ed98b)
 2018-03-22 12:28:34 -05:00
Konstantin V Shvachko 21db4e9efe HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan. 
(cherry picked from commit 8d898ab25f)
 2018-03-21 16:51:33 -07:00
weiy c804b28462 HDFS-13215. RBF: Move Router to its own module. 2018-03-19 23:05:33 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0bd99621c5 Preparing for 3.1.0 development 2018-03-19 15:16:54 -07:00
Inigo Goiri 08ff1586d5 HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 2018-03-16 10:29:19 -07:00
Xiaoyu Yao 78b05fde6c HDFS-13280. WebHDFS: Fix NPE in get snasphottable directory list call. Contributed by Lokesh Jain. 2018-03-15 11:02:27 -07:00
Inigo Goiri e71bc00a47 HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 2018-03-15 10:32:30 -07:00
Xiaoyu Yao da777a5498 HDFS-13251. Avoid using hard coded datanode data dirs in unit tests(addendum). Contributed by Ajay Kumar. 2018-03-15 10:18:44 -07:00
Inigo Goiri 6de135169e HDFS-12723. TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently. Contributed by Ajay Kumar. 2018-03-15 10:14:35 -07:00
fang zhenyi 5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Xiao Chen 5ff22d4c3a HDFS-13246. FileInputStream redundant closes in readReplicasFromCache. Contributed by liaoyuxiangqin. 2018-03-14 21:59:51 -07:00
Yiqun Lin 3a0f4bc0d5 HDFS-13261. Fix incorrect null value check. Contributed by Jianfei Jiang. 2018-03-15 11:06:04 +08:00
Inigo Goiri 41fc7f80be HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router. Contributed by Wei Yan. 2018-03-14 13:14:36 -07:00
Xiaoyu Yao f83716b7f2 HDFS-13251. Avoid using hard coded datanode data dirs in unit tests. Contributed by Ajay Kumar. 2018-03-14 11:26:58 -07:00
Xiaoyu Yao 0be0f1ce1d HDFS-13275. Adding log for BlockPoolManager#refreshNamenodes failures. Contributed by Ajay Kumar. 2018-03-14 10:38:00 -07:00
Arpit Agarwal 4c57fb0cd9 HDFS-13257. Code cleanup: INode never throws QuotaExceededException. Contributed by Tsz Wo Nicholas Sze. 2018-03-14 09:42:48 -07:00
Xiaoyu Yao a34d83d2fd HDFS-13249. Document webhdfs support for getting snapshottable directory list. Contributed by Lokesh Jain. 2018-03-14 09:14:26 -07:00
Sammi Chen ad1b988a82 HDFS-11600. Refactor TestDFSStripedOutputStreamWithFailure test classes. Contributed by Sammi Chen. 2018-03-14 14:39:53 +08:00
Yiqun Lin 76be6cbf6c HDFS-12773. RBF: Improve State Store FS implementation. Contributed by Inigo Goiri. 2018-03-14 11:20:59 +08:00
Arpit Agarwal 9714fc1dd4 HDFS-336. dfsadmin -report should report number of blocks from datanode. Contributed by Bharat Viswanadham. 2018-03-13 16:39:17 -07:00
Arpit Agarwal 39537b7c84 HDFS-13235. DiskBalancer: Update Documentation to add newly added options. Contributed by Bharat Viswanadham. 2018-03-13 16:35:51 -07:00
Xiao Chen 8211a3d469 HDFS-12505. Extend TestFileStatusWithECPolicy with a random EC policy. Contributed by Takanobu Asanuma. 2018-03-13 10:48:45 -07:00