Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,011 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7071 Commits

Author SHA1 Message Date
Inigo Goiri 4b47daa640 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e0)
(cherry picked from commit 9bf4f3d614)
 2018-11-05 16:53:28 -08:00
Akira Ajisaka 2ae21d5387
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 
(cherry picked from commit dd70b1f538)
 2018-10-31 17:03:47 +09:00
Xiao Chen 20c9a12bc7 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java

(cherry picked from commit 399645ebc1)
 2018-10-29 19:14:12 -07:00
Wei-Chiu Chuang 0f34ff7722 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fa01a8fa0e.
 2018-10-28 17:07:29 -07:00
Anu Engineer c5b704a34c HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.

(cherry picked from commit 4f10d7e23f)
 2018-10-26 16:55:55 -07:00
Arpit Agarwal f6469adbb1 HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd)
 2018-10-25 15:14:39 -07:00
Wei-Chiu Chuang fa01a8fa0e HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
(cherry picked from commit fd02c501c0)
(cherry picked from commit ef9f8ca13d)
 2018-10-25 09:38:26 -07:00
Ajay Kumar 2caf69debd HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Wei-Chiu Chuang. 2018-10-24 22:35:06 -07:00
Inigo Goiri 1d5390679e HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd)
 2018-10-24 16:14:22 -07:00
Xiao Chen e8832418bf Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
(cherry picked from commit 96cedb87b9)
 2018-10-24 16:10:51 -07:00
Inigo Goiri 6aef514795 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2)
 2018-10-24 09:38:48 -07:00
Wei-Chiu Chuang e4464f9598 Revert "HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar." 
This reverts commit d8cd07d5f5.
 2018-10-23 05:58:10 -07:00
Arpit Agarwal d8cd07d5f5 HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d)
 2018-10-22 14:44:42 -07:00
drankye b34c650a41 HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 
(cherry picked from commit 975d60685e)
 2018-10-18 08:31:12 -07:00
Inigo Goiri ea19a36132 HDFS-14000. RBF: Documentation should reflect right scripts for v3.0 and above. Contributed by CR Hota. 
(cherry picked from commit 7347fa2df3)
 2018-10-17 16:43:05 -07:00
Xiao Chen b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a025)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Inigo Goiri 53b522af6d HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db)
 2018-10-12 10:12:07 -07:00
Hrishikesh Gadre b170de8be5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 323b76bccf)
 2018-10-09 19:54:56 -07:00
Inigo Goiri 8a9f61be4e HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e)
 2018-10-06 10:42:51 -07:00
Brahma Reddy Battula b28dacf60c HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10)
 2018-09-26 23:55:02 +05:30
Siyao Meng 2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Chen Liang 78e0c0a1bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:51:23 -07:00
Arpit Agarwal ffef819ee8 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b151)
 2018-09-19 13:49:57 -07:00
Inigo Goiri 3fc3a5c01c HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:42 -07:00
Inigo Goiri b93aeb43a4 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0)
 2018-09-19 09:31:58 -07:00
Wei-Chiu Chuang 823b1fceee HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a72)
(cherry picked from commit 7442b7941d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2018-09-19 03:20:26 -07:00
Wei-Chiu Chuang 3a32e84d3e HDFS-13886. HttpFSFileSystem.getFileStatus() doesn't return "snapshot enabled" bit. Contributed by Siyao Meng. 
(cherry picked from commit 44857476fa)
(cherry picked from commit 595ce94570)
 2018-09-18 15:48:05 -07:00
Wei-Chiu Chuang 2ef3911f7c HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5)
(cherry picked from commit 829399a9f3)
 2018-09-18 15:48:04 -07:00
Inigo Goiri f0537b5161 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:39:31 -07:00
Inigo Goiri 1314dc2397 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a984)
 2018-09-13 09:18:11 -07:00
Brahma Reddy Battula 0c8aab690c HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 
(cherry picked from commit b2432d254c)
 2018-09-12 06:35:29 +05:30
Xiao Chen a5da4df73c HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 
(cherry picked from commit 8e54da1511)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java

(cherry picked from commit 2dd27c999b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2018-09-10 23:00:41 -07:00
Inigo Goiri a9392bdb17 HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng. 
(cherry picked from commit 527288ef89)
 2018-09-06 16:49:14 -07:00
Wei-Chiu Chuang e99ed05444 HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e780556ae9)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java

(cherry picked from commit d1ea7df43d)
 2018-09-05 10:18:56 -07:00
Hrishikesh Gadre 25337c045a HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d)
 2018-09-04 21:54:56 -07:00
Inigo Goiri 54d3189cfa HDFS-13857. RBF: Choose to enable the default nameservice to read/write files. Contributed by yanghuafeng. 
(cherry picked from commit 54f2044595)
 2018-09-04 12:18:08 -07:00
Wei-Chiu Chuang d06717ef94 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit d89a485b25.
 2018-08-30 11:59:39 -07:00
Vinayakumar B 67a5491962 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 
(cherry picked from commit 65977e5d81)
 2018-08-30 22:17:00 +05:30
Yiqun Lin 6a547856ef HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 
(cherry picked from commit 582cb10ec7)
 2018-08-30 11:24:25 +08:00
Brahma Reddy Battula 3b01279612 HDFS-13854. RBF: The ProcessingAvgTime and ProxyAvgTime should display by JMX with ms unit. Contributed by yanghuafeng. 
(cherry picked from commit 64ad0298d4)
 2018-08-29 08:32:05 +05:30
Wei-Chiu Chuang 72e1e1904b HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-28 16:36:30 -07:00
Zsolt Venczel 2d1f819979 HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 
(cherry picked from commit 3e18b957eb)
 2018-08-28 15:14:57 -07:00
Wei-Chiu Chuang d89a485b25 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit 26c2a97c56)
(cherry picked from commit 41732e7b28)
 2018-08-27 16:04:16 -07:00
Konstantin V Shvachko 27abdb390b HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 
(cherry picked from commit a4121c71c2)
 2018-08-24 18:51:42 -07:00
Yiqun Lin 4ef3025964 HDFS-13821. RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache. Contributed by Fei Hui. 2018-08-22 13:01:13 +08:00
Wei-Chiu Chuang bd46906bd8 HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 
(cherry picked from commit 865650052b)
 2018-08-21 15:44:08 -07:00
Yiqun Lin 5e2e68d4bb HDFS-13750. RBF: Router ID in RouterRpcClient is always null. Contributed by Takanobu Asanuma. 
(cherry picked from commit 01ff817814)
 2018-08-20 23:03:19 +08:00
Xiao Chen 185c8f2abc HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 
(cherry picked from commit c67b0650ea)
 2018-08-16 23:15:58 -07:00
Daniel Templeton 27508086de HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
(cherry picked from commit 8512e1a91b)
(cherry picked from commit fbedf89377)
 2018-08-16 15:02:18 -07:00
Xiao Chen 5d16301ad9 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 
(cherry picked from commit cede33997f)
 2018-08-14 11:59:05 -07:00
Wei-Chiu Chuang 6f2a46578f HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 
(cherry picked from commit 61a9b4f58b)
(cherry picked from commit 7a115e8de4)
 2018-08-14 11:52:55 -07:00
Wei-Chiu Chuang d502abe947 HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 
(cherry picked from commit 4023eeba05)
(cherry picked from commit 74d4fd975a)
 2018-08-13 17:37:36 -07:00
Wei-Chiu Chuang 4b606eb9fd HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 
(cherry picked from commit 23854443ef)
(cherry picked from commit b38649c59a)
 2018-08-13 16:14:01 -07:00
Vinayakumar B eecfbaff3b HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena. 
(cherry picked from commit 2b0f977241)
 2018-08-13 13:58:37 -07:00
Chen Liang 2d985b6c81 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:47:17 -07:00
Chao Sun 07054c5994 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 
(cherry picked from commit a2a8c48699)
 2018-08-10 16:12:05 -07:00
Wangda Tan 8b4392769c Make 3.1.1 awared by other branches 
(cherry picked from commit 49c687608b)
(cherry picked from commit 781620a44d)
 2018-08-08 13:07:09 -07:00
Xiao Chen 8e5081569f HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell. 
(cherry picked from commit 6677717c68)
 2018-08-07 22:05:59 -07:00
Brahma Reddy Battula abc32c863e HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani 
(cherry picked from commit 3108d27edd)
 2018-07-30 16:08:49 +05:30
Arpit Agarwal eb40d4fab4 HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:11 -07:00
Anu Engineer faaf72dcb3 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception. 
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
 2018-07-27 09:35:02 -07:00
Wei-Chiu Chuang 0af04ad89f HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 
(cherry picked from commit e95c5e9f62)
(cherry picked from commit 964f3454d1)
 2018-07-25 11:05:36 -07:00
Daniel Templeton 3210b3d8aa HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica 
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
(cherry picked from commit 00c476abd8)
 2018-07-24 16:13:30 -07:00
Yiqun Lin 097ae7fdc8 HDFS-13743. RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver. Contributed by Takanobu Asanuma. 
(cherry picked from commit 7b25fb949b)
 2018-07-20 17:32:27 +08:00
Yiqun Lin 34afb9f946 HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF document. Contributed by Takanobu Asanuma. 
(cherry picked from commit 1af87df242)
 2018-07-17 10:48:36 +08:00
Konstantin V Shvachko 475c282939 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal. 
(cherry picked from commit 0a1e922f3d)
 2018-07-16 18:33:36 -07:00
Wei-Chiu Chuang 0d2b983dc2 HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng. 
(cherry picked from commit 88b2794244)
(cherry picked from commit ac9155d6cb)
 2018-07-16 10:53:30 -07:00
Inigo Goiri 3d47ef1d31 HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed by Chao Sun. 
(cherry picked from commit 359ea4e181)
 2018-07-16 09:47:22 -07:00
Xiao Chen 883df537d6 HDFS-12837. Intermittent failure in TestReencryptionWithKMS. 
(cherry picked from commit b37074be5a)
 2018-07-11 21:03:53 -07:00
Akira Ajisaka 4d5473c89d
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota. 
(cherry picked from commit 418cc7f3ae)
 2018-07-11 14:48:12 -04:00
Wei-Chiu Chuang 74aa4be92d HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng. 
(cherry picked from commit 162228e8db)
(cherry picked from commit f9fa3cb157)
 2018-07-11 10:04:38 -07:00
Yiqun Lin 241f7bb7ad HDFS-13726. RBF: Fix RBF configuration links. Contributed by Takanobu Asanuma. 
(cherry picked from commit 2ae13d41dc)
 2018-07-11 22:19:32 +08:00
Andrew Wang 0e57e81b0c HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 
(cherry picked from commit eecb5baaaa)
 2018-07-09 15:17:48 +02:00
Wei-Chiu Chuang 68290606ea HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel. 
(cherry picked from commit 0247cb6318)
(cherry picked from commit f5f4d0b7e7)
 2018-07-06 15:01:35 -07:00
Andrew Wang 896b51578f HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas. 
(cherry picked from commit 344f324710)
 2018-07-03 11:08:05 +02:00
Inigo Goiri 98f2444a7d HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 7296b644f7)
 2018-07-02 16:37:36 -07:00
Wei-Chiu Chuang 563d56be62 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota. 
(cherry picked from commit fef20a446f)
(cherry picked from commit 2ce892fcf5)
 2018-07-02 09:44:16 -07:00
Andrew Wang d34260f122 HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon. 
(cherry picked from commit 5d748bd056)
 2018-07-02 12:11:14 +02:00
Andrew Wang 7cd769b31c HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon. 
(cherry picked from commit 6ba9974108)
 2018-07-02 12:02:27 +02:00
Yiqun Lin db1f37a711 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham. 
(cherry picked from commit 30728aced4)
 2018-06-22 10:55:09 +08:00
Xiao Chen aca66f6155 HDFS-13682. Cannot create encryption zone after KMS auth token expires. 
(cherry picked from commit 32f867a6a9)
 2018-06-20 15:59:45 -07:00
Surendra Singh Lilhore 5667582bc4 HDFS-12857. StoragePolicyAdmin should support schema based path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 30941d99c9)
 2018-06-19 10:09:05 -07:00
Inigo Goiri 11d3db2dba HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 8762e9cf10)
 2018-06-15 16:50:20 -07:00
Inigo Goiri 69abd1fc5c HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak. 
(cherry picked from commit d31a3ce767)
 2018-06-15 16:00:16 -07:00
Wei-Chiu Chuang 73badce597 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth. 
(cherry picked from commit c966a3837a)
(cherry picked from commit 975d4b3d60)
 2018-06-15 13:38:11 -07:00
Inigo Goiri fe1926a50f HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit eebeb6033f)
 2018-06-15 13:08:47 -07:00
Inigo Goiri 94e0b9839f HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit 43d994e4a6)
 2018-06-15 10:34:33 -07:00
Inigo Goiri a3c0c03899 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 8d4926f38b)
 2018-06-14 09:59:50 -07:00
Yiqun Lin b94b071b10 HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun. 
(cherry picked from commit 8e7548d33b)
 2018-06-13 20:24:12 +08:00
Rohith Sharma K S 96e7b7e8ae YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:25:22 +05:30
Yongjun Zhang 8202c334fe Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release. 
(cherry picked from commit 5a82f10e32)
 2018-06-11 14:28:44 -07:00
Inigo Goiri 6eea6d28de HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
 2018-06-11 11:18:04 -07:00
Bharat Viswanadham 7c144e4ae4 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar 
(cherry picked from commit ccfb816d39)
 2018-06-09 16:44:47 -07:00
Arpit Agarwal ea05171ceb HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:58:02 -07:00
Chao Sun d62b0f84f1 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun. 
(cherry picked from commit fba1c42adc)
 2018-06-08 16:42:18 -07:00
Brahma Reddy Battula a45e9812d6 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9654dd1f47)
(cherry picked from commit 630f43faf7)
 2018-06-07 09:18:20 -07:00
Wei-Chiu Chuang 8e423a7779 HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e39b113db0)
(cherry picked from commit b2e919dc1a)
 2018-06-07 08:32:53 -07:00
Xiao Chen 3843abe482 HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel. 
(cherry picked from commit ba4011d64f)
 2018-06-05 21:19:19 -07:00
Akira Ajisaka cc735146dc
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia. 
(cherry picked from commit 0e3c31579d)
 2018-06-05 17:55:33 +09:00
Inigo Goiri e794ff2dcb HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 8d31ddcfeb)
 2018-06-04 18:29:12 -07:00
Inigo Goiri 076dc95c76 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 16316b6011)
 2018-06-04 18:22:35 -07:00
Inigo Goiri 2e5f849ea5 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu. 
(cherry picked from commit a28848104f)
 2018-06-04 18:13:50 -07:00
Rushabh Shah 4ecb7aaf77 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah 
(cherry picked from commit e2289c8d14)
 2018-06-04 09:58:38 -05:00
Wei-Chiu Chuang 63fc8f82cb HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel. 
(cherry picked from commit bccdfeee0a)
(cherry picked from commit 302c36ffdc)
 2018-06-04 07:04:32 -07:00
Xiao Chen 3ee5fd16f4 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel. 
(cherry picked from commit 9efb4b7db0)
 2018-06-03 22:37:36 -07:00
Inigo Goiri a274d21ea3 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit e308ac88d3)
 2018-06-02 20:13:55 -07:00
Inigo Goiri d40c49257a HDFS-13637. RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota. 
(cherry picked from commit e11d674049)
 2018-06-01 17:00:12 -07:00
Chao Sun c6becec392 HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun. 
(cherry picked from commit ff013d2c95)
 2018-05-31 17:42:41 -07:00
Konstantin V Shvachko 3d7e345823 HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 15:00:58 -07:00
Inigo Goiri 33c4f60b06 HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu. 
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
 2018-05-30 10:07:16 -07:00
Yongjun Zhang f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Inigo Goiri 128cfa101a HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9dbf4f0166)
 2018-05-28 16:35:32 -07:00
Inigo Goiri 1c1553670d HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu. 
(cherry picked from commit 8605a38514)
 2018-05-25 19:46:16 -07:00
Inigo Goiri ee31d9d0dc HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 13d2528907)
 2018-05-25 13:33:30 -07:00
Inigo Goiri 3b0da097a4 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
 2018-05-25 09:11:34 -07:00
Andrew Wang 731abae557 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient. 
(cherry picked from commit c9b63deb53)
 2018-05-24 09:56:40 -07:00
Inigo Goiri 506f61e186 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen. 
(cherry picked from commit cddbbe5f69)
 2018-05-23 12:13:06 -07:00
Inigo Goiri c5f29d62eb HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c13dea87d9)
 2018-05-23 11:37:27 -07:00
Inigo Goiri 8d8ef081a2 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit c0c9b7a8ef)
 2018-05-23 09:47:32 -07:00
Sammi Chen 6db710b9d8 HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen. 
(cherry picked from commit 34e8b9f9a8)
(cherry picked from commit 7d71b3a1cc)
 2018-05-23 19:20:32 +08:00
Andrew Wang 5b3a6e3678 HDFS-13601. Optimize ByteString conversions in PBHelper. 
(cherry picked from commit 1d2640b613)
(cherry picked from commit d47c0fc39a)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
 2018-05-22 23:57:05 -07:00
Tsz-Wo Nicholas Sze 2acd8e4ecc HDFS-12813. RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1. Contributed by Mukul Kumar Singh 2018-05-19 11:12:13 -07:00
Inigo Goiri 97aa781d3d HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9775ecb235)
 2018-05-18 09:47:22 -07:00
Inigo Goiri d923e05798 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu. 
(cherry picked from commit 57b893de3d)
 2018-05-18 09:38:48 -07:00
Inigo Goiri cdc16b3a2f HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 53b807a6a8)
 2018-05-17 17:04:23 -07:00
Inigo Goiri ccd7d50faf HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit a97a2042f2)
 2018-05-17 16:54:17 -07:00
Inigo Goiri 6b2d86debe HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 65476458fa)
 2018-05-17 14:49:03 -07:00
Inigo Goiri d3457b38e6 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 26f1e22fc9)
 2018-05-17 13:36:16 -07:00
Inigo Goiri fb0a258b3c HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c0ec061e28)
 2018-05-17 09:22:48 -07:00
Kihwal Lee 0943112d85 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0ce6290de6)
 2018-05-17 09:28:17 -05:00
Inigo Goiri fcf0b6a8dd HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu. 
(cherry picked from commit 454de3b543)
 2018-05-16 19:24:00 -07:00
Inigo Goiri 16e89a2771 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 55d5549087)
 2018-05-16 10:10:08 -07:00
Inigo Goiri 0a2cdd2730 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 6b67161ca3)
 2018-05-16 09:32:39 -07:00
Inigo Goiri eadfb55195 HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu. 
(cherry picked from commit 63480976a0)
 2018-05-15 15:36:30 -07:00
Inigo Goiri 1b4c39fd2f HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 2d6195c0af)
 2018-05-15 11:22:12 -07:00
Inigo Goiri d1b17f997c HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu. 
(cherry picked from commit eaca7980ad)
 2018-05-15 10:29:04 -07:00
Inigo Goiri 9a722150c6 HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 92ebd466c7)
 2018-05-15 10:22:52 -07:00
Hanisha Koneru b00c0a08a3 HDFS-13544. Improve logging for JournalNode in federated cluster. 
(cherry picked from commit 6beb25ab7e)
 2018-05-14 10:16:06 -07:00
Xiao Chen 220e38ec02 HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure. 
(cherry picked from commit 960940e0e0)
 2018-05-14 09:29:22 -07:00
Sammi Chen a700d05f72 Add 2.9.1 release notes and changes documents 
(cherry picked from commit f3f544b004)
 2018-05-14 15:18:19 +08:00
Inigo Goiri 3b57a236ed HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu. 
(cherry picked from commit d50c4d71dc)
 2018-05-11 09:49:21 -07:00
Inigo Goiri d72a4b6303 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun." 
This reverts commit 1049e180a7.
 2018-05-10 09:24:45 -07:00
Inigo Goiri e466b4e2a7 HDFS-13537. TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang. 
(cherry picked from commit 914b98a713)
 2018-05-09 16:53:09 -07:00
Inigo Goiri 1049e180a7 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit 63803e7051)
 2018-05-09 13:56:58 -07:00
Yongjun Zhang 21bf02023c Revert "HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem" 
This reverts commit 218a21a0f0.
 2018-05-08 22:22:06 -07:00
Xiao Chen fe66ed7da5 HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi. 
(cherry picked from commit 696a4be0da)
 2018-05-07 15:39:33 -07:00
Xiao Chen 22384580b2 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445." 
This reverts commit 9f19c3b03e.
 2018-05-07 13:56:26 -07:00
Inigo Goiri 21442dcaa3 HDFS-13525. RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin. 
(cherry picked from commit a3b416f69d)
 2018-05-03 11:26:00 -07:00
Yiqun Lin 221db515aa HDFS-13488. RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri. 
(cherry picked from commit 37269261d1)
(cherry picked from commit 5fef28d0d4)
 2018-05-02 15:06:53 +08:00