Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,855 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

9195 Commits

Author SHA1 Message Date
litao cfca024190 HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:03:27 -07:00
Lei Yang 4f85c9a73b HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:28:38 -07:00
Masatake Iwasaki f8314cd469 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a244)
 2022-03-28 08:08:59 +00:00
GuoPhilipse 6300fc1fcd
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:24:44 +09:00
qinyuren 55f4baf3e4 HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:05:24 +09:00
Owen O'Malley 1175acbb75 HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-21 10:24:54 -07:00
Viraj Jasani 712d9bece8
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1c0bc35305)
 2022-03-16 09:40:59 +08:00
Owen O'Malley c52b97d084 HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:30:29 -07:00
Takanobu Asanuma 52fb9d7ce2 HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>

Cherry-picked from 9c887e5b by Owen O'Malley
 2022-03-14 10:29:37 -07:00
litao 496657c63f HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 
Cherry-picked from 5b05068f by Owen O'Malley
 2022-03-14 10:29:30 -07:00
litao f9d40ed7b7 HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Cherry-picked from 359b03c8 by Owen O'Malley
 2022-03-14 10:29:11 -07:00
Hui Fei 2479d4ab6c HDFS-15630. RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. Contributed by Chengwei Wang 
Cherry-picked from 264c948e by Owen O'Malley
 2022-03-14 10:29:04 -07:00
Hui Fei 8e129e5b8d HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer (#2363) 
Cherry-picked from 518a212c by Owen O'Malley
 2022-03-14 10:28:55 -07:00
Thinker313 0801fe450e
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-03-12 19:45:13 +08:00
Chao Sun b174aaed57 Make upstream aware of 3.3.2 release 2022-03-02 19:10:30 -08:00
litao 74f5f90615 HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 
(cherry picked from commit 6b07c851f3)
 2022-02-26 02:22:28 +09:00
litao 9c0bdc5aea
HDFS-16371. Exclude slow disks when choosing volume (#3753) (#4031) 2022-02-26 02:21:07 +09:00
litao 5601d0848a HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718) 
Authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit 32353eb38a)
 2022-02-25 11:19:27 +09:00
Ayush Saxena fa30224e95
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
 2022-02-23 11:41:19 +08:00
Chentao Yu d14a7c6ee5 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:48:13 -08:00
litao db67952f9f HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0c194f2157)
 2022-02-16 09:45:07 +09:00
Takanobu Asanuma 4c57fb4d6b
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. (#3992) 
(cherry picked from commit 1cd96e8dd8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-16 09:42:43 +09:00
luoyuan3471 752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a)
 2022-02-11 23:18:25 +08:00
singer-bin ce7cabb771
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
 2022-02-06 13:36:57 +08:00
daimin 709e617a84
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
 2022-02-06 13:06:35 +08:00
KevinWikant 7171e2190e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)
 2022-01-31 13:19:40 +09:00
KevinWikant 5e2eac6c41
HDFS-16303. Improve handling of datanode lost while decommissioning (#3921) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:18:36 +09:00
secfree 110104da38 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:05:32 +09:00
litao b5d2e00f81 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:44:03 +09:00
litao ef1a2b478b HDFS-16398. Reconfig block report parameters for datanode (#3831) 
(cherry picked from commit c2ff39006f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2022-01-26 17:36:17 +09:00
Wei-Chiu Chuang ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages (#3883) (#3924) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithHANameNodes.java

Co-authored-by: qinyuren <1476659627@qq.com>
 2022-01-26 11:54:13 +08:00
Bryan Beaudreault bd13d73334 HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 
(cherry picked from commit 94b884ae55)
 2022-01-25 11:43:47 +00:00
daimin 728ed10a7c
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
 2022-01-25 09:58:20 +08:00
liubingxing d6ff60df65
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
 2022-01-20 18:47:00 +08:00
litao f9c0bc094a HDFS-16399. Reconfig cache report parameters for datanode (#3841) 
(cherry picked from commit e355646330)
 2022-01-19 18:43:15 +09:00
litao 11fe5279b0 HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit f02374df92)
 2022-01-19 18:42:48 +09:00
litao cdaf4d89f9 HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 52ec65fd10)
 2022-01-19 18:40:41 +09:00
Viraj Jasani 831c11c47a HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit b038042ece)
 2022-01-19 16:01:00 +09:00
qinyuren 1c71d6e9fe HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:02 +09:00
Xiangyi Zhu b5e7f59e53
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 23:18:05 +08:00
Jackson Wang 926222a0d0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:40:58 +09:00
Ashutosh Gupta 6b83fe4a00 HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:00:23 -08:00
liever18 3a82899493 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 15:46:21 +00:00
Wei-Chiu Chuang 350b51f287 Make upstream aware of 3.3.1 release 2022-01-04 14:48:49 -08:00
Ashutosh Gupta bad3a0964c HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:25:56 +09:00
jianghuazhu b623acff5a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:25:00 +09:00
wangzhaohui cc7b7e174c HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 11:46:23 +00:00
jianghuazhu a4496a0187 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 11:32:01 +00:00
litao e60b8a0358 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:51:40 +09:00
litao 75af1d67e4 HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2021-12-16 13:31:17 +09:00