Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,511 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5045 Commits

Author SHA1 Message Date
Masatake Iwasaki f241a09c30 HADOOP-18088. Replace log4j 1.x with reload4j. 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-08 20:13:52 +09:00
wangzhaohui 0fdcfe2beb HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:32:22 +09:00
Xing Lin 4bd2f58a83
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS (#4124) 
* HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.

To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02d)
 2022-03-31 20:58:29 +00:00
Lei Yang ebaaf3c6dd HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:31:50 -07:00
Owen O'Malley ce208b1a1e HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-23 15:06:22 -07:00
Manoj Govindassamy ef8582bfa8 HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 
(cherry picked from commit 133d7ca76e)
Fixes #4015
 2022-03-15 17:04:50 -07:00
Chris Douglas 20ff294714 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 
(cherry picked from commit 1f3bc63e67)
 2022-03-15 15:40:17 -07:00
Ayush Saxena f36d61a0c2
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
(cherry picked from commit 551f8d3ddc)
 2022-02-23 11:42:01 +08:00
Xiao Chen cb5af0012e HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2022-02-15 16:25:25 -08:00
Ashutosh Gupta b567dcd05f
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:16:08 +09:00
Akira Ajisaka 7d88a54d40 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml

(cherry picked from commit 94ca965e21)
 2021-12-10 01:47:31 +09:00
Akira Ajisaka eda934025b HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 3854b59bd1)
 2021-12-10 01:27:38 +09:00
litao e648b3e34b HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:27:17 +09:00
litao feb5971f49 HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:24:56 +09:00
jianghuazhu 64cd82e472 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:46:50 +09:00
jianghuazhu f7ec214568 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:57:35 +09:00
Michael Kuchenbecker 79d1818879 HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:54:05 -07:00
Wei-Chiu Chuang 6d11f7ae8c HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
(cherry picked from commit bb08de559a)
(cherry picked from commit 978ec57445)
(cherry picked from commit cd7c34f9b4)
 2021-09-24 09:15:48 -07:00
Kihwal Lee 30f48c4d06 HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8b)
 2021-09-15 14:45:28 -05:00
wangzhaohui 79c1a93bd7 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14)
(cherry picked from commit 928dbac0a6)
(cherry picked from commit 9b63f601d3)
 2021-09-15 10:10:33 -07:00
Surendra Singh Lilhore edc138186f HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f0)
(cherry picked from commit 2e939515df)
 2021-09-10 15:35:49 -07:00
Ahmed Hussein 7f32a31abf HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 21:34:04 +00:00
Abhishek Das 99b952fcb4 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260, #3218) 
(cherry picked from commit 1dd03cc4b5)
 2021-07-20 18:26:24 -07:00
Jim Brennan 39c8297350 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 21:13:25 +00:00
Artem Smotrakov 9241470dfd HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 15:06:37 +09:00
Jim Brennan 7d231f4b98 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 18:06:37 +00:00
S O'Donnell caa59d7a58 HDFS-15150. Introduce read write lock to Datanode. Contributed by Stephen O'Donnell and Ahmed Hussein 2021-06-22 16:24:19 +01:00
Kihwal Lee ab6b5681e8 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2021-06-16 11:38:30 -05:00
Kihwal Lee e7c7fb2896 HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fd)
 2021-06-15 19:57:21 -05:00
Konstantin V Shvachko 149325484f HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 20:19:44 -07:00
Konstantin V Shvachko 8715a18f95 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)

# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2021-05-26 16:00:18 -07:00
Hui Fei 77d9c6d0f7 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151)
 2021-04-29 12:56:57 -07:00
Viraj Jasani 3c1c1b40d8
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:48:27 -07:00
Akira Ajisaka f207d9e620
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin. Contributed by Ahmed Hussein. 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
 2021-03-29 09:44:28 +09:00
touchida 7183e269ac
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 16:34:52 +09:00
Viraj Jasani 3d0a0ed92b HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:16:18 +09:00
tomscut 68cccba893 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:22:51 -08:00
Arpit Agarwal 9f713bb825 HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq and Stephen O'Donnell. 
(cherry picked from commit 102c8fca10f3c626ab8bc47f818c8391a5c35289)
(cherry picked from commit 4a1d51dea2)
 2021-03-02 20:31:33 +00:00
Konstantin V Shvachko cdd3982db4 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:30:40 -08:00
S O'Donnell 628ccf81ef HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 2021-02-24 14:47:36 +00:00
He Xiaoqiao ef14e16d36
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-07 15:50:10 +08:00
Symious 5dd6db1856
HDFS-15801. Backport HDFS-14582 to branch-2.10 (Failed to start DN with ArithmeticException when NULL checksum used) (#2659) 
Co-authored-by: Symious <yiyang0203@gmail.com>
 2021-02-01 15:43:44 -08:00
He Xiaoqiao b30e9198bb
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Felix N, Haibin Huang. 2021-01-30 12:11:44 +08:00
Jim Brennan c1a3e8158d [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 17:46:06 +00:00
Wei-Chiu Chuang e48b91c83c HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng. 
(cherry picked from commit 162228e8db)
 2021-01-21 17:48:14 +00:00
Akira Ajisaka db5b804238
HDFS-15717. Improve fsck logging. Contributed by Kihwal Lee and Ahmed Hussein (#2620) 
(cherry picked from commit be35fa186c)
 2021-01-15 10:28:11 +09:00
S O'Donnell 35c99be3da HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-14 12:37:26 +00:00
Jim Brennan 8af247df85 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) Contributed by Ahmed Hussein 2020-12-08 20:32:31 +00:00
Konstantin V Shvachko a9bf374380 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 14:39:38 -08:00
Wei-Chiu Chuang e7e165a927 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 2020-10-28 09:22:47 -07:00