Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,512 Commits 326 Branches 371 Tags
Commit Graph

5244 Commits

Author SHA1 Message Date
wangzhaohui
0fdcfe2beb HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a73a18cf05aeb2c40429c34c4b9b0732)
 2022-04-06 17:32:22 +09:00
Xing Lin
4bd2f58a83
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS (#4124) 
* HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.

To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
(cherry picked from commit 8b8158f02df18424b2406fd66b34c1bdf3d7ab55)
 2022-03-31 20:58:29 +00:00
Lei Yang
ebaaf3c6dd HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache 
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-28 13:31:50 -07:00
Owen O'Malley
ce208b1a1e HDFS-13248: Namenode needs to use the actual client IP when going through the 
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
 2022-03-23 15:06:22 -07:00
Manoj Govindassamy
ef8582bfa8 HADOOP-13055. Implement linkMergeSlash and linkFallback for ViewFileSystem 
(cherry picked from commit 133d7ca76e3d4b60292d57429d4259e80bec650a)
Fixes #4015
 2022-03-15 17:04:50 -07:00
Chris Douglas
20ff294714 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 
(cherry picked from commit 1f3bc63e6772be81bc9a6a7d93ed81d2a9e066c0)
 2022-03-15 15:40:17 -07:00
Ayush Saxena
f36d61a0c2
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409bc1d62f23efef485d1c6f1ff21e86c)
(cherry picked from commit fa30224e95d920adaea0ec65c360359044e3f754)
(cherry picked from commit 551f8d3ddc674d94343eacd63f11ce95564c7138)
 2022-02-23 11:42:01 +08:00
Xiao Chen
cb5af0012e HDFS-10650. DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission. Contributed by John Zhuge. 2022-02-15 16:25:25 -08:00
Ashutosh Gupta
b567dcd05f
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a869ca5c2bdc457375461ae89cdf97315)
 2022-01-11 02:16:08 +09:00
Akira Ajisaka
7d88a54d40 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83beb4f68e1ff18907589b062a0467e8)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml

(cherry picked from commit 94ca965e21067f1d957d21f19ae8e6abca9da971)
 2021-12-10 01:47:31 +09:00
Akira Ajisaka
eda934025b HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 3854b59bd1b06e9740cf33efc7844b66eec30ba4)
 2021-12-10 01:27:38 +09:00
litao
e648b3e34b HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616f34d039cf9246da9613914aa870515)
 2021-11-12 10:27:17 +09:00
litao
feb5971f49 HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039518bab7e5b581ef23bc5a2e8c81544)
 2021-11-10 19:24:56 +09:00
litao
899c4802fc HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d80db478f7e49749065c69da7e241bf0)
 2021-11-10 19:19:31 +09:00
jianghuazhu
64cd82e472 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a584b207782124bee0a377b2dbfc13df0)
 2021-11-10 11:46:50 +09:00
jianghuazhu
f7ec214568 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1d6d16249c1073c31f1a078d4e118f6)
 2021-11-04 16:57:35 +09:00
Michael Kuchenbecker
79d1818879 HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915ae3e0ced1b4c814a94845f51e655a0c)
 2021-10-21 13:54:05 -07:00
Symious
b7bb43f6db
HDFS-16257. Set initialCapacity for guava cache to solve performance issue (#3524) 
* HDFS-16257. Set initialCapacity for guava cache to solve performance issue

* HDFS-16257. Add comments

Co-authored-by: Symious <yiyang0203@gmail.com>
 2021-10-18 19:58:00 +09:00
Wei-Chiu Chuang
6d11f7ae8c HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
(cherry picked from commit 978ec57445f4178b31a6bfc383c6be032692c288)
(cherry picked from commit cd7c34f9b4005d27886f73e58bef88e706fcccf9)
 2021-09-24 09:15:48 -07:00
Kihwal Lee
30f48c4d06 HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
 2021-09-15 14:45:28 -05:00
wangzhaohui
79c1a93bd7 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
(cherry picked from commit 928dbac0a627c5198b3d5884ef5a7dee3d975855)
(cherry picked from commit 9b63f601d3285c3885b2941c5874f225bfa9ab31)
 2021-09-15 10:10:33 -07:00
Surendra Singh Lilhore
edc138186f HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f070bbfe65c736f6f3eca721b9d227f5)
(cherry picked from commit 2e939515dfbaf26ca466c8a755cedde0ce4e9c1a)
 2021-09-10 15:35:49 -07:00
Ahmed Hussein
7f32a31abf HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 21:34:04 +00:00
Akira Ajisaka
d07955ce50
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:46:30 +09:00
Abhishek Das
99b952fcb4 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260, #3218) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-20 18:26:24 -07:00
Jim Brennan
39c8297350 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 21:13:25 +00:00
Artem Smotrakov
9241470dfd HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 15:06:37 +09:00
Jim Brennan
7d231f4b98 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 18:06:37 +00:00
S O'Donnell
caa59d7a58 HDFS-15150. Introduce read write lock to Datanode. Contributed by Stephen O'Donnell and Ahmed Hussein 2021-06-22 16:24:19 +01:00
Kihwal Lee
ab6b5681e8 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2021-06-16 11:38:30 -05:00
Kihwal Lee
e7c7fb2896 HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fdcf1e61b39651a51f8223cfb8ce2e31)
 2021-06-15 19:57:21 -05:00
Takanobu Asanuma
a690e8ae59 HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682270bc6ba151875549d64d81917f425)
 2021-06-15 17:24:27 +09:00
Konstantin V Shvachko
149325484f HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 20:19:44 -07:00
Konstantin V Shvachko
8715a18f95 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f4f236674ce929164cc460037730abb)

# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2021-05-26 16:00:18 -07:00
Hui Fei
77d9c6d0f7 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:56:57 -07:00
Viraj Jasani
3c1c1b40d8
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:48:27 -07:00
Akira Ajisaka
f207d9e620
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin. Contributed by Ahmed Hussein. 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
 2021-03-29 09:44:28 +09:00
touchida
7183e269ac
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 16:34:52 +09:00
Viraj Jasani
3d0a0ed92b HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:16:18 +09:00
tomscut
68cccba893 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:22:51 -08:00
Arpit Agarwal
9f713bb825 HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq and Stephen O'Donnell. 
(cherry picked from commit 102c8fca10f3c626ab8bc47f818c8391a5c35289)
(cherry picked from commit 4a1d51dea2149e8f458467467798e81b126b7cc5)
 2021-03-02 20:31:33 +00:00
Konstantin V Shvachko
cdd3982db4 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:30:40 -08:00
S O'Donnell
628ccf81ef HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 2021-02-24 14:47:36 +00:00
He Xiaoqiao
ef14e16d36
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-07 15:50:10 +08:00
Symious
5dd6db1856
HDFS-15801. Backport HDFS-14582 to branch-2.10 (Failed to start DN with ArithmeticException when NULL checksum used) (#2659) 
Co-authored-by: Symious <yiyang0203@gmail.com>
 2021-02-01 15:43:44 -08:00
He Xiaoqiao
b30e9198bb
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Felix N, Haibin Huang. 2021-01-30 12:11:44 +08:00
Jim Brennan
c1a3e8158d [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f670c181d1fda99e76729e920fff287e)
 2021-01-28 17:46:06 +00:00
Wei-Chiu Chuang
e48b91c83c HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng. 
(cherry picked from commit 162228e8db937d4bdb9cf61d15ed555f1c96368f)
 2021-01-21 17:48:14 +00:00
Akira Ajisaka
db5b804238
HDFS-15717. Improve fsck logging. Contributed by Kihwal Lee and Ahmed Hussein (#2620) 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)
 2021-01-15 10:28:11 +09:00
S O'Donnell
35c99be3da HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-14 12:37:26 +00:00