Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,708 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7581 Commits

Author SHA1 Message Date
S O'Donnell 27b2c1ea7b HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 
(cherry picked from commit 5f34e3214e)
 2020-09-22 10:51:16 +01:00
hemanthboyina 43b113de69 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 
(cherry picked from commit aa582ccc2a)
 2020-09-17 10:17:30 +05:30
Uma Maheswara Rao G 0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:16:47 -07:00
S O'Donnell 0f234667dd HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 16:05:00 +01:00
Chen Liang e54128205c HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 2020-08-20 16:28:46 -07:00
Ayush Saxena 4af5fc86b2 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:24:41 +05:30
Mingliang Liu 1a53bc6ecf
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:13:41 -07:00
Uma Maheswara Rao G 1768618ab9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51a)
 2020-07-31 01:42:22 -07:00
Uma Maheswara Rao G 544602e3d1 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e70006639)
 2020-07-31 01:42:03 -07:00
Uma Maheswara Rao G 7084b273ac HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2)
 2020-07-31 01:41:45 -07:00
Uma Maheswara Rao G ab43b7bcfb HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc)
 2020-07-31 01:41:21 -07:00
Uma Maheswara Rao G 49a7f9ff7b HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5)
 2020-07-31 01:40:57 -07:00
Ayush Saxena 4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Chen Liang 1e33696bc8 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
(cherry picked from commit 370efc6d95)
 2020-07-20 15:24:07 -07:00
Akira Ajisaka 635067d03d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65a)
(cherry picked from commit 0d949d375e)
 2020-07-20 13:37:08 -07:00
He Xiaoqiao b611559755 HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-17 00:08:42 +08:00
S O'Donnell 1a11c4bc71 HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-07-16 12:43:29 +01:00
Ahmed Hussein b7b9bd32db HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 12:19:42 -05:00
Ahmed Hussein 34a6cbb5bb HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
(cherry picked from commit 43a865dc07)
 2020-07-15 11:01:34 -05:00
Eric Badger 2de0f99539 Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell." 
This reverts commit 0bfd1b5a94.
 2020-07-13 21:12:04 +00:00
He Xiaoqiao 0bfd1b5a94 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:41:22 +08:00
Uma Maheswara Rao G 4e1abe61a2 HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5)
 2020-07-09 16:29:36 -07:00
Shanyu Zhao d787b9373a HDFS-15451. Do not discard non-initial block report for provided storage. (#2119). Contributed by Shanyu Zhao. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2020-07-07 12:01:23 +08:00
Ayush Saxena 1e0424f39a HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell. 2020-07-04 12:27:12 +05:30
Akira Ajisaka d26dcf6686
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 16:43:32 +09:00
Virajith Jalaparti c4e0098bd6 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a374)
 2020-06-27 16:25:26 -07:00
Uma Maheswara Rao G 30d1d29076 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155)
 2020-06-27 16:09:36 -07:00
Uma Maheswara Rao G 81e62613fb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889b)
 2020-06-27 16:09:16 -07:00
Uma Maheswara Rao G e17770dec6 HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa60)
 2020-06-27 16:08:57 -07:00
Uma Maheswara Rao G 208701329c HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 16:08:38 -07:00
Uma Maheswara Rao G 100c13967e HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def95)
(cherry picked from commit 120ee793fc)
 2020-06-27 16:07:28 -07:00
Ayush Saxena f7e590aaff HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 
(cherry picked from commit cc671b16f7)
(cherry picked from commit bee2846bee)
 2020-06-27 16:06:52 -07:00
Uma Maheswara Rao G 64dbb39e71 HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 76fa0222f0)
(cherry picked from commit 418580446b)
 2020-06-27 16:06:25 -07:00
Uma Maheswara Rao G 2e1dfc152b HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd)
(cherry picked from commit 0b5e202614)
 2020-06-27 16:06:04 -07:00
Uma Maheswara Rao G 6ae92962d9 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b)
(cherry picked from commit 8e71e85af7)
 2020-06-27 16:05:36 -07:00
Abhishek Das 7015589f58 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec74453)
(cherry picked from commit 5b248de42d)
 2020-06-27 16:05:09 -07:00
Uma Maheswara Rao G c4857bb9c1 HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d9)
(cherry picked from commit 544996c857)
 2020-06-27 16:04:23 -07:00
S O'Donnell 9d891f08c2 HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 
(cherry picked from commit 123777823e)
 2020-06-18 12:59:47 +01:00
Eric Yang a8d3d4a2a8
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0)
 2020-06-15 11:13:06 +09:00
Takanobu Asanuma c2f908482f HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 
(cherry picked from commit f41a144077)
 2020-06-15 09:18:55 +09:00
Ayush Saxena 9f02fea756 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:49:17 +05:30
Toshihiro Suzuki f701d7bdd1 HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit e5a02a7499)
 2020-06-05 12:36:05 +01:00
Takanobu Asanuma 7168938223 HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:35:30 +09:00
Chen Liang 8f53545f9d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:08:08 -07:00
Akira Ajisaka de4ac41528
Revert "SPNEGO TLS verification" 
This reverts commit 4b7d6667fd.
 2020-05-13 17:16:15 +09:00
Uma Maheswara Rao G fda93334dc HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-12 21:33:49 -07:00
Virajith Jalaparti bec5d7baa0 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 19:37:40 -07:00
Ayush Saxena d24e1bc388 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:45:52 +05:30
Babneet Singh 42997d14a9 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
(cherry picked from commit d1e5e393c3)
(cherry picked from commit 44fdee351d)
 2020-05-11 10:46:21 -07:00
Ayush Saxena 86c15b8287 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:45:13 +05:30