S O'Donnell
52adda89c4
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
2020-12-14 11:37:08 +00:00
Jim Brennan
eec10a087e
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
2020-12-11 18:10:24 +00:00
Neil
9123f9245c
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
...
(cherry picked from commit 9bd3c9bc50
)
(cherry picked from commit bc5458bbd4
)
2020-12-10 13:50:42 -08:00
Jim Brennan
76316c4bc0
HDFS-15711. Add Metrics to HttpFS Server. ( #2521 ) Contributed by Ahmed Hussein and Kihwal Lee
2020-12-10 21:30:03 +00:00
Ahmed Hussein
bb23d4f14b
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
...
(cherry picked from commit 01383a2172
)
2020-12-08 19:41:54 +00:00
crossfire
31a6b45117
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
...
(cherry picked from commit 40f7543a6d
)
(cherry picked from commit edd9b659ca
)
2020-12-07 15:50:47 -08:00
Jim Brennan
d7c955b646
HDFS-15707. NNTop counts don't add up as expected. ( #2516 ) Contributed by Ahmed Hussein and Daryn Sharp
...
(cherry picked from commit 6a5864ee4a
)
2020-12-07 21:22:15 +00:00
Chao Sun
3b2a131885
HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 ( #2517 )
2020-12-04 10:42:46 -08:00
Hui Fei
6e215953e6
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-04 17:11:21 +08:00
Ahmed Hussein
f547cd43d1
HDFS-15706. HttpFS: Log more information on request failures. ( #2515 )
...
(cherry picked from commit 07655a7040
)
(cherry picked from commit a7dbd3b6f9
)
2020-12-03 15:15:31 -08:00
Ahmed Hussein
91310102d7
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit 60201cbf69
)
2020-12-02 21:52:52 +00:00
Ahmed Hussein
6cc0824072
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit 2b5b556dd7
)
2020-12-02 20:31:04 +00:00
Ahmed Hussein
625f85f40f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
...
(cherry picked from commit 918ba9e82f
)
2020-12-01 21:31:18 +00:00
Takanobu Asanuma
3ee0a174dd
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
...
(cherry picked from commit f56cd88d7d
)
2020-11-13 11:08:57 +09:00
Yizhong Zhang
52c2ce53c2
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
...
(cherry picked from commit 82de07c794
)
(cherry picked from commit 12bfd9194a
)
2020-11-10 09:14:25 -08:00
Giovanni Matteo Fumarola
4af5f55623
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor.
2020-11-10 14:44:34 +08:00
Ayush Saxena
ce883fe1bb
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:53:28 +05:30
Konstantin V Shvachko
23dcd8edb6
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit d07dc7afb4
)
2020-11-03 12:04:38 -08:00
Ahmed Hussein
dedf0d42bc
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4
)
2020-10-27 20:52:23 +09:00
Ahmed Hussein
244fd5eed2
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails ( #2404 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d332
)
2020-10-27 17:09:20 +09:00
Hui Fei
b32926f110
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
...
(cherry picked from commit 12c908c827
)
2020-10-26 22:34:06 +08:00
Wei-Chiu Chuang
87666e2438
HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.
...
(cherry picked from commit 74634eb002
)
(cherry picked from commit c8eb86ee3d
)
2020-10-23 12:06:36 -07:00
Inigo Goiri
7119af2c6d
HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.
...
(cherry picked from commit e7aa4da0a2
)
2020-10-23 09:49:46 -07:00
Kihwal Lee
f363c3b315
HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.
...
(cherry picked from commit da1b6e3cc2
)
2020-10-22 19:48:42 -05:00
Kihwal Lee
f3b2d85690
HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.
...
(cherry picked from commit cf932a7e2d
)
2020-10-22 14:59:09 -05:00
Takanobu Asanuma
0d6958962c
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. ( #2394 )
...
(cherry picked from commit 30f06e0c74
)
2020-10-20 19:13:20 +09:00
Kihwal Lee
7e4572e56a
HDFS-15627. Log delete audits before collecting blocks.
...
(cherry picked from commit 740a2c4635
)
2020-10-16 11:17:34 -05:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development
2020-10-16 14:52:41 +08:00
Kihwal Lee
392b26c23b
HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein.
...
(cherry picked from commit e45407128d
)
2020-10-14 18:19:16 -05:00
Akira Ajisaka
69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c2
)
2020-10-13 18:09:14 +09:00
Konstantin V Shvachko
23b6ab67f7
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit b3786d6c3c
)
2020-10-12 17:42:58 -07:00
Doroszlai, Attila
61563885f8
HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
2020-10-08 23:44:33 -07:00
Stephen O'Donnell
71d006130e
HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-09-28 12:45:28 -07:00
Kihwal Lee
25a51f2f39
HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross.
...
(cherry picked from commit dfc2682213
)
2020-09-22 11:12:43 -05:00
S O'Donnell
5f34e3214e
HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell
2020-09-22 10:34:05 +01:00
hemanthboyina
aa582ccc2a
HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.
2020-09-17 10:15:18 +05:30
Ayush Saxena
544d6da3ab
HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. ( #2261 )
2020-09-06 07:56:16 +05:30
Kitti Nanasi
9b8eaa0572
HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 14282e311b
)
2020-08-30 10:21:53 +05:30
Kitti Nanasi
3ba40d0ccf
HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5f15a60e32
)
2020-08-30 10:21:53 +05:30
Kitti Nanasi
0dbcb81a41
HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 951cdd7e4c
)
2020-08-30 10:21:53 +05:30
Kitti Nanasi
a78430abf4
HDFS-12946. Add a tool to check rack configuration against EC policies. Contributed by Kitti Nanasi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd5e7c6b72
)
2020-08-30 10:21:35 +05:30
Surendra Singh Lilhore
3bacea2e5e
HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena.
...
(cherry picked from commit 788e7473a4
)
2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172
HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported)
2020-08-26 09:33:51 -07:00
Uma Maheswara Rao G
978ce5a1ee
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false ( #2205 )
...
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false
* Used LambdaTestUtils#intercept in test
(cherry picked from commit 99b120a06e
)
2020-08-25 12:05:31 -07:00
S O'Donnell
a3f12f7f20
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
...
(cherry picked from commit 6a49bf9bff
)
2020-08-25 15:49:54 +01:00
Chen Liang
4ab14692e6
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
...
(cherry picked from commit f734455e5d
)
2020-08-20 15:26:44 -07:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
2020-08-19 17:15:06 +01:00
Gautham B A
ff907b310d
HADOOP-17196. Fix C/C++ standard warnings ( #2208 )
...
* Passing C/C++ standard flags -std is
not cross-compiler friendly as not all
compilers support all values.
* Thus, we need to make use of the
appropriate flags provided by CMake in
order to specify the C/C++ standards.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3
)
2020-08-11 16:47:26 +09:00
Ayush Saxena
d6a9ed8140
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:23:02 +05:30
Mingliang Liu
41c211d762
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion ( #2188 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-08-04 23:01:04 -07:00
Uma Maheswara Rao G
cd5efe91d9
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. ( #2160 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ac9a07b51a
)
2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G
65778cdd47
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links ( #2132 ). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 3e70006639
)
2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G
1369e41c65
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit dc0626b5f2
)
2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G
512d1d6d27
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 55a2ae80dc
)
2020-07-31 01:38:18 -07:00
Uma Maheswara Rao G
655b39cc30
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 1f2a80b5e5
)
2020-07-31 01:37:14 -07:00
Yuan
737bbab90a
HDFS-14950. fix missing libhdfspp lib in dist-package ( #1947 )
...
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit e756fe3590
)
2020-07-31 16:52:14 +09:00
Ayush Saxena
27a97e4f28
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-22 18:39:49 +05:30
Chen Liang
370efc6d95
HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.
...
(cherry picked from commit 3833c616e0
)
2020-07-20 14:07:10 -07:00
Akira Ajisaka
0d949d375e
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.
...
(cherry picked from commit 1defe3a65a
)
2020-07-20 13:20:34 -07:00
He Xiaoqiao
6d003f67dd
HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-16 23:59:05 +08:00
Ahmed Hussein
8fd3dcc9ce
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71
)
2020-07-15 12:05:49 -05:00
Ahmed Hussein
43a865dc07
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f
)
2020-07-15 10:18:47 -05:00
Shashikant Banerjee
acaee85a69
HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.
...
(cherry picked from commit 82343790ee
)
2020-07-15 13:38:29 +01:00
Eric Badger
37032bcaf6
Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell."
...
This reverts commit edb45cac82
.
2020-07-13 21:08:12 +00:00
He Xiaoqiao
edb45cac82
HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-13 14:40:21 +08:00
Uma Maheswara Rao G
7607f24c9e
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 3ca15292c5
)
2020-07-09 16:29:02 -07:00
Shanyu Zhao
8d4028864d
HDFS-15451. Do not discard non-initial block report for provided storage. ( #2119 ). Contributed by Shanyu Zhao.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-07-07 11:53:29 +08:00
Ayush Saxena
642aafb106
HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.
2020-07-04 12:25:54 +05:30
Akira Ajisaka
37e0a1208c
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
...
(cherry picked from commit c71ce7ac33
)
2020-06-28 16:16:49 +09:00
Virajith Jalaparti
704b273ec8
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable ( #2100 )
...
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable
* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests
* Address Uma's comments on PR#2100
* Sort lists in test to match without concern to order
* Address comments, fix checkstyle and fix failing tests
* Fix checkstyle
(cherry picked from commit bed0a3a374
)
2020-06-27 16:25:04 -07:00
Uma Maheswara Rao G
d030f4b2a6
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit d5e1bb6155
)
2020-06-27 15:56:10 -07:00
Uma Maheswara Rao G
80111fe5bb
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 7c02d1889b
)
2020-06-27 15:55:41 -07:00
Uma Maheswara Rao G
6c22210baa
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit b27810aa60
)
2020-06-27 15:55:11 -07:00
Uma Maheswara Rao G
4a9ec4d143
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 93b121a971
)
2020-06-27 15:54:37 -07:00
Uma Maheswara Rao G
aa8de2f43b
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 785b1def95
)
(cherry picked from commit 120ee793fc
)
2020-06-27 15:53:07 -07:00
Ayush Saxena
e80421820a
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
...
(cherry picked from commit cc671b16f7
)
(cherry picked from commit bee2846bee
)
2020-06-27 15:52:47 -07:00
Uma Maheswara Rao G
e7cead1143
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 76fa0222f0
)
(cherry picked from commit 418580446b
)
2020-06-27 15:52:18 -07:00
Uma Maheswara Rao G
852ee71354
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ed83c865dd
)
(cherry picked from commit 0b5e202614
)
2020-06-27 15:51:51 -07:00
Uma Maheswara Rao G
84ceb6d520
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 4734c77b4b
)
(cherry picked from commit 8e71e85af7
)
2020-06-27 15:51:28 -07:00
Abhishek Das
7cf9601987
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.
...
(cherry picked from commit ce4ec74453
)
(cherry picked from commit 5b248de42d
)
2020-06-27 15:50:41 -07:00
Uma Maheswara Rao G
3dcc9aed4d
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit ac4a2e11d9
)
(cherry picked from commit 544996c857
)
2020-06-27 15:49:48 -07:00
S O'Donnell
240974932b
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
...
(cherry picked from commit 123777823e
)
2020-06-18 12:44:42 +01:00
Eric Yang
e9cf454ac2
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0
)
2020-06-15 11:12:40 +09:00
Takanobu Asanuma
695d20615d
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
...
(cherry picked from commit f41a144077
)
2020-06-15 09:18:22 +09:00
Ayush Saxena
452d801f1b
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:48:48 +05:30
Toshihiro Suzuki
e5a02a7499
HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
(cherry picked from commit 545a0a147c
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
2020-06-05 12:26:52 +01:00
Takanobu Asanuma
d6de530bbb
HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.
...
(cherry picked from commit 4d22d1c58f
)
2020-05-23 17:31:59 +09:00
Chen Liang
1813d25bf2
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang
2020-05-18 12:05:47 -07:00
Akira Ajisaka
dfb094d581
Revert "SPNEGO TLS verification"
...
This reverts commit 9db6e96db3
.
2020-05-13 17:15:46 +09:00
Ayush Saxena
b706ac11f7
HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.
2020-05-12 18:44:56 +05:30
Uma Maheswara Rao G
0d0aac1a2a
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.
...
(cherry picked from commit 9c8236d04d
)
2020-05-11 15:29:26 -07:00
Virajith Jalaparti
e5e824bd6f
HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun.
2020-05-11 12:50:14 -07:00
Babneet Singh
44fdee351d
HDFS-15270. Account for *env == NULL in hdfsThreadDestructor ( #1951 )
...
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.
The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.
Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b
)
(cherry picked from commit d1e5e393c3
)
2020-05-11 10:46:10 -07:00
Ayush Saxena
06f43fc628
HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest.
2020-05-10 11:44:43 +05:30
Ayush Saxena
c3fffa8451
HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.
2020-05-08 00:57:24 +05:30
Inigo Goiri
53f6d26500
HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao.
...
(cherry picked from commit 4a212242d9
)
2020-05-04 18:04:07 -07:00
Konstantin V Shvachko
e95a4ddbb1
HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit ebb878bab9
)
2020-05-04 10:47:07 -07:00
Ayush Saxena
d451c9e2a4
Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.
2020-05-03 19:17:44 +05:30
Eric Yang
9db6e96db3
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454
)
2020-05-02 21:51:41 +09:00
Akira Ajisaka
5154f6eef4
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.
...
(cherry picked from commit 5e0eda5d5f
)
2020-04-28 06:19:17 +09:00
Mingliang Liu
57c1e7ad9b
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena
2020-04-25 15:44:27 -07:00
Dhiraj
6225fce50d
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address ( #1964 )
...
Contributed by Dhiraj Hegde.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:08:32 -07:00
Wei-Chiu Chuang
9e3b4c9ebd
HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.
...
(cherry picked from commit 453771f170
)
(cherry picked from commit d6c48f8100
)
2020-04-24 11:54:20 -07:00
Mingliang Liu
e154d9d9f7
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina
2020-04-22 13:56:15 -07:00
Inigo Goiri
7d64d44515
HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena.
...
(cherry picked from commit efdfe679d6
)
2020-04-21 09:50:20 -07:00
Stephen O'Donnell
041c93a26c
HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a
)
(cherry picked from commit aaad947c74
)
2020-04-16 20:19:14 -07:00
Eric Badger
17fc8e4a64
Revert "HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago."
...
This reverts commit 26b51f3e22
.
2020-04-02 19:24:42 +00:00
Chen Liang
10fe37b704
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.
2020-03-27 13:01:00 -07:00
Ayush Saxena
6cd9290401
HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li.
...
(cherry picked from commit 62deab17a3
)
2020-03-25 17:12:52 -07:00
Eric Yang
ba6b3a3848
HDFS-14434. Ignore user.name query parameter in secure WebHDFS.
...
Contributed by KWON BYUNGCHANG
(cherry picked from commit d78854b928
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java
2020-03-25 17:12:01 -07:00
Giovanni Matteo Fumarola
eca7bc7ac4
HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota.
...
(cherry picked from commit 00d5e631b5
)
2020-03-25 17:06:17 -07:00
Isa Hekmatizadeh
152cbc6457
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error ( #1902 ). Contributed by zhengchenyu.
...
(cherry picked from commit d9c4f1129c
)
2020-03-25 16:52:52 -07:00
Ayush Saxena
1e3b0df6ab
HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun.
...
(cherry picked from commit 6191d4b4a0
)
2020-03-25 16:41:48 -07:00
Yiqun Lin
26b51f3e22
HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
...
(cherry picked from commit 2b452b4e60
)
2020-03-25 16:31:42 -07:00
Ayush Saxena
853eafa81a
HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.
...
(cherry picked from commit bb41ddaf1e
)
2020-03-25 16:31:34 -07:00
Akira Ajisaka
1bbbef38d0
HDFS-15232. Fix libhdfspp test failures with GCC 7. ( #1906 )
...
(cherry picked from commit f59f6891c8
)
2020-03-24 11:38:12 +09:00
Wei-Chiu Chuang
1d55e1e229
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
...
Contributed by Wei-Chiu Chuang.
Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
(cherry picked from commit 096533c2dc
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties
2020-03-18 09:58:35 -07:00
Surendra Singh Lilhore
e9916052b1
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
2020-03-15 20:53:45 +05:30
Ayush Saxena
87532b6e3f
HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang.
2020-03-15 16:31:20 +05:30
S O'Donnell
b7f33e2e37
HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.
...
(cherry picked from commit 25497799cb
)
2020-03-13 15:34:20 +00:00
Surendra Singh Lilhore
269aa25a73
HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.
2020-03-12 19:10:27 +05:30
Surendra Singh Lilhore
57f887a529
HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree.
2020-03-10 23:56:42 +05:30
Inigo Goiri
1d89ba72f6
HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.
...
(cherry picked from commit 1d4d0fcbe1
)
2020-03-04 10:04:22 -08:00
Akshay Nehe
369f4f9c58
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. ( #1871 )
...
(cherry picked from commit bbd704bb82
)
2020-03-04 11:12:00 +09:00
Surendra Singh Lilhore
d33aec93f1
HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.
2020-02-28 18:43:57 +05:30
Ayush Saxena
9acbbd885c
HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui.
...
(cherry picked from commit df622cf4a3
)
2020-02-27 16:36:53 -08:00
Masatake Iwasaki
e1ff3b2f97
HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1
)
2020-02-27 16:36:46 -08:00
Konstantin V Shvachko
c82e20f9f8
HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit ecbcb058b8
)
2020-02-27 15:29:31 -08:00
Istvan Fajth
e42ac486e7
HDFS-14668 Support Fuse with Users from multiple Security Realms ( #1739 )
...
(cherry picked from commit 57aa048516
)
2020-02-27 07:49:31 -08:00
Kihwal Lee
27cfda708e
HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed
...
by Ahmed Hussein.
2020-02-27 09:17:44 -06:00
Kihwal Lee
de7edf58bd
Revert "HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by"
...
Incorrect commit message
This reverts commit 749d7c0027
.
2020-02-27 09:16:55 -06:00
Kihwal Lee
749d7c0027
HADOOP-16888. [JDK11] Support JDK11 in the precommit job. Contributed by
...
Ahmed Hussein.
2020-02-27 09:13:20 -06:00
Ayush Saxena
b182dd809c
HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.
2020-02-27 04:02:55 +05:30
Stephen O'Donnell
2377649cdb
HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b990
)
2020-02-25 13:29:03 -08:00
Wei-Chiu Chuang
f9437b1817
HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.
...
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
2020-02-24 13:37:48 -08:00
Ayush Saxena
af46658baf
HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li.
2020-02-24 20:17:31 +05:30
Konstantin V Shvachko
769397201a
HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit 6f84269bcd
)
2020-02-21 10:54:20 -08:00
Masatake Iwasaki
610805ec72
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. ( #1758 )
...
(cherry picked from commit 2338d25dc7
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
2020-02-21 13:40:11 +09:00
xiaoxiaopan118
b5321f1600
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' ( #1849 )
...
(cherry picked from commit 439d935e1d
)
2020-02-17 15:57:15 +09:00
Ayush Saxena
9a351e837f
HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena.
2020-02-14 22:15:12 +05:30
Surendra Singh Lilhore
e13ea1b094
HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina.
2020-02-13 22:52:50 +05:30
Ayush Saxena
9a1ce410f6
HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun
2020-02-12 20:29:09 +05:30
Kihwal Lee
f1840669cf
HDFS-14758. Make lease hard limit configurable and reduce the default.
...
Contributed by hemanthboyina.
(cherry picked from commit 9b8a78d97b
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java
2020-02-11 14:50:10 -06:00
Ayush Saxena
7be104a9be
HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang
2020-02-08 10:41:15 +05:30
Chen Liang
226cd3977b
HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.
2020-02-04 12:34:19 -08:00
Kihwal Lee
5257afb153
HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan.
...
(cherry picked from commit 10a60fbe20
)
2020-02-04 12:24:12 -06:00
Stephen O'Donnell
e89436b183
HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d93
)
2020-01-31 16:13:42 -08:00
Inigo Goiri
2c8c6e8e22
HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein.
...
(cherry picked from commit 1839c467f6
)
2020-01-31 11:24:37 -08:00
Kihwal Lee
7034482222
HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails
...
intermittently. Contributed by Ahmed Hussein.
(cherry picked from commit 799d4c1cf4
)
2020-01-29 11:01:12 -06:00
Chen Liang
9813ba2ebc
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang
2020-01-28 16:36:27 -08:00