Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,676 Commits 326 Branches 371 Tags
Commit Graph

8475 Commits

Author SHA1 Message Date
Ahmed Hussein
91310102d7 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69044a0770390f674d93f57e4bab3880)
 2020-12-02 21:52:52 +00:00
Ahmed Hussein
6cc0824072 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7f17f095548b4a4485e7b7218af74be)
 2020-12-02 20:31:04 +00:00
Ahmed Hussein
625f85f40f HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82fb0bc6f96301c7ca74a32018b60c49e)
 2020-12-01 21:31:18 +00:00
Takanobu Asanuma
3ee0a174dd HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d9381d13329d25bb4f367e5378b6b5b)
 2020-11-13 11:08:57 +09:00
Yizhong Zhang
52c2ce53c2 HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794fe119946bec2e563c1937f312e8ac9)
(cherry picked from commit 12bfd9194a1b6ff89fb31f1f49e69c727f844155)
 2020-11-10 09:14:25 -08:00
Giovanni Matteo Fumarola
4af5f55623
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2020-11-10 14:44:34 +08:00
Ayush Saxena
ce883fe1bb HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:53:28 +05:30
Konstantin V Shvachko
23dcd8edb6 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4aa0d6cc9f9be530802e54610776a4d)
 2020-11-03 12:04:38 -08:00
Ahmed Hussein
dedf0d42bc
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac43a480a010d40c182c1d4cc5346db8f)
 2020-10-27 20:52:23 +09:00
Ahmed Hussein
244fd5eed2
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d3325e0b454475fa01cc50f68007b0e15c)
 2020-10-27 17:09:20 +09:00
Hui Fei
b32926f110 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827c80ea37ed5207a65831d0b699c8381)
 2020-10-26 22:34:06 +08:00
Wei-Chiu Chuang
87666e2438 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002717a9684d00f0e9dc263ab7eb49246)
(cherry picked from commit c8eb86ee3d4f997b72e87e139e6a4dafa1ff4941)
 2020-10-23 12:06:36 -07:00
Inigo Goiri
7119af2c6d HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a26988fcf5880311067e62a15e05fc1e)
 2020-10-23 09:49:46 -07:00
Kihwal Lee
f363c3b315 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc286db00b385f3280627d2b2063b4e59)
 2020-10-22 19:48:42 -05:00
Kihwal Lee
f3b2d85690 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d6182471df4eba1333737912a32534b)
 2020-10-22 14:59:09 -05:00
Takanobu Asanuma
0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:20 +09:00
Kihwal Lee
7e4572e56a HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c46353f8005dbed6f5bc15f21acfc4a6a23)
 2020-10-16 11:17:34 -05:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Kihwal Lee
392b26c23b HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d4e9a9804c777c8f845ad41e1280177)
 2020-10-14 18:19:16 -05:00
Akira Ajisaka
69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c22ccd60a8560a80e2d84129b359e630)
 2020-10-13 18:09:14 +09:00
Konstantin V Shvachko
23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:42:58 -07:00
Doroszlai, Attila
61563885f8 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:33 -07:00
Stephen O'Donnell
71d006130e HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-09-28 12:45:28 -07:00
Kihwal Lee
25a51f2f39 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 
(cherry picked from commit dfc268221352880992c55abec9e6b7b73044b0f1)
 2020-09-22 11:12:43 -05:00
S O'Donnell
5f34e3214e HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 2020-09-22 10:34:05 +01:00
hemanthboyina
aa582ccc2a HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 10:15:18 +05:30
Ayush Saxena
544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Kitti Nanasi
9b8eaa0572 HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 14282e311be6ffcaddd2f74fa8e67c4e98a32291)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi
3ba40d0ccf HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5f15a60e329b8d0a495a977dc5255126814c5271)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi
0dbcb81a41 HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 951cdd7e4cbe68284620f6805f85c51301150c58)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi
a78430abf4 HDFS-12946. Add a tool to check rack configuration against EC policies. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd5e7c6b7239a93f2391beaa11181e442a387db4)
 2020-08-30 10:21:35 +05:30
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Uma Maheswara Rao G
978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:05:31 -07:00
S O'Donnell
a3f12f7f20 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bffbe9bad1c719fc3a3b734f0060df70a)
 2020-08-25 15:49:54 +01:00
Chen Liang
4ab14692e6 HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d76c75b7d3a0b751023f5bd02ba38d2)
 2020-08-20 15:26:44 -07:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Gautham B A
ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3e956bdd01b9d24ed2108f93bb94953)
 2020-08-11 16:47:26 +09:00
Ayush Saxena
d6a9ed8140 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:23:02 +05:30
Mingliang Liu
41c211d762
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:01:04 -07:00
Uma Maheswara Rao G
cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G
65778cdd47 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e700066394fb9f516e23537d8abb4661409cae1)
 2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G
1369e41c65 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2f2ba0bd3919650ea231cedd424f77a)
 2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G
512d1d6d27 HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc9b45413febd33840b8a653e3e29440)
 2020-07-31 01:38:18 -07:00
Uma Maheswara Rao G
655b39cc30 HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5024aeb7fb1f8c31b8fdd0fdb88bb66)
 2020-07-31 01:37:14 -07:00
Yuan
737bbab90a
HDFS-14950. fix missing libhdfspp lib in dist-package (#1947) 
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit e756fe3590906bfd8ffe4ab5cc8b9b24a9b2b4b2)
 2020-07-31 16:52:14 +09:00
Ayush Saxena
27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Chen Liang
370efc6d95 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
 2020-07-20 14:07:10 -07:00
Akira Ajisaka
0d949d375e HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
 2020-07-20 13:20:34 -07:00
He Xiaoqiao
6d003f67dd HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-16 23:59:05 +08:00