Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,488 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

963 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. (#3054) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-08 19:32:34 -07:00
Yiqun Lin ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2021-05-24 10:48:48 +08:00
Wei-Chiu Chuang fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
Karthik Palanisamy 00c3f3c42f HDFS-15865. Interrupt DataStreamer thread if no ack (#2728) 
(cherry picked from commit bd3da73a0f)
 2021-05-04 11:11:36 +01:00
litao f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 (#2940) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-27 22:39:25 +09:00
litao 7a7e194228 HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9e2d5d6d05)
 2021-04-05 17:49:27 +09:00
Gautham B A e4d04a640b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74)
 2021-04-05 11:42:52 +09:00
Ayush Saxena 5187bd37ae HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 
(cherry picked from commit 37d6582223)
 2021-04-02 14:06:52 +08:00
Borislav Iordanov c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
Akira Ajisaka a407f7de5a HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 
(cherry picked from commit 32895f4f7e)
 2021-03-30 22:29:49 +08:00
Stephen O'Donnell 56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
leosunli b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf42)
 2021-03-25 13:25:17 +00:00
sunlisheng a8c0083b1b HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 
(cherry picked from commit 7025f39944)
 2021-03-24 14:04:47 +08:00
Ayush Saxena 94766fdb13 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 
(cherry picked from commit ff84a57483)
 2021-03-24 14:04:47 +08:00
imbajin 4f79df4da1 HDFS-15551. Tiny Improve for DeadNode detector (#2265) 
Contributed by imbajin.

Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 89428f142f)
 2021-03-24 14:04:47 +08:00
sunlisheng 3ecd02ac78 HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 
(cherry picked from commit f8769e0f4b)
 2021-03-24 14:04:47 +08:00
Ayush Saxena b26d75ce37 HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena. 
(cherry picked from commit e0ae232f66)
 2021-03-23 15:34:48 +00:00
Ayush Saxena 9665a0be73 HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:18 +05:30
Surendra Singh Lilhore a6f86af39f HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
 2021-02-26 10:49:16 +08:00
Steve Loughran 98e4d516ea
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.

Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
 2021-02-10 10:31:22 +00:00
Jim Brennan 62389a5a04 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 22:16:36 +00:00
Ayush Saxena 8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Chao Sun 4628647067 HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:43:25 -08:00
Akira Ajisaka be108159b1
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. (#2465) 
(cherry picked from commit b57f04cd5b)
 2020-11-17 11:01:23 +09:00
Takanobu Asanuma 0bb1f0df27 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:12:26 +09:00
Konstantin V Shvachko b6423d2780 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:38:42 -07:00
Akira Ajisaka 800b1ed1c2 Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-12 11:59:17 -07:00
Doroszlai, Attila 13e0c5f6e0 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-10-08 23:44:56 -07:00
Uma Maheswara Rao G 2ce5846bfa HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:23:35 +05:30
Uma Maheswara Rao G 1fc1b34633 HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured (#2275) Contributed by Uma Maheswara Rao G. 2020-09-18 15:23:12 +05:30
Uma Maheswara Rao G ba0eca6a2c HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. (#2229). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dd013f2fdf)
 2020-08-25 12:00:52 -07:00
Ayush Saxena e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Steve Loughran a51d72f0c6 HDFS-13934. Multipart uploaders to be created through FileSystem/FileContext. 
Contributed by Steve Loughran.

Change-Id: Iebd34140c1a0aa71f44a3f4d0fee85f6bdf123a3
 2020-07-13 13:32:04 +01:00
Uma Maheswara Rao G 3cddd0be29 HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a971)
 2020-06-27 15:39:38 -07:00
Eric Yang d73cdb1c86
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0)
 2020-06-15 11:12:08 +09:00
Vinayakumar B 534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Ayush Saxena 043628dcf1 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:09:34 +05:30
Wei-Chiu Chuang 240cba7e6d HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
(cherry picked from commit 2abcf7762a)
 2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang 289f8acc64 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit ad9a6a0ee3.
 2020-05-18 09:41:03 -07:00
pustota2009 ad9a6a0ee3 HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

(cherry picked from commit 86e6aa8eec)
 2020-05-18 07:05:03 -07:00
Surendra Singh Lilhore 7cef7109de YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:17:13 +05:30
Ayush Saxena 74887ab2ad YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:09 +05:30
Akira Ajisaka ee58c112d0
Revert "SPNEGO TLS verification" 
This reverts commit 0f27c04c23.
 2020-05-13 17:14:51 +09:00
S O'Donnell 433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena 61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Ayush Saxena 6ccb1cd5f1 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Ayush Saxena 4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Eric Yang 0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:51:12 +09:00
Akira Ajisaka dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
Chen Liang f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00