Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,668 Commits 326 Branches 371 Tags
Commit Graph

4990 Commits

Author SHA1 Message Date
Ayush Saxena
4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Chen Liang
1e33696bc8 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
(cherry picked from commit 370efc6d95fde8065dd5ba742446d0e9e570a5e9)
 2020-07-20 15:24:07 -07:00
Akira Ajisaka
635067d03d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
(cherry picked from commit 0d949d375e3e7602978f094f7ef7952783ab8421)
 2020-07-20 13:37:08 -07:00
Mukund Thakur
cb3e378782 HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-07-17 10:40:01 +01:00
Ahmed Hussein
b7b9bd32db HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
 2020-07-15 12:19:42 -05:00
Erik Krogen
0de48933c5 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:33:38 -07:00
Siyao Meng
47df2245d3 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9d59cc46882df506dcea386020b1e4d)
 2020-07-09 16:32:07 -07:00
Uma Maheswara Rao G
4e1abe61a2 HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3ca15292c5584ec220b3eeaf76da85d228bcbd8b)
 2020-07-09 16:29:36 -07:00
Abhishek Das
5f84093cf8 HADOOP-17032. Fix getContentSummary in ViewFileSystem to handle multiple children mountpoints pointing to different filesystems (#2060). Contributed by Abhishek Das. 2020-07-01 13:02:00 +05:30
Virajith Jalaparti
c4e0098bd6 HDFS-15436. Default mount table name used by ViewFileSystem should be configurable (#2100) 
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable

* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests

* Address Uma's comments on PR#2100

* Sort lists in test to match without concern to order

* Address comments, fix checkstyle and fix failing tests

* Fix checkstyle

(cherry picked from commit bed0a3a37404e9defda13a5bffe5609e72466e46)
 2020-06-27 16:25:26 -07:00
Uma Maheswara Rao G
30d1d29076 HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit d5e1bb6155496cf9d82e121dd1b65d0072312197)
 2020-06-27 16:09:36 -07:00
Uma Maheswara Rao G
81e62613fb HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 7c02d1889bbeabc73c95a4c83f0cd204365ff410)
 2020-06-27 16:09:16 -07:00
Uma Maheswara Rao G
e17770dec6 HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit b27810aa6015253866ccc0ccc7247ad7024c0730)
 2020-06-27 16:08:57 -07:00
Uma Maheswara Rao G
208701329c HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 93b121a9717bb4ef5240fda877ebb5275f6446b4)
 2020-06-27 16:08:38 -07:00
Ayush Saxena
b720d7770d HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 
(cherry picked from commit a8610c15c498531bf3c011f1b0ace8eddddf61f2)
 2020-06-27 16:08:10 -07:00
Abhishek Das
0e9b6b0124 HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 
(cherry picked from commit e7dd02768b658b2a1f216fbedc65938d9b6ca6e9)
 2020-06-27 16:07:49 -07:00
Uma Maheswara Rao G
100c13967e HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 785b1def959fab6b8b7ffff66410bcd240feee13)
(cherry picked from commit 120ee793fc4bcbf9d1945d5e38e3ad5b2b290a0e)
 2020-06-27 16:07:28 -07:00
Uma Maheswara Rao G
2e1dfc152b HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ed83c865dd0b4e92f3f89f79543acc23792bb69c)
(cherry picked from commit 0b5e202614f0bc20a0db6656f924fa4d2741d00c)
 2020-06-27 16:06:04 -07:00
Uma Maheswara Rao G
6ae92962d9 HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 4734c77b4b64b7c6432da4cc32881aba85f94ea1)
(cherry picked from commit 8e71e85af70c17f2350f794f8bc2475eb1e3acea)
 2020-06-27 16:05:36 -07:00
Abhishek Das
7015589f58 HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das. 
(cherry picked from commit ce4ec7445345eb94c6741d416814a4eac319f0a6)
(cherry picked from commit 5b248de42d2ae42710531a1514a21d60a1fca4b2)
 2020-06-27 16:05:09 -07:00
Uma Maheswara Rao G
c4857bb9c1 HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac4a2e11d98827c7926a34cda27aa7bcfd3f36c1)
(cherry picked from commit 544996c85702af7ae241ef2f18e2597e2b4050be)
 2020-06-27 16:04:23 -07:00
Kihwal Lee
400c5accd9 HADOOP-16255. Add ChecksumFs.rename(path, path, boolean) 
Contributed by Jungtaek Lim

Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
(cherry picked from commit bb0b922a71cba9ceaf00588e9f3e3b2a3c2e3eab)
 2020-06-08 17:25:15 -05:00
Mingliang Liu
ac0e928564
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:32:32 -07:00
Mingliang Liu
101ce83f37
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:51:36 -07:00
Dhiraj
2c40f20deb
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:07:25 -07:00
S O'Donnell
0cd4459f4e HADOOP-7002. Wrong description of copyFromLocal and copyToLocal in documentation. Contributed by Andras Bokor. 
(cherry picked from commit 19f26a020e2e5cec2ceb28d796c63c83bc8ac506)
 2020-05-29 14:53:59 +01:00
S O'Donnell
1f674e26ba HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 2020-05-29 12:25:35 +01:00
Akira Ajisaka
533e51f97e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79ed053ce978ac18a2c5706d32e58019f)
 2020-05-18 15:38:01 +09:00
Uma Maheswara Rao G
fda93334dc HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04dfc3d4cefe7a00b63625f60ee232cfe)
 2020-05-12 21:33:49 -07:00
Virajith Jalaparti
bec5d7baa0 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 19:37:40 -07:00
Ayush Saxena
41df5bf835 HADOOP-16886. Add hadoop.http.idle_timeout.ms to core-default.xml. Contributed by Lisheng Sun. 2020-05-01 14:28:45 +05:30
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Dhiraj
68dcdad666
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:53:32 -07:00
Mingliang Liu
52ea5a0227
HADOOP-17001. The suffix name of the unified compression class. Contributed by bianqi 2020-04-22 13:08:03 -07:00
Yiqun Lin
f1a19b7a3f HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2020-03-25 16:31:54 -07:00
Kihwal Lee
b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708ef66dfbe5f52a5f1e716298a294f3f7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Takanobu Asanuma
241c5a14c5 HADOOP-16841. The description of hadoop.http.authentication.signature.secret.file contains outdated information. Contributed by Xieming Li. 
(cherry picked from commit 5cbc4c54611f062690c3bfcf78d09d37c78ffd12)
 2020-02-25 11:10:22 +09:00
Wei-Chiu Chuang
b28347d282 HADOOP-16868. ipc.Server readAndProcess threw NullPointerException. Contributed by Tsz-wo Sze. 
(cherry picked from commit cb3f3cca01d6ab9f4befb1bcd31f384c37c0231a)
(cherry picked from commit d411a849e10910d24cf1bbc8d64405470e00f510)
 2020-02-18 21:54:00 -08:00
Kihwal Lee
a628ab0863 HADOOP-15787. TestIPC.testRTEDuringConnectionSetup fails with jdk8u242. Contributed by Zsolt Venczel. 
(cherry picked from commit a463cf75a0ab1f0dbb8cfa16c39a4e698bc1a625)
(cherry picked from commit 0b19c8e0860bf558d995951ab9ac4cd8a1674759)
 2020-02-05 13:16:34 -06:00
Chen Liang
466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 
(cherry picked from commit 226cd3977bfc5948afff6a2d38a41e341089d5fe)
 2020-02-04 12:36:46 -08:00
sunlisheng
96c653d0d5 Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure() 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit d887e49dd4ed2b94bbb53b7608586f5da6cee037)
 2020-01-21 18:32:14 +09:00
Szilard Nemeth
3a09593a73 HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal 2020-01-14 11:19:52 +01:00
Wei-Chiu Chuang
286c94f13d HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 
(cherry picked from commit 6b86a5110e4b3f45bf55f97bd402680bf26cefb9)
(cherry picked from commit aa3c96474f52fba01d999fa75880e0948affbb01)
 2020-01-13 10:49:39 -08:00
Chen Liang
5490061885 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)
 2020-01-10 17:19:05 -08:00
Daryn Sharp
da45684185 HADOOP-16749. Configuration parsing of CDATA values are blank 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit c4fb43c94259546f2c96b50ceae9cd3adb726166)
 2020-01-10 10:39:18 -06:00
Rakesh Radhakrishnan
8b748c0308 HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 
(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)
 2020-01-02 12:19:38 +05:30
Chandra Sanivarapu
4acbe1c05d
HADOOP-16042. Update the link to HadoopJavaVersion (#1784) 
(cherry picked from commit d80d86264d8e4b1a5cfc6fc12eec8f9dfcd76b3f)
 2019-12-27 17:21:29 +09:00
Erik Krogen
b06965ae32 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30abc08a4642614fb35165e79792608b)
(cherry picked from 3402c87353c18730cb659ac1d8b43b4e21c634f6)
 2019-12-11 09:41:11 -08:00
Mingliang Liu
941a98eb28
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:52:25 -08:00
Aravindan Vijayan
e73fa45e5b HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Siyao Meng <smeng@cloudera.com>
(cherry picked from commit f1ab7f18c423a9cfc59292d25fa178e73715b85b)
(cherry picked from commit f0c1403ec382a7a8c25b0311db0c88749576c308)
 2019-12-03 15:48:25 -08:00