Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,944 Commits 326 Branches 371 Tags
Commit Graph

528 Commits

Author SHA1 Message Date
Weiwei Yang
6c8cfd135e HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:53:28 +08:00
Inigo Goiri
9aae8b05a7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:03:58 -07:00
Wei-Chiu Chuang
79f8e4d5bc HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
(cherry picked from commit fc8983a5b8c94746d3e58cc203974aa09c1b640b)
 2019-06-17 20:20:21 -07:00
huzheng
2b9c4fccf4 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
(cherry picked from commit b082628e5a39fc1bc9d6094beb9c253cc096c701)
 2019-06-17 07:21:46 -07:00
Takanobu Asanuma
abb06c9d7c HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb48ca26a629d4d5f3f07e2858d09594)
 2019-03-27 03:30:12 +09:00
Erik Krogen
9a5daa5ec6 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
(cherry picked from commit fec7c5f3ebbaea7b290e0904570c3a485b541a22)
(cherry picked from commit cb76fc5618c582887e4e71b98f3065c5a916334a)
 2019-03-13 13:21:40 -07:00
Dinesh Chitlangia
36ff7a4d02 HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cfcf1ae3355cd880402159cf31e94a8a)
(cherry picked from commit eb77e60163e318ca42548e134fe79dba7289fdc6)
(cherry picked from commit b63941516f78e0d6e22ecc629fc82b16fd17ad87)
 2019-01-25 07:38:49 -08:00
Brahma Reddy Battula
6895550043 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 
(cherry picked from commit f534736867eed962899615ca1b7eb68bcf591d17)
 2018-11-30 00:21:07 +05:30
Inigo Goiri
c4d390a775 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 
(cherry picked from commit 887244de4adebe27693ed4ad3296a6f700cfa8c1)
 2018-11-06 11:21:19 -08:00
Xiao Chen
20c9a12bc7 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824a36b90ba1c8e9b2fba1162771700fe)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java

(cherry picked from commit 399645ebc162371a63eb81840e01ed52261f43bc)
 2018-10-29 19:14:12 -07:00
Xiao Chen
e8832418bf Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
(cherry picked from commit 96cedb87b94c07c11152580bf36978186d622b50)
 2018-10-24 16:10:51 -07:00
Xiao Chen
b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen
ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Hrishikesh Gadre
b170de8be5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 323b76bccfa153ef5ba52dc14876283d05618739)
 2018-10-09 19:54:56 -07:00
Siyao Meng
2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Wei-Chiu Chuang
823b1fceee HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a725cd06d28fb51361c49bb45464f5368)
(cherry picked from commit 7442b7941dc57ebf4bd0ded4d46cfbf5af6fea81)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2018-09-19 03:20:26 -07:00
Wei-Chiu Chuang
2ef3911f7c HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5fb9af403db2a02814575f0940d5f39)
(cherry picked from commit 829399a9f38db40b5bb247cb9c0d5f930c094e37)
 2018-09-18 15:48:04 -07:00
Wei-Chiu Chuang
d06717ef94 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit d89a485b2545c759a0f62810ff9757ede6d622f2.
 2018-08-30 11:59:39 -07:00
Wei-Chiu Chuang
d89a485b25 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit 26c2a97c566969f50eb8e8432009724c51152a98)
(cherry picked from commit 41732e7b28bb70f7ba56cf6131ac10c062f93fc3)
 2018-08-27 16:04:16 -07:00
Konstantin V Shvachko
27abdb390b HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 
(cherry picked from commit a4121c71c29d13866a605d9c0d013e5de9c147c3)
 2018-08-24 18:51:42 -07:00
Xiao Chen
185c8f2abc HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 
(cherry picked from commit c67b0650ea10896c6289703595faef0d262c00b3)
 2018-08-16 23:15:58 -07:00
Daniel Templeton
3210b3d8aa HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica 
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db187224095b13fe297a4d7377fbb9d2cd)
(cherry picked from commit 00c476abd8f1d34414b646219856859477558458)
 2018-07-24 16:13:30 -07:00
Wei-Chiu Chuang
68290606ea HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel. 
(cherry picked from commit 0247cb6318507afe06816e337a19f396afc53efa)
(cherry picked from commit f5f4d0b7e72eaf2ea9c9e791f71d0ea95f5b4b20)
 2018-07-06 15:01:35 -07:00
Andrew Wang
896b51578f HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas. 
(cherry picked from commit 344f324710522ffb27852c1a673c4f7d3d6eac4b)
 2018-07-03 11:08:05 +02:00
Wei-Chiu Chuang
563d56be62 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota. 
(cherry picked from commit fef20a446f7bf9f29e0f0ee690987fb6fc78a031)
(cherry picked from commit 2ce892fcf57b7d32af4f423d0ae9025f7c39b6e3)
 2018-07-02 09:44:16 -07:00
Andrew Wang
d34260f122 HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon. 
(cherry picked from commit 5d748bd056a32f2c6922514cd0c5b31d866a9919)
 2018-07-02 12:11:14 +02:00
Andrew Wang
7cd769b31c HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon. 
(cherry picked from commit 6ba99741086170b83c38d3e7e715d9e8046a1e00)
 2018-07-02 12:02:27 +02:00
Inigo Goiri
6eea6d28de HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
 2018-06-11 11:18:04 -07:00
Arpit Agarwal
ea05171ceb HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:58:02 -07:00
Chao Sun
d62b0f84f1 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun. 
(cherry picked from commit fba1c42adc1c8ae57951e1865ec2ab05c8707bdf)
 2018-06-08 16:42:18 -07:00
Xiao Chen
3843abe482 HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel. 
(cherry picked from commit ba4011d64fadef3bee5920ccedbcdac01794cc23)
 2018-06-05 21:19:19 -07:00
Yongjun Zhang
f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Andrew Wang
731abae557 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient. 
(cherry picked from commit c9b63deb533274ca8ef4939f6cd13f728a067f7b)
 2018-05-24 09:56:40 -07:00
Sammi Chen
6db710b9d8 HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen. 
(cherry picked from commit 34e8b9f9a86fb03156861482643fba11bdee1dd4)
(cherry picked from commit 7d71b3a1cc0f2572d8643bc2faeb878dfe028b8b)
 2018-05-23 19:20:32 +08:00
Andrew Wang
5b3a6e3678 HDFS-13601. Optimize ByteString conversions in PBHelper. 
(cherry picked from commit 1d2640b6132e8308c07476badd2d1482be68a298)
(cherry picked from commit d47c0fc39a48dace6c7a2332ee041885cc4bb89c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/dev-support/findbugsExcludeFile.xml
 2018-05-22 23:57:05 -07:00
Tsz-Wo Nicholas Sze
2acd8e4ecc HDFS-12813. RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1. Contributed by Mukul Kumar Singh 2018-05-19 11:12:13 -07:00
Xiao Chen
220e38ec02 HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure. 
(cherry picked from commit 960940e0e08f7839775f2d8a352b444d104d36b4)
 2018-05-14 09:29:22 -07:00
Inigo Goiri
d72a4b6303 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun." 
This reverts commit 1049e180a7dbc85e17c17240d231ed9a4659ed09.
 2018-05-10 09:24:45 -07:00
Inigo Goiri
1049e180a7 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit 63803e7051dcfcd8e3d28a8b3217b7735a00b7d8)
 2018-05-09 13:56:58 -07:00
Arpit Agarwal
e8f62357c0 HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal. 2018-04-23 10:09:16 -07:00
Shashikant Banerjee
2c4b1c2b15
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247da77d03c6015de4f18be75924fcb22)
 2018-04-19 08:36:59 +09:00
Wei-Chiu Chuang
dc01e323ed HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota. 
(cherry picked from commit e66e287efe2b43e710137a628f03c7df3ebdf498)
(cherry picked from commit ca8bb322be5e21ce00bb49cecfc4f515b49d2a37)
 2018-04-13 12:20:01 -07:00
Lei Xu
3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Inigo Goiri
fe7a70e586 Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun." 
This reverts commit 877f963a059d21e942b0b5cbff5f60ce23c5b5fe.
 2018-04-10 10:08:34 -07:00
Inigo Goiri
877f963a05 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit ac32b3576da4cc463dff85118163ccfff02215fc)
 2018-04-09 09:18:03 -07:00
Xiaoyu Yao
2a93ba690c HDFS-12455. WebHDFS - Adding "snapshot enabled" status to ListStatus query result. Contributed by Ajay Kumar. 
(cherry picked from commit 107c177782a24a16c66113841f2fc5144f56207b)
 2018-03-12 20:23:52 -07:00
Hanisha Koneru
0b1e9665df HDFS-13225. StripeReader#checkMissingBlocks() 's IOException info is incomplete. Contributed by lufei. 
(cherry picked from commit 19ae4429f9a2807548bb4b9ff296521fbdbbb2bc)
 2018-03-07 14:22:01 -08:00
Xiaoyu Yao
b74df5523c HDFS-13109. Support fully qualified hdfs path in EZ commands. Contributed by Hanisha Koneru. 
(cherry picked from commit edf9445708ffb7a9e59cb933e049b540f99add1e)
 2018-03-06 18:30:10 -08:00
Xiaoyu Yao
21d4b5fd24 HDFS-13052. WebHDFS: Add support for snasphot diff. Contributed by Lokesh Jain. 
(cherry picked from commit 1e84e46f1621fe694f806bfc41d3b2a06c9500b6)
 2018-02-23 19:41:28 -08:00
Xiao Chen
1fbd419957 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException. 
(cherry picked from commit 51088d323359587dca7831f74c9d065c2fccc60d)
 2018-02-23 14:00:25 -08:00