Commit Graph

569 Commits

Author SHA1 Message Date
Weiwei Yang 9c3806cf1b HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:46:27 +08:00
Inigo Goiri aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
(cherry picked from commit 0966407ad6)
2019-06-24 19:03:26 -07:00
Wei-Chiu Chuang fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu.
(cherry picked from commit 62ad9885ea)
(cherry picked from commit 45543ffd6a)
2019-06-17 20:20:08 -07:00
huzheng b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
(cherry picked from commit ea3b0a1844)
(cherry picked from commit 9b61ecfcbe)
2019-06-17 07:21:07 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Srinivasu Majeti e599e3b546 HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3e5e5b028a)
(cherry picked from commit 85f3664bb8)
2019-05-17 19:22:03 +02:00
Takanobu Asanuma 1fc74178eb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory.
(cherry picked from commit 55fb3c32fb)
2019-03-27 03:29:42 +09:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
2019-03-13 13:20:51 -07:00
Dinesh Chitlangia b63941516f HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cf)
(cherry picked from commit eb77e60163)
2019-01-25 07:38:11 -08:00
Wangda Tan 31ea2f7806 Preparing for 3.1.3 development
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
2019-01-07 10:04:58 -08:00
Brahma Reddy Battula 0b83c95ff4 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena.
(cherry picked from commit f534736867)
2018-11-30 00:27:37 +05:30
Inigo Goiri f65324aa47 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen.
(cherry picked from commit 887244de4a)
2018-11-06 11:20:33 -08:00
Xiao Chen 399645ebc1 HDFS-14027. DFSStripedOutputStream should implement both hsync methods.
(cherry picked from commit db7e636824)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
2018-10-29 19:13:51 -07:00
Xiao Chen 96cedb87b9 Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B.
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
2018-10-24 16:10:25 -07:00
Xiao Chen d1749fbaae HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
(cherry picked from commit b6fc72a025)
2018-10-15 11:00:30 -07:00
Xiao Chen 6a1ce74fb1 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.

(cherry picked from commit 5ec86b445c)
(cherry picked from commit e93e401f492c7fc112da3d0c63b3b186c1b196ec)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
2018-10-12 12:08:26 -07:00
Hrishikesh Gadre 323b76bccf HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads.
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
2018-10-09 19:54:34 -07:00
Todd Lipcon 89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers
(cherry picked from commit 7a3c6e9c3c)
2018-10-09 13:50:00 -05:00
Wei-Chiu Chuang 7442b7941d HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh.
(cherry picked from commit 28ceb34a72)
2018-09-19 03:12:55 -07:00
Wei-Chiu Chuang 829399a9f3 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
(cherry picked from commit a1de8cbac5)
2018-09-14 05:24:24 +08:00
Wei-Chiu Chuang 9f0cd2ade2 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng."
This reverts commit 41732e7b28.
2018-08-30 11:58:42 -07:00
Wei-Chiu Chuang 41732e7b28 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng.
(cherry picked from commit 26c2a97c56)
2018-08-27 16:03:52 -07:00
Konstantin V Shvachko 60fd202a59 HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
(cherry picked from commit a4121c71c2)
2018-08-24 18:44:04 -07:00
Xiao Chen c0ac0a5337 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-20 14:52:28 -07:00
Xiao Chen 06f0d5e257 HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi.
(cherry picked from commit c67b0650ea)
2018-08-16 23:15:48 -07:00
Wangda Tan 5583711419 Preparing for 3.1.2 release
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
2018-07-31 13:08:55 -07:00
Daniel Templeton 00c476abd8 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
2018-07-24 16:12:43 -07:00
Wei-Chiu Chuang f5f4d0b7e7 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel.
(cherry picked from commit 0247cb6318)
2018-07-06 15:01:10 -07:00
Andrew Wang 3b266abdad HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas.
(cherry picked from commit 344f324710)
2018-07-03 11:07:57 +02:00
Wei-Chiu Chuang 2ce892fcf5 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota.
(cherry picked from commit fef20a446f)
2018-07-02 09:43:44 -07:00
Andrew Wang 2d2639beaa HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon.
(cherry picked from commit 5d748bd056)
2018-07-02 12:11:11 +02:00
Andrew Wang 0268b954e0 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon.
(cherry picked from commit 6ba9974108)
2018-07-02 12:02:48 +02:00
Xiao Chen 3b646a9836 HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota.
(cherry picked from commit 774c1f199e)
2018-06-12 11:17:54 -07:00
Inigo Goiri f0150f024f HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam.
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
2018-06-11 11:17:37 -07:00
Arpit Agarwal 33b6a68c81 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:30 -07:00
Chao Sun d827ffe245 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun.
(cherry picked from commit fba1c42adc)
2018-06-08 16:40:13 -07:00
Xiao Chen f1c91e9fda HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel.
(cherry picked from commit ba4011d64f)
2018-06-05 21:19:11 -07:00
Sean Mackrory 58b5b3aa75 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel.
(cherry picked from commit 1361030e59)
2018-05-31 22:23:19 -06:00
Andrew Wang 31d597f0c7 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient.
(cherry picked from commit c9b63deb53)
2018-05-24 09:56:37 -07:00
Bharat Viswanadham d0d3d5b2ad HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru
(cherry picked from commit bc6d9d4c79)
2018-05-23 10:23:45 -07:00
Sammi Chen 7d71b3a1cc HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen.
(cherry picked from commit 34e8b9f9a8)
2018-05-23 19:12:04 +08:00
Andrew Wang d47c0fc39a HDFS-13601. Optimize ByteString conversions in PBHelper.
(cherry picked from commit 1d2640b613)
2018-05-22 23:55:35 -07:00
Arpit Agarwal 45dd200a2c HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar. 2018-05-16 11:28:51 -07:00
Xiao Chen 6e731eb20d HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure.
(cherry picked from commit 960940e0e0)
2018-05-14 09:29:14 -07:00
Inigo Goiri c649c76566 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun.
(cherry picked from commit 63803e7051)
2018-05-09 13:56:06 -07:00
Arpit Agarwal 1f486a0649 HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal. 2018-04-23 10:09:02 -07:00
Shashikant Banerjee a13b3edfd8
HADOOP-15396. Some java source files are executable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
2018-04-19 08:36:32 +09:00
Xiao Chen 805e33b62c HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-16 11:58:10 -07:00
Wei-Chiu Chuang ca8bb322be HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota.
(cherry picked from commit e66e287efe)
2018-04-13 12:18:49 -07:00