Commit Graph

3637 Commits

Author SHA1 Message Date
sunlisheng 4f09a6bcfd Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9
Signed-off-by: stack <stack@apache.org>
2019-07-09 08:54:37 -07:00
stack bb7e143d34 HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9
Reapply but include JIRA # in the commit message this time.

Revert "Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9""

This reverts commit 1dd54163fc.
2019-07-01 11:08:05 -07:00
stack 1dd54163fc Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9"
This reverts commit bccb0fd846.
2019-07-01 11:07:43 -07:00
sunlisheng bccb0fd846 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-07-01 10:59:58 -07:00
Takanobu Asanuma 64102e865f HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.
(cherry picked from commit 54f9f75a44)
2019-06-14 10:21:30 +09:00
Erik Krogen e9eaf21874 HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.
(cherry-picked from 76b94c274f)
(cherry-picked from f607efd195)
(cherry-picked from cf3c1b9ffc)
(cherry-picked from d43219c21d)
2019-06-07 14:23:23 -07:00
Takanobu Asanuma e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:41:21 +09:00
Akira Ajisaka cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Alexis Daboville b70fdc64d3
HADOOP-16248. MutableQuantiles leak memory under heavy load.
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac7)
2019-05-17 15:27:47 +01:00
Yuan Gao 18f66a05cf
HADOOP-16205 Backport ABFS driver from trunk to branch 2.0: Fix build and test failures.
Contributed by Yuan Gao.
2019-04-25 18:36:14 +01:00
Steve Loughran 94d0de2eab HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
Contributed by Steve Loughran.
2019-04-24 21:46:25 +00:00
Thomas Marquardt 3bd089ddca HADOOP-15663. ABFS: Simplify configuration.
Contributed by Da Zhou.
2019-04-24 21:46:25 +00:00
Thomas Marquardt b537d8c442 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
Contributed by Steve Loughran and Da Zhou.
2019-04-24 21:46:25 +00:00
Steve Loughran 1b64b0f924 HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2019-04-24 21:46:25 +00:00
Chen Liang 7935485eec HDFS-14205. Backport HDFS-6440 to branch-2. Contributed by Chao Sun. 2019-03-26 12:12:14 -07:00
Erik Krogen c3973c9ae6 HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
(cherry-picked from d4fbbc83ad)
(cherry-picked from e172fc62ce)
2019-03-18 09:00:40 -07:00
Steve Loughran 9b8044d00b
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589)
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
2019-03-13 15:33:58 +00:00
Tsz Wo Nicholas Sze 5bed24ad5f HADOOP-16126. ipc.Client.stop() may sleep too long to wait for all connections.
(cherry picked from commit 0edb0c51dc)
2019-03-01 17:52:53 +09:00
Eric Yang bae607f734 Revert "HADOOP-13707. Skip authorization for anonymous user to access Hadoop"
This reverts commit 439422fff9.
2019-02-21 17:36:59 -05:00
Eric Yang 4a45a9b57e Revert "HADOOP-13707. Skip authorization for anonymous user to access Hadoop"
This reverts commit a4356bb7a5.
2019-02-21 17:36:02 -05:00
Eric Yang da854234a2 Revert "HADOOP-13707. Skip authorization for anonymous user to access Hadoop"
This reverts commit 7993fb5b81.
2019-02-21 17:35:19 -05:00
Daryn Sharp 665cad03f3 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
(cherry picked from commit 704330a616)
(cherry picked from commit 4eccf2a3cc)
2019-02-20 18:22:48 -08:00
Chen Liang d3b06d179d HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-17 13:30:42 -08:00
Chen Liang 1dbe6c9780 HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 2019-01-11 12:57:30 -08:00
Akira Ajisaka 1684301485
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper
(cherry picked from commit d4ca907da6)
2019-01-11 13:05:58 +09:00
Jason Lowe d62bfaf1a4 HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 2018-12-20 10:02:18 -06:00
shwetayakkali@cloudera.com 500b2a0ca6
HADOOP-16008. Fix typo in CommandsManual.md
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-12-18 13:27:14 +09:00
Inigo Goiri 5cef413a8e HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 13:23:10 -08:00
Inigo Goiri a29eb49f85 Revert "HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak."
This reverts commit b98ffbe3f2.
2018-12-03 13:22:14 -08:00
Giovanni Matteo Fumarola b98ffbe3f2 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:46:30 -08:00
Steve Loughran a3470c65d8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 2018-11-30 11:33:19 +00:00
Akira Ajisaka 6cae964f6c
Add 2.9.2 release notes and changes documents.
(cherry picked from commit 1a00b4e325)
2018-11-20 14:30:09 +09:00
Steve Loughran f23f339a6c
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility.
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 52cb766ad0)
2018-10-30 09:58:59 +09:00
Akira Ajisaka ca699774a2
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe.
(cherry picked from commit 8f97d6f2cd)

Conflicts:
	dev-support/docker/Dockerfile
2018-10-24 14:00:34 +09:00
Jason Lowe e412d8f6cd HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe
(cherry picked from commit 9abda83947)
2018-10-17 14:45:56 -05:00
Steve Loughran 5da49631b6
HADOOP-15679. ShutdownHookManager shutdown time needs to be configurable & extended. 2018-10-11 12:43:09 +01:00
Jason Lowe d7e9ea37bf HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe
(cherry picked from commit f13e231025)
2018-10-05 09:30:50 -05:00
Chen Liang 3a6ad9cd3d HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-20 10:00:13 -07:00
Steve Loughran 5b77991548
HADOOP-15748. S3 listing inconsistency can raise NPE in globber.
Contributed by Steve Loughran.

(cherry picked from commit 646874c326)
2018-09-20 13:07:07 +01:00
Jason Lowe 88e2ca405a HADOOP-15755. StringUtils#createStartupShutdownMessage throws NPE when args is null. Contributed by Lokesh Jain and Dinesh Chitlangia
(cherry picked from commit e71f61ecb8)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
2018-09-18 16:05:47 -05:00
Inigo Goiri 1b12925d69 HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng.
(cherry picked from commit 527288ef89)
2018-09-06 16:50:09 -07:00
Vrushali C f6e4b13d5f HADOOP-15657 Registering MutableQuantiles via Metric annotation. Contributed by Sushil Ks 2018-09-05 21:09:16 -07:00
Steve Loughran 5902c0658e HADOOP-10219. ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns.
Contributed by Kihwal Lee and Lukas Majercak.

(cherry picked from commit 9e96ac666d)
2018-09-04 11:58:56 -07:00
Konstantin V Shvachko 64681a362d HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
(cherry picked from commit a4121c71c2)
2018-08-24 19:00:21 -07:00
Takanobu Asanuma 947b9a27e2 HADOOP-14314. The OpenSolaris taxonomy link is dead in InterfaceClassification.md. Contributed by Rui Gao.
(cherry picked from commit bed8cb6979)
2018-08-23 21:56:30 +09:00
Xiao Chen 8056472879 HADOOP-15674. Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite. Contributed by Szilard Nemeth.
(cherry picked from commit 8d7c93186e)
2018-08-17 10:20:29 -07:00
Chen Liang d2ec117435 HADOOP-15637. LocalFs#listLocatedStatus does not filter out hidden .crc files. Contributed by Erik Krogen. 2018-07-31 11:44:08 -07:00
Kihwal Lee cf255dc01f HADOOP-15614. TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails. Contributed by Weiwei Yang.
(cherry picked from commit ccf2db7fc2)
2018-07-19 11:29:58 -05:00
Hanisha Koneru 4c54ddd1c2 HADOOP-15121. Encounter NullPointerException when using DecayRpcScheduler. Contributed by Tao Jie.
(cherry picked from commit 3fde0f1db5)
2018-07-09 15:23:33 -05:00
Eric E Payne f951d92d46 HADOOP-15548: Randomize local dirs. Contributed by Jim Brennan. 2018-06-29 20:18:55 +00:00
Inigo Goiri 1d457673c8 HADOOP-15458. TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows. Contributed by Xiao Liang.
(cherry picked from commit abc3e4bad9)
2018-06-25 09:52:15 -07:00
Eric E Payne d084c1a4bc YARN-8444: NodeResourceMonitor crashes on bad swapFree value. Contributed by Jim Brennan.
(cherry picked from commit 6432128622)
2018-06-22 17:53:07 +00:00
Surendra Singh Lilhore 59686179ae HADOOP-15523. Shell command timeout given is in seconds whereas it is taken as millisec while scheduling. Contributed by Bilwa S T.
(cherry picked from commit 3905fdb793)
2018-06-17 12:20:11 +05:30
Inigo Goiri b3f4c27661 HADOOP-15532. TestBasicDiskValidator fails with NoSuchFileException. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit aeaf9fec62)
2018-06-12 14:21:45 -07:00
Rohith Sharma K S 0af3bea05d YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri.
(cherry picked from commit 2df73dace0)
2018-06-12 17:30:02 +05:30
Kihwal Lee 6d7d192e47 Additional check when unpacking archives. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 65e55097da)
2018-05-29 14:33:31 -05:00
Inigo Goiri 09fbbff695 HADOOP-15497. TestTrash should use proper test path to avoid failing on Windows. Contributed by Anbang Hu.
(cherry picked from commit 3c75f8e493)
2018-05-29 09:12:47 -07:00
Karthik Palanisamy 961c0461fe
HADOOP-15449. Increase default timeout of ZK session to avoid frequent NameNode failover
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61df174e8b)
2018-05-28 19:43:03 +09:00
Xiao Chen fe32d65211 HADOOP-15473. Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997. Contributed by Gabor Bota.
(cherry picked from commit 02322de3f9)
2018-05-25 09:42:16 -07:00
Miklos Szegedi 81ee9938c5 YARN-8310. Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formats. Contributed by Robert Kanter. 2018-05-24 15:50:17 -07:00
Arpit Agarwal 50c39db1ad HADOOP-15486. Make NetworkTopology#netLock fair. Contributed by Nanda kumar. 2018-05-23 10:30:48 -07:00
Akira Ajisaka cedc28d4ab
Additional check when unpacking archives. Contributed by Jason Lowe and Akira Ajisaka. 2018-05-23 17:20:20 +09:00
Kihwal Lee 5dcd57cbea HADOOP-15450. Avoid fsync storm triggered by DiskChecker and handle disk full situation. Contributed by Arpit Agarwal.
(cherry picked from commit bcc8e76bad)
2018-05-22 11:29:54 -05:00
Rushabh Shah 734cbdd6b2 Skip the proxy user check if the ugi has not been initialized. Contributed by Daryn Sharp
(cherry picked from commit 73e9120ad7)
2018-05-21 12:53:54 -05:00
Inigo Goiri c4d4ae62cb HDFS-13586. Fsync fails on directories on Windows. Contributed by Lukas Majercack.
(cherry picked from commit 8783613696)
2018-05-17 19:28:28 -07:00
Inigo Goiri a5ec44c655 HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu.
(cherry picked from commit 53b807a6a8)
2018-05-17 17:04:59 -07:00
Sammi Chen eabc588d22 Add 2.9.1 release notes and changes documents 2018-05-14 15:36:30 +08:00
Rushabh Shah 1bc12310d8 HADOOP-15441. Log kms url and token service at debug level. Contributed by Gabor Bota
(cherry picked from commit a5449d3631)
2018-05-12 12:30:59 -05:00
Inigo Goiri ed592a210a HADOOP-15454. TestRollingFileSystemSinkWithLocal fails on Windows. Contributed by Xiao Liang.
(cherry picked from commit 1da8d4190d)
2018-05-10 09:42:39 -07:00
Xiao Chen 723afa0978 Revert "HADOOP-14445. Delegation tokens are not shared between KMS instances. Contributed by Xiao Chen and Rushabh S Shah."
This reverts commit 95cedc5587.

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSTokenRenewer.java
2018-05-07 14:11:13 -07:00
Robert Kanter 312462532e HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)
(cherry picked from commit 7ab08a9c37)
(cherry picked from commit 74e5b4b437)
2018-05-07 12:39:57 -07:00
Rushabh Shah d261645177 HADOOP-14841 Kms client should disconnect if unable to get output stream from connection. Contributed by Rushabh S Shah
(cherry picked from commit 4cdbdce752)
2018-05-04 15:55:12 -05:00
Brahma Reddy Battula cbcd16e6b7 HADOOP-15180. branch-2 : daemon processes' sysout overwrites 'ulimit -a' in daemon's out file. Contributed by Ranith Sardar 2018-04-18 23:01:50 +05:30
Inigo Goiri 7a2edb588c HADOOP-15394. Backport PowerShell NodeFencer HADOOP-14309 to branch-2. Contributed by Inigo Goiri. 2018-04-17 11:33:09 -07:00
Virajith Jalaparti b9dc0e597e HDFS-13311. RBF: TestRouterAdminCLI#testCreateInvalidEntry fails on Windows. Contributed by Inigo Goiri 2018-04-16 16:08:36 -07:00
Xiao Chen 95cedc5587 HADOOP-14445. Delegation tokens are not shared between KMS instances. Contributed by Xiao Chen and Rushabh S Shah. 2018-04-10 15:46:46 -07:00
Jason Lowe 0fb1457d86 HADOOP-15357. Configuration.getPropsWithPrefix no longer does variable substitution. Contributed by Jim Brennan 2018-04-10 16:59:31 -05:00
Wei Yan 689a8252aa HDFS-13353. RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma.
(cherry picked from commit 3121e8c293)
2018-04-05 13:23:07 -07:00
Xiao Chen b8d8c25a18 HADOOP-14987. Improve KMSClientProvider log around delegation token checking. Contributed by Xiaoyu Yao and Xiao Chen.
(cherry picked from commit 59d78a5088)
2018-04-03 11:14:26 -07:00
Xiao Chen 117a8d6573 HADOOP-15355. TestCommonConfigurationFields is broken by HADOOP-15312. Contributed by LiXin Ge.
(cherry picked from commit f614b44fe583033947cc78d8f04cfe6fb64fc892)
2018-04-02 23:10:35 -07:00
Xiao Chen 251fc1fb40 HADOOP-15317. Improve NetworkTopology chooseRandom's loop.
(cherry picked from commit 57374c4737)
2018-04-02 22:55:03 -07:00
Konstantin V Shvachko de419e2777 HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie.
(cherry picked from commit acfd764fcc)
2018-03-30 19:08:12 -07:00
Konstantin V Shvachko c138682e04 HADOOP-12862. LDAP Group Mapping over SSL can not specify trust store. Contributed by Wei-Chiu Chuang and Konstantin Shvachko.
(cherry picked from commit 2216bde322)
2018-03-29 18:03:32 -07:00
Wei-Chiu Chuang 827d2015c4 HADOOP-15312. Undocumented KeyProvider configuration keys. Contributed by LiXin Ge.
(cherry picked from commit 3fe41c65d8)
(cherry picked from commit 1176a128d6)
(cherry picked from commit c6939e9330)
2018-03-27 13:38:06 -07:00
Inigo Goiri d6df90f7b5 HDFS-13268. TestWebHdfsFileContextMainOperations fails on Windows. Contributed by Xiao Liang.
(cherry picked from commit fe224ff972)
2018-03-20 11:21:44 -07:00
Inigo Goiri 30d9a5db29 HDFS-13296. GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases. Contributed by Xiao Liang.
(cherry picked from commit e65ff1c8be)
2018-03-19 17:21:55 -07:00
Inigo Goiri 36451f2d54 HADOOP-15308. TestConfiguration fails on Windows because of paths. Contributed by Xiao Liang.
(cherry picked from commit 427fd027a3)
2018-03-13 20:06:07 -07:00
Chris Douglas 340cd5f1b1 HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
(cherry picked from commit a03c8ea61f)
2018-03-12 20:08:46 -07:00
Nanda kumar 45ec3e18e3
HADOOP-15275. Incorrect javadoc for return type of RetryPolicy#shouldRetry
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 55d04a6db1)
2018-03-01 19:04:57 +09:00
Arpit Agarwal 0945f0eb23 HADOOP-15266. [branch-2] Upper/Lower case conversion support for group names in LdapGroupsMapping. Contributed by Nanda kumar. 2018-02-26 13:45:51 -08:00
Jason Lowe b1d5e73c1f HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka
(cherry picked from commit 0898ff42e9)
2018-02-16 15:00:10 -06:00
Kihwal Lee 34466c6b16 HADOOP-15212. Add independent secret manager method for logging expired tokens. Contributed by Daryn Sharp.
(cherry picked from commit 2dd960de98)
2018-02-06 15:36:32 -06:00
Xiao Chen 0922ea0aaf HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi.
(cherry picked from commit b63dcd583f)
(cherry picked from commit 75be15cf02)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
2018-01-28 22:30:14 -08:00
Steve Loughran 312c571694 HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common
Contributed by Genmao Yu
2018-01-27 20:31:21 -08:00
Chris Douglas 9bd439e2c5 MapFile.fix creates a wrong index file in case of block-compressed data file. Contributed by Grigori Rybkine
(cherry picked from commit 91db424c4360d7556660e8c57ac9a266e6688e01)
(cherry picked from commit 7f35487788)
2018-01-26 09:15:53 -08:00
Wei Yan cefe4b7bb6 HDFS-13049. RBF: Inconsistent Router OPTS config in branch-2 and branch-3. Contributed by Wei Yan. 2018-01-25 12:58:16 -08:00
Masahiro Tanaka ab482ebe73
HADOOP-15181. Typo in SecureMode.md
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e27b20b4d)
2018-01-22 17:15:31 +09:00
Brahma Reddy Battula 8181367c62 HADOOP-15150. in FsShell, UGI params should be overidden through env vars(-D arg). Contributed by Brahma Reddy Battula.
(cherry picked from commit 08332e12d0)
2018-01-18 10:56:58 +05:30
Miklos Szegedi 679ad2ced0 HADOOP-15060. TestShellBasedUnixGroupsMapping.testFiniteGroupResolutionTime flaky. Contributed by Miklos Szegedi.
(cherry picked from commit 12d0645)
2018-01-10 11:15:58 -08:00
Jitendra Pandey cb60c05c3a HADOOP-15143. NPE due to Invalid KerberosTicket in UGI. Contributed by Mukul Kumar Singh. 2018-01-01 21:51:48 -08:00