Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,903 Commits 326 Branches 371 Tags
Commit Graph

4720 Commits

Author SHA1 Message Date
Akira Ajisaka
524dc4e662
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d2153884bf78a159534ea711c0b70acc1c05)
 2019-04-02 10:55:27 +09:00
Erik Krogen
e172fc62ce HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6bef369fef6a8364f0c0764eba90e44a)
(cherry-picked from 0de8b55a095ada2b98c0a41899651bd8e524f42e)
(cherry-picked from d4fbbc83ad8c4d818deccf62b4c54cead1d17a8f)
 2019-03-18 08:57:41 -07:00
Erik Krogen
9a5daa5ec6 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
(cherry picked from commit fec7c5f3ebbaea7b290e0904570c3a485b541a22)
(cherry picked from commit cb76fc5618c582887e4e71b98f3065c5a916334a)
 2019-03-13 13:21:40 -07:00
Steve Loughran
7f70a78ce3
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589) 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
 2019-03-12 11:36:32 +00:00
Daryn Sharp
4eccf2a3cc HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432609b7a35a2abd6410b02e8a2ffc974)
(cherry picked from commit ae8839e6e8cc3e8f8d5e50525d3302038ada484b)
(cherry picked from commit 704330a616c17256b3e39370f999928ba1c463e6)
 2019-02-20 18:22:20 -08:00
Todd Lipcon
02cfd5fa35
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.

(cherry picked from commit 1f21be2d024de724990a0b5737c0c65c6e59d7f3)
 2019-02-05 19:57:08 +00:00
Chen Liang
80b9fecaaf HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:45:22 -08:00
Inigo Goiri
174fa73f99 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685605d8b3639e6c5cbe83cd9acd8cbe7)
 2019-01-11 18:16:44 -08:00
Akira Ajisaka
9fd5a6b66b
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da636892e4ab98e232fe0f7f77f1f7aac)
 2019-01-11 13:05:31 +09:00
Jason Lowe
642213a74a Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit a4f84213241c647c973acfa460ac4f4b0895aa2d.
 2019-01-09 17:56:32 -06:00
Pranay Singh
a4f8421324 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6ad43555031aed032e6ba7a14a17d7bc)
(cherry picked from commit 1f39eae7e6f59206b86f96063ffb2ebe15a9cbe1)
(cherry picked from commit e8e55839a0c2b5479d7a25256ed1db751e09c122)
 2019-01-03 13:44:57 -08:00
Jason Lowe
c9fdf7503e HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 2018-12-20 10:00:17 -06:00
Takanobu Asanuma
e38e184a19 HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d9635759182b614a1dd5034c30978e7c4be8d0dd)
 2018-12-20 10:06:26 +09:00
Steve Loughran
38335e0b7d
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c65fdff3acf83b4c4f0a79fe48270487)
 2018-12-04 15:37:36 +00:00
Giovanni Matteo Fumarola
25fdf2ba6c HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:45:51 -08:00
Akira Ajisaka
c9c4511a0e
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274)
 2018-11-30 13:31:13 +09:00
Steve Loughran
065a1e72f2
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.

(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
 2018-11-29 17:58:15 +00:00
Vinayakumar B
c2e65cac5e HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c2253bcb3be5229d3c5eb1e165369070c0)
 2018-11-26 18:14:30 +05:30
Jason Lowe
f3e6682ba7 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c7265db5bcbb446c2f779289023d454b81)
 2018-11-06 08:59:47 -06:00
Wei-Chiu Chuang
0f34ff7722 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fa01a8fa0e345c533e843504743baa00020be706.
 2018-10-28 17:07:29 -07:00
Wei-Chiu Chuang
fa01a8fa0e HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fce019130e10964a644b94cddbab1c06)
(cherry picked from commit fd02c501c0cea3ec55956e11b390111519cedc00)
(cherry picked from commit ef9f8ca13d8dca8031c045bad801fceb3e00aa15)
 2018-10-25 09:38:26 -07:00
Akira Ajisaka
e402791a51
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe. 
(cherry picked from commit 8f97d6f2cdfccefba5457ae3d561e9ce0109da3f)

Conflicts:
	dev-support/docker/Dockerfile
 2018-10-24 13:56:19 +09:00
Steve Loughran
a069dade5c
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
 2018-10-22 11:09:04 +01:00
Siyao Meng
6a7bf9f850 HDFS-14009. HttpFS: FileStatus#setSnapShotEnabledFlag throws InvocationTargetException when attribute set is emptySet. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-18 17:02:38 -07:00
Jason Lowe
0aee3a0c3f HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe 
(cherry picked from commit 9abda83947a5babfe5a650b3409ad952f6782105)
 2018-10-17 14:44:39 -05:00
Steve Loughran
ae42d59eba
HADOOP-15861. Move DelegationTokenIssuer to the right path. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 41b3603b5bcb74a7d78a314a4a5c177d941af27f)
 2018-10-17 11:04:05 +01:00
Xiao Chen
b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen
ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Jonathan Eagles
2b6d8c1e29 HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles) 2018-10-10 10:32:43 -05:00
Jason Lowe
bbceae10b9 HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe 
(cherry picked from commit f13e231025333ebf80b30bbdce1296cef554943b)
 2018-10-05 09:29:41 -05:00
Yiqun Lin
d993a1fc3d HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. 
(cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d)
 2018-10-04 10:36:49 +08:00
Siyao Meng
2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Steve Loughran
fc6d851f70
HADOOP-15748. S3 listing inconsistency can raise NPE in globber. 
Contributed by Steve Loughran.

(cherry picked from commit 646874c326139457b79cf8cfa547b3c91a78c7b4)
 2018-09-20 13:06:20 +01:00
Vinayakumar B
d2655ea60c HADOOP-15736. Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Contributed by Ayush Saxena. 
(cherry picked from commit 7ad27e97f05b13b33fdcef9cb63ace9c1728bfb5)
(cherry picked from commit 2aa385463305beaae8770c8deb22faed6943ef00)
 2018-09-20 09:34:37 +05:30
Chen Liang
78e0c0a1bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:51:23 -07:00
Jason Lowe
f2eeba5c5b HADOOP-15755. StringUtils#createStartupShutdownMessage throws NPE when args is null. Contributed by Lokesh Jain and Dinesh Chitlangia 
(cherry picked from commit e71f61ecb87e04727a5a76e578a75714c9db6706)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
 2018-09-18 16:03:42 -05:00
Inigo Goiri
a9392bdb17 HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng. 
(cherry picked from commit 527288ef891dc26019d003bd85ddfd50eb4f3b7b)
 2018-09-06 16:49:14 -07:00
Wei-Chiu Chuang
e99ed05444 HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e780556ae9229fe7a90817eb4e5449d7eed35dd8)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java

(cherry picked from commit d1ea7df43d6a5623b139acbbc9992831d5c7e484)
 2018-09-05 10:18:56 -07:00
Steve Loughran
9ee9d68e4b HADOOP-10219. ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns. 
Contributed by Kihwal Lee and Lukas Majercak.

(cherry picked from commit 9e96ac666d783376a8cdea9c3cc84098c5bdcb56)
(cherry picked from commit 142d878c902da6bc02d0c21cdd5ca8674da7f5c8)
 2018-09-04 22:25:24 -07:00
Daniel Templeton
9a02c4ff65 HADOOP-15706. Typo in compatibility doc: SHOUD -> SHOULD 
(Contributed by Laszlo Kollar via Daniel Templeton)

Change-Id: I6e2459d0700df7f3bad4eac8297a11690191c3ba
(cherry picked from commit f2c2a68ec208f640e778fc41f95f0284fcc44729)
(cherry picked from commit 68301ad7d36d516e265d34dd47c344239e3c6c7d)
 2018-08-30 09:15:25 -07:00
Kitti Nanasi
fa32269cee HADOOP-15698. KMS log4j is not initialized properly at startup. Contributed by Kitti Nanasi. 
(cherry picked from commit 781437c219dc3422797a32dc7ba72cd4f5ee38e2)
 2018-08-29 22:09:13 -07:00
Daniel Templeton
fbaa11ef4c HADOOP-15705. Typo in the definition of "stable" in the interface classification 
Change-Id: I3eae2143400a534903db4f186400561fc8d2bd56
(cherry picked from commit d53a10b0a552155de700e396fd7f450a4c5f9c22)
(cherry picked from commit 90c8cca78389f534429ebf81576ce7ebba9356f7)
 2018-08-29 14:01:19 -07:00
John Zhuge
2260718e24
HADOOP-15633. fs.TrashPolicyDefault: Can't create trash directory. Contributed by Fei Hui. 
(cherry picked from commit f9c6fd94711458b77ecf3fa425aad7fda5089376)
 2018-08-27 15:22:19 -07:00
Konstantin V Shvachko
27abdb390b HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 
(cherry picked from commit a4121c71c29d13866a605d9c0d013e5de9c147c3)
 2018-08-24 18:51:42 -07:00
Takanobu Asanuma
cd8ac03357 HADOOP-14314. The OpenSolaris taxonomy link is dead in InterfaceClassification.md. Contributed by Rui Gao. 
(cherry picked from commit bed8cb6979e0460141ed77e3b15d4f18db098a8e)
 2018-08-23 21:55:55 +09:00
Steve Loughran
53ce37ad29
HADOOP-15679. ShutdownHookManager shutdown time needs to be configurable & extended. Contributed by Steve Loughran. 
Change-Id: Ifd4a6e3f796b4dc88e97f63066289e2534c77a29
(cherry picked from commit 95fcdc04355920697337db1136ffe463d4794616)
 2018-08-20 19:18:37 -07:00
Xiao Chen
a41f18098b HADOOP-15674. Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite. Contributed by Szilard Nemeth. 
(cherry picked from commit 8d7c93186e3090b19aa59006bb6b32ba929bd8e6)
 2018-08-17 10:20:20 -07:00
Xiao Chen
7730a85233 HADOOP-15638. KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit b94c8874e2a634637b1ef5d837f05bc5c5e8e6a6)
 2018-08-13 10:41:14 -07:00
Wangda Tan
8b4392769c Make 3.1.1 awared by other branches 
(cherry picked from commit 49c687608b65b772faeed614700ece8e526432e8)
(cherry picked from commit 781620a44db1eea511406f6c300f5b12b3a24daf)
 2018-08-08 13:07:09 -07:00
Chen Liang
207f08bab4 HADOOP-15637. LocalFs#listLocatedStatus does not filter out hidden .crc files. Contributed by Erik Krogen. 2018-07-30 17:00:51 -07:00