Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,993 Commits 326 Branches 371 Tags
Commit Graph

4842 Commits

Author SHA1 Message Date
Erik Krogen
d4fbbc83ad HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6bef369fef6a8364f0c0764eba90e44a)
(cherry-picked from 0de8b55a095ada2b98c0a41899651bd8e524f42e)
 2019-03-18 08:53:13 -07:00
Erik Krogen
cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
(cherry picked from commit fec7c5f3ebbaea7b290e0904570c3a485b541a22)
 2019-03-13 13:20:51 -07:00
Steve Loughran
1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Daryn Sharp
704330a616 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432609b7a35a2abd6410b02e8a2ffc974)
(cherry picked from commit ae8839e6e8cc3e8f8d5e50525d3302038ada484b)
 2019-02-20 18:22:00 -08:00
Vinayakumar B
c450f9f2c4 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:46:04 +05:30
Wangda Tan
a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa221a336ef0dbb1a17358b1ad6f9356)
(cherry picked from commit 649da5af04d57ee910dbe7fb0f4a7d9cb4c40eb0)
 2019-02-05 14:09:33 -08:00
Todd Lipcon
1f21be2d02
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.
 2019-02-05 19:56:04 +00:00
Steve Loughran
9a31dd4c67
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac8cf85b094f41b3dd68e02c69e5afbc)
 2019-02-01 14:34:39 +00:00
Eric Yang
ba3f559724 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218fd4d6c660fd064e893be3112c546c9f)
 2019-01-22 19:39:39 -05:00
Chen Liang
0cfc2d8475 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:48:26 -08:00
luhuachao
4297a3b7ac
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621aba4f1fc9ca880e294a66862493805b0)
 2019-01-15 13:16:50 +00:00
Inigo Goiri
b05d8de10e HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685605d8b3639e6c5cbe83cd9acd8cbe7)
 2019-01-11 18:10:40 -08:00
Akira Ajisaka
ced32754bb
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da636892e4ab98e232fe0f7f77f1f7aac)
 2019-01-11 13:05:06 +09:00
Jason Lowe
d8aac6c215 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit e8e55839a0c2b5479d7a25256ed1db751e09c122.
 2019-01-09 17:48:16 -06:00
Wangda Tan
31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Eric Yang
2ba9c9f7c1 HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin

(cherry picked from commit d43af8b3db4743b4b240751b6f29de6c20cfd6e5)
(cherry picked from commit febafd0e4f6e7b0a95045565e149e3ed2bee4b6c)
 2019-01-04 18:04:11 -05:00
Sunil G
d3fdbbc2ab HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 51427cbdfb39cb6f5774b7b70009d7ee4388edfc)
 2019-01-04 21:51:00 +05:30
Pranay Singh
e8e55839a0 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6ad43555031aed032e6ba7a14a17d7bc)
(cherry picked from commit 1f39eae7e6f59206b86f96063ffb2ebe15a9cbe1)
 2019-01-03 09:57:34 -08:00
Surendra Singh Lilhore
ef94eeebb4 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. 
(cherry picked from commit 763e96ea2b18f584ba102b68271263be8c5d621c)
 2018-12-22 12:46:37 +05:30
Jason Lowe
5c5c4438ad HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 
(cherry picked from commit 3961690037fa4274a3640104b3c344e8626caa10)
 2018-12-20 09:56:12 -06:00
Takanobu Asanuma
9f30916a1b HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d9635759182b614a1dd5034c30978e7c4be8d0dd)
 2018-12-20 10:05:47 +09:00
Steve Loughran
1192fad0d2
HADOOP-15808. Harden Token service loader use. 
Contributed by Steve Loughran.

Note that the patch for TestToken is much simpler than the 3.2+ patch.
In later branches,it modifies a test case which is not present in this branch.

(cherry picked from commit 202926ac3301298753abd0e6e1f324caf0202ec6)
 2018-12-11 17:39:55 +00:00
Steve Loughran
844f51217b
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c65fdff3acf83b4c4f0a79fe48270487)
 2018-12-04 15:37:13 +00:00
Giovanni Matteo Fumarola
023e42fa05 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:45:05 -08:00
Eric Yang
e534be892c HADOOP-15922. Revert patch 004. 
(cherry picked from commit 38ea3814bd0641d895e5d3b7415c6308e7f8491e)
 2018-11-30 12:52:36 -05:00
Akira Ajisaka
298c76c485
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274)
 2018-11-30 13:30:52 +09:00
Steve Loughran
65e0d6ff46
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.

(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
 2018-11-29 17:57:24 +00:00
Eric Yang
20a7f3b50d HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. 
Contributed by He Xiaoqiao

(cherry picked from commit a9d96948ede11d4d8a8fb58b31b150af84e118d9)
 2018-11-26 13:53:16 -05:00
Vinayakumar B
07529a209d HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c2253bcb3be5229d3c5eb1e165369070c0)
 2018-11-26 18:11:55 +05:30
Surendra Singh Lilhore
c97f8b6d95 HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 75291e6d53c13debf45493a870a898b63779914b)
 2018-11-13 21:40:00 +05:30
Steve Loughran
29752ed580
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. 
Contributed by LiXin Ge.

(cherry picked from commit 3c9d97b8f7d6eb75f08fc6d37cee37c22760bb86)
 2018-11-12 12:59:15 +00:00
Jason Lowe
7335d940de YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c7265db5bcbb446c2f779289023d454b81)
 2018-11-06 08:52:29 -06:00
Xiaoyu Yao
0577c89015 HADOOP-15339. Support additional key/value propereties in JMX bean registration. Contributed by Elek Marton. 2018-11-02 10:44:41 +01:00
Steve Loughran
e060e90baa
HADOOP-15855. Review hadoop credential doc, including object store details. 
Contributed by Steve Loughran.

(cherry picked from commit 62d98ca92aee15d1790d169bfdf0043b05b748ce)
 2018-10-30 15:59:00 +00:00
Wei-Chiu Chuang
35e09fe594 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit ef9f8ca13d8dca8031c045bad801fceb3e00aa15.
 2018-10-28 17:06:40 -07:00
Wei-Chiu Chuang
ef9f8ca13d HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fce019130e10964a644b94cddbab1c06)
(cherry picked from commit fd02c501c0cea3ec55956e11b390111519cedc00)
 2018-10-25 09:37:31 -07:00
Akira Ajisaka
080e9d0f9b
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe. 
(cherry picked from commit 8f97d6f2cdfccefba5457ae3d561e9ce0109da3f)

Conflicts:
	dev-support/docker/Dockerfile
 2018-10-24 13:54:30 +09:00
Steve Loughran
f8c7073bd0
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
 2018-10-22 11:08:47 +01:00
Jason Lowe
65b27f8ed2 HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe 
(cherry picked from commit 9abda83947a5babfe5a650b3409ad952f6782105)
 2018-10-17 14:43:19 -05:00
Steve Loughran
9c350785d4
HADOOP-15861. Move DelegationTokenIssuer to the right path. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 41b3603b5bcb74a7d78a314a4a5c177d941af27f)
 2018-10-17 11:03:22 +01:00
Xiao Chen
d1749fbaae HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)
 2018-10-15 11:00:30 -07:00
Xiao Chen
6a1ce74fb1 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.

(cherry picked from commit 5ec86b445cc492f52c33639efb6a09a0d2f27475)
(cherry picked from commit e93e401f492c7fc112da3d0c63b3b186c1b196ec)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
 2018-10-12 12:08:26 -07:00
Steve Loughran
2a305efd4e
HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys. 
Contributed by Steve Loughran.

(cherry picked from commit cdc4350718055189fef8c70e31314607001d4009)
 2018-10-10 17:03:06 +01:00
Eric Yang
5813c1de56 YARN-8845. Removed unused hadoop.registry.rm reference. 
Contributed by Íñigo Goiri

(cherry picked from commit bf04f194568f9e81f5481b25a84ad903e3c307cf)
 2018-10-09 15:56:38 -04:00
Todd Lipcon
89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3cd9ffdc71946fd12f5c3d59718c4939)
 2018-10-09 13:50:00 -05:00
Jason Lowe
df7ec58ced HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe 
(cherry picked from commit f13e231025333ebf80b30bbdce1296cef554943b)
 2018-10-05 09:27:23 -05:00
Yiqun Lin
1a890b17b9 HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. 
(cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d)
 2018-10-04 10:35:16 +08:00
Steve Loughran
9dc180d14a
HADOOP-15748. S3 listing inconsistency can raise NPE in globber. 
Contributed by Steve Loughran.

(cherry picked from commit 646874c326139457b79cf8cfa547b3c91a78c7b4)
 2018-09-20 13:05:46 +01:00
Vinayakumar B
2aa3854633 HADOOP-15736. Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Contributed by Ayush Saxena. 
(cherry picked from commit 7ad27e97f05b13b33fdcef9cb63ace9c1728bfb5)
 2018-09-20 09:33:22 +05:30
Chen Liang
b4069343bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:28:35 -07:00