Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,018 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4847 Commits

Author SHA1 Message Date
Siyao Meng 8414353daa HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69)
(cherry picked from commit 095815220d)
 2019-05-06 15:48:14 -07:00
Erik Krogen 9da70a1b4b HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f)
(cherry-picked from 28ff96f367)
 2019-04-22 08:19:25 -07:00
David Mollitor 41f1bc2056
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350b)
 2019-04-04 21:13:09 +01:00
Steve Loughran 3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka 10d6e68324
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:55:07 +09:00
Erik Krogen d4fbbc83ad HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
 2019-03-18 08:53:13 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Steve Loughran 1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Daryn Sharp 704330a616 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
 2019-02-20 18:22:00 -08:00
Vinayakumar B c450f9f2c4 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:46:04 +05:30
Wangda Tan a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
(cherry picked from commit 649da5af04)
 2019-02-05 14:09:33 -08:00
Todd Lipcon 1f21be2d02
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.
 2019-02-05 19:56:04 +00:00
Steve Loughran 9a31dd4c67
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac)
 2019-02-01 14:34:39 +00:00
Eric Yang ba3f559724 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218f)
 2019-01-22 19:39:39 -05:00
Chen Liang 0cfc2d8475 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:48:26 -08:00
luhuachao 4297a3b7ac
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621ab)
 2019-01-15 13:16:50 +00:00
Inigo Goiri b05d8de10e HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:10:40 -08:00
Akira Ajisaka ced32754bb
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da6)
 2019-01-11 13:05:06 +09:00
Jason Lowe d8aac6c215 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit e8e55839a0.
 2019-01-09 17:48:16 -06:00
Wangda Tan 31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Eric Yang 2ba9c9f7c1 HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin

(cherry picked from commit d43af8b3db)
(cherry picked from commit febafd0e4f)
 2019-01-04 18:04:11 -05:00
Sunil G d3fdbbc2ab HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 51427cbdfb)
 2019-01-04 21:51:00 +05:30
Pranay Singh e8e55839a0 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a)
(cherry picked from commit 1f39eae7e6)
 2019-01-03 09:57:34 -08:00
Surendra Singh Lilhore ef94eeebb4 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. 
(cherry picked from commit 763e96ea2b)
 2018-12-22 12:46:37 +05:30
Jason Lowe 5c5c4438ad HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 
(cherry picked from commit 3961690037)
 2018-12-20 09:56:12 -06:00
Takanobu Asanuma 9f30916a1b HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d963575918)
 2018-12-20 10:05:47 +09:00
Steve Loughran 1192fad0d2
HADOOP-15808. Harden Token service loader use. 
Contributed by Steve Loughran.

Note that the patch for TestToken is much simpler than the 3.2+ patch.
In later branches,it modifies a test case which is not present in this branch.

(cherry picked from commit 202926ac33)
 2018-12-11 17:39:55 +00:00
Steve Loughran 844f51217b
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c6)
 2018-12-04 15:37:13 +00:00
Giovanni Matteo Fumarola 023e42fa05 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:45:05 -08:00
Eric Yang e534be892c HADOOP-15922. Revert patch 004. 
(cherry picked from commit 38ea3814bd)
 2018-11-30 12:52:36 -05:00
Akira Ajisaka 298c76c485
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f)
 2018-11-30 13:30:52 +09:00
Steve Loughran 65e0d6ff46
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d27.

(cherry picked from commit d0edd37269)
 2018-11-29 17:57:24 +00:00
Eric Yang 20a7f3b50d HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. 
Contributed by He Xiaoqiao

(cherry picked from commit a9d96948ed)
 2018-11-26 13:53:16 -05:00
Vinayakumar B 07529a209d HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c225)
 2018-11-26 18:11:55 +05:30
Surendra Singh Lilhore c97f8b6d95 HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 75291e6d53)
 2018-11-13 21:40:00 +05:30
Steve Loughran 29752ed580
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. 
Contributed by LiXin Ge.

(cherry picked from commit 3c9d97b8f7)
 2018-11-12 12:59:15 +00:00
Jason Lowe 7335d940de YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c726)
 2018-11-06 08:52:29 -06:00
Xiaoyu Yao 0577c89015 HADOOP-15339. Support additional key/value propereties in JMX bean registration. Contributed by Elek Marton. 2018-11-02 10:44:41 +01:00
Steve Loughran e060e90baa
HADOOP-15855. Review hadoop credential doc, including object store details. 
Contributed by Steve Loughran.

(cherry picked from commit 62d98ca92a)
 2018-10-30 15:59:00 +00:00
Wei-Chiu Chuang 35e09fe594 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit ef9f8ca13d.
 2018-10-28 17:06:40 -07:00
Wei-Chiu Chuang ef9f8ca13d HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
(cherry picked from commit fd02c501c0)
 2018-10-25 09:37:31 -07:00
Akira Ajisaka 080e9d0f9b
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe. 
(cherry picked from commit 8f97d6f2cd)

Conflicts:
	dev-support/docker/Dockerfile
 2018-10-24 13:54:30 +09:00
Steve Loughran f8c7073bd0
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
 2018-10-22 11:08:47 +01:00
Jason Lowe 65b27f8ed2 HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe 
(cherry picked from commit 9abda83947)
 2018-10-17 14:43:19 -05:00
Steve Loughran 9c350785d4
HADOOP-15861. Move DelegationTokenIssuer to the right path. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 41b3603b5b)
 2018-10-17 11:03:22 +01:00
Xiao Chen d1749fbaae HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a025)
 2018-10-15 11:00:30 -07:00
Xiao Chen 6a1ce74fb1 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.

(cherry picked from commit 5ec86b445c)
(cherry picked from commit e93e401f492c7fc112da3d0c63b3b186c1b196ec)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
 2018-10-12 12:08:26 -07:00
Steve Loughran 2a305efd4e
HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys. 
Contributed by Steve Loughran.

(cherry picked from commit cdc4350718)
 2018-10-10 17:03:06 +01:00
Eric Yang 5813c1de56 YARN-8845. Removed unused hadoop.registry.rm reference. 
Contributed by Íñigo Goiri

(cherry picked from commit bf04f19456)
 2018-10-09 15:56:38 -04:00
Todd Lipcon 89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3c)
 2018-10-09 13:50:00 -05:00