Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,547 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4959 Commits

Author SHA1 Message Date
Siyao Meng 095815220d HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69)
 2019-05-06 15:47:58 -07:00
Erik Krogen 28ff96f367 HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f)
 2019-04-22 08:18:17 -07:00
Akira Ajisaka b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang. 
(cherry picked from commit ab645b3caa)
 2019-04-08 15:19:17 +09:00
David Mollitor 50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350b)
 2019-04-04 16:16:57 +01:00
Steve Loughran b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999)
 2019-04-03 21:35:19 +01:00
Akira Ajisaka b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str 
(cherry picked from commit aaaf856f4b)
 2019-04-03 13:18:36 +09:00
Akira Ajisaka 8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:54:16 +09:00
David Mollitor 397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup. 
Author:    David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859eb)
 2019-03-22 10:30:21 +00:00
David Mollitor 9a449ac075
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>

(cherry picked from commit 246ab77f28)
 2019-03-22 10:27:17 +00:00
Erik Krogen 0de8b55a09 HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
 2019-03-18 08:46:53 -07:00
Ben Roling 43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-14 19:46:34 +00:00
Erik Krogen fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
 2019-03-13 13:19:18 -07:00
Steve Loughran b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 11:15:25 +00:00
Praveen Krishna 451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. 
Author:    Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f)
 2019-03-07 11:08:48 +00:00
Stephen O'Donnell 3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

(cherry picked from commit 686c0141ef)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:11:49 +00:00
Daryn Sharp ae8839e6e8 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
 2019-02-20 18:21:14 -08:00
Akira Ajisaka 3e3275a80b
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit dabfeab785)
 2019-02-15 14:53:36 +09:00
Vinayakumar B 49af36b25c HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:45:34 +05:30
Wangda Tan 649da5af04 Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
 2019-02-05 14:05:51 -08:00
Steve Loughran bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f8483)
 2019-02-04 17:10:19 +00:00
Steve Loughran 3b47371fe2
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac)
 2019-02-01 14:34:11 +00:00
Eric Yang 65d2b13dc2 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218f)
 2019-01-22 19:39:05 -05:00
Chen Liang 54d4abdfe9 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:54:20 -08:00
luhuachao 18becec548
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621ab)
 2019-01-15 13:16:07 +00:00
Inigo Goiri c1c15c1bd8 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:08:14 -08:00
Akira Ajisaka d345b1f441
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da6)
 2019-01-11 13:04:40 +09:00
Jason Lowe 6236de9e23 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1f39eae7e6.
 2019-01-09 17:41:52 -06:00
Eric Yang febafd0e4f HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin

(cherry picked from commit d43af8b3db)
 2019-01-04 18:02:30 -05:00
Sunil G 7d752c1b9b HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 51427cbdfb)
 2019-01-04 21:49:59 +05:30
Pranay Singh 1f39eae7e6 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a)
 2019-01-03 09:57:08 -08:00
Surendra Singh Lilhore 5ab79f6377 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. 
(cherry picked from commit 763e96ea2b)
 2018-12-22 12:39:51 +05:30
Jason Lowe 9c89e2ea76 HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 
(cherry picked from commit 3961690037)
 2018-12-20 09:33:22 -06:00
Takanobu Asanuma 1b941a680d HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d963575918)
 2018-12-20 10:05:02 +09:00
Steve Loughran bbe1ddd309
HADOOP-15808. Harden Token service loader use. 
Contributed by Steve Loughran.

(cherry picked from commit 202926ac33)
 2018-12-11 17:34:16 +00:00
Steve Loughran 05e41dd4fe
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c6)
 2018-12-04 15:36:50 +00:00
Giovanni Matteo Fumarola 7cfcabc473 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:44:19 -08:00
Eric Yang 69be6e7352 HADOOP-15922. Revert patch 004. 
(cherry picked from commit 38ea3814bd)
 2018-11-30 12:51:58 -05:00
Akira Ajisaka b722bfd6f5
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f)
 2018-11-30 13:30:32 +09:00
Steve Loughran 1a448565a8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d27.

(cherry picked from commit d0edd37269)
 2018-11-29 17:53:23 +00:00
Eric Yang f089796251 HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. 
Contributed by He Xiaoqiao

(cherry picked from commit a9d96948ed)
 2018-11-26 13:52:42 -05:00
Vinayakumar B 1b937d701d HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c225)
 2018-11-26 18:13:09 +05:30
Akira Ajisaka f6227367fb
Add 2.9.2 release notes and changes documents. 
(cherry picked from commit 1a00b4e325)
 2018-11-20 14:25:48 +09:00
Surendra Singh Lilhore 86deff0a06 HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 75291e6d53)
 2018-11-13 21:26:22 +05:30
Steve Loughran d4afeac285
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. 
Contributed by LiXin Ge.

(cherry picked from commit 3c9d97b8f7)
 2018-11-12 12:58:53 +00:00
Jason Lowe 9265934201 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c726)
 2018-11-06 08:47:30 -06:00
Lars Francke a15bcf97a4
HADOOP-15687. Credentials class should allow access to aliases. 
Author:    Lars Francke <lars.francke@gmail.com>

(cherry picked from commit cb8d679c95)
 2018-11-03 16:29:56 +00:00
Steve Loughran d79ea3844c
HADOOP-15855. Review hadoop credential doc, including object store details. 
Contributed by Steve Loughran.

(cherry picked from commit 62d98ca92a)
 2018-10-30 15:58:44 +00:00
Wei-Chiu Chuang 5f8dd40647 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fd02c501c0.
 2018-10-28 17:05:45 -07:00
Wei-Chiu Chuang fd02c501c0 HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
 2018-10-25 09:36:28 -07:00
Akira Ajisaka 63396beab4
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe. 
(cherry picked from commit 8f97d6f2cd)
 2018-10-24 13:50:30 +09:00