Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,559 Commits 326 Branches 371 Tags
Commit Graph

4822 Commits

Author SHA1 Message Date
Alexis Daboville
3f05b699ef
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac703b10de59a8ca371fe957a4a338601)
 2019-05-17 15:16:59 +01:00
David Mollitor
6eb48542f1
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor

(cherry picked from commit 2713dcf6e9ef308ffe6102532c90b27c52d27f7c)
 2019-05-16 16:03:06 +02:00
Bharat Viswanadham
2c72f48527
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 
(cherry picked from commit d4c8858586eeed2820f3ab21da79603b52c64594)
 2019-05-15 17:56:22 -07:00
Siyao Meng
095815220d HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69ea4e07a998f2a4065f238f9574921b)
 2019-05-06 15:47:58 -07:00
Erik Krogen
28ff96f367 HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f6a4985f4d0baadbb183899226cff68)
 2019-04-22 08:18:17 -07:00
Akira Ajisaka
b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang. 
(cherry picked from commit ab645b3caa0313f08a26bdd70fa14434d09e6cb5)
 2019-04-08 15:19:17 +09:00
David Mollitor
50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350ba158c7872a39426e7a29c5e5e0bb48)
 2019-04-04 16:16:57 +01:00
Steve Loughran
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d9990ef9059b6ac9a19ad24310d6f36d04)
 2019-04-03 21:35:19 +01:00
Akira Ajisaka
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str 
(cherry picked from commit aaaf856f4b7b53d424eb1eab010311de0d5fbe1e)
 2019-04-03 13:18:36 +09:00
Akira Ajisaka
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d2153884bf78a159534ea711c0b70acc1c05)
 2019-04-02 10:54:16 +09:00
David Mollitor
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup. 
Author:    David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859ebfc46a18fd9140b42fb95f1da96380e)
 2019-03-22 10:30:21 +00:00
David Mollitor
9a449ac075
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>

(cherry picked from commit 246ab77f281614b2206bfb1dae70a8dec62147a7)
 2019-03-22 10:27:17 +00:00
Erik Krogen
0de8b55a09 HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6bef369fef6a8364f0c0764eba90e44a)
 2019-03-18 08:46:53 -07:00
Ben Roling
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-14 19:46:34 +00:00
Erik Krogen
fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
 2019-03-13 13:19:18 -07:00
Steve Loughran
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 11:15:25 +00:00
Praveen Krishna
451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. 
Author:    Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f17548e5ef6af8e1e934a94031bb1dc)
 2019-03-07 11:08:48 +00:00
Stephen O'Donnell
3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

(cherry picked from commit 686c0141eff0886c285b8e52fddade43c1ce4570)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:11:49 +00:00
Daryn Sharp
ae8839e6e8 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432609b7a35a2abd6410b02e8a2ffc974)
 2019-02-20 18:21:14 -08:00
Akira Ajisaka
3e3275a80b
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit dabfeab7854aab9b1eacf05bca954f2cf4e5ab89)
 2019-02-15 14:53:36 +09:00
Vinayakumar B
49af36b25c HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:45:34 +05:30
Wangda Tan
649da5af04 Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa221a336ef0dbb1a17358b1ad6f9356)
 2019-02-05 14:05:51 -08:00
Steve Loughran
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f848345f6d34a2ac2d570d49e9787a0df6a)
 2019-02-04 17:10:19 +00:00
Steve Loughran
3b47371fe2
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac8cf85b094f41b3dd68e02c69e5afbc)
 2019-02-01 14:34:11 +00:00
Eric Yang
65d2b13dc2 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218fd4d6c660fd064e893be3112c546c9f)
 2019-01-22 19:39:05 -05:00
Chen Liang
54d4abdfe9 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:54:20 -08:00
luhuachao
18becec548
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621aba4f1fc9ca880e294a66862493805b0)
 2019-01-15 13:16:07 +00:00
Inigo Goiri
c1c15c1bd8 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685605d8b3639e6c5cbe83cd9acd8cbe7)
 2019-01-11 18:08:14 -08:00
Akira Ajisaka
d345b1f441
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da636892e4ab98e232fe0f7f77f1f7aac)
 2019-01-11 13:04:40 +09:00
Jason Lowe
6236de9e23 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit 1f39eae7e6f59206b86f96063ffb2ebe15a9cbe1.
 2019-01-09 17:41:52 -06:00
Eric Yang
febafd0e4f HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin

(cherry picked from commit d43af8b3db4743b4b240751b6f29de6c20cfd6e5)
 2019-01-04 18:02:30 -05:00
Sunil G
7d752c1b9b HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 51427cbdfb39cb6f5774b7b70009d7ee4388edfc)
 2019-01-04 21:49:59 +05:30
Pranay Singh
1f39eae7e6 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6ad43555031aed032e6ba7a14a17d7bc)
 2019-01-03 09:57:08 -08:00
Surendra Singh Lilhore
5ab79f6377 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. 
(cherry picked from commit 763e96ea2b18f584ba102b68271263be8c5d621c)
 2018-12-22 12:39:51 +05:30
Jason Lowe
9c89e2ea76 HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 
(cherry picked from commit 3961690037fa4274a3640104b3c344e8626caa10)
 2018-12-20 09:33:22 -06:00
Takanobu Asanuma
1b941a680d HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d9635759182b614a1dd5034c30978e7c4be8d0dd)
 2018-12-20 10:05:02 +09:00
Steve Loughran
bbe1ddd309
HADOOP-15808. Harden Token service loader use. 
Contributed by Steve Loughran.

(cherry picked from commit 202926ac3301298753abd0e6e1f324caf0202ec6)
 2018-12-11 17:34:16 +00:00
Steve Loughran
05e41dd4fe
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c65fdff3acf83b4c4f0a79fe48270487)
 2018-12-04 15:36:50 +00:00
Giovanni Matteo Fumarola
7cfcabc473 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:44:19 -08:00
Eric Yang
69be6e7352 HADOOP-15922. Revert patch 004. 
(cherry picked from commit 38ea3814bd0641d895e5d3b7415c6308e7f8491e)
 2018-11-30 12:51:58 -05:00
Akira Ajisaka
b722bfd6f5
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f4b15a3a367db51147d0dc2075ca274)
 2018-11-30 13:30:32 +09:00
Steve Loughran
1a448565a8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d271f05afb209ddc834cd4a0e85492eda.

(cherry picked from commit d0edd37269bb40290b409d583bcf3b70897c13e0)
 2018-11-29 17:53:23 +00:00
Eric Yang
f089796251 HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. 
Contributed by He Xiaoqiao

(cherry picked from commit a9d96948ede11d4d8a8fb58b31b150af84e118d9)
 2018-11-26 13:52:42 -05:00
Vinayakumar B
1b937d701d HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c2253bcb3be5229d3c5eb1e165369070c0)
 2018-11-26 18:13:09 +05:30
Akira Ajisaka
f6227367fb
Add 2.9.2 release notes and changes documents. 
(cherry picked from commit 1a00b4e325146988375c9ce5b11016c45f059a4e)
 2018-11-20 14:25:48 +09:00
Surendra Singh Lilhore
86deff0a06 HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 75291e6d53c13debf45493a870a898b63779914b)
 2018-11-13 21:26:22 +05:30
Steve Loughran
d4afeac285
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. 
Contributed by LiXin Ge.

(cherry picked from commit 3c9d97b8f7d6eb75f08fc6d37cee37c22760bb86)
 2018-11-12 12:58:53 +00:00
Jason Lowe
9265934201 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c7265db5bcbb446c2f779289023d454b81)
 2018-11-06 08:47:30 -06:00
Lars Francke
a15bcf97a4
HADOOP-15687. Credentials class should allow access to aliases. 
Author:    Lars Francke <lars.francke@gmail.com>

(cherry picked from commit cb8d679c95642842efacc5d38ccf2a61b043c689)
 2018-11-03 16:29:56 +00:00
Steve Loughran
d79ea3844c
HADOOP-15855. Review hadoop credential doc, including object store details. 
Contributed by Steve Loughran.

(cherry picked from commit 62d98ca92aee15d1790d169bfdf0043b05b748ce)
 2018-10-30 15:58:44 +00:00