Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,072 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4860 Commits

Author SHA1 Message Date
Inigo Goiri 3dd4000884 HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. 
(cherry picked from commit f4e18242bd)
 2019-06-20 13:38:48 -07:00
Shweta Yakkali 0b3a2eb8a7 HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd4a7633ec)
(cherry picked from commit 99abd35a75)
 2019-06-17 23:48:40 -07:00
Adam Antal 15689ce0e8 HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9f47a2e4a9845b84cc963708c7924b92df2964f6)
(cherry picked from commit 2667177248)
 2019-06-17 22:56:18 -07:00
Wei-Chiu Chuang 8f4204dc3b HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 
(cherry picked from commit f1c239c6a4)
(cherry picked from commit e8db39b0e9)
 2019-06-17 21:56:04 -07:00
Takanobu Asanuma 94e66ba338 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a44)
 2019-06-14 10:19:50 +09:00
Sean Mackrory fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Erik Krogen cf3c1b9ffc HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274f)
(cherry-picked from f607efd195)
 2019-06-07 14:22:30 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Shweta a6ff20d25d HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93d7630062)
(cherry picked from commit 7a5d9ab9ee)
 2019-05-24 18:43:49 +02:00
Alexis Daboville 55cc35c0e1
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac7)
 2019-05-17 15:17:55 +01:00
David Mollitor cf0d5a0e6e
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor

(cherry picked from commit 2713dcf6e9)
 2019-05-16 16:03:34 +02:00
Bharat Viswanadham d4b4d90a52
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 
(cherry picked from commit d4c8858586)
 2019-05-15 17:50:56 -07:00
Siyao Meng 8414353daa HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69)
(cherry picked from commit 095815220d)
 2019-05-06 15:48:14 -07:00
Erik Krogen 9da70a1b4b HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f)
(cherry-picked from 28ff96f367)
 2019-04-22 08:19:25 -07:00
David Mollitor 41f1bc2056
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350b)
 2019-04-04 21:13:09 +01:00
Steve Loughran 3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka 10d6e68324
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:55:07 +09:00
Erik Krogen d4fbbc83ad HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
 2019-03-18 08:53:13 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Steve Loughran 1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Daryn Sharp 704330a616 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
 2019-02-20 18:22:00 -08:00
Vinayakumar B c450f9f2c4 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:46:04 +05:30
Wangda Tan a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
(cherry picked from commit 649da5af04)
 2019-02-05 14:09:33 -08:00
Todd Lipcon 1f21be2d02
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.
 2019-02-05 19:56:04 +00:00
Steve Loughran 9a31dd4c67
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac)
 2019-02-01 14:34:39 +00:00
Eric Yang ba3f559724 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218f)
 2019-01-22 19:39:39 -05:00
Chen Liang 0cfc2d8475 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:48:26 -08:00
luhuachao 4297a3b7ac
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621ab)
 2019-01-15 13:16:50 +00:00
Inigo Goiri b05d8de10e HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:10:40 -08:00
Akira Ajisaka ced32754bb
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da6)
 2019-01-11 13:05:06 +09:00
Jason Lowe d8aac6c215 Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit e8e55839a0.
 2019-01-09 17:48:16 -06:00
Wangda Tan 31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Eric Yang 2ba9c9f7c1 HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop. 
Contributed by Bolke de Bruin

(cherry picked from commit d43af8b3db)
(cherry picked from commit febafd0e4f)
 2019-01-04 18:04:11 -05:00
Sunil G d3fdbbc2ab HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 51427cbdfb)
 2019-01-04 21:51:00 +05:30
Pranay Singh e8e55839a0 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a)
(cherry picked from commit 1f39eae7e6)
 2019-01-03 09:57:34 -08:00
Surendra Singh Lilhore ef94eeebb4 HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan. 
(cherry picked from commit 763e96ea2b)
 2018-12-22 12:46:37 +05:30
Jason Lowe 5c5c4438ad HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 
(cherry picked from commit 3961690037)
 2018-12-20 09:56:12 -06:00
Takanobu Asanuma 9f30916a1b HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d963575918)
 2018-12-20 10:05:47 +09:00
Steve Loughran 1192fad0d2
HADOOP-15808. Harden Token service loader use. 
Contributed by Steve Loughran.

Note that the patch for TestToken is much simpler than the 3.2+ patch.
In later branches,it modifies a test case which is not present in this branch.

(cherry picked from commit 202926ac33)
 2018-12-11 17:39:55 +00:00
Steve Loughran 844f51217b
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c6)
 2018-12-04 15:37:13 +00:00
Giovanni Matteo Fumarola 023e42fa05 HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:45:05 -08:00
Eric Yang e534be892c HADOOP-15922. Revert patch 004. 
(cherry picked from commit 38ea3814bd)
 2018-11-30 12:52:36 -05:00
Akira Ajisaka 298c76c485
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f)
 2018-11-30 13:30:52 +09:00
Steve Loughran 65e0d6ff46
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d27.

(cherry picked from commit d0edd37269)
 2018-11-29 17:57:24 +00:00
Eric Yang 20a7f3b50d HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter. 
Contributed by He Xiaoqiao

(cherry picked from commit a9d96948ed)
 2018-11-26 13:53:16 -05:00
Vinayakumar B 07529a209d HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c225)
 2018-11-26 18:11:55 +05:30
Surendra Singh Lilhore c97f8b6d95 HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 75291e6d53)
 2018-11-13 21:40:00 +05:30
Steve Loughran 29752ed580
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds. 
Contributed by LiXin Ge.

(cherry picked from commit 3c9d97b8f7)
 2018-11-12 12:59:15 +00:00
Jason Lowe 7335d940de YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c726)
 2018-11-06 08:52:29 -06:00