Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,155 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4880 Commits

Author SHA1 Message Date
Gopal V 2c4d8c343e HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679 
Contributed by Gopal V and Atilla Magyar.

Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
(cherry picked from commit 76d2cd2283)
 2019-07-17 17:13:03 +01:00
Konstantin V Shvachko 321e26ced1 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:43 -07:00
Chao Sun 8f36bdc3b9 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Erik Krogen 6c1bf61628 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-12 12:24:40 -07:00
Chen Liang c23755a792 HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. 2019-07-12 12:24:40 -07:00
Erik Krogen 67c4768825 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-12 12:24:36 -07:00
Chao Sun c4d944aeb3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-12 12:23:16 -07:00
Chao Sun 8100dcbb34 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-07-12 12:11:18 -07:00
Chen Liang 91803c8216 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-07-12 11:50:51 -07:00
Chen Liang 8ec17f6aea HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-12 11:50:03 -07:00
Chen Liang ee9977cc69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-12 11:50:01 -07:00
Plamen Jeliazkov cfb4399b36 HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. 2019-07-12 11:05:08 -07:00
Chao Sun 9840dae72f HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 11:05:07 -07:00
Erik Krogen abb8614dcf HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:04:24 -07:00
Plamen Jeliazkov 7d0c3ba700 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:31 -07:00
Erik Krogen 07a6510e6a HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen b999e96c9a HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:30 -07:00
Plamen Jeliazkov 21a886a4ef HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:27 -07:00
Inigo Goiri 3ab475377a HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena. 
(cherry picked from commit aa9f0850e8)
 2019-07-02 13:26:17 -07:00
Wei-Chiu Chuang e4a6976c30 HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 840d02ca5b)
 2019-06-20 19:58:28 -07:00
Inigo Goiri 3dd4000884 HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke. 
(cherry picked from commit f4e18242bd)
 2019-06-20 13:38:48 -07:00
Shweta Yakkali 0b3a2eb8a7 HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd4a7633ec)
(cherry picked from commit 99abd35a75)
 2019-06-17 23:48:40 -07:00
Adam Antal 15689ce0e8 HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9f47a2e4a9845b84cc963708c7924b92df2964f6)
(cherry picked from commit 2667177248)
 2019-06-17 22:56:18 -07:00
Wei-Chiu Chuang 8f4204dc3b HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor. 
(cherry picked from commit f1c239c6a4)
(cherry picked from commit e8db39b0e9)
 2019-06-17 21:56:04 -07:00
Takanobu Asanuma 94e66ba338 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a44)
 2019-06-14 10:19:50 +09:00
Sean Mackrory fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Erik Krogen cf3c1b9ffc HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274f)
(cherry-picked from f607efd195)
 2019-06-07 14:22:30 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Shweta a6ff20d25d HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93d7630062)
(cherry picked from commit 7a5d9ab9ee)
 2019-05-24 18:43:49 +02:00
Alexis Daboville 55cc35c0e1
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac7)
 2019-05-17 15:17:55 +01:00
David Mollitor cf0d5a0e6e
HADOOP-16307. Intern User Name and Group Name in FileStatus. 
Author:    David Mollitor

(cherry picked from commit 2713dcf6e9)
 2019-05-16 16:03:34 +02:00
Bharat Viswanadham d4b4d90a52
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy. 
(cherry picked from commit d4c8858586)
 2019-05-15 17:50:56 -07:00
Siyao Meng 8414353daa HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69)
(cherry picked from commit 095815220d)
 2019-05-06 15:48:14 -07:00
Erik Krogen 9da70a1b4b HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f)
(cherry-picked from 28ff96f367)
 2019-04-22 08:19:25 -07:00
David Mollitor 41f1bc2056
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350b)
 2019-04-04 21:13:09 +01:00
Steve Loughran 3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka 10d6e68324
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:55:07 +09:00
Erik Krogen d4fbbc83ad HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
 2019-03-18 08:53:13 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Steve Loughran 1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Daryn Sharp 704330a616 HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
 2019-02-20 18:22:00 -08:00
Vinayakumar B c450f9f2c4 HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena. 2019-02-13 16:46:04 +05:30
Wangda Tan a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
(cherry picked from commit 649da5af04)
 2019-02-05 14:09:33 -08:00
Todd Lipcon 1f21be2d02
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.
 2019-02-05 19:56:04 +00:00
Steve Loughran 9a31dd4c67
HADOOP-16079. Token.toString faulting if any token listed can't load. 
Contributed by Steve Loughran.

(cherry picked from commit 7f46d13dac)
 2019-02-01 14:34:39 +00:00
Eric Yang ba3f559724 HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user. 
Contributed by He Xiaoqiao

(cherry picked from commit 0dd35e218f)
 2019-01-22 19:39:39 -05:00
Chen Liang 0cfc2d8475 HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:48:26 -08:00
luhuachao 4297a3b7ac
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration. 
Contributed by luhuachao.

(cherry picked from commit ccc43621ab)
 2019-01-15 13:16:50 +00:00
Inigo Goiri b05d8de10e HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:10:40 -08:00