Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,999 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4745 Commits

Author SHA1 Message Date
Christopher Gregorian ec00431eaa HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
 2019-07-30 14:46:11 -07:00
Erik Krogen f7e517ee24 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb63006)
(cherry-picked from 5ec34bf43a)
(cherry-picked from e2aab0efef9d504e4695a9c1df2835ed60f6c0c7)
 2019-07-26 11:43:34 -07:00
Konstantin V Shvachko 380668559f HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun 9aecfdcb52 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Erik Krogen 8b8ec65e65 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang 2e7610a029 HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng. 2019-07-25 10:07:05 -07:00
Erik Krogen 7b1e3c49d1 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun 6916ee71fd HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chao Sun 11897618b9 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang 56af83c961 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Chen Liang 613c9e4f7b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Chen Liang 60f0e4df69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov ac0cdb0a8d HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Chao Sun 13e86926a5 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Erik Krogen db9cec6ad2 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov d502e48a7c HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Erik Krogen 849e1f018e HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Erik Krogen c2585f7e28 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov ccad9ce59b HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Takanobu Asanuma 1be7c42cb7 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a44)
 2019-06-14 10:20:33 +09:00
Erik Krogen d43219c21d HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274f)
(cherry-picked from f607efd195)
(cherry-picked from cf3c1b9ffc)
 2019-06-07 14:22:48 -07:00
Sean Mackrory c352b0011e HADOOP-16212. Update guava to 27.0-jre in hadoop-project branch-3.0. Contributed by Gabor Bota. 2019-06-03 07:45:56 -06:00
Alexis Daboville 3b09902dd2
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac7)
 2019-05-17 15:18:19 +01:00
Erik Krogen c20d61c34d HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f)
(cherry-picked from 28ff96f367)
(cherry-picked from 9da70a1b4b)
 2019-04-22 08:20:17 -07:00
Steve Loughran bdf8be6374
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 22:40:09 +01:00
Akira Ajisaka 524dc4e662
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d21538)
 2019-04-02 10:55:27 +09:00
Erik Krogen e172fc62ce HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6b)
(cherry-picked from 0de8b55a09)
(cherry-picked from d4fbbc83ad)
 2019-03-18 08:57:41 -07:00
Erik Krogen 9a5daa5ec6 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
(cherry picked from commit cb76fc5618)
 2019-03-13 13:21:40 -07:00
Steve Loughran 7f70a78ce3
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589) 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
 2019-03-12 11:36:32 +00:00
Daryn Sharp 4eccf2a3cc HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432)
(cherry picked from commit ae8839e6e8)
(cherry picked from commit 704330a616)
 2019-02-20 18:22:20 -08:00
Todd Lipcon 02cfd5fa35
HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU consumption. 
Contributed by Todd Lipcon.

(cherry picked from commit 1f21be2d02)
 2019-02-05 19:57:08 +00:00
Chen Liang 80b9fecaaf HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian. 2019-01-18 11:45:22 -08:00
Inigo Goiri 174fa73f99 HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 35fa3bd685)
 2019-01-11 18:16:44 -08:00
Akira Ajisaka 9fd5a6b66b
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper 
(cherry picked from commit d4ca907da6)
 2019-01-11 13:05:31 +09:00
Jason Lowe 642213a74a Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh." 
This reverts commit a4f8421324.
 2019-01-09 17:56:32 -06:00
Pranay Singh a4f8421324 HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a)
(cherry picked from commit 1f39eae7e6)
(cherry picked from commit e8e55839a0)
 2019-01-03 13:44:57 -08:00
Jason Lowe c9fdf7503e HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne 2018-12-20 10:00:17 -06:00
Takanobu Asanuma e38e184a19 HDFS-13661. Ls command with e option fails when the filesystem is not HDFS. 
(cherry picked from commit d963575918)
 2018-12-20 10:06:26 +09:00
Steve Loughran 38335e0b7d
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset. 
Contributed by Steve Loughran

(cherry picked from commit db2d8b01c6)
 2018-12-04 15:37:36 +00:00
Giovanni Matteo Fumarola 25fdf2ba6c HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak. 2018-12-03 12:45:51 -08:00
Akira Ajisaka c9c4511a0e
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin. 
(cherry picked from commit c9bfca217f)
 2018-11-30 13:31:13 +09:00
Steve Loughran 065a1e72f2
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple" 
This reverts commit 829a2e4d27.

(cherry picked from commit d0edd37269)
 2018-11-29 17:58:15 +00:00
Vinayakumar B c2e65cac5e HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena. 
(cherry picked from commit 23b441c225)
 2018-11-26 18:14:30 +05:30
Jason Lowe f3e6682ba7 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c726)
 2018-11-06 08:59:47 -06:00
Wei-Chiu Chuang 0f34ff7722 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fa01a8fa0e.
 2018-10-28 17:07:29 -07:00
Wei-Chiu Chuang fa01a8fa0e HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
(cherry picked from commit fd02c501c0)
(cherry picked from commit ef9f8ca13d)
 2018-10-25 09:38:26 -07:00
Akira Ajisaka e402791a51
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe. 
(cherry picked from commit 8f97d6f2cd)

Conflicts:
	dev-support/docker/Dockerfile
 2018-10-24 13:56:19 +09:00
Steve Loughran a069dade5c
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
 2018-10-22 11:09:04 +01:00
Siyao Meng 6a7bf9f850 HDFS-14009. HttpFS: FileStatus#setSnapShotEnabledFlag throws InvocationTargetException when attribute set is emptySet. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-18 17:02:38 -07:00
Jason Lowe 0aee3a0c3f HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe 
(cherry picked from commit 9abda83947)
 2018-10-17 14:44:39 -05:00