2d8799f4bc
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2019-08-27 19:08:39 +00:00
1b30b3fbe7
HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 901652e26321832ac42ce54daa18ebc1813772b5)
2019-08-18 18:43:52 -07:00
d8a6098a96
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-08-16 10:50:00 -07:00
f507bc059d
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-08-16 10:50:00 -07:00
ca15968d09
HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang
2019-08-16 10:50:00 -07:00
22d81f7fd0
HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang.
2019-08-16 10:49:59 -07:00
e4b650f91e
HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian.
...
(cherry picked from 129576f628
2019-08-14 13:33:33 -07:00
d7ca016d63
HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04
2019-08-14 08:45:10 +09:00
0c02217d9b
HDFS-14691. Wrong usage hint for hadoop fs command "test".
...
Contributed by Jianfei Jiang.
Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
(cherry picked from commit 71aad60e51
2019-08-05 13:09:39 +01:00
490b18fd4f
HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat.
...
(cherry picked from commit e20b19543b
2019-08-01 17:36:48 -07:00
a405db8014
HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f86de6f76a
2019-08-01 14:56:38 -07:00
d4492bdd9e
HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
...
This commit also includes the follow-on commit 827a84778af96a2df38d
2019-07-30 14:41:36 -07:00
f921c72fda
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.
...
(cherry picked from commit cbfa3f3e98
2019-07-29 17:39:05 -07:00
5ec34bf43a
HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.
...
(cherry-picked from 62efb63006
2019-07-26 11:39:14 -07:00
7418764a67
HADOOP-9844. NPE when trying to create an error message response of SASL RPC
...
This closes #55
Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 07530314c2
2019-07-26 17:54:32 +09:00
23b470ee92
HADOOP-16443. Improve help text for setfacl --set option.
...
Contributed by S O'Donnell.
Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
2019-07-23 10:25:59 +01:00
76d2cd2283
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
...
Contributed by Gopal V and Atilla Magyar.
Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
2019-07-17 13:52:58 +01:00
1dbaf74985
HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.
...
(cherry picked from commit aa9f0850e8
2019-07-02 13:23:55 -07:00
c936562c72
HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko.
2019-06-28 14:23:14 -07:00
eb7d7fa5e2
HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun.
2019-06-28 14:23:13 -07:00
e917ac2854
HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun.
2019-06-28 14:23:12 -07:00
790831824f
HDFS-14138. [SBN read] Description errors in the comparison logic of transaction ID. Contributed by xiangheng.
2019-06-28 14:23:12 -07:00
c1c061d767
HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko.
2019-06-28 14:23:12 -07:00
45d6834d4c
HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun.
2019-06-28 14:23:10 -07:00
cc70e81931
HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun.
2019-06-28 14:23:09 -07:00
3998bab3da
HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen.
2019-06-28 14:23:05 -07:00
4c4ca86286
HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang.
2019-06-28 14:22:11 -07:00
e2337bfc57
HDFS-13767. Add msync server implementation. Contributed by Chen Liang.
2019-06-28 14:22:10 -07:00
7e0a71dd7f
HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov.
2019-06-28 14:22:08 -07:00
a97f00e1c6
HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun.
2019-06-28 14:22:08 -07:00
b13110a6f0
HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen.
2019-06-28 14:22:07 -07:00
a3521c53fe
HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov.
2019-06-28 14:22:03 -07:00
49d7bb6a92
HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
2019-06-28 14:20:01 -07:00
c0ca2bb853
HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov.
2019-06-28 14:20:01 -07:00
1eeca2d9fb
HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov.
2019-06-28 14:20:00 -07:00
77e391fd8b
HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 840d02ca5b
2019-06-20 19:58:17 -07:00
f046b881f6
HADOOP-16028. Fix NetworkTopology chooseRandom function to support excluded nodes. Contributed by Sihai Ke.
...
(cherry picked from commit f4e18242bd
2019-06-20 13:38:00 -07:00
99abd35a75
HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dd4a7633ec
2019-06-17 23:48:29 -07:00
2667177248
HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9f47a2e4a9845b84cc963708c7924b92df2964f6)
2019-06-17 22:56:04 -07:00
e8db39b0e9
HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor.
...
(cherry picked from commit f1c239c6a4
2019-06-17 21:55:55 -07:00
73829589b7
HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.
...
(cherry picked from commit 54f9f75a44
2019-06-14 10:18:46 +09:00
e0b3cbd221
HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota.
2019-06-13 07:53:40 -06:00
f607efd195
HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.
...
(cherry-picked from 76b94c274f
2019-06-07 14:22:08 -07:00
a9a3450560
HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:34:16 +09:00
855dc997d6
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:27:33 +09:00
7a5d9ab9ee
HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93d7630062
2019-05-24 18:43:30 +02:00
3f05b699ef
HADOOP-16248. MutableQuantiles leak memory under heavy load.
...
Contributed by Alexis Daboville,
(cherry picked from commit 4cb3da6ac7
2019-05-17 15:16:59 +01:00
6eb48542f1
HADOOP-16307. Intern User Name and Group Name in FileStatus.
...
Author: David Mollitor
(cherry picked from commit 2713dcf6e9
2019-05-16 16:03:06 +02:00
2c72f48527
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.
...
(cherry picked from commit d4c8858586
2019-05-15 17:56:22 -07:00
095815220d
HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 93f2283a69
2019-05-06 15:47:58 -07:00
28ff96f367
HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.
...
(cherry-picked from 1ddb48872f
2019-04-22 08:18:17 -07:00
b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
...
(cherry picked from commit ab645b3caa
2019-04-08 15:19:17 +09:00
50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
(cherry picked from commit c90736350b
2019-04-04 16:16:57 +01:00
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
2019-04-03 21:35:19 +01:00
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
...
(cherry picked from commit aaaf856f4b
2019-04-03 13:18:36 +09:00
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
...
(cherry picked from commit ebd0d21538
2019-04-02 10:54:16 +09:00
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859eb
2019-03-22 10:30:21 +00:00
9a449ac075
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit 246ab77f28
2019-03-22 10:27:17 +00:00
0de8b55a09
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
...
(cherry-picked from 8c95cb9d6b
2019-03-18 08:46:53 -07:00
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
fec7c5f3eb
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
...
(cherry picked from commit 66357574ae
2019-03-13 13:19:18 -07:00
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host.
...
Author: Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f
2019-03-07 11:08:48 +00:00
3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately.
...
Contributed by Stephen O'Donnell.
(cherry picked from commit 686c0141ef
2019-03-05 14:11:49 +00:00
ae8839e6e8
HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432
2019-02-20 18:21:14 -08:00
3e3275a80b
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg.
...
(cherry picked from commit dabfeab785
2019-02-15 14:53:36 +09:00
49af36b25c
HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena.
2019-02-13 16:45:34 +05:30
649da5af04
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa
2019-02-05 14:05:51 -08:00
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
(cherry picked from commit c1d24f8483
2019-02-04 17:10:19 +00:00
3b47371fe2
HADOOP-16079. Token.toString faulting if any token listed can't load.
...
Contributed by Steve Loughran.
(cherry picked from commit 7f46d13dac
2019-02-01 14:34:11 +00:00
65d2b13dc2
HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user.
...
Contributed by He Xiaoqiao
(cherry picked from commit 0dd35e218f
2019-01-22 19:39:05 -05:00
54d4abdfe9
HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian.
2019-01-18 11:54:20 -08:00
18becec548
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration.
...
Contributed by luhuachao.
(cherry picked from commit ccc43621ab
2019-01-15 13:16:07 +00:00
c1c15c1bd8
HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak.
...
(cherry picked from commit 35fa3bd685
2019-01-11 18:08:14 -08:00
d345b1f441
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper
...
(cherry picked from commit d4ca907da6
2019-01-11 13:04:40 +09:00
6236de9e23
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1f39eae7e6
2019-01-09 17:41:52 -06:00
febafd0e4f
HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop.
...
Contributed by Bolke de Bruin
(cherry picked from commit d43af8b3db
2019-01-04 18:02:30 -05:00
7d752c1b9b
HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang.
...
(cherry picked from commit 51427cbdfb
2019-01-04 21:49:59 +05:30
1f39eae7e6
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a
2019-01-03 09:57:08 -08:00
5ab79f6377
HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan.
...
(cherry picked from commit 763e96ea2b
2018-12-22 12:39:51 +05:30
9c89e2ea76
HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne
...
(cherry picked from commit 3961690037
2018-12-20 09:33:22 -06:00
1b941a680d
HDFS-13661. Ls command with e option fails when the filesystem is not HDFS.
...
(cherry picked from commit d963575918
2018-12-20 10:05:02 +09:00
bbe1ddd309
HADOOP-15808. Harden Token service loader use.
...
Contributed by Steve Loughran.
(cherry picked from commit 202926ac33
2018-12-11 17:34:16 +00:00
05e41dd4fe
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset.
...
Contributed by Steve Loughran
(cherry picked from commit db2d8b01c6
2018-12-04 15:36:50 +00:00
7cfcabc473
HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak.
2018-12-03 12:44:19 -08:00
69be6e7352
HADOOP-15922. Revert patch 004.
...
(cherry picked from commit 38ea3814bd
2018-11-30 12:51:58 -05:00
b722bfd6f5
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
...
(cherry picked from commit c9bfca217f
2018-11-30 13:30:32 +09:00
1a448565a8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
...
This reverts commit 829a2e4d27d0edd37269
2018-11-29 17:53:23 +00:00
f089796251
HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter.
...
Contributed by He Xiaoqiao
(cherry picked from commit a9d96948ed
2018-11-26 13:52:42 -05:00
1b937d701d
HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena.
...
(cherry picked from commit 23b441c225
2018-11-26 18:13:09 +05:30
f6227367fb
Add 2.9.2 release notes and changes documents.
...
(cherry picked from commit 1a00b4e325
2018-11-20 14:25:48 +09:00
86deff0a06
HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore
...
(cherry picked from commit 75291e6d53
2018-11-13 21:26:22 +05:30
d4afeac285
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds.
...
Contributed by LiXin Ge.
(cherry picked from commit 3c9d97b8f7
2018-11-12 12:58:53 +00:00
9265934201
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
...
(cherry picked from commit ab6aa4c726
2018-11-06 08:47:30 -06:00
a15bcf97a4
HADOOP-15687. Credentials class should allow access to aliases.
...
Author: Lars Francke <lars.francke@gmail.com>
(cherry picked from commit cb8d679c95
2018-11-03 16:29:56 +00:00
d79ea3844c
HADOOP-15855. Review hadoop credential doc, including object store details.
...
Contributed by Steve Loughran.
(cherry picked from commit 62d98ca92a
2018-10-30 15:58:44 +00:00
5f8dd40647
Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao."
...
This reverts commit fd02c501c0
2018-10-28 17:05:45 -07:00
fd02c501c0
HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao.
...
(cherry picked from commit fb2b72e6fc
2018-10-25 09:36:28 -07:00
63396beab4
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe.
...
(cherry picked from commit 8f97d6f2cd
2018-10-24 13:50:30 +09:00
85bf008405
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility.
...
Contributed by Wei-Chiu Chuang.
(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
2018-10-22 11:08:30 +01:00
Akira Ajisaka
David Mollitor
Chen Liang
Christopher Gregorian
Masatake Iwasaki
Jianfei Jiang
Wei-Chiu Chuang
He Xiaoqiao
Erik Krogen
Steve Loughran
S O'Donnell
Gopal V
Inigo Goiri
Konstantin V Shvachko
Chao Sun
Plamen Jeliazkov
Shweta Yakkali
Adam Antal
Takanobu Asanuma
Sean Mackrory
Erik Krogen
Alexis Daboville
Bharat Viswanadham
Siyao Meng
Ben Roling
Praveen Krishna
Daryn Sharp
Vinayakumar B
Wangda Tan
Eric Yang
luhuachao
Jason Lowe
Sunil G
Pranay Singh
Surendra Singh Lilhore
Giovanni Matteo Fumarola
Lars Francke