b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
...
(cherry picked from commit ab645b3caa
2019-04-08 15:19:17 +09:00
50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
(cherry picked from commit c90736350b
2019-04-04 16:16:57 +01:00
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
2019-04-03 21:35:19 +01:00
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
...
(cherry picked from commit aaaf856f4b
2019-04-03 13:18:36 +09:00
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
...
(cherry picked from commit ebd0d21538
2019-04-02 10:54:16 +09:00
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859eb
2019-03-22 10:30:21 +00:00
9a449ac075
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit 246ab77f28
2019-03-22 10:27:17 +00:00
0de8b55a09
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
...
(cherry-picked from 8c95cb9d6b
2019-03-18 08:46:53 -07:00
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
fec7c5f3eb
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
...
(cherry picked from commit 66357574ae
2019-03-13 13:19:18 -07:00
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host.
...
Author: Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f
2019-03-07 11:08:48 +00:00
3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately.
...
Contributed by Stephen O'Donnell.
(cherry picked from commit 686c0141ef
2019-03-05 14:11:49 +00:00
ae8839e6e8
HADOOP-15813. Enable more reliable SSL connection reuse. Contributed by Daryn Sharp.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a87e458432
2019-02-20 18:21:14 -08:00
3e3275a80b
YARN-9308. fairscheduler-statedump.log gets generated regardless of service again after the merge of HDFS-7240. Contributed by Wilfred Spiegelenburg.
...
(cherry picked from commit dabfeab785
2019-02-15 14:53:36 +09:00
49af36b25c
HADOOP-16108. Tail Follow Interval Should Allow To Specify The Sleep Interval To Save Unnecessary RPC's. Contributed by Ayush Saxena.
2019-02-13 16:45:34 +05:30
649da5af04
Make upstream aware of 3.1.2 release
...
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa
2019-02-05 14:05:51 -08:00
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
(cherry picked from commit c1d24f8483
2019-02-04 17:10:19 +00:00
3b47371fe2
HADOOP-16079. Token.toString faulting if any token listed can't load.
...
Contributed by Steve Loughran.
(cherry picked from commit 7f46d13dac
2019-02-01 14:34:11 +00:00
65d2b13dc2
HADOOP-15922. Fixed DelegationTokenAuthenticator URL decoding for doAs user.
...
Contributed by He Xiaoqiao
(cherry picked from commit 0dd35e218f
2019-01-22 19:39:05 -05:00
54d4abdfe9
HADOOP-15481. Emit FairCallQueue stats as metrics. Contributed by Christopher Gregorian.
2019-01-18 11:54:20 -08:00
18becec548
HADOOP-16019. ZKDelegationTokenSecretManager won't log exception message occured in function setJaasConfiguration.
...
Contributed by luhuachao.
(cherry picked from commit ccc43621ab
2019-01-15 13:16:07 +00:00
c1c15c1bd8
HADOOP-16045. Don't run TestDU on Windows. Contributed by Lukas Majercak.
...
(cherry picked from commit 35fa3bd685
2019-01-11 18:08:14 -08:00
d345b1f441
HADOOP-16016. TestSSLFactory#testServerWeakCiphers fails on Java 1.8.0_191 or upper
...
(cherry picked from commit d4ca907da6
2019-01-11 13:04:40 +09:00
6236de9e23
Revert "HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh."
...
This reverts commit 1f39eae7e6
2019-01-09 17:41:52 -06:00
febafd0e4f
HADOOP-15996. Improved Kerberos username mapping strategy in Hadoop.
...
Contributed by Bolke de Bruin
(cherry picked from commit d43af8b3db
2019-01-04 18:02:30 -05:00
7d752c1b9b
HADOOP-15997. KMS client uses wrong UGI after HADOOP-14445. Contributed by Wei-Chiu Chuang.
...
(cherry picked from commit 51427cbdfb
2019-01-04 21:49:59 +05:30
1f39eae7e6
HDFS-14084. Need for more stats in DFSClient. Contributed by Pranay Singh.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ecdeaa7e6a
2019-01-03 09:57:08 -08:00
5ab79f6377
HDFS-14166. Ls with -e option not giving the result in proper format. Contributed by Shubham Dewan.
...
(cherry picked from commit 763e96ea2b
2018-12-22 12:39:51 +05:30
9c89e2ea76
HADOOP-15973. Configuration: Included properties are not cached if resource is a stream. Contributed by Eric Payne
...
(cherry picked from commit 3961690037
2018-12-20 09:33:22 -06:00
1b941a680d
HDFS-13661. Ls command with e option fails when the filesystem is not HDFS.
...
(cherry picked from commit d963575918
2018-12-20 10:05:02 +09:00
bbe1ddd309
HADOOP-15808. Harden Token service loader use.
...
Contributed by Steve Loughran.
(cherry picked from commit 202926ac33
2018-12-11 17:34:16 +00:00
05e41dd4fe
HADOOP-15966. Hadoop Kerberos broken on macos as java.security.krb5.realm is reset.
...
Contributed by Steve Loughran
(cherry picked from commit db2d8b01c6
2018-12-04 15:36:50 +00:00
7cfcabc473
HADOOP-15950. Failover for LdapGroupsMapping. Contributed by Lukas Majercak.
2018-12-03 12:44:19 -08:00
69be6e7352
HADOOP-15922. Revert patch 004.
...
(cherry picked from commit 38ea3814bd
2018-11-30 12:51:58 -05:00
b722bfd6f5
MAPREDUCE-7165. mapred-site.xml is misformatted in single node setup document. Contributed by Zhaohui Xin.
...
(cherry picked from commit c9bfca217f
2018-11-30 13:30:32 +09:00
1a448565a8
HADOOP-15959. Revert "HADOOP-12751. While using kerberos Hadoop incorrectly assumes names with '@' to be non-simple"
...
This reverts commit 829a2e4d27d0edd37269
2018-11-29 17:53:23 +00:00
f089796251
HADOOP-15922. Fixed doAsUser decoding for DelegationTokenAuthenticationFilter.
...
Contributed by He Xiaoqiao
(cherry picked from commit a9d96948ed
2018-11-26 13:52:42 -05:00
1b937d701d
HADOOP-15948. Inconsistency in get and put syntax if filename/dirname contains space. Contributed by Ayush Saxena.
...
(cherry picked from commit 23b441c225
2018-11-26 18:13:09 +05:30
f6227367fb
Add 2.9.2 release notes and changes documents.
...
(cherry picked from commit 1a00b4e325
2018-11-20 14:25:48 +09:00
86deff0a06
HADOOP-15869. BlockDecompressorStream#decompress should not return -1 in case of IOException. Contributed by Surendra Singh Lilhore
...
(cherry picked from commit 75291e6d53
2018-11-13 21:26:22 +05:30
d4afeac285
HADOOP-15110. Gauges are getting logged in exceptions from AutoRenewalThreadForUserCreds.
...
Contributed by LiXin Ge.
(cherry picked from commit 3c9d97b8f7
2018-11-12 12:58:53 +00:00
9265934201
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
...
(cherry picked from commit ab6aa4c726
2018-11-06 08:47:30 -06:00
a15bcf97a4
HADOOP-15687. Credentials class should allow access to aliases.
...
Author: Lars Francke <lars.francke@gmail.com>
(cherry picked from commit cb8d679c95
2018-11-03 16:29:56 +00:00
d79ea3844c
HADOOP-15855. Review hadoop credential doc, including object store details.
...
Contributed by Steve Loughran.
(cherry picked from commit 62d98ca92a
2018-10-30 15:58:44 +00:00
5f8dd40647
Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao."
...
This reverts commit fd02c501c0
2018-10-28 17:05:45 -07:00
fd02c501c0
HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao.
...
(cherry picked from commit fb2b72e6fc
2018-10-25 09:36:28 -07:00
63396beab4
HADOOP-15822. zstd compressor can fail with a small output buffer. Contributed by Jason Lowe.
...
(cherry picked from commit 8f97d6f2cd
2018-10-24 13:50:30 +09:00
85bf008405
HADOOP-15866. Renamed HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT keys break compatibility.
...
Contributed by Wei-Chiu Chuang.
(cherry picked from commit cb8d11e7bf5055508949a6a4f074efacc8cb1305)
2018-10-22 11:08:30 +01:00
bde4fd5ed9
Preparing for 3.2.0 release
2018-10-18 17:07:45 +05:30
30fc5966a2
HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe
...
(cherry picked from commit 9abda83947
2018-10-17 14:40:57 -05:00
88ea877ac0
HADOOP-15861. Move DelegationTokenIssuer to the right path.
...
Contributed by Wei-Chiu Chuang.
(cherry picked from commit 41b3603b5b
2018-10-17 11:02:47 +01:00
7b06383163
HADOOP-15857. Addendum. Remove ozonefs class name definition from core-default.xml. Contributed by Elek, Marton.
2018-10-16 15:51:59 +02:00
c5bf07154f
HADOOP-15857. Remove ozonefs class name definition from core-default.xml. Contributed by Elek, Marton.
2018-10-16 18:56:49 +05:30
ced2596094
HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
(cherry picked from commit b6fc72a025
2018-10-15 11:00:21 -07:00
9cb0654fbb
HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances.
...
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
(cherry picked from commit 5ec86b445c
2018-10-12 12:08:08 -07:00
1ee09906ce
HADOOP-15839. Review + update cloud store sensitive keys in hadoop.security.sensitive-config-keys.
...
Contributed by Steve Loughran.
(cherry picked from commit cdc4350718
2018-10-10 17:02:00 +01:00
5f97c0cd76
HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe
...
(cherry picked from commit f13e231025
2018-10-10 07:00:19 -07:00
614d6cf99c
YARN-8845. Removed unused hadoop.registry.rm reference.
...
Contributed by Íñigo Goiri
(cherry picked from commit bf04f19456
2018-10-09 15:56:04 -04:00
71faacd175
HADOOP-15818. Fix deprecated maven-surefire-plugin configuration in hadoop-kms module
...
This closes #425
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a30b1d1824
2018-10-09 12:57:06 +09:00
2c392da8aa
HADOOP-15791. Remove Ozone related sources from the 3.2 branch. Contributed by Elek, Marton.
2018-10-05 16:10:47 +05:30
e185ae2d17
HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles.
...
(cherry picked from commit 81f635f47f
2018-10-04 11:04:39 +08:00
6fab6886f6
HDFS-13936. Multipart upload to HDFS to support 0 byte upload. Contributed by Ewan Higgs.
2018-10-02 14:05:35 +02:00
5c8d90763c
HADOOP-15764. Addendum patch: Fix NPE in SecurityUtil.
2018-09-28 10:03:22 +09:00
c62b440421
HADOOP-15693. Typo in Secure Mode documentation. Contributed by Thomas Levine.
2018-09-26 10:30:43 +09:00
93b0f540ed
HADOOP-15783. [JDK10] TestSFTPFileSystem.testGetModifyTime fails.
2018-09-25 16:03:28 +09:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
0def61482b
Merge branch 'HADOOP-15407' into trunk
2018-09-22 21:19:12 -06:00
429a07e08c
HADOOP-15764. [JDK10] Migrate from sun.net.dns.ResolverConfiguration to the replacement. Contributed by Akira Ajisaka.
2018-09-20 15:13:55 +02:00
3da94a36e2
HADOOP-15756. [JDK10] Migrate from sun.net.util.IPAddressUtil to the replacement. Contributed by Akira Ajisaka.
2018-09-20 14:53:21 +02:00
646874c326
HADOOP-15748. S3 listing inconsistency can raise NPE in globber.
...
Contributed by Steve Loughran.
2018-09-20 13:04:52 +01:00
7ad27e97f0
HADOOP-15736. Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Contributed by Ayush Saxena.
2018-09-20 09:31:35 +05:30
a30b4f9e71
HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen.
2018-09-19 13:22:37 -07:00
e71f61ecb8
HADOOP-15755. StringUtils#createStartupShutdownMessage throws NPE when args is null. Contributed by Lokesh Jain and Dinesh Chitlangia
2018-09-18 15:57:33 -05:00
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
...
Contributed by Steve Loughran.
2018-09-18 10:56:56 +01:00
f796cfde76
HADOOP-15304. [JDK10] Migrate from com.sun.tools.doclets to the replacement. Contributed by Akira Ajisaka.
2018-09-18 18:48:25 +09:00
ee051ef9fe
HADOOP-15742. Log if ipc backoff is enabled in CallQueueManager. Contributed by Ryan Wu.
2018-09-18 11:10:33 +08:00
9475fd902a
HADOOP-15745. ABFS: Add ABFS configuration to ConfigRedactor.
2018-09-17 19:54:01 +00:00
81dc4a995c
HADOOP-15663. ABFS: Simplify configuration.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
ce03a93f78
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
...
Contributed by Steve Loughran and Da Zhou.
2018-09-17 19:54:01 +00:00
f044deedbb
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
...
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2018-09-17 19:54:01 +00:00
ef5c776a42
HADOOP-15733. Correct the log when Invalid emptier Interval configured. Contributed by Ayush Saxena
2018-09-14 07:32:27 +05:30
291dcf2247
HDDS-423. Introduce an ozone specific log4j.properties. Contributed by Elek, Marton.
2018-09-13 16:37:44 -07:00
5d084d7eca
HADOOP-12760. sun.misc.Cleaner has moved to a new location in OpenJDK 9. Contributed by Akira Ajisaka.
2018-09-13 20:42:09 +09:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
c44088ac19
YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan.
2018-09-12 16:01:01 +05:30
2475fb0a1e
YARN-6856. [YARN-3409] Support CLI for Node Attributes Mapping. Contributed by Naganarasimha G R.
2018-09-12 16:00:59 +05:30
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
4b8c2b1c1c
HADOOP-14671. Upgrade Apache Yetus to 0.8.0
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-09-11 08:55:34 -07:00
527288ef89
HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng.
2018-09-06 16:47:54 -07:00
962089ab59
HADOOP-15657 Registering MutableQuantiles via Metric annotation. Contributed by Sushil Ks
2018-09-05 20:53:21 -07:00
9af96d4ed4
HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak.
2018-09-05 10:50:25 -07:00
e780556ae9
HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang.
2018-09-05 09:52:35 -07:00
9e96ac666d
HADOOP-10219. ipc.Client.setupIOstreams() needs to check for ClientCache.stopClient requested shutdowns.
...
Contributed by Kihwal Lee and Lukas Majercak.
2018-09-04 16:46:12 +01:00
f2c2a68ec2
HADOOP-15706. Typo in compatibility doc: SHOUD -> SHOULD
...
(Contributed by Laszlo Kollar via Daniel Templeton)
Change-Id: I6e2459d0700df7f3bad4eac8297a11690191c3ba
2018-08-30 09:12:36 -07:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
781437c219
HADOOP-15698. KMS log4j is not initialized properly at startup. Contributed by Kitti Nanasi.
2018-08-29 22:07:49 -07:00
d53a10b0a5
HADOOP-15705. Typo in the definition of "stable" in the interface classification
...
Change-Id: I3eae2143400a534903db4f186400561fc8d2bd56
2018-08-29 13:59:32 -07:00
f9c6fd9471
HADOOP-15633. fs.TrashPolicyDefault: Can't create trash directory. Contributed by Fei Hui.
2018-08-27 09:22:59 -07:00
a4121c71c2
HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko.
2018-08-24 18:27:30 -07:00
Akira Ajisaka
David Mollitor
Steve Loughran
Erik Krogen
Ben Roling
Praveen Krishna
Stephen O'Donnell
Daryn Sharp
Vinayakumar B
Wangda Tan
Eric Yang
Chen Liang
luhuachao
Inigo Goiri
Jason Lowe
Sunil G
Pranay Singh
Surendra Singh Lilhore
Takanobu Asanuma
Giovanni Matteo Fumarola
Lars Francke
Wei-Chiu Chuang
Márton Elek
Xiao Chen
Vidura Mudalige
Yiqun Lin
Ewan Higgs
Sean Mackrory
Ewan Higgs
Thomas Marquardt
Brahma Reddy Battula
Hanisha Koneru
bibinchundatt
Naganarasimha
Mingliang Liu
Allen Wittenauer
Vrushali C
Daniel Templeton
Kitti Nanasi
John Zhuge
Konstantin V Shvachko