Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,016 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

943 Commits

Author SHA1 Message Date
Akira Ajisaka b45813b259
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e)
 2019-04-15 10:58:41 +09:00
Masatake Iwasaki fd90af892c HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a1)
 2019-04-12 12:03:11 +09:00
Steve Loughran 3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka 875435dc7d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1d)
 2019-04-03 19:40:23 +09:00
Weiwei Yang f34d6b937f HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b6)
 2019-03-19 10:11:17 +08:00
Steve Loughran 1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Steve Loughran a1d383fc14
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:58:34 +00:00
bibinchundatt 616e138adb YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 
(cherry picked from commit 779dae4de7)
 2019-02-20 00:38:27 +05:30
bibinchundatt cc0ef524d9 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 2019-02-14 23:00:56 +05:30
Eric E Payne 49d54633e0 HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.
 2019-02-07 23:15:18 +00:00
Eric E Payne 352ebc6ed9 Revert "HADOOP-15281. Distcp to add no-rename copy option." 
Revert "HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson."
This reverts commit d2765ffc2e.
 2019-02-07 20:18:32 +00:00
Andrew Olson d2765ffc2e
HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.

(cherry picked from commit de804e53b9)
 2019-02-07 10:09:55 +00:00
Kai Xie fee33057f0
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7e)
 2019-01-08 13:35:17 +00:00
Wangda Tan 31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Weiwei Yang 5819f91be0 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c4)
 2019-01-07 16:15:58 +08:00
Sunil G 87ad2615b0
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae4.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb.

(cherry picked from commit 1f425271a7)
 2019-01-05 17:50:11 +09:00
Weiwei Yang 3f9fdd2344 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 
(cherry picked from commit 040a202b20)
 2019-01-03 21:55:25 +08:00
Steve Loughran a7d3f22b4f
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0c)
 2018-11-29 17:56:29 +00:00
Weiwei Yang f94d40341f HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 
(cherry picked from commit 5ff0cf86a9)
 2018-11-23 14:27:15 +08:00
Weiwei Yang fdb95ef150 HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 
(cherry picked from commit be0708c6eb)
 2018-11-19 14:32:58 +08:00
Arpit Agarwal 3e57adee0a HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f)
 2018-11-15 13:58:38 -08:00
Sammi Chen 5d532cfc6f HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce)
(cherry picked from commit 64cb97fb44)
 2018-11-14 13:12:22 +08:00
Steve Loughran cd641f3218
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran.

Missed final change to ITestAssumeRole
 2018-11-01 12:55:19 +00:00
Steve Loughran 3f35946c6e
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran

This is a subset of c07715e378 to track changes in the AWS STS error messages.
If the later AWS SDK is ever backported to older branches, the rest of that patch will need
cherry-picking
 2018-11-01 12:52:48 +00:00
Sammi Chen 366541d834 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538)
 2018-10-26 15:28:20 +08:00
Ted Yu 84726679f0 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e)
(cherry picked from commit a7dd244a49)
 2018-10-19 13:22:31 -07:00
Steve Loughran f1fbc56371
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-12 09:48:17 +01:00
Todd Lipcon 89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3c)
 2018-10-09 13:50:00 -05:00
Sammi Chen 5da3e83597 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb)
 2018-09-25 19:50:39 +08:00
Steve Loughran 1c2c0ed209
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.

(cherry picked from commit 1eef0acfe5)
 2018-09-10 14:25:42 +01:00
Steve Loughran a0766bf66a
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.

(cherry picked from commit 5a0babf765)
 2018-08-30 15:23:08 +01:00
Steve Loughran 6f939d4294
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.

(cherry picked from commit e8d138ca7c)
 2018-08-30 14:37:07 +01:00
Wangda Tan 5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Sammi Chen 42e34dae57 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 
(cherry picked from commit 0857f116b7)
(cherry picked from commit f9aedf32ec)
 2018-07-30 11:00:30 +08:00
bibinchundatt d2212c20c5 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 
(cherry picked from commit 3d3158cea4)
 2018-07-25 19:00:12 +05:30
Steve Loughran 749fff577e
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.
 2018-07-19 12:29:21 -07:00
Wangda Tan 44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4)
 2018-07-16 11:04:08 -07:00
Steve Loughran caf38532f3
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting 
connection in case of timeout. Contributed by Sean Mackrory.

(cherry picked from commit d503f65b66)
 2018-07-11 14:55:11 +01:00
Steve Loughran d54241e9c9
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan. 
Contributed by Steve Loughran.

(cherry picked from commit ca8b80bf59)
 2018-07-10 10:50:40 +01:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Chris Douglas 7e655b642d HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603e)
 2018-06-17 23:12:59 -07:00
Inigo Goiri 8be28376a9 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594f)
 2018-06-12 11:00:22 -07:00
Steve Loughran baac7c2b28 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be6795)
 2018-06-11 15:33:03 -07:00
Inigo Goiri 6918d9e9c8 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 8fdc993a99)
 2018-05-28 16:46:21 -07:00
Steve Loughran a89e65faa0 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c9)
 2018-05-21 11:13:18 +01:00
Steve Loughran 170f1040d4 HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest. 
Contributed by Steve Loughran.
 2018-05-17 19:08:05 +01:00
Steve Loughran a0cb1e0941 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name. 
Contributed by Sean Mackrory.

(cherry picked from commit b670837469)
 2018-05-15 16:15:13 +01:00
Steve Loughran 057a4a6bb7 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt
 2018-05-07 11:56:00 +01:00
Weiwei Yang 386c39cd84 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:44:35 +08:00
Yiqun Lin 213f733604 HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 
(cherry picked from commit 1312f9ae4c)
 2018-04-18 16:06:39 +08:00