Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,528 Commits 326 Branches 371 Tags
Commit Graph

960 Commits

Author SHA1 Message Date
Akira Ajisaka
2c84ea96e3
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:13 +09:00
Akira Ajisaka
39c6849d34
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor. 
(cherry picked from commit f777cd398f1d48898ddc4a9a5ab4e7e310e3027a)
 2019-12-20 13:12:46 +09:00
Mingliang Liu
a2336d35f6
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:54:56 -08:00
Mingliang Liu
941a98eb28
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:52:25 -08:00
Szilard Nemeth
0a4e6b7b62 YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal 2019-11-19 21:22:01 +01:00
Duo Zhang
6097e909ec HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-10-17 16:52:09 -07:00
Ranith Sardar
a400f396a6 HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 
Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70efebb828389f609a89b123c4ee51f867)
(cherry picked from commit c5eca3f7ee095d6a261eb411ad97aba654d67d13)

 Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-10-01 16:34:00 -07:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Akira Ajisaka
42c6119e47
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 
(cherry picked from commit 84b1982060422760702eca6f1ef515c6ad3e85a5)
 2019-08-23 14:40:34 +09:00
bibinchundatt
3547261749 YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 
(cherry picked from commit 10ec31d20ee1b6a0b1da915acb6b6ec33f2cd415)
 2019-08-21 13:59:57 +05:30
KAI XIE
c1a2b29c0f HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)
(cherry picked from commit b3c14d4132ed6aa871bb88c4f84f3e3d90da6f93)

Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-08-18 18:55:59 -07:00
Ayush Saxena
befafe9408 HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun. 2019-07-20 13:43:38 +05:30
Takanobu Asanuma
2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:18:00 -07:00
Takanobu Asanuma
8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Weiwei Yang
82430eec5a HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:07:40 -07:00
Rajat Khandelwal
8e0fdaeb38
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die. 
Contributed by Rajat Khandelwal

(cherry picked from commit 591ca698230f25217c10c7549aff8097baa11f1e)
 2019-05-09 21:15:09 +01:00
Akira Ajisaka
b45813b259
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)
 2019-04-15 10:58:41 +09:00
Masatake Iwasaki
fd90af892c HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)
 2019-04-12 12:03:11 +09:00
Steve Loughran
3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka
875435dc7d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:40:23 +09:00
Weiwei Yang
f34d6b937f HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:11:17 +08:00
Steve Loughran
1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Steve Loughran
a1d383fc14
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:58:34 +00:00
bibinchundatt
616e138adb YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 
(cherry picked from commit 779dae4de7e518938d58badcef065ea457be911c)
 2019-02-20 00:38:27 +05:30
bibinchundatt
cc0ef524d9 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 2019-02-14 23:00:56 +05:30
Eric E Payne
49d54633e0 HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.
 2019-02-07 23:15:18 +00:00
Eric E Payne
352ebc6ed9 Revert "HADOOP-15281. Distcp to add no-rename copy option." 
Revert "HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson."
This reverts commit d2765ffc2e3f6ce144bb0ca6066801d79cd7217d.
 2019-02-07 20:18:32 +00:00
Andrew Olson
d2765ffc2e
HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.

(cherry picked from commit de804e53b9d20a2df75a4c7252bf83ed52011488)
 2019-02-07 10:09:55 +00:00
Kai Xie
fee33057f0
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7eaec71433548dc47f5a9a995cbea15a)
 2019-01-08 13:35:17 +00:00
Wangda Tan
31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Weiwei Yang
5819f91be0 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c46704dcdb63089dd971e2a5ba1deaac)
 2019-01-07 16:15:58 +08:00
Sunil G
87ad2615b0
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae46a3c04fc56897986a4ab4e5aa98503.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9.

(cherry picked from commit 1f425271a73fff1fdbe3fbfdb71e906fd1ac0574)
 2019-01-05 17:50:11 +09:00
Weiwei Yang
3f9fdd2344 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 
(cherry picked from commit 040a202b202a37f3b922cd321eb0a8ded457d88b)
 2019-01-03 21:55:25 +08:00
Steve Loughran
a7d3f22b4f
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
 2018-11-29 17:56:29 +00:00
Weiwei Yang
f94d40341f HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 
(cherry picked from commit 5ff0cf86a940fd83f1425794921cc075b19f1108)
 2018-11-23 14:27:15 +08:00
Weiwei Yang
fdb95ef150 HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 
(cherry picked from commit be0708c6eb3c759e43639e0f8df7c9910308fb97)
 2018-11-19 14:32:58 +08:00
Arpit Agarwal
3e57adee0a HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f14ddff581b63f0902b7760e4d2bd56)
 2018-11-15 13:58:38 -08:00
Sammi Chen
5d532cfc6f HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce84dcf68bcd7de5a5ed1c7d904796e9)
(cherry picked from commit 64cb97fb4467513f73fde18f96f391ad34e3bb0a)
 2018-11-14 13:12:22 +08:00
Steve Loughran
cd641f3218
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran.

Missed final change to ITestAssumeRole
 2018-11-01 12:55:19 +00:00
Steve Loughran
3f35946c6e
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran

This is a subset of c07715e37895a2e406 to track changes in the AWS STS error messages.
If the later AWS SDK is ever backported to older branches, the rest of that patch will need
cherry-picking
 2018-11-01 12:52:48 +00:00
Sammi Chen
366541d834 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538e838f40581519080d7c8621c65e53b)
 2018-10-26 15:28:20 +08:00
Ted Yu
84726679f0 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
(cherry picked from commit a7dd244a4995cbf20feb054530fc5913ccf384ac)
 2018-10-19 13:22:31 -07:00
Steve Loughran
f1fbc56371
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-12 09:48:17 +01:00
Todd Lipcon
89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3cd9ffdc71946fd12f5c3d59718c4939)
 2018-10-09 13:50:00 -05:00
Sammi Chen
5da3e83597 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9)
 2018-09-25 19:50:39 +08:00
Steve Loughran
1c2c0ed209
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.

(cherry picked from commit 1eef0acfe5725d08d3a4eb76c5138076ff1857f1)
 2018-09-10 14:25:42 +01:00
Steve Loughran
a0766bf66a
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.

(cherry picked from commit 5a0babf76550f63dad4c17173c4da2bf335c6532)
 2018-08-30 15:23:08 +01:00
Steve Loughran
6f939d4294
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.

(cherry picked from commit e8d138ca7c1b695688515d816ac693437c87df62)
 2018-08-30 14:37:07 +01:00
Wangda Tan
5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00