Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,783 Commits 326 Branches 371 Tags
Commit Graph

974 Commits

Author SHA1 Message Date
Ankit Kumar
b89aff6db4
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320a9d11d2fd8c86fb5fa4e2d1c45d9ef)
 2020-12-09 10:54:39 +09:00
Eric E Payne
052b9799c0 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 31154fdde5d81d0bf4593087507b926b73543a68)
 2020-11-02 17:59:58 +00:00
Steve Loughran
5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
Ayush Saxena
4af5fc86b2 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:24:41 +05:30
Ayush Saxena
4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Steve Loughran
31755c91ff Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b1f9dde858ff34df8111e5340115511.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:49:25 +01:00
Steve Loughran
eebd96636e Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf495f5cf1bb15b1fe6f6e9587d3de0a59.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:49:13 +01:00
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Sean Mackrory
dde8417972 HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota. 2020-03-23 13:53:18 -07:00
Sean Mackrory
770e2aaf83 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 
(cherry picked from commit b089a06793d94d42b7da1b7566e366ceb748e081)
 2020-03-23 13:52:08 -07:00
Steve Loughran
caec6a1945 HADOOP-16775. DistCp reuses the same temp file within the task for different files. 
Contributed by Amir Shenavandeh.

This avoids overwrite consistency issues with S3 and other stores

Change-Id: Ic4d05ef3397e963ba28fd9f775bb362b0da36ad9
 2020-03-13 19:34:50 +00:00
Weiwei Yang
79e5633867 HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71eb3721e9be3fc42349a81c4b013ada1)
 2020-03-08 22:01:59 -07:00
Wei-Chiu Chuang
58b025c8f4 HADOOP-16882. Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10. Contributed by Lisheng Sun. 2020-02-28 16:24:38 -08:00
Mukund Thakur
ce6614565e HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running 
This uses the length of the file known at the start of the copy to determine the amount of data to copy.

* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
  distcp will now fail. Until now these failures were not detected.

Contributed by Mukund Thakur.

Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
(cherry picked from commit 51c64b357d4bd1a0038e61df3d4b8ea0a3ad7449)
 2020-02-27 16:37:36 -08:00
Akira Ajisaka
2c84ea96e3
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:13 +09:00
Akira Ajisaka
39c6849d34
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor. 
(cherry picked from commit f777cd398f1d48898ddc4a9a5ab4e7e310e3027a)
 2019-12-20 13:12:46 +09:00
Mingliang Liu
a2336d35f6
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:54:56 -08:00
Mingliang Liu
941a98eb28
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:52:25 -08:00
Szilard Nemeth
0a4e6b7b62 YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal 2019-11-19 21:22:01 +01:00
Duo Zhang
6097e909ec HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-10-17 16:52:09 -07:00
Ranith Sardar
a400f396a6 HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 
Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70efebb828389f609a89b123c4ee51f867)
(cherry picked from commit c5eca3f7ee095d6a261eb411ad97aba654d67d13)

 Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-10-01 16:34:00 -07:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Akira Ajisaka
42c6119e47
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 
(cherry picked from commit 84b1982060422760702eca6f1ef515c6ad3e85a5)
 2019-08-23 14:40:34 +09:00
bibinchundatt
3547261749 YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 
(cherry picked from commit 10ec31d20ee1b6a0b1da915acb6b6ec33f2cd415)
 2019-08-21 13:59:57 +05:30
KAI XIE
c1a2b29c0f HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)
(cherry picked from commit b3c14d4132ed6aa871bb88c4f84f3e3d90da6f93)

Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-08-18 18:55:59 -07:00
Ayush Saxena
befafe9408 HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun. 2019-07-20 13:43:38 +05:30
Takanobu Asanuma
2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:18:00 -07:00
Takanobu Asanuma
8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Weiwei Yang
82430eec5a HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:07:40 -07:00
Rajat Khandelwal
8e0fdaeb38
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die. 
Contributed by Rajat Khandelwal

(cherry picked from commit 591ca698230f25217c10c7549aff8097baa11f1e)
 2019-05-09 21:15:09 +01:00
Akira Ajisaka
b45813b259
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)
 2019-04-15 10:58:41 +09:00
Masatake Iwasaki
fd90af892c HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)
 2019-04-12 12:03:11 +09:00
Steve Loughran
3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Akira Ajisaka
875435dc7d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:40:23 +09:00
Weiwei Yang
f34d6b937f HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:11:17 +08:00
Steve Loughran
1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Steve Loughran
a1d383fc14
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:58:34 +00:00
bibinchundatt
616e138adb YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 
(cherry picked from commit 779dae4de7e518938d58badcef065ea457be911c)
 2019-02-20 00:38:27 +05:30
bibinchundatt
cc0ef524d9 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 2019-02-14 23:00:56 +05:30
Eric E Payne
49d54633e0 HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.
 2019-02-07 23:15:18 +00:00
Eric E Payne
352ebc6ed9 Revert "HADOOP-15281. Distcp to add no-rename copy option." 
Revert "HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson."
This reverts commit d2765ffc2e3f6ce144bb0ca6066801d79cd7217d.
 2019-02-07 20:18:32 +00:00
Andrew Olson
d2765ffc2e
HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.

(cherry picked from commit de804e53b9d20a2df75a4c7252bf83ed52011488)
 2019-02-07 10:09:55 +00:00
Kai Xie
fee33057f0
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7eaec71433548dc47f5a9a995cbea15a)
 2019-01-08 13:35:17 +00:00
Wangda Tan
31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Weiwei Yang
5819f91be0 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c46704dcdb63089dd971e2a5ba1deaac)
 2019-01-07 16:15:58 +08:00
Sunil G
87ad2615b0
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae46a3c04fc56897986a4ab4e5aa98503.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9.

(cherry picked from commit 1f425271a73fff1fdbe3fbfdb71e906fd1ac0574)
 2019-01-05 17:50:11 +09:00
Weiwei Yang
3f9fdd2344 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 
(cherry picked from commit 040a202b202a37f3b922cd321eb0a8ded457d88b)
 2019-01-03 21:55:25 +08:00
Steve Loughran
a7d3f22b4f
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
 2018-11-29 17:56:29 +00:00
Weiwei Yang
f94d40341f HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 
(cherry picked from commit 5ff0cf86a940fd83f1425794921cc075b19f1108)
 2018-11-23 14:27:15 +08:00