7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1de2904f1230b278502e2
2021-03-27 00:40:24 +09:00
162923e799
HADOOP-17594. DistCp: Expose the JobId for applications executing through run method ( #2786 ). Contributed by Ayush Saxena.
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-19 14:21:02 +05:30
4d323cd47c
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 21:39:04 +00:00
7bcb1a83a3
HADOOP-16764. Rewrite Python example codes using Python3 ( #1762 )
...
(cherry picked from commit fd7de2b82a
2021-02-20 19:16:07 +09:00
b89aff6db4
YARN-10491. Fix deprecation warnings in SLSWebApp.java ( #2519 )
...
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320
2020-12-09 10:54:39 +09:00
052b9799c0
YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan).
...
(cherry picked from commit 31154fdde5
2020-11-02 17:59:58 +00:00
5f0004e012
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210
...
This backports the listing-side changes of HADOOP-13230.
With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.
It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"
The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.
Read the documentation!
Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
2020-08-28 17:55:18 +01:00
4af5fc86b2
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:24:41 +05:30
4592af898b
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-22 19:05:13 +05:30
31755c91ff
Revert "HADOOP-8143. Change distcp to have -pb on by default."
...
This reverts commit dd65eea74b
2020-05-15 13:49:25 +01:00
eebd96636e
Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)."
...
This reverts commit 44350fdf49
2020-05-15 13:49:13 +01:00
ec6d2a8402
Preparing for 3.1.5 development
...
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
2020-04-29 11:18:18 +02:00
dde8417972
HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota.
2020-03-23 13:53:18 -07:00
770e2aaf83
HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota.
...
(cherry picked from commit b089a06793
2020-03-23 13:52:08 -07:00
caec6a1945
HADOOP-16775. DistCp reuses the same temp file within the task for different files.
...
Contributed by Amir Shenavandeh.
This avoids overwrite consistency issues with S3 and other stores
Change-Id: Ic4d05ef3397e963ba28fd9f775bb362b0da36ad9
2020-03-13 19:34:50 +00:00
79e5633867
HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu.
...
(cherry picked from commit 6dfe00c71e
2020-03-08 22:01:59 -07:00
58b025c8f4
HADOOP-16882. Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10. Contributed by Lisheng Sun.
2020-02-28 16:24:38 -08:00
ce6614565e
HDFS-13660. DistCp job fails when new data is appended in the file while the DistCp copy job is running
...
This uses the length of the file known at the start of the copy to determine the amount of data to copy.
* If a file is appended to during the copy, the original bytes are copied.
* If a file is truncated during a copy, or the attempt to read the data fails with a truncated stream,
distcp will now fail. Until now these failures were not detected.
Contributed by Mukund Thakur.
Change-Id: I576a49d951fa48d37a45a7e4c82c47488aa8e884
(cherry picked from commit 51c64b357d
2020-02-27 16:37:36 -08:00
2c84ea96e3
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li.
...
(cherry picked from commit f6d20daf40
2020-01-21 18:05:13 +09:00
39c6849d34
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor.
...
(cherry picked from commit f777cd398f
2019-12-20 13:12:46 +09:00
a2336d35f6
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml ( #1753 )
...
Contributed by Mingliang Liu
2019-12-11 11:54:56 -08:00
941a98eb28
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu
...
This closes #1733
2019-12-05 17:52:25 -08:00
0a4e6b7b62
YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal
2019-11-19 21:22:01 +01:00
6097e909ec
HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1
...
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-10-17 16:52:09 -07:00
a400f396a6
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on.
...
Contributed by Ranith Sardar.
(cherry picked from commit 546c5d70efc5eca3f7ee
2019-10-01 16:34:00 -07:00
ef79d98788
Preparing for 3.1.4 development
2019-09-04 16:11:36 +08:00
42c6119e47
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya.
...
(cherry picked from commit 84b1982060
2019-08-23 14:40:34 +09:00
3547261749
YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi.
...
(cherry picked from commit 10ec31d20e
2019-08-21 13:59:57 +05:30
c1a2b29c0f
HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel ( #919 )
...
* DistCp to support checksum validation when copy blocks in parallel
* address review comments
* add checksums comparison test for combine mode
(cherry picked from commit c765584eb2b3c14d4132
2019-08-18 18:55:59 -07:00
befafe9408
HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.
2019-07-20 13:43:38 +05:30
2f6f42bc7b
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643
2019-06-20 20:18:00 -07:00
8098ddaf40
HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:39:49 +09:00
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:28:21 +09:00
82430eec5a
HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu.
...
(cherry picked from commit 2d8282bb82
2019-05-14 14:07:40 -07:00
8e0fdaeb38
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
(cherry picked from commit 591ca69823
2019-05-09 21:15:09 +01:00
b45813b259
HADOOP-16227. Upgrade checkstyle to 8.19
...
(cherry picked from commit 4b4fef2f0e
2019-04-15 10:58:41 +09:00
fd90af892c
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
...
(cherry picked from commit bbdbc7a9a1
2019-04-12 12:03:11 +09:00
3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:42:08 +01:00
875435dc7d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
...
(cherry picked from commit 8b6deebb1d
2019-04-03 19:40:23 +09:00
f34d6b937f
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
...
(cherry picked from commit 568d3ab8b6
2019-03-19 10:11:17 +08:00
1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 16:53:06 +00:00
a1d383fc14
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
2019-02-21 13:58:34 +00:00
616e138adb
YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T.
...
(cherry picked from commit 779dae4de7
2019-02-20 00:38:27 +05:30
cc0ef524d9
YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt.
2019-02-14 23:00:56 +05:30
49d54633e0
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 23:15:18 +00:00
352ebc6ed9
Revert "HADOOP-15281. Distcp to add no-rename copy option."
...
Revert "HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson."
This reverts commit d2765ffc2e
2019-02-07 20:18:32 +00:00
d2765ffc2e
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
(cherry picked from commit de804e53b9
2019-02-07 10:09:55 +00:00
fee33057f0
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
...
Contributed by Kai Xie.
(cherry picked from commit 188bebbe7e
2019-01-08 13:35:17 +00:00
31ea2f7806
Preparing for 3.1.3 development
...
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
2019-01-07 10:04:58 -08:00
5819f91be0
HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu.
...
(cherry picked from commit f87b3b11c4
2019-01-07 16:15:58 +08:00
Akira Ajisaka
Ayush Saxena
Jim Brennan
Kengo Seki
Ankit Kumar
Eric E Payne
Steve Loughran
Gabor Bota
Sean Mackrory
Weiwei Yang
Wei-Chiu Chuang
Mukund Thakur
Mingliang Liu
Szilard Nemeth
Duo Zhang
Ranith Sardar
Zhankun Tang
bibinchundatt
KAI XIE
Takanobu Asanuma
Rajat Khandelwal
Masatake Iwasaki
Steve Loughran
Eric E Payne
Andrew Olson
Wangda Tan