Commit Graph

1504 Commits

Author SHA1 Message Date
Steve Loughran 85d3bba555
HADOOP-17476. ITestAssumeRole.testAssumeRoleBadInnerAuth failure. (#2777)
Contributed by Steve Loughran.
2021-03-24 16:47:55 +00:00
Steve Loughran 04880f076d
HADOOP-13551. AWS metrics wire-up (#2778)
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.

This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.

Explicitly setting the endpoint will save the cost of these probe
HTTP requests.

Contributed by Steve Loughran.
2021-03-24 13:32:54 +00:00
Ayush Saxena 03cfc85279
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732). Contributed by Ayush Saxena.
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-24 02:36:26 +05:30
Jack Jiang d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Ayush Saxena 4781761dc2
HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-19 14:19:49 +05:30
Chao Sun 9b2f812996 HADOOP-17532. Yarn Job execution get failed when LZ4 Compression Codec is used. Contributed Bhavik Patel. 2021-03-14 21:15:08 -07:00
sumangala-patki fe633d4739
HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter (#2729)
Contributed by Sumangala Patki.
2021-03-14 13:35:02 +05:30
Steve Loughran bcd9c67082
HADOOP-16721. Improve S3A rename resilience (#2742)
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
2021-03-11 12:47:39 +00:00
Akira Ajisaka 23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753)
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
Pierrick Hymbert ebfba0b6fa
[HADOOP-17567] typo in MagicCommitTracker (#2749)
Contributed by Pierrick Hymbert
2021-03-10 15:39:55 +00:00
Chao Sun 176bd88890
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. (#2522)
Contributed by Chao Sun.

(Cherry-picked via PR #2575)
2021-03-09 20:01:29 +00:00
Peter Bacsko c3aa413ee3 YARN-10679. Better logging of uncaught exceptions throughout SLS. Contributed by Szilard Nemeth. 2021-03-09 14:02:12 +01:00
Peter Bacsko 099f58f8f4 YARN-10681. Fix assertion failure message in BaseSLSRunnerTest. Contributed by Szilard Nemeth. 2021-03-09 13:22:48 +01:00
Peter Bacsko 7f522c92fa YARN-10677. Logger of SLSFairScheduler is provided with the wrong class. Contributed by Szilard Nemeth. 2021-03-09 12:53:32 +01:00
Peter Bacsko ea90cd3556 YARN-10678. Try blocks without catch blocks in SLS scheduler classes can swallow other exceptions. Contributed by Szilard Nemeth. 2021-03-09 12:03:53 +01:00
Ahmed Hussein e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
sumangala-patki 7f64030314
HADOOP-17537. ABFS: Correct assertion reversed in HADOOP-13327
Contributed Sumangala Patki.
2021-02-22 11:45:58 +00:00
Akira Ajisaka 9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)"
This reverts commit 4cf3531583.
2021-02-19 11:09:10 +09:00
Akira Ajisaka 4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
Anoop Sam John 1bb4101b59
HADOOP-17038 Support disabling buffered reads in ABFS positional reads. (#2646)
- Contributed by @anoopsjohn
2021-02-16 22:27:52 +05:30
Steve Loughran 78905d7e3f
HADOOP-16906. Abortable (#2684)
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.
2021-02-11 17:37:20 +00:00
Steve Loughran 798df6d699
HADOOP-13327 Output Stream Specification. (#2587)
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
bilaharith 5f34271bb1
HADOOP-17475. ABFS : add high performance listStatusIterator (#2548)
The ABFS connector now implements listStatusIterator() with
asynchronous prefetching of the next page(s) of results.
For listing large directories this can provide tangible speedups.

If for any reason this needs to be disabled, set
fs.azure.enable.abfslistiterator to false.

Contributed by Bilahari T H.
2021-02-04 13:36:19 +00:00
Steve Loughran 26b9d480e8
HADOOP-17337. S3A NetworkBinding has a runtime dependency on shaded httpclient. (#2599)
Contributed by Steve Loughran.
2021-02-03 14:29:56 +00:00
Steve Loughran f37bf65199
HADOOP-15710. ABFS checkException to map 403 to AccessDeniedException. (#2648)
When 403 is returned from an ABFS HTTP call, an AccessDeniedException is raised.
The exception text is unchanged, for any application string matching on the getMessage() contents.

Contributed by Steve Loughran.
2021-02-02 18:13:41 +00:00
Steve Loughran 0bb52a42e5
HADOOP-17483. Magic committer is enabled by default. (#2656)
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
2021-01-27 19:04:22 +00:00
Steve Loughran 28cc912a5c
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649)
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.
2021-01-27 16:39:29 +00:00
Steve Loughran 80c7404b51
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530)
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.
2021-01-26 19:30:51 +00:00
Ayush Saxena e40f99f6d5 HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun. 2021-01-26 14:25:27 +05:30
Steve Loughran 06a5d3437f
HADOOP-17480. Document that AWS S3 is consistent and that S3Guard is not needed (#2636)
Contributed by Steve Loughran.
2021-01-25 13:21:34 +00:00
Maksim Bober e2f8503ebd
HADOOP-17484. Typo in hadop-aws index.md (#2634)
Contributed by Maksim Bober.
2021-01-21 17:30:58 +00:00
Steve Loughran 68bc721841
HADOOP-17433. Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole. (#2600)
Contributed by Steve Loughran.
2021-01-19 17:19:27 +00:00
Szilard Nemeth 6cd540e964 YARN-7200. SLS generates a realtimetrack.json file but that file is missing the closing ']'. Contributed by Agshin Kazimli 2021-01-15 22:32:30 +01:00
Steve Loughran 724edb0354
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16830 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.

Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
2021-01-12 17:30:32 +00:00
Steve Loughran 05c9c2ed02 Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)"
This reverts commit d3014e01f3.
(fixing commit text before it is frozen)
2021-01-12 17:29:59 +00:00
Steve Loughran d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16380 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.
2021-01-12 17:25:14 +00:00
Mehakmeet Singh 0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604)
Contributed by Mehakmeet Singh.
2021-01-12 15:48:09 +00:00
bilaharith 612330661b
HADOOP-17459. ADLS Gen1: Fixes for rename contract tests #2607
Contributed by Bilaharith
2021-01-12 14:00:48 +00:00
Sneha Vijayarajan b612c310c2
HADOOP-17404. ABFS: Small write - Merge append and flush
- Contributed by Sneha Vijayarajan
2021-01-06 10:43:37 -08:00
bilaharith d21c1c6576
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2551)
Contributed by bilaharith
2021-01-06 14:32:13 +00:00
Gabor Bota 42eb9ff68e
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593)
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, ebcause the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
2021-01-05 15:43:01 +01:00
Ayush Saxena 77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
bilaharith 1448add08f
HADOOP-17347. ABFS: Read optimizations
- Contributed by Bilahari T H
2021-01-02 10:37:10 -08:00
Sneha Vijayarajan 5ca1ea89b3
HADOOP-17407. ABFS: Fix NPE on delete idempotency flow
- Contributed by Sneha Vijayarajan
2021-01-02 10:22:10 -08:00
Steve Loughran 617af28e80
HADOOP-17271. S3A connector to support IOStatistics. (#2580)
S3A connector to support the IOStatistics API of HADOOP-16830,

This is a major rework of the S3A Statistics collection to

* Embrace the IOStatistics APIs
* Move from direct references of S3AInstrumention statistics
  collectors to interface/implementation classes in new packages.
* Ubiquitous support of IOStatistics, including:
  S3AFileSystem, input and output streams, RemoteIterator instances
  provided in list calls.
* Adoption of new statistic names from hadoop-common

Regarding statistic collection, as well as all existing
statistics, the connector now records min/max/mean durations
of HTTP GET and HEAD requests, and those of LIST operations.

Contributed by Steve Loughran.
2020-12-31 21:55:39 +00:00
Sumangala a35fc3871b
HADOOP-17422: ABFS: Set default ListMaxResults to max server limit (#2535)
Contributed by Sumangala Patki

TEST RESULTS:

namespace.enabled=true
auth.type=SharedKey
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 24
Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
-------------------
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 90, Failures: 0, Errors: 0, Skipped: 0
Tests run: 462, Failures: 0, Errors: 0, Skipped: 70
Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
2020-12-21 06:40:36 +00:00
yzhangal 3d2193cd64
HADOOP-17338. Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2497)
Yongjun Zhang <yongjunzhang@pinterest.com>
2020-12-18 19:08:10 +00:00
bilaharith 4c033bafa0
HADOOP-17191. ABFS: Run the tests with various combinations of configurations and publish a consolidated results
- Contributed by Bilahari T H
2020-12-16 10:34:59 -08:00
Sneha Vijayarajan 5bf977e6b1
Hadoop-17413. Release elastic byte buffer pool at close
- Contributed by Sneha Vijayarajan
2020-12-14 20:45:37 -08:00
Ankit Kumar aaf9e3d320
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519)
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
2020-12-09 10:52:31 +09:00