Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.
This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.
Contributed By: Mukund Thakur
This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint
is set in fs.s3a.endpoint, S3A filesystem creation may fail on
non-EC2 deployments, depending on the local host environment setup.
* If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region
is null, the region is set to "us-east-1".
* If fs.s3a.endpoint.region is explicitly set to "" then the client
falls back to the SDK region resolution chain; this works on EC2
* Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+
* Also contains some minor restructuring of troubleshooting.md
Contributed by Steve Loughran.
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.
The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.
If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.
Contributed by Steve Loughran.
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.
This is needed when using AWS Private Link, as
the region cannot be automatically determined.
Contributed by Mehakmeet Singh
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:
org.apache.hadoop.fs.statistics.IOStatisticsLogging
Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).
Contributed by: Mehakmeet Singh
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.
Adds a new option fs.s3a.downgrade.syncable.exceptions
When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter
With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
JIRA.
Contributed by Steve Loughran.
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.
This includes the min/max/mean durations of all REST API calls.
Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
This moves the mock account name --which is required to never exist-- from
"mockAccount" to an account name containing a static UUID.
Contributed by Steve Loughran.
* HADOOP-16948. Support single writer dirs.
* HADOOP-16948. Fix findbugs and checkstyle problems.
* HADOOP-16948. Fix remaining checkstyle problems.
* HADOOP-16948. Add DurationInfo, retry policy for acquiring lease, and javadocs
* HADOOP-16948. Convert ABFS client to use an executor for lease ops
* HADOOP-16948. Fix ABFS lease test for non-HNS
* HADOOP-16948. Fix checkstyle and javadoc
* HADOOP-16948. Address review comments
* HADOOP-16948. Use daemon threads for ABFS lease ops
* HADOOP-16948. Make lease duration configurable
* HADOOP-16948. Add error messages to test assertions
* HADOOP-16948. Remove extra isSingleWriterKey call
* HADOOP-16948. Use only infinite lease duration due to cost of renewal ops
* HADOOP-16948. Remove acquire/renew/release lease methods
* HADOOP-16948. Rename single writer dirs to infinite lease dirs
* HADOOP-16948. Fix checkstyle
* HADOOP-16948. Wait for acquire lease future
* HADOOP-16948. Add unit test for acquire lease failure
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.
This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.
Explicitly setting the endpoint will save the cost of these probe
HTTP requests.
Contributed by Steve Loughran.
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.
When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.
Contributed by Steve Loughran.
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Mehakmeet Singh
Anoop Sam John
litao
Mukund Thakur
sumangala-patki
Masatake Iwasaki
Zamil Majdy
Steve Loughran
Takanobu Asanuma
snehavarma
Petre Bogdan Stolojan
Viraj Jasani
Akira Ajisaka
zhengchenyu
bilaharith
Ayush Saxena
billierinaldi
Jack Jiang
Chao Sun
Pierrick Hymbert
Chao Sun