This feature adds methods for ranged vectored read operations
in PositionedReadable.
All stream which implement that interface support the new API.
The default implementation reads each range in the vector
sequentially.
However, specific implementations may provide higher performance
versions. This is done in two places
* Local FileSystem/Checksum FileSystem
* The S3A client.
The S3A client first coalesces adjacent and "nearby" ranges
together, then fetches each range in separate HTTP GET requests,
executed in parallel. As such it delivers significant speedups
to applications reading separate blocks of data from the same
file, columnar data format libraries in particular.
This is the merge commit of the feature branch; the work is in
HADOOP-11867. Add a high-performance vectored read API.
HADOOP-18104. S3A: Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads.
HADOOP-18107. Adding scale test for vectored reads for large file
HADOOP-18105. Implement buffer pooling with weak references.
HADOOP-18106. Handle memory fragmentation in S3A Vectored IO.
Contributed By: Owen O'Malley and Mukund Thakur
part of HADOOP-18103.
Handling memory fragmentation in S3A vectored IO implementation by
allocating smaller user range requested size buffers and directly
filling them from the remote S3 stream and skipping undesired
data in between ranges.
This patch also adds aborting active vectored reads when stream is
closed or unbuffer() is called.
Contributed By: Mukund Thakur
part of HADOOP-18103.
Required for vectored IO feature. None of current buffer pool
implementation is complete. ElasticByteBufferPool doesn't use
weak references and could lead to memory leak errors and
DirectBufferPool doesn't support caller preferences of direct
and heap buffers and has only fixed length buffer implementation.
Contributed By: Mukund Thakur
Part of HADOOP-18103.
Introducing fs.s3a.vectored.read.min.seek.size and fs.s3a.vectored.read.max.merged.size
to configure min seek and max read during a vectored IO operation in S3A connector.
These properties actually define how the ranges will be merged. To completely
disable merging set fs.s3a.max.readsize.vectored.read to 0.
Contributed By: Mukund Thakur
part of HADOOP-18103.
Add support for multiple ranged vectored read api in PositionedReadable.
The default iterates through the ranges to read each synchronously,
but the intent is that FSDataInputStream subclasses can make more
efficient readers especially in object stores implementation.
Also added implementation in S3A where smaller ranges are merged and
sliced byte buffers are returned to the readers. All the merged ranged are
fetched from S3 asynchronously.
Contributed By: Owen O'Malley and Mukund Thakur
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.
Contributed by zhengchenyu
Speed up the magic committer with key changes being
* Writes under __magic always retain directory markers
* File creation under __magic skips all overwrite checks,
including the LIST call intended to stop files being
created over dirs.
* mkdirs under __magic probes the path for existence
but does not look any further.
Extra parallelism in task and job commit directory scanning
Use of createFile and openFile with parameters which all for
HEAD checks to be skipped.
The committer can write the summary _SUCCESS file to the path
`fs.s3a.committer.summary.report.directory`, which can be in a
different file system/bucket if desired, using the job id as
the filename.
Also: HADOOP-15460. S3A FS to add `fs.s3a.create.performance`
Application code can set the createFile() option
fs.s3a.create.performance to true to disable the same
safety checks when writing under magic directories.
Use with care.
The createFile option prefix `fs.s3a.create.header.`
can be used to add custom headers to S3 objects when
created.
Contributed by Steve Loughran.
Bump cos_api-bundle to 5.6.69
All copies of httpclient, including shaded ones in libraries used
by the s3a, gs and cos cloud connectors, turn out to load their
TLD list from the same resource mozilla/public-suffix-list.txt
Updating the hadoop-cos dependency ensures that its version
of public-suffix-list.txt is up to date -and so the s3a connector
able to talk to s3 resources if the cos-api-bundle JAR is where
the resource is loaded from.
Contributed by André Fonseca
* Maven runs the ember build script.
The environment variable TMPDIR was
set as per bash syntax.
* This failed on Windows since the
Windows command prompt doesn't
support bash syntax.
* We're now detecting the OS and
setting a Maven property
"emberBuildScript" in a cross
platform compatible way.
* jnihelper.c in HDFS native client uses
dirent.h. This header file isn't available
on Windows.
* This PR provides a cross platform
compatible implementation for dirent
under the XPlatform library.
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.
Contributed by Steve Loughran
Adds a new option fs.s3a.create.storage.class which can
be used to set the storage class for files created in AWS S3.
Consult the documentation for details and instructions on how
disable the relevant tests when testing against third-party
stores.
Contributed by Monthon Klongklaew
Igor Dvorzhak
Steve Loughran
Mukund Thakur
9uapaw
Samrat
Ashutosh Gupta
Christian Bartolomäus
Viraj Jasani
Szilard Nemeth
slfan1989
KevinWikant
zhengchenyu
Benjamin Teke
xuzq
caozhiqiang
jianghuazhu
André Fonseca
章锡平
Gautham B A
Steve Vaughan
monthonk