hadoop/hadoop-tools
Steve Loughran 9db61adeda
HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3)
S3A input stream support for the few fs.option.openfile settings.
As well as supporting the read policy option and values,
if the file length is declared in fs.option.openfile.length
then no HEAD request will be issued when opening a file.
This can cut a few tens of milliseconds off the operation.

The patch adds a new openfile parameter/FS configuration option
fs.s3a.input.async.drain.threshold (default: 16000).
It declares the number of bytes remaining in the http input stream
above which any operation to read and discard the rest of the stream,
"draining", is executed asynchronously.
This asynchronous draining offers some performance benefit on seek-heavy
file IO.

Contributed by Steve Loughran.

Change-Id: I9b0626bbe635e9fd97ac0f463f5e7167e0111e39
2022-04-27 19:23:56 +01:00
..
hadoop-aliyun HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-archive-logs HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 2022-04-27 10:30:07 +01:00
hadoop-archives HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-aws HADOOP-16202. Enhanced openFile(): hadoop-aws changes. (#2584/3) 2022-04-27 19:23:56 +01:00
hadoop-azure HADOOP-17682. ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters (#2975) 2022-04-27 19:22:49 +01:00
hadoop-azure-datalake HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-datajoin HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-distcp HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 2022-04-27 19:23:25 +01:00
hadoop-dynamometer HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-extras HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-fs2img HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-gridmix HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-kafka HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-openstack HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-pipes HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-resourceestimator HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-rumen HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-sls HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
hadoop-streaming HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 2022-04-27 19:23:25 +01:00
hadoop-tools-dist HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00
pom.xml HADOOP-18198. Preparing for 3.3.4 development 2022-04-12 14:09:08 +01:00