hadoop/hadoop-tools
Steve Loughran 7f9ca101e2
HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead (#5103)
* HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead

Adds new config option to turn off readahead
* also allows it to be passed in through openFile(),
* extends ITestAbfsReadWriteAndSeek to use the option, including one
  replicated test...that shows that turning it off is slower.

Important: this does not address the critical data corruption issue
HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

What is does do is provide a way to completely bypass the ReadBufferManager.
To mitigate the problem, either fs.azure.enable.readahead needs to be set to false,
or set "fs.azure.readaheadqueue.depth" to 0 -this still goes near the (broken)
ReadBufferManager code, but does't trigger the bug.

For safe reading of files through the ABFS connector, readahead MUST be disabled
or the followup fix to HADOOP-18521 applied

Contributed by Steve Loughran
2022-11-08 11:43:04 +00:00
..
hadoop-aliyun HADOOP-18313: AliyunOSSBlockOutputStream should not mark the temporary file for deletion (#4502) 2022-07-06 14:23:46 +08:00
hadoop-archive-logs HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-archives HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-aws HADOOP-18482. ITestS3APrefetchingInputStream to skip if CSV test file unavailable (#4983) 2022-10-31 21:19:34 +00:00
hadoop-azure HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead (#5103) 2022-11-08 11:43:04 +00:00
hadoop-azure-datalake HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 (#4229) 2022-05-21 02:53:14 +09:00
hadoop-benchmark HADOOP-18507. VectorIO FileRange type to support a "reference" field (#5076) 2022-10-31 21:12:13 +00:00
hadoop-datajoin HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-distcp HADOOP-18442. Remove openstack support (#4855) 2022-10-06 11:49:38 +01:00
hadoop-dynamometer HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108) 2022-04-06 18:17:02 +09:00
hadoop-extras HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-federation-balance HDFS-16256. Minor fix in HDFS Fedbalance document (#4192) 2022-05-02 08:08:12 +08:00
hadoop-fs2img HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-gridmix HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 2022-03-08 17:27:04 +09:00
hadoop-kafka HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-openstack HADOOP-18442. Remove openstack support (#4855) 2022-10-06 11:49:38 +01:00
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 2022-04-28 14:18:19 +09:00
hadoop-rumen HADOOP-18469. Add secure XML parser factories to XMLUtils (#4940) 2022-10-06 19:30:51 +01:00
hadoop-sls YARN-10680. Revisit try blocks without catch blocks but having finally blocks. Contributed by Susheel Gupta 2022-10-15 21:51:08 +02:00
hadoop-streaming HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2) 2022-04-24 17:33:05 +01:00
hadoop-tools-dist HADOOP-18442. Remove openstack support (#4855) 2022-10-06 11:49:38 +01:00
pom.xml HADOOP-11867. Add a high-performance vectored read API. (#3904) 2022-06-22 17:29:32 +01:00