hadoop/hadoop-tools/hadoop-azure
Steve Loughran 7f9ca101e2
HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead (#5103)
* HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead

Adds new config option to turn off readahead
* also allows it to be passed in through openFile(),
* extends ITestAbfsReadWriteAndSeek to use the option, including one
  replicated test...that shows that turning it off is slower.

Important: this does not address the critical data corruption issue
HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests

What is does do is provide a way to completely bypass the ReadBufferManager.
To mitigate the problem, either fs.azure.enable.readahead needs to be set to false,
or set "fs.azure.readaheadqueue.depth" to 0 -this still goes near the (broken)
ReadBufferManager code, but does't trigger the bug.

For safe reading of files through the ABFS connector, readahead MUST be disabled
or the followup fix to HADOOP-18521 applied

Contributed by Steve Loughran
2022-11-08 11:43:04 +00:00
..
dev-support HADOOP-17767. ABFS: Update test scripts (#3124) 2022-10-20 18:07:04 +01:00
src HADOOP-18517. ABFS: Add fs.azure.enable.readahead option to disable readahead (#5103) 2022-11-08 11:43:04 +00:00
.gitignore HADOOP-17767. ABFS: Update test scripts (#3124) 2022-10-20 18:07:04 +01:00
pom.xml HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3699) 2022-09-06 11:00:52 +01:00