Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593) 

Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, because the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
This commit is contained in:
Gabor Bota 2021-01-05 15:43:01 +01:00 committed by Steve Loughran
parent 33f9ceb3cc
commit 1b2bc77923
No known key found for this signature in database
GPG Key ID: D22CF846DBB162A0
3 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
View File

@ -518,7 +518,7 @@ public final class Constants {
* will be validated using {@code S3AFileSystem.verifyBucketExistsV2()}. * will be validated using {@code S3AFileSystem.verifyBucketExistsV2()}.
* Value: {@value} * Value: {@value}
*/ */
public static final int S3A_BUCKET_PROBE_DEFAULT = 2; public static final int S3A_BUCKET_PROBE_DEFAULT = 0;
/** /**
* How long a directory listing in the MS is considered as authoritative. * How long a directory listing in the MS is considered as authoritative.

4
hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
View File

@ -1040,9 +1040,9 @@ options are covered in [Testing](./testing.md).
<property> <property>
<name>fs.s3a.bucket.probe</name> <name>fs.s3a.bucket.probe</name>
<value>2</value> <value>0</value>
<description> <description>
The value can be 0, 1 or 2 (default). The value can be 0 (default), 1 or 2.
When set to 0, bucket existence checks won't be done When set to 0, bucket existence checks won't be done
during initialization thus making it faster. during initialization thus making it faster.
Though it should be noted that when the bucket is not available in S3, Though it should be noted that when the bucket is not available in S3,

1
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AConfiguration.java
View File

@ -159,6 +159,7 @@ public class ITestS3AConfiguration {
return intercept(clazz, return intercept(clazz,
() -> { () -> {
fs = S3ATestUtils.createTestFileSystem(conf); fs = S3ATestUtils.createTestFileSystem(conf);
fs.listFiles(new Path("/"), false);
return "expected failure creating FS " + text + " got " + fs; return "expected failure creating FS " + text + " got " + fs;
}); });
} }