7f40e6688a
Contributed by Steve Loughran. This fixes two problems with S3Guard authoritative mode and the auth directory flags which are stored in DynamoDB. 1. mkdirs was creating dir markers without the auth bit, forcing needless scans on newly created directories and files subsequently added; it was only with the first listStatus call on that directory that the dir would be marked as authoritative -even though it would be complete already. 2. listStatus(path) would reset the authoritative status bit of all child directories even if they were already marked as authoritative. Issue #2 is possibly the most expensive, as any treewalk using listStatus (e.g globfiles) would clear the auth bit for all child directories before listing them. And this would happen every single time... essentially you weren't getting authoritative directory listings. For the curious, that the major bug was actually found during testing -we'd all missed it during reviews. A lesson there: the better the tests the fewer the bugs. Maybe also: something obvious and significant can get by code reviews. modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/BulkOperationState.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/LocalMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/NullMetadataStore.java modified: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3Guard.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3GuardWriteBack.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestRestrictedReadAccess.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestPartialDeleteFailures.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStore.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreAuthoritativeMode.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestDynamoDBMetadataStoreScale.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardFsck.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/MetadataStoreTestBase.java modified: hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestS3Guard.java Change-Id: Ic3ffda13f2af2430afedd50fd657b595c83e90a7 |
||
|---|---|---|
| .github | ||
| dev-support | ||
| hadoop-assemblies | ||
| hadoop-build-tools | ||
| hadoop-client-modules | ||
| hadoop-cloud-storage-project | ||
| hadoop-common-project | ||
| hadoop-dist | ||
| hadoop-hdfs-project | ||
| hadoop-mapreduce-project | ||
| hadoop-maven-plugins | ||
| hadoop-minicluster | ||
| hadoop-project | ||
| hadoop-project-dist | ||
| hadoop-tools | ||
| hadoop-yarn-project | ||
| licenses | ||
| licenses-binary | ||
| .gitattributes | ||
| .gitignore | ||
| BUILDING.txt | ||
| Jenkinsfile | ||
| LICENSE-binary | ||
| LICENSE.txt | ||
| NOTICE-binary | ||
| NOTICE.txt | ||
| README.txt | ||
| pom.xml | ||
| start-build-env.sh | ||
README.txt
For the latest information about Hadoop, please visit our website at: http://hadoop.apache.org/ and our wiki, at: https://cwiki.apache.org/confluence/display/HADOOP/