Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,905 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1381 Commits

Author SHA1 Message Date
Kai Xie 188bebbe7e HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-01-08 11:57:57 +00:00
Akira Ajisaka 7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens." 
This reverts commit d7152332b3.
 2019-01-08 14:51:30 +09:00
Steve Loughran d7152332b3
HADOOP-14556. S3A to support Delegation Tokens. 
Contributed by Steve Loughran.
 2019-01-07 13:18:03 +00:00
Weiwei Yang f87b3b11c4 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 2019-01-07 15:56:49 +08:00
Sunil G 1f425271a7
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae4.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb.
 2019-01-05 17:35:29 +09:00
Weiwei Yang 040a202b20 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 2019-01-03 21:37:49 +08:00
Sean Mackrory 13f0ee21f2 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. 2019-01-02 09:33:09 -07:00
Sean Mackrory d8f670ff28 HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal. 2018-12-27 08:19:25 -07:00
Eric Yang f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Steve Loughran a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies. 
Contributed by Steve Loughran.
 2018-12-20 18:09:01 +00:00
Vishwajeet Dusane 784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress. 
Contributed by Vishwajeet Dusane.
 2018-12-20 18:00:05 +00:00
Da Zhou 346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.
 2018-12-17 11:15:20 +00:00
Da Zhou 62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb)
 2018-12-17 11:10:22 +00:00
Da Zhou b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2018-12-17 11:04:40 +00:00
Sean Mackrory 82b798581d HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota. 2018-12-12 09:30:13 -07:00
Sean Mackrory 1a25bbe9ec HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota. 2018-12-11 08:33:13 -07:00
Sean Mackrory c35de95a22 HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota. 2018-12-11 08:29:39 -07:00
Sean Mackrory 3ff8580f22 HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota) 2018-12-10 14:03:08 -07:00
Inigo Goiri 8fc0d04517 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-12-07 18:30:36 -08:00
Da Zhou a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2018-12-04 13:39:10 +00:00
Da Zhou 7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.
 2018-11-30 10:12:41 +00:00
Sean Mackrory 7eb0d3a324 HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota. 2018-11-29 09:36:39 -07:00
Ewan Higgs c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.
 2018-11-29 15:12:17 +00:00
Sean Mackrory 085f10e75d HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory e148c3ff09 HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory 5d96b74f33 HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Steve Loughran 4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.
 2018-11-27 20:39:54 +00:00
Da Zhou 300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d9)
 2018-11-27 18:13:07 +00:00
Da Zhou a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2018-11-23 14:17:36 +00:00
Weiwei Yang 5ff0cf86a9 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 2018-11-23 13:56:54 +08:00
Weiwei Yang be0708c6eb HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 2018-11-19 14:00:54 +08:00
Arpit Agarwal 914b0cf15f HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 2018-11-15 13:58:13 -08:00
Akira Ajisaka 66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 2018-11-15 16:30:24 +09:00
Sammi Chen 3fade865ce HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 2018-11-14 12:58:57 +08:00
Da Zhou a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2018-11-13 21:46:18 +00:00
Da Zhou 9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2018-11-09 11:06:23 +00:00
Junhua Gu 724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9)
 2018-11-08 17:21:40 +00:00
Sammi Chen 7574d18538 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 2018-10-26 15:19:56 +08:00
Daniel Templeton e374584479 HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2018-10-24 14:18:35 -07:00
Robert Kanter 5e294e950b YARN-8919. Some tests fail due to NoClassDefFoundError for OperatorCreationException (tasanuma0829 via rkanter) 2018-10-23 13:37:17 -07:00
Ted Yu e2cecb681e HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-19 13:21:06 -07:00
Giovanni Matteo Fumarola cc09b2b0c7 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:24:23 -07:00
Steve Loughran d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong. 
Contributed by Steve Loughran and Ewan Higgs.
 2018-10-16 20:02:54 +01:00
Steve Loughran ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console. 
Contributed by Vishwajeet Dusane.
 2018-10-15 22:53:55 +01:00
bibinchundatt b4a38e7b3e YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt. 2018-10-15 16:10:25 +05:30
Steve Loughran e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus. 
Contributed by Ted Yu.
 2018-10-12 09:59:19 +01:00
Steve Loughran ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-11 14:57:38 +01:00
Steve Loughran bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2018-10-09 20:02:12 +01:00
Jason Lowe 4b5b1ac3d1 MAPREDUCE-7130. Rumen crashes trying to handle MRAppMaster recovery events. Contributed by Peter Bacsko 2018-10-09 13:27:03 -05:00
Steve Loughran 7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard. 
Contributed by Gabor Bota
 2018-10-09 10:46:41 +01:00
Steve Loughran 273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2018-10-05 11:29:03 +01:00
Steve Loughran 7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2018-10-03 12:52:53 +01:00
Aaron Fabbri 4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-03 00:24:29 -07:00
Aaron Fabbri 046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-02 21:22:49 -07:00
Steve Loughran e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2018-10-02 18:57:20 +01:00
Sunil G 58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Steve Loughran a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2018-10-02 11:42:52 +01:00
Steve Loughran 43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2018-10-02 11:37:28 +01:00
Steve Loughran a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2018-09-25 20:58:32 +01:00
Steve Loughran d5da9928c9 HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2018-09-25 19:13:10 +01:00
Sammi Chen 2b635125fb HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 2018-09-25 19:48:30 +08:00
Mingliang Liu c07715e378 HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran 2018-09-24 12:53:21 -07:00
Sunil G d060cbea48 HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran. 2018-09-24 21:19:47 +05:30
Sean Mackrory 0def61482b Merge branch 'HADOOP-15407' into trunk 2018-09-22 21:19:12 -06:00
Steve Loughran d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2018-09-21 11:06:24 +01:00
Steve Loughran a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2018-09-20 17:36:18 +01:00
Sean Mackrory 8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2018-09-19 13:57:39 +01:00
Steve Loughran df2166a643
HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2018-09-18 12:20:52 +01:00
Steve Loughran 51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2018-09-18 12:09:25 +01:00
Steve Loughran 524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. 
Contributed by Da Zhou
 2018-09-18 11:48:46 +01:00
Steve Loughran 1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch. 
Contributed by Steve Loughran.
 2018-09-18 10:56:56 +01:00
Steve Loughran 26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config. 
Contributed by Gabor Bota.
 2018-09-17 22:40:08 +01:00
Thomas Marquardt b4c23043d3 HADOOP-15757. ABFS: remove dependency on common-codec Base64. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 26211019c8 HADOOP-15753. ABFS: support path "abfs://mycluster/file/path" 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt e5593cbd83 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 13c70e9ba3 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. 
Contributed by Sneha Varma.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 6801b30733 HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation 
for storage account with no namespace feature.

Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 347a52a866 Fixing findbugs and license issues related to: 
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 97f06b3fc7 HADOOP-15703. ABFS - Implement client-side throttling. 
Contributed by Sneha Varma and Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 4410eacba7 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 81dc4a995c HADOOP-15663. ABFS: Simplify configuration. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt df57c6c3b1 HADOOP-15692. ABFS: extensible support for custom oauth. 
Contributed by Junhua Gu and Rajeev Bansal.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt dd2b22fa31 HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. 
Contributed by Thomas Marquardt.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 6b6f8cc2be HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. 
Contributed by Sean Mackrory.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 9c1e4e8139 HADOOP-15661. ABFS: Add support for ACL. 
Contributed by Junhua Gu and Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt 9149b9703e HADOOP-15660. ABFS: Add support for OAuth 
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt d6a4f39bd5 HADOOP-15669. ABFS: Improve HTTPS Performance. 
Contributed by Vishwajeet Dusane.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt cc5cc60c41 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2018-09-17 19:54:01 +00:00
Thomas Marquardt b54b0c1b67 HADOOP-15659. Code changes for bug fix and new tests. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Thomas Marquardt ce03a93f78 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. 
Contributed by Steve Loughran and Da Zhou.
 2018-09-17 19:54:01 +00:00
Steve Loughran a271fd0eca HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies. 
Contributed by Da Zhou.
 2018-09-17 19:54:01 +00:00
Steve Loughran f044deedbb HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit. 
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
 2018-09-17 19:54:01 +00:00
Steve Loughran d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran) 2018-09-12 21:04:49 -07:00
Aaron Fabbri d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine) 2018-09-12 16:36:01 -07:00
Mingliang Liu 1f6c4545cf HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran 2018-09-12 10:58:39 -07:00
Sean Mackrory 47b72c87eb HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded. 
Contributed by Gabor Bota.
 2018-09-12 09:12:38 -06:00
bibinchundatt c44088ac19 YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Naganarasimha eb08543c7a YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
Sunil G 89b3ebd11e YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Mingliang Liu 87f63b6479 HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran 2018-09-11 17:18:42 -07:00
Steve Loughran 1eef0acfe5
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.
 2018-09-10 14:24:41 +01:00
Gabor Bota 36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota) 2018-09-07 10:25:20 -07:00
Steve Loughran 5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.
 2018-08-30 14:49:53 +01:00
Steve Loughran e8d138ca7c
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.
 2018-08-30 14:36:00 +01:00
Steve Loughran 2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length. 
Contributed by Ewan Higgs
 2018-08-30 14:33:16 +01:00
Yufei Gu 8563fd67be YARN-8632. Threads in SLS quit without logging exception. Contributed by Xianghao Lu. 2018-08-24 11:23:17 -07:00
Surendra Singh Lilhore 96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Aaron Fabbri d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota) 2018-08-17 10:15:39 -07:00
Steve Loughran 0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375. 
Contributed by Steve Loughran.
 2018-08-16 09:58:46 -07:00
Akira Ajisaka 3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Ewan Higgs a13929ddcb HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran. 2018-08-13 12:57:45 +02:00
Steve Loughran da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support. 
Contributed by Steve Loughran.
 2018-08-08 22:57:24 -07:00
Ewan Higgs 2ec97abb2e HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran. 2018-08-08 13:50:23 +02:00
Sean Mackrory 7862f1523f HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota) 2018-08-07 20:13:09 -06:00
Steve Loughran 48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append. 
Contributed by Steve Loughran.
 2018-08-03 16:06:00 -07:00
Inigo Goiri 9fea5c9ee7 YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi. 2018-07-31 09:36:34 -07:00
Sammi Chen 0857f116b7 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 2018-07-30 10:53:44 +08:00
Sean Mackrory 59adeb8d7f HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota. 2018-07-27 18:23:29 -06:00
bibinchundatt 3d3158cea4 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 2018-07-25 18:59:05 +05:30
Steve Loughran 45d9568aaa
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.

(cherry picked from commit 749fff577e)
 2018-07-19 12:31:19 -07:00
Wangda Tan 752dcce5f4 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
 2018-07-16 10:54:41 -07:00
Sean Mackrory a08812a1b1 HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota. 2018-07-12 17:24:01 +02:00
Sean Mackrory d503f65b66 HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting 
connection in case of timeout.
 2018-07-10 17:52:57 +02:00
Steve Loughran ca8b80bf59
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan. 
Contributed by Steve Loughran.
 2018-07-10 10:43:59 +01:00
Aaron Fabbri 93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota) 2018-07-03 13:50:11 -07:00
Inigo Goiri 1804a31515 HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-07-02 10:48:20 -07:00
Inigo Goiri 73746c5da7 HDFS-13707. [PROVIDED Storage] Fix failing integration tests in ITestProvidedImplementation. Contributed by Virajith Jalaparti. 2018-06-29 09:56:13 -07:00
Akira Ajisaka 2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma. 2018-06-28 14:37:22 +09:00
Todd Lipcon 7a3c6e9c3c HADOOP-15550. Avoid static initialization of ObjectMappers 2018-06-25 15:36:45 -07:00
Sean Mackrory c687a6617d HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota. 2018-06-25 14:59:41 -06:00
Anu Engineer e16e5b307d HDDS-177. Create a releasable ozonefs artifact 
Contributed by Marton, Elek.
 2018-06-24 01:05:04 -07:00
Anu Engineer ca14fec02f HDDS-184. Upgrade common-langs version to 3.7 in hadoop-tools/hadoop-ozone. 
Contributed by Takanobu Asanuma.
 2018-06-23 15:49:44 -07:00
Sean Mackrory 55fad6a3de HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota. 2018-06-22 11:36:56 -06:00
Sean Mackrory b089a06793 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 2018-06-20 16:45:08 -06:00
Chris Douglas f34744603e HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 2018-06-17 23:12:18 -07:00
Chris Douglas 980031bb04 HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs 2018-06-17 11:54:26 -07:00
Jitendra Pandey f4c7c91123 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-13 00:36:02 -07:00
Inigo Goiri c35481594f YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 2018-06-12 10:59:50 -07:00
Steve Loughran d901be6795
HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.
 2018-06-06 18:28:14 +01:00
Anu Engineer 3b34148c4f HDDS-88. Create separate message structure to represent ports in DatanodeDetails. 
Contributed by Nanda Kumar.
 2018-05-30 08:52:07 -07:00
Aaron Fabbri 5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota) 2018-05-29 19:20:22 -07:00
Aaron Fabbri 30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota. 2018-05-29 13:38:15 -07:00
Inigo Goiri 8fdc993a99 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 2018-05-28 16:45:42 -07:00
Johan Gustavsson d14e26b31f
HADOOP-15477. Make unjar in RunJar overrideable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-05-28 17:29:59 +09:00
Steve Loughran ba842847c9 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.
 2018-05-21 11:12:34 +01:00
Steve Loughran cc3600aabd HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest. 
Contributed by Steve Loughran.

(cherry picked from commit 170f1040d4)
 2018-05-17 19:08:55 +01:00
Steve Loughran b670837469 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name. 
Contributed by Sean Mackrory.
 2018-05-15 16:13:56 +01:00
Steve Loughran ba051b0686
HADOOP-15354. hadoop-aliyun & hadoop-azure modules to mark hadoop-common as provided 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-05-10 18:38:08 +09:00
Anu Engineer 46e0f27862 HDDS-18. Ozone Shell should use RestClient and RpcClient. Contributed by Lokesh Jain. 2018-05-09 23:00:13 -07:00
Sean Mackrory 1cfe7506f7 HADOOP-15356. Make HTTP timeout configurable in ADLS connector. Contributed by Atul Sikaria and Sean Mackrory. 2018-05-09 15:15:51 -06:00
Mukul Kumar Singh 343b51dcf4 HDDS-28. Ozone:Duplicate declaration in hadoop-tools/hadoop-ozone/pom.xml. Contributed by Tsz Wo Nicholas Sze. 2018-05-09 20:20:21 +05:30
Mukul Kumar Singh 208b97e969 HDDS-20. Ozone: Add support for rename key within a bucket for rpc client. Contributed by Lokesh Jain. 2018-05-09 19:06:07 +05:30
Aaron Fabbri 8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota) 2018-05-08 18:58:42 -07:00
Aaron Fabbri 69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota) 2018-05-08 15:29:54 -07:00
Steve Loughran 5b11b9fd41 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt
 2018-05-07 11:54:08 +01:00
Aaron Fabbri 96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri) 2018-05-04 11:34:45 -07:00
Sean Mackrory 85381c7b60 HADOOP-15434. Upgrade to ADLS SDK that exposes current timeout. 2018-05-02 21:30:31 -06:00
Aaron Fabbri 919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota. 2018-04-30 16:02:57 -07:00
Weiwei Yang ba2db39c65 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:36:21 +08:00
Aaron Fabbri 7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota) 2018-04-26 20:41:29 -07:00
Owen O'Malley 2adda92de1
HDFS-7240 git commit fixup 
Signed-off-by: Owen O'Malley <omalley@apache.org>
 2018-04-26 13:52:17 -07:00
Mukul Kumar Singh ae8ac7f082 HDFS-13424. Ozone: Refactor MiniOzoneClassicCluster. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar 025058f251 HDFS-13446. Ozone: Fix OzoneFileSystem contract test failures. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer 8b832f3c35 HDFS-13405. Ozone: Rename HDSL to HDDS. 
Contributed by Ajay Kumar, Elek Marton, Mukul Kumar Singh, Shashikant Banerjee and Anu Engineer.
 2018-04-26 05:36:04 -07:00
Nanda kumar 7ae9543a57 HDFS-13391. Ozone: Make dependency of internal sub-module scope as provided in maven. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 04e1287771 HDFS-13298. Ozone: Make ozone/hdsl/cblock modules turned off by default. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects. 
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
 2018-04-26 05:36:04 -07:00
Anu Engineer 5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key. 
Contributed by Mukul Kumar Singh.
 2018-04-26 05:36:04 -07:00
Anu Engineer 61651dcf5c HDFS-13108. Ozone: OzoneFileSystem: Simplified url schema for Ozone File System. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh cd807c9890 HDFS-13133. Ozone: OzoneFileSystem: Calling delete with non-existing path shouldn't be logged on ERROR level. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh 9272e1021d HDFS-12636. Ozone: OzoneFileSystem: Implement seek functionality for rpc client. Contributed by Lokesh Jain. 2018-04-26 05:36:04 -07:00
Anu Engineer 4460ac9c57 HDFS-12756. Ozone: Add datanodeID to heartbeat responses and container protocol. Contributed by Anu Engineer. 2018-04-26 05:36:04 -07:00
Anu Engineer 740a06cdd7 HDFS-12719. Ozone: Fix checkstyle, javac, whitespace issues in HDFS-7240 branch. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Chen Liang e395244e95 HDFS-12758. Ozone: Correcting assertEquals argument order in test cases. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 2ec24a7542 HDFS-12774. Ozone: OzoneClient: Moving OzoneException from hadoop-hdfs to hadoop-hdfs-client. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Chen Liang 895a4cc60a HDFS-12632. Ozone: OzoneFileSystem: Add contract tests to OzoneFileSystem. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer d90ab3365d HDFS-12608. Ozone: Remove Warnings when building. Contributed by Bharat Viswanadham. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 9445a9267f HDFS-12572. Ozone: OzoneFileSystem: delete/list status/rename/mkdir APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer 0b3270a8de HDFS-12513. Ozone: Create UI page to show Ozone configs by tag. Contributed by Ajay Kumar. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao 5e4a6b686c HDFS-12425. Ozone: OzoneFileSystem: read/write/create/open/getFileInfo APIs. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Xiaoyu Yao f2d9360b9d HDFS-12199. Ozone: OzoneFileSystem: OzoneFileystem initialization code. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Akira Ajisaka 3e2ca23a53 HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 2018-04-26 05:36:04 -07:00
Aaron Fabbri 989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota) 2018-04-23 15:26:53 -07:00
Inigo Goiri fd24fd0ff7 YARN-8137. Parallelize node addition in SLS. Contributed by Abhishek Modi. 2018-04-20 09:09:12 -07:00
Yiqun Lin 1312f9ae4c HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 2018-04-18 16:04:00 +08:00
Inigo Goiri 78860372bd YARN-8134. Support specifying node resources in SLS. Contributed by Abhishek Modi. 2018-04-17 13:11:34 -07:00
Aaron Fabbri b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:55:38 -07:00
Xiao Chen 7c9cdad6d0 HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-10 21:31:48 -07:00
Aaron Fabbri ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota. 2018-04-05 20:23:17 -07:00
Steve Loughran 5a174f8ac6 HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException. 
Contributed by Gabor Bota.
 2018-04-03 16:31:34 +01:00
Sammi Chen 6542d17ea4 HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 2018-03-30 20:23:05 +08:00
Sean Mackrory 9d7a9031a5 HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 2018-03-29 10:13:40 -06:00
Chris Douglas 081c350188 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 2018-03-28 11:58:59 -07:00
Haibo Chen fcea5a4d79 YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen) 2018-03-26 14:30:11 -07:00
Sean Mackrory 82665a7887 HADOOP-15299. Bump Jackson 2 version to Jackson 2.9.x. 2018-03-26 14:05:15 -06:00
Sean Mackrory 7ce6b41509 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-03-20 21:12:20 -07:00
Sammi Chen d67a5e2dec HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 2018-03-19 15:02:37 +08:00
Wangda Tan f47659fb97 MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda) 
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
 2018-03-15 13:26:01 -07:00
Steve Loughran 1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories. 
Contributed by Steve Loughran.
 2018-03-15 18:05:14 +00:00
fang zhenyi 5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Chris Douglas 45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Chris Douglas 45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional. 
Contributed by Steve Loughran.
 2018-03-12 14:01:42 +00:00
Steve Loughran 7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.
 2018-03-08 11:24:06 +00:00
Steve Loughran 3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down. 
Contributed by Virajith Jalaparti.
 2018-03-08 11:15:46 +00:00
Steve Loughran e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d)
 2018-03-07 18:51:10 +00:00
Steve Loughran 2e1e049bd0 HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile. 
Contributed by Steve Loughran.
 2018-03-05 18:12:38 +00:00
Steve Loughran 8110d6a0d5 HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions. 
Contributed by Aaron Fabbri.
 2018-03-05 14:06:20 +00:00
Gergely Novák e99a1e7767
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-02 16:15:04 -08:00
Ray Chiang 923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
fang zhenyi 29233c3c4a
HADOOP-15274. Move hadoop-openstack to slf4j 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-01 15:08:30 -08:00
Steve Loughran edc9f1451b HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran
 2018-02-28 14:42:25 +00:00
Weiwei Yang 7af4f34de5 YARN-7929. Support to set container execution type in SLS. (Jiandan Yang via Weiwei Yang) 2018-02-28 17:58:09 +08:00
Arpit Agarwal 2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Xuan Gong 68ce193efc MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 2018-02-23 14:37:26 -08:00
Carlo Curino 84cea0011f YARN-7732. Support Generic AM Simulator from SynthGenerator. (Contributed by Young Chen via curino) 2018-02-20 17:00:34 -08:00
fang zhenyi 4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Steve Loughran 7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. 
Contributed by Steve Loughran.
 2018-02-16 16:37:06 +00:00
Steve Loughran 481d79fedc HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b7)
 2018-02-15 16:27:31 +00:00
Steve Loughran 9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client. 
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb9)
 2018-02-15 15:57:10 +00:00
Steve Loughran b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document. 
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6)
 2018-02-15 14:57:56 +00:00
Steve Loughran c9a373fb14 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a)
 2018-02-15 14:26:00 +00:00
Steve Loughran f20dc0d577 HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-14 16:20:14 +00:00
Steve Loughran 8cf88fcd1f HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit bd5ab59125)
 2018-02-12 15:13:00 +00:00
Wangda Tan 60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Anu Engineer 4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Sunil G 9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Arun Suresh 38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Steve Loughran 7fd287b4af HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit 95a96b13e2)
 2018-01-29 09:48:14 -08:00
Jason Lowe ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Yufei Gu 16be42d309 YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen. 2018-01-25 13:06:01 -08:00
Steve Loughran f63d13f10d HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.
 2018-01-23 09:09:11 -08:00
Steve Loughran 1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2018-01-18 13:13:58 +00:00
Steve Loughran f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits." 
This reverts commit 35ad9b1dd2.
 2018-01-18 12:35:57 +00:00
Aaron Fabbri 268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Sammi Chen 9195a6e302 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 2018-01-17 15:55:59 +08:00
Steve Loughran a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing 
after OutputCommitter patch.
Contributed by Steve Loughran
 2018-01-15 11:33:32 +00:00
Mukul Kumar Singh 2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:45:51 +09:00
Steve Loughran 1a09da7400 HADOOP-15163. Fix S3ACommitter documentation 
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
 2018-01-10 15:37:07 +00:00
Steve Loughran b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics 
Contributed by Sean Mackrory
 2018-01-09 18:46:52 +00:00
Steve Loughran 52babbb4a0 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
 2017-12-22 11:39:55 +00:00
Steve Loughran c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls. 
Contributed by Steve Loughran
 2017-12-21 14:58:58 +00:00
Steve Loughran ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance. 
Contributed by Steve Loughran.
 2017-12-21 14:15:53 +00:00
Akira Ajisaka 5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java. 2017-12-21 10:04:34 +09:00
Steve Loughran 1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran 35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2017-12-18 21:19:06 +00:00
Chris Douglas 4b3a785914 HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti 2017-12-15 17:51:42 -08:00
Chris Douglas e515103a83 Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter" 
This reverts commit c1bf2654b0.
 2017-12-15 17:51:42 -08:00
Virajith Jalaparti 8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 962b5e722b HDFS-12903. [READ] Fix closing streams in ImageWriter 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00
Virajith Jalaparti 9c35be86e1 HDFS-12713. [READ] Refactor FileRegion and BlockAliasMap to separate out HDFS metadata and PROVIDED storage metadata. Contributed by Ewan Higgs 2017-12-15 17:51:40 -08:00
Virajith Jalaparti a027055dd2 HDFS-12885. Add visibility/stability annotations. Contributed by Chris Douglas 2017-12-15 17:51:40 -08:00
Virajith Jalaparti 352f994b64 HDFS-12665. [AliasMap] Create a version of the AliasMap that runs in memory in the Namenode (leveldb). Contributed by Ewan Higgs. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti 4d59dabb7f HDFS-12809. [READ] Fix the randomized selection of locations in {{ProvidedBlocksBuilder}}. 2017-12-15 17:51:40 -08:00
Virajith Jalaparti 3d3be87e30 HDFS-12778. [READ] Report multiple locations for PROVIDED blocks 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 3b1d30301b HDFS-12775. [READ] Fix reporting of Provided volumes 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 6cd80b2521 HDFS-12779. [READ] Allow cluster id to be specified to the Image generation tool 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 90d1b47a2a HDFS-12776. [READ] Increasing replication for PROVIDED files should create local replicas 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 87dc026bee HDFS-12789. [READ] Image generation tool does not close an opened stream 2017-12-15 17:51:39 -08:00
Virajith Jalaparti c293cc8e9b HDFS-12671. [READ] Test NameNode restarts when PROVIDED is configured 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 71d0a82571 HDFS-12607. [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED 2017-12-15 17:51:39 -08:00
Virajith Jalaparti 98f5ed5aa3 HDFS-11902. [READ] Merge BlockFormatProvider and FileRegionProvider. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti d6a9a89973 HDFS-12605. [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti 17052c4aff HDFS-12584. [READ] Fix errors in image generation tool from latest rebase 2017-12-15 17:51:38 -08:00
Virajith Jalaparti aca023b72c HDFS-12289. [READ] HDFS-12091 breaks the tests for provided block reads 2017-12-15 17:51:38 -08:00
Virajith Jalaparti 546b95f484 HDFS-11673. [READ] Handle failures of Datanode with PROVIDED storage 2017-12-15 17:51:38 -08:00
Virajith Jalaparti 4851f06bc2 HDFS-11791. [READ] Test for increasing replication of provided files. 2017-12-15 17:51:38 -08:00
Virajith Jalaparti aa5ec85f7f HDFS-11663. [READ] Fix NullPointerException in ProvidedBlocksBuilder 2017-12-15 17:51:37 -08:00
Virajith Jalaparti d65df0f273 HDFS-11190. [READ] Namenode support for data stored in external stores. 2017-12-15 17:51:37 -08:00
Virajith Jalaparti 8da3a6e314 HDFS-10706. [READ] Add tool generating FSImage from external store 2017-12-15 17:51:37 -08:00
Kai Zheng f86c81d923 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-14 11:14:52 +08:00
Kai Zheng 28792b6b7f Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu" 
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
 2017-12-14 11:05:11 +08:00
Surendra Singh Lilhore 00129c5314 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-12 00:28:02 +05:30
Kai Zheng ce04340ec7 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 2017-12-08 23:03:54 +08:00
Kai Zheng 73b86979d6 HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu 2017-12-06 12:01:36 +08:00
Aaron Fabbri 6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory. 2017-12-05 11:06:32 -08:00
Steve Loughran 3150c019ae HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes. 
Contributed by Steve Loughran
 2017-12-05 15:05:41 +00:00
Steve Loughran 0ea182d0fa HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan
 2017-11-28 11:52:59 +00:00
Rohith Sharma K S 3cd75845da MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran. 2017-11-27 22:01:00 +05:30
Steve Loughran de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
Steve Loughran c326fc89b0 HADOOP-15024 Support user agent configuration and include that & Hadoop version information to oss server. 
Contributed by Sammi Chen.
 2017-11-20 18:56:42 +00:00
Akira Ajisaka 6903cf096e HADOOP-13514. Upgrade maven surefire plugin to 2.20.1 
Signed-off-by: Allen Wittenauer <aw@apache.org>
 2017-11-19 12:39:37 -08:00
Kai Zheng 18621af7ae HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu 2017-11-14 17:58:37 +08:00
Daniel Templeton ba8136615a YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton) 
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
 2017-11-09 12:09:48 -08:00
John Zhuge bf6a660232 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-11-09 10:16:12 -08:00
Subru Krishnan f2df6b8983 HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 2017-11-08 18:07:12 -08:00
Arun Suresh 53c0fb7efe HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 2017-11-02 17:15:33 -07:00
Akira Ajisaka cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 2017-11-02 18:32:24 +09:00
Chris Douglas 139cc752d2 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 2017-10-27 21:59:27 -07:00
Yufei Gu 99880d0a16 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 2017-10-27 14:41:37 -07:00
Subru Krishnan 625039ef20 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 2017-10-25 15:51:27 -07:00
Wangda Tan 281d83604d YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 2017-10-20 15:54:15 -07:00
Steve Loughran f36cbc8475 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.
 2017-10-20 22:27:04 +01:00
Haibo Chen 46eb1033a8 YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen) 2017-10-17 16:04:19 -07:00
Steve Loughran 9fcc3a1fc8 HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak
 2017-10-16 16:01:47 +01:00
Steve Loughran 3de574413c HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.
 2017-10-11 18:06:43 +01:00
Steve Loughran 572cdb5463 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-06 17:43:38 +01:00
Haibo Chen c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Steve Loughran 9288206cb3
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-10-05 15:05:23 +01:00
Yufei Gu 79e37dc6db YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu) 2017-10-03 15:10:50 -07:00
Akira Ajisaka c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
John Zhuge 8aca46e53f HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge. 2017-09-29 12:15:06 -07:00
Steve Loughran a530e7ab3b
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-09-28 19:52:56 +01:00
Aaron Fabbri 47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 15:59:38 -07:00
Jason Lowe b133dc5700 MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 2017-09-22 15:21:42 -05:00
Steve Loughran 2d2d97fa7d
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure 
Contributed by Steve Loughran
 2017-09-15 17:03:01 +01:00
Aaron Fabbri 49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 14:10:48 -07:00
ChenSammi e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
John Zhuge a4661850c1 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 2017-09-08 11:51:03 -07:00
Steve Loughran 5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A. 
Contributed by Aaron Fabbri
 2017-09-08 12:07:02 +01:00
Steve Loughran 83449abb40
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.
 2017-09-07 19:55:56 +01:00
Steve Loughran 13eda50003
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov
 2017-09-07 18:35:03 +01:00
Xiaoyu Yao 63720ef574 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-05 23:34:55 -07:00
John Zhuge 50506e90a8 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-05 23:26:57 -07:00
Steve Loughran 792eff9ea7
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS. 
Contributed by Sivaguru Sankaridurg
 2017-09-05 14:16:57 +01:00
Andrew Wang 0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Allen Wittenauer a3fee475f7
HADOOP-14414. Calling maven-site-plugin directly for docs profile is unnecessary (Andras Bokor via aw) 2017-09-01 08:38:30 -07:00
Steve Loughran 621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
 2017-09-01 14:13:41 +01:00
John Zhuge 7ecc6dbed6 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria. 2017-08-31 21:13:22 -07:00
Allen Wittenauer 71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Haibo Chen d04f85f387 MAPREDUCE-6892. Issues with the count of failed/killed tasks in the jhist file. (Peter Bacsko via Haibo Chen) 2017-08-30 10:07:48 -07:00
Yufei Gu 26fafc3597 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-29 16:33:51 -07:00
Steve Loughran 9374f38820
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory 
Contributed by Esfandiar Manii and Thomas Marquardt.
 2017-08-29 19:03:44 +01:00
Steve Loughran 021974f4cb
HADOOP-14802. Add support for using container saskeys for all accesses. 
Contributed by Sivaguru Sankaridurg
 2017-08-29 19:02:43 +01:00
Andrew Wang f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Arpit Agarwal 8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Steve Loughran 27ab5f7385
HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator. 
Contributed by Genmao Yu
 2017-08-22 11:55:48 +01:00
Kai Zheng 267e19a09f HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu 2017-08-21 13:36:28 +08:00
Arun Suresh 8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Steve Loughran c6b4e656b7
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted. 
Contributed by Thomas Marquardt
 2017-08-18 14:13:40 +01:00
Andrew Wang dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Chris Douglas 645a8f2a4d HADOOP-14726. Mark FileStatus::isDir as final 2017-08-14 21:57:20 -07:00
John Zhuge 7769e96149 HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 2017-08-13 00:22:34 -07:00
Allen Wittenauer 4222c97108
HADOOP-10392. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) (ajisakaa via aw) 2017-08-11 09:25:56 -07:00
Steve Loughran 54356b1e83
HADOOP-14183. Remove service loader config file for wasb fs. 
Contributed by Esfandiar Manii.
 2017-08-10 16:46:33 +01:00
Steve Loughran f4e1aa0508
HADOOP-14715. TestWasbRemoteCallHelper failing. 
Contributed by Esfandiar Manii.
 2017-08-08 23:37:47 +01:00
Chris Douglas 55a181f845 HADOOP-14730. Support protobuf FileStatus in AdlFileSystem. 2017-08-07 21:31:28 -07:00
Steve Loughran d91b7a8451
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek. 
Contributed by Thomas Marquardt
 2017-08-06 20:19:23 +01:00
Steve Loughran 7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran

(cherry picked from commit ced547d5f0)
 2017-08-04 11:09:08 +01:00
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Steve Loughran 778d4edd9a
HADOOP-14660 wasb: improve throughput by 34% when account limit exceeded. 
Contributed by Thomas Marquardt.
 2017-08-01 21:36:28 +01:00
John Zhuge f735ad1b67 HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge. 2017-07-28 09:46:54 -07:00
Jitendra Pandey a92bf39e23 HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:26:48 -07:00