5c37a0b841
HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
...
Contributed by Rajesh Balamohan
(cherry picked from commit 0ea182d0fa
2017-11-28 11:57:51 +00:00
30ab9b6aef
HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2. Contributed by Sammi Chen
...
The consolidated commits in this backport are as follows:
HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
2017-11-13 09:29:33 +08:00
5991c218a0
HADOOP-15026. Rebase ResourceEstimator start/stop scripts for branch-2. (Rui Li via Subru).
2017-11-09 00:32:34 -08:00
46a740a82e
HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru).
...
(cherry picked from commit f2df6b8983
2017-11-08 18:09:02 -08:00
0200fbac5b
HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh)
...
(cherry picked from commit 53c0fb7efe
2017-11-02 17:18:03 -07:00
82fc80a987
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi.
...
(cherry picked from commit cc3f3eca40
2017-11-02 18:33:25 +09:00
71a55bebfb
Preparing for 2.10.0 development
2017-10-31 12:51:14 -07:00
ee89241282
HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li
...
(cherry picked from commit 139cc752d277ad73501c
2017-10-27 22:01:41 -07:00
3fbcf76324
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure.
...
Contributed by Steve Loughran.
(backport JIRA HADOOP-14870)
2017-10-27 12:55:12 +01:00
9897538a96
HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru).
...
(cherry picked from commit 625039ef20
2017-10-26 19:14:24 -07:00
13f1a7e0ed
Revert "HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru)."
...
This reverts commit da7f989d86
2017-10-26 15:33:53 +09:00
da7f989d86
HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru).
...
(cherry picked from commit 625039ef20
2017-10-25 17:05:35 -07:00
062d9c9bab
HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
...
Contributed by Andras Bokor.
(cherry picked from commit f36cbc8475
2017-10-20 22:27:59 +01:00
d894f910d0
YARN-6608. Backport all SLS improvements from trunk to branch-2. (Carlo Curino via wangda)
2017-10-17 19:45:09 -07:00
8902178d3e
HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
...
Contributed by Santhosh G Nayak
Cherry-picked from commit 9fcc3a1fc8
2017-10-16 16:06:41 +01:00
ccfd1a4a56
HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
...
Contributed by Varada Hemeswari.
Cherry-picked from commit ref 3de574413c
2017-10-11 18:09:13 +01:00
34040e8d73
HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
...
Contributed by Kannapiran Srinivasan.
2017-10-07 11:20:18 +01:00
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
2017-10-05 16:03:13 +01:00
28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests.
...
Contributed by Steve Loughran
2017-10-03 20:06:45 +01:00
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
...
Contributed by Varada Hemeswari
2017-10-02 14:28:59 +01:00
a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A.
...
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
(cherry-picked from/based on commit 621b43e254
2017-09-29 11:29:22 +01:00
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Contributed by Sivaguru Sankaridurg
2017-09-29 11:16:31 +01:00
27a41848ac
HADOOP-14890 Move up to AWS SDK 1.11.199.
...
Contributed by Steve Loughran.
2017-09-26 20:11:10 +01:00
2c10abaddd
MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari
...
(cherry picked from commit b133dc5700
2017-09-22 15:30:53 -05:00
12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory.
...
Contributed by Esfandiar Manii.
2017-09-19 11:02:10 +01:00
b669aca518
HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin.
2017-09-11 11:40:48 -07:00
87af3f4991
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
...
Contributed by Georgi Chalakov
2017-09-11 14:15:58 +01:00
1421196d20
HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria.
...
(cherry picked from commit a4661850c1
2017-09-08 14:15:51 -07:00
6e20ae954c
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter.
...
Contributed by Yonger.
(cherry picked from commit 83449abb40
2017-09-07 19:58:40 +01:00
4c34c03b27
HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge.
2017-09-06 01:03:18 -07:00
f382f66255
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS.
...
Contributed by Sivaguru Sankaridurg
2017-09-05 14:18:43 +01:00
2442a8d716
HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria.
...
(cherry picked from commit 7ecc6dbed6
2017-08-31 21:13:50 -07:00
626fa7c8e1
YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu)
2017-08-30 23:08:02 -07:00
25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses.
...
Contributed by Sivaguru Sankaridurg
(cherry picked from commit 021974f4cb
2017-08-30 22:40:50 +01:00
f8f193c8e8
YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh)
...
(cherry picked from commit 8410d862d3
2017-08-20 08:31:17 -07:00
c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted.
...
Contributed by Thomas Marquardt
(cherry picked from commit c6b4e656b7
2017-08-18 14:38:04 +01:00
d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded.
...
Contributed by Thomas Marquardt.
2017-08-16 13:49:47 +01:00
0efc590e6d
HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria.
...
(cherry picked from commit 7769e96149
2017-08-13 00:27:33 -07:00
96befde480
HADOOP-14183. Remove service loader config file for wasb fs.
...
Contributed by Esfandiar Manii.
(cherry picked from commit 54356b1e83
2017-08-10 16:49:51 +01:00
bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing.
...
Contributed by Esfandiar Manii.
2017-08-10 16:49:19 +01:00
7f7ab0302f
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek.
...
Contributed by Thomas Marquardt
(cherry picked from commit d91b7a8451
2017-08-06 20:30:40 +01:00
ced547d5f0
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws.
...
Contributed by Steve Loughran
2017-08-04 11:07:34 +01:00
ba94c68e0b
HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge.
...
(cherry picked from commit f735ad1b67
2017-07-28 09:51:32 -07:00
554c3cd21d
HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt.
2017-07-25 16:35:40 -07:00
ab95086ed1
HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov.
2017-07-25 08:31:35 -07:00
23b920cd7a
HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg.
2017-07-19 11:37:36 -07:00
b9f2effaf6
HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak.
2017-07-17 16:25:05 -07:00
df69b13b47
HADOOP-14658. branch-2 compilation is broken in hadoop-azure
...
Contributed by Sunil G.
Signed-off-by: Steve Loughran <stevel@apache.org>
2017-07-13 18:25:38 +01:00
3dfd107070
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
...
Contributed by Varada Hemeswari
(fix for java 7 compilation)
2017-07-12 11:07:22 +01:00
1e69e52603
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
...
Contributed by Varada Hemeswari
2017-07-12 10:35:59 +01:00
41e83b2ca2
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
...
Contributed by Thomas Marquardt
(cherry picked from commit d670c3a4da
2017-07-11 21:45:24 +01:00
d94b30cb03
HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak.
2017-07-10 17:30:31 -07:00
2d69952925
HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
...
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
(cherry picked from commit 72993b33b7
2017-06-29 17:08:51 -07:00
ba76897392
HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
...
(cherry picked from commit 990aa34de2
2017-06-28 14:20:44 -07:00
44d43a8848
HADOOP-14190. Add more on S3 regions to the s3a documentation.
...
Contributed by Steve Loughran
(cherry picked from commit ee243e5289
2017-06-28 10:26:05 +01:00
ce57458ec5
HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
...
(cherry picked from commit 7e031c2c18
2017-06-27 17:32:56 -07:00
1b1065a163
HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
...
(cherry picked from commit 686a634f01
2017-06-27 16:49:48 -07:00
ea402c364f
HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
...
(cherry picked from commit bc4dfe9c9c
2017-06-27 13:20:03 -07:00
3a87591df8
HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu.
...
(cherry picked from commit 48f4a229a9
2017-06-26 15:22:44 -07:00
239b382153
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he.
...
(cherry picked from commit a594f96fb1
2017-06-26 16:41:44 +09:00
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
...
Contributed by Thomas.
(cherry picked from commit c6bd73c6c5
2017-06-23 21:40:05 +01:00
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools.
2017-06-20 13:21:12 +09:00
ac058c75a4
YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka.
2017-06-19 12:41:13 +05:30
c953e0ce8b
HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
...
(cherry picked from commit 325163f23f
2017-06-09 13:07:35 -07:00
de4e4ec774
HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan
...
(cherry picked from commit df35e4cc3b
2017-06-07 13:55:11 -07:00
e3a72c2b5c
HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
...
(cherry picked from commit 536f057158
2017-06-06 11:13:42 -07:00
bacfd1baae
HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu
...
(cherry picked from commit 6b5285bbcb
2017-06-06 11:07:42 -07:00
95c2978640
YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
...
Contributed by Carlo Curino. Backport by Ye Zhou.
(cherry picked from commit e07f3c87cd
2017-06-05 16:13:51 -07:00
53e90a8064
HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan
...
(cherry picked from commit 5fd9742c83
2017-06-05 16:01:27 -07:00
867026d94e
HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu
...
(cherry picked from commit 6aeda55bb8
2017-06-05 11:31:13 -07:00
0b78db35fd
HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu
...
(cherry picked from commit ece33208b8
2017-06-01 11:52:37 -07:00
f72f194e24
HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
...
(cherry picked from commit 6c6a7a5962
2017-05-28 22:29:03 -07:00
9d7e67ccf9
HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari
...
(cherry picked from commit 89bb8bfe58
2017-05-26 17:59:14 -07:00
dd552a97b7
HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)
2017-05-24 19:05:40 -07:00
49dc547252
HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li.
2017-05-24 18:54:51 -07:00
97317b1aaa
Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li."
...
This reverts commit c411adac32
2017-05-24 18:54:51 -07:00
39474ed50d
Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)"
...
This reverts commit fe185e2c3a
2017-05-24 18:54:50 -07:00
ec346d6377
YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang.
2017-05-23 07:57:36 +05:30
fe185e2c3a
HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)
2017-05-19 21:11:38 -07:00
2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
...
Contributed by Steve Loughran.
2017-05-18 15:38:46 +01:00
f3015425d0
HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg
...
(cherry picked from commit b415c6fe74
2017-05-16 11:28:12 -07:00
18b89e902b
HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy.
2017-05-13 21:41:11 -07:00
586126956b
HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak
...
(cherry picked from commit eed731496f
2017-05-10 15:05:54 -07:00
753954f7be
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong.
...
(cherry picked from commit c60164fb60
2017-05-09 05:39:02 -05:00
616413ecb2
HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran
...
(cherry picked from commit 83dded556d
2017-05-03 11:23:15 -07:00
e84588eb03
HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak
...
(cherry picked from commit 8b5f2c372e
2017-04-26 13:51:58 -07:00
5ceb61cb43
HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu
...
(cherry picked from commit e4c32bccde
2017-04-25 20:55:56 -07:00
d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
...
Contributed by Steve Moist.
2017-04-25 17:42:30 +01:00
3b7bb7b94b
HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran
2017-04-20 17:07:57 -07:00
1ae6be46b4
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
...
Contributed by Steve Loughran
2017-04-19 10:19:43 +01:00
c411adac32
HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li.
2017-04-14 10:14:02 -07:00
5756256280
YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda)
...
(cherry picked from commit eac6b4c35c
2017-04-13 13:16:31 -07:00
03aa142e35
HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg
...
(cherry picked from commit 0cab57223e
2017-04-12 16:09:20 -07:00
508389ee54
HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu
...
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
2017-04-12 14:30:54 -07:00
1bfb382296
HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu.
2017-04-12 10:07:00 -07:00
ff2870310e
HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge.
...
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a20
2017-04-07 15:01:29 -07:00
72aa0a7ab3
HADOOP-13626. Remove distcp dependency on FileStatus serialization
...
(cherry picked from commit a1a0281e12
2017-04-05 15:47:46 -07:00
8ee3e7e4b1
HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak
...
(cherry picked from commit 56ab02eed9
2017-04-04 11:16:48 -07:00
ce8cdc109b
HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak
...
(cherry picked from commit 27a44b60c1
2017-04-04 11:04:42 -07:00
8c21b2a259
HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka
...
(cherry picked from commit 5faa949b78
2017-04-03 11:07:56 -07:00
bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
...
(cherry picked from commit 0d053eeb30
2017-03-30 18:13:01 +09:00
ef43a62482
HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge.
...
(cherry picked from commit 253e3e78ab
2017-03-27 23:07:08 -07:00
100b8ce05b
HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge.
...
(cherry picked from commit d1b7439b48
2017-03-24 13:41:09 -07:00
62b8665726
HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge
...
(cherry picked from commit c55195588f
2017-03-23 12:17:42 -07:00
05148e33ce
HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria
...
(cherry picked from commit 7d4884a57bcaf2ab1ad120c436f9c7e6739dbc2a)
2017-03-23 12:13:33 -07:00
b24b416db5
HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu
...
(cherry picked from commit 2e30aa72e0
2017-03-23 11:37:14 -07:00
20878d052c
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject().
...
Contributed by Yuanbo Liu
2017-03-23 12:52:22 +00:00
84e2c11f96
HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
2017-03-21 08:23:30 -07:00
b4481fef41
HADOOP-14204 S3A multipart commit failing,
...
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
Contributed by Steve Loughran
(cherry picked from commit 595501c9dd
2017-03-21 13:14:43 +00:00
22930b09a0
HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
...
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e24059
2017-03-20 16:37:57 -07:00
90e3283a0a
HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran.
2017-03-20 11:29:30 -07:00
ed0d426a88
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
2017-03-14 15:00:59 -07:00
0da850a472
HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge.
...
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
(cherry picked from commit 5a40bafdae
2017-03-13 08:36:46 -07:00
0d9030b732
HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu
...
(cherry picked from commit 881ec4d97b
2017-03-10 13:22:24 -08:00
851c0df5d3
HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran
...
(cherry picked from commit e60c6543d5
2017-03-10 13:22:02 -08:00
f298b992f4
HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge.
...
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded4
2017-03-09 19:53:23 -08:00
acf20c8318
HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane
...
(cherry picked from commit 924def7854
2017-03-09 15:16:28 -08:00
a146866802
HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane
...
(cherry picked from commit f432999025
2017-03-09 15:16:28 -08:00
ed4388f903
HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei)
...
(cherry picked from commit 71c23c9fc9
2017-03-09 15:16:28 -08:00
d3a6124ffa
HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei)
...
(cherry picked from commit ccf2d66244
2017-03-09 15:16:28 -08:00
01624f0c1b
HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei)
...
(cherry picked from commit e015b56319
2017-03-09 15:16:28 -08:00
881b63499e
HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane
...
(cherry picked from commit ef34bf2bb9
2017-03-09 15:16:28 -08:00
7fd0556b2b
HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane
...
(cherry picked from commit 4113ec5fa5
2017-03-09 15:16:28 -08:00
edf149b979
HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane
2017-03-09 15:16:28 -08:00
f1fd20be82
Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
...
This reverts commit 53a12fa72198142d2f72
2017-03-08 11:46:27 -08:00
d0e26075b8
HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
...
(cherry picked from commit 28daaf0eb2
2017-03-07 14:57:41 -08:00
d2d0872f4d
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri.
...
(cherry picked from commit f597f4c43e
2017-03-07 15:16:18 +09:00
ff050866bf
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
...
(cherry picked from commit 686823529b
2017-03-06 17:35:06 -08:00
8323e651a1
Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
...
This reverts commit b26870c58e
2017-03-06 17:18:36 -08:00
2d059b5a40
HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
...
(cherry picked from commit c571cda5c7
2017-03-06 17:01:01 -08:00
b26870c58e
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-03 17:09:05 -08:00
51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
...
Contributed by Steve Loughran
2017-03-02 21:41:49 +00:00
16ca70553c
HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang)
...
(cherry picked from commit d235dcdf0b
2017-02-27 20:46:17 -08:00
29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
...
Contributed by Steve Loughran.
2017-02-25 15:31:26 +00:00
c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
...
(cherry picked from commit ff87ca8441
2017-02-24 10:40:58 +00:00
650ff95e00
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
...
(cherry picked from commit 0a55bd841e
2017-02-23 17:22:25 -08:00
b0f8de184a
HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
...
(cherry picked from commit 82607fce39
2017-02-23 14:11:08 -08:00
db955f0844
HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu
...
(cherry picked from commit 0013090fb4
2017-02-22 14:38:57 -08:00
171b18693a
HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran.
2017-02-22 11:41:24 +00:00
6843659689
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
2017-02-21 18:29:30 +00:00
274c02d2fa
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
2017-02-20 16:21:00 +00:00
8a88e8ea57
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
...
(cherry picked from commit b1c1f05b1d
2017-02-18 18:17:54 +00:00
b23e93d465
HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
...
(cherry picked from commit 658702efff
2017-02-16 16:59:31 -08:00
29e1007172
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
...
(cherry picked from commit bdad8b7b97
2017-02-16 16:42:56 -08:00
25ac54a52e
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
...
(cherry picked from commit b9f8491252
2017-02-15 01:46:59 +09:00
6d62d0ea87
HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
2017-02-11 15:02:19 -08:00
c88ec54588
HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus.
2017-02-10 10:09:39 -08:00
b204a1eb5f
HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows.
...
(cherry picked from commit 98c35bb1a4
2017-01-21 17:35:57 +09:00
b2704702f9
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fc
2017-01-19 11:25:20 +00:00
ec2f5b18ab
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
2017-01-11 17:24:25 +00:00
447f71326b
YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
...
(cherry picked from commit 287d3d6804
2017-01-09 08:42:38 -08:00
7437252102
YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
...
(cherry picked from commit 0aafc122d4
2017-01-06 11:15:09 -08:00
9c61858ee7
HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
...
(cherry picked from commit 4c8f9e1302
2017-01-05 15:05:01 -08:00
3e2d26a97f
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
...
(cherry picked from commit 01d31fe938
2017-01-04 14:15:00 +09:00
ec646b1a9d
HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
...
(cherry picked from commit c6a5b689db
2016-12-29 16:24:38 -08:00
edac82737a
HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
...
(cherry picked from commit e92a77099b
2016-12-22 20:33:51 -08:00
522bc98026
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
...
(cherry picked from commit ae401539ea
2016-12-23 04:06:53 +09:00
867389247f
HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
...
(cherry picked from commit f92913c35b
2016-12-15 12:38:31 -08:00
2dbd86bcf4
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
...
(cherry picked from commit c6a3923245
2016-12-12 15:06:39 -08:00
89eaa94c59
HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
...
(cherry picked from commit 15dd1f3381
2016-12-05 12:06:35 -08:00
c37e814fae
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
...
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
2016-12-02 13:48:38 -08:00
a4e0539636
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
...
(cherry picked from commit 2ff84a0040
2016-12-02 13:39:23 -08:00
39b74b6f9e
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
...
(cherry picked from commit 3fd844b99f
2016-11-30 13:04:44 -08:00
1b831d8e01
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
...
(cherry picked from commit d60a60be8a
2016-11-28 16:35:18 -08:00
9701ab48e8
HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay.
2016-11-25 13:34:11 +00:00
f3fff5c2e1
HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran
2016-11-23 21:37:04 +00:00
cae4a8c05d
HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran
2016-11-22 14:05:58 -08:00
9af7d1e298
HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
2016-11-22 13:19:32 -08:00
4e0fcff8ab
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
...
(cherry picked from commit 5af572b644
2016-11-15 11:36:08 -08:00
d064d5dbb2
HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth
...
(cherry picked from commit 89354f0475
2016-11-10 08:59:07 -08:00
8bb264c70d
YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri
...
(cherry picked from commit 3f93ac0733
2016-11-08 22:16:53 +00:00
3b279fe426
HADOOP-13798. TestHadoopArchives times out.
...
(cherry picked from commit b970446b2c
2016-11-07 19:54:50 +09:00
c4ccafdaff
HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi.
2016-10-31 20:52:49 +00:00
7e918637de
HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
...
(cherry picked from commit 0bdd263d82
2016-10-27 15:54:03 +09:00
082d69ee66
HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth.
...
(cherry picked from commit 1f8490a5ba
2016-10-26 08:57:29 -07:00
67e01f7218
HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran.
...
(cherry picked from commit 9cad3e2350
2016-10-26 08:27:33 -07:00
e29d0a2fd8
YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton.
...
(cherry picked from commit d3bb69a667
2016-10-26 15:34:40 +09:00
ce72314657
HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang.
...
(cherry picked from commit 0f0c15f7a5
2016-10-25 12:29:28 -07:00
7bc170ba26
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
...
(cherry picked from commit 8650cc84f2
2016-10-25 11:32:26 -07:00
05d772e297
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
...
(cherry picked from commit 309a43925c
2016-10-25 09:03:11 -07:00
5c2f67bdae
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
...
(cherry picked from commit 3372e94030
2016-10-24 21:54:16 -07:00
5b7cbb5a3c
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
...
(cherry picked from commit d8fa1cfa67
2016-10-24 21:22:46 -07:00
385c1daa46
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-21 19:11:31 +01:00
f0af3dee25
Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran."
...
This reverts commit 42f8a1d6eb
2016-10-20 22:48:26 +01:00
42f8a1d6eb
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-20 12:47:57 -07:00
c640ba5fc9
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
2016-10-20 14:50:30 +01:00
bc176961e6
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
2016-10-18 19:33:38 +01:00
a5a56c3564
HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw)
...
(cherry picked from commit 422c73a865
2016-10-17 16:35:32 -07:00
4ed7cf3b36
HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei.
...
(cherry picked from commit 987ee51141
2016-10-17 13:32:39 -07:00
ad69baf6a9
HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin.
...
(cherry picked from commit 0a85d07983
2016-10-13 13:26:02 -07:00
2de2d0df1d
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
...
(cherry picked from commit 69620f9559
2016-10-07 11:41:31 -07:00
a6bb21eec4
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
2016-10-07 12:47:14 +01:00
6ed335a018
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
2016-10-05 15:01:15 +01:00
213f0ad7f0
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
...
(cherry picked from commit 88b9444a81
2016-10-04 10:37:04 -07:00
896df3f55a
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
2016-09-29 17:00:34 +01:00
1f1e47e411
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
...
(cherry picked from commit 47f80922dc
2016-09-28 16:21:00 -07:00
5737d04c5f
HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore
...
(cherry picked from commit e17a4970be
2016-09-27 10:47:07 +05:30
247cdbb86a
HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri.
2016-09-23 10:00:32 +01:00
e19f910245
HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan.
...
(cherry picked from commit 98bdb51397
2016-09-19 15:16:56 -07:00
3b9268cfba
HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran
2016-09-14 15:44:45 +01:00
Steve Loughran
Kai Zheng
Subru Krishnan
Subru Krishnan
Arun Suresh
Akira Ajisaka
Chris Douglas
Wangda Tan
Jason Lowe
Xiaoyu Yao
John Zhuge
Yufei Gu
Jitendra Pandey
Mingliang Liu
Naganarasimha
Yongjun Zhang
Sunil G
Andrew Wang
Chris Nauroth
Arpit Agarwal
Lei Xu
Lei Xu
John Zhuge
Masatake Iwasaki
Anu Engineer
Allen Wittenauer
Jing Zhao
Brahma Reddy Battula