Steve Loughran
851c0df5d3
HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran
...
(cherry picked from commit e60c6543d5
)
2017-03-10 13:22:02 -08:00
John Zhuge
f298b992f4
HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge.
...
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded4
)
2017-03-09 19:53:23 -08:00
Mingliang Liu
acf20c8318
HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane
...
(cherry picked from commit 924def7854
)
2017-03-09 15:16:28 -08:00
Mingliang Liu
a146866802
HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane
...
(cherry picked from commit f432999025
)
2017-03-09 15:16:28 -08:00
Lei Xu
ed4388f903
HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei)
...
(cherry picked from commit 71c23c9fc9
)
2017-03-09 15:16:28 -08:00
Lei Xu
d3a6124ffa
HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei)
...
(cherry picked from commit ccf2d66244
)
2017-03-09 15:16:28 -08:00
Lei Xu
01624f0c1b
HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei)
...
(cherry picked from commit e015b56319
)
2017-03-09 15:16:28 -08:00
Mingliang Liu
881b63499e
HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane
...
(cherry picked from commit ef34bf2bb9
)
2017-03-09 15:16:28 -08:00
Mingliang Liu
7fd0556b2b
HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane
...
(cherry picked from commit 4113ec5fa5
)
2017-03-09 15:16:28 -08:00
Chris Douglas
edf149b979
HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane
2017-03-09 15:16:28 -08:00
John Zhuge
f1fd20be82
Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
...
This reverts commit 53a12fa721
.
(cherry picked from commit 98142d2f72
)
Change-Id: Ia5fa901fa97391cd46d41f59cd05f7a19c7c7532
2017-03-08 11:46:27 -08:00
Mingliang Liu
d0e26075b8
HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
...
(cherry picked from commit 28daaf0eb2
)
2017-03-07 14:57:41 -08:00
Akira Ajisaka
d2d0872f4d
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri.
...
(cherry picked from commit f597f4c43e
)
2017-03-07 15:16:18 +09:00
Mingliang Liu
ff050866bf
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
...
(cherry picked from commit 686823529b
)
2017-03-06 17:35:06 -08:00
Mingliang Liu
8323e651a1
Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
...
This reverts commit b26870c58e
.
2017-03-06 17:18:36 -08:00
Mingliang Liu
2d059b5a40
HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
...
(cherry picked from commit c571cda5c7
)
2017-03-06 17:01:01 -08:00
Mingliang Liu
b26870c58e
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-03 17:09:05 -08:00
Steve Loughran
51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
...
Contributed by Steve Loughran
2017-03-02 21:41:49 +00:00
Yongjun Zhang
16ca70553c
HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang)
...
(cherry picked from commit d235dcdf0b
)
2017-02-27 20:46:17 -08:00
Steve Loughran
29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
...
Contributed by Steve Loughran.
2017-02-25 15:31:26 +00:00
Steve Loughran
c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
...
(cherry picked from commit ff87ca8441
)
2017-02-24 10:40:58 +00:00
Wangda Tan
650ff95e00
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
...
(cherry picked from commit 0a55bd841e
)
2017-02-23 17:22:25 -08:00
Mingliang Liu
b0f8de184a
HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
...
(cherry picked from commit 82607fce39
)
2017-02-23 14:11:08 -08:00
Mingliang Liu
db955f0844
HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu
...
(cherry picked from commit 0013090fb4
)
2017-02-22 14:38:57 -08:00
Steve Loughran
171b18693a
HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran.
2017-02-22 11:41:24 +00:00
Steve Loughran
6843659689
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
2017-02-21 18:29:30 +00:00
Steve Loughran
274c02d2fa
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
2017-02-20 16:21:00 +00:00
Steve Loughran
8a88e8ea57
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
...
(cherry picked from commit b1c1f05b1d
)
2017-02-18 18:17:54 +00:00
Mingliang Liu
b23e93d465
HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
...
(cherry picked from commit 658702efff
)
2017-02-16 16:59:31 -08:00
Mingliang Liu
29e1007172
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
...
(cherry picked from commit bdad8b7b97
)
2017-02-16 16:42:56 -08:00
Akira Ajisaka
25ac54a52e
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
...
(cherry picked from commit b9f8491252
)
2017-02-15 01:46:59 +09:00
Lei Xu
6d62d0ea87
HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
2017-02-11 15:02:19 -08:00
Arpit Agarwal
c88ec54588
HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus.
2017-02-10 10:09:39 -08:00
Akira Ajisaka
b204a1eb5f
HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows.
...
(cherry picked from commit 98c35bb1a4
)
2017-01-21 17:35:57 +09:00
Steve Loughran
b2704702f9
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fc
)
2017-01-19 11:25:20 +00:00
Steve Loughran
ec2f5b18ab
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
2017-01-11 17:24:25 +00:00
Wangda Tan
447f71326b
YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
...
(cherry picked from commit 287d3d6804
)
2017-01-09 08:42:38 -08:00
Arun Suresh
7437252102
YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
...
(cherry picked from commit 0aafc122d4
)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
2017-01-06 11:15:09 -08:00
Chris Nauroth
9c61858ee7
HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
...
(cherry picked from commit 4c8f9e1302
)
2017-01-05 15:05:01 -08:00
Akira Ajisaka
3e2d26a97f
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
...
(cherry picked from commit 01d31fe938
)
2017-01-04 14:15:00 +09:00
Mingliang Liu
ec646b1a9d
HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
...
(cherry picked from commit c6a5b689db
)
2016-12-29 16:24:38 -08:00
Mingliang Liu
edac82737a
HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
...
(cherry picked from commit e92a77099b
)
2016-12-22 20:33:51 -08:00
Akira Ajisaka
522bc98026
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
...
(cherry picked from commit ae401539ea
)
2016-12-23 04:06:53 +09:00
Mingliang Liu
867389247f
HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
...
(cherry picked from commit f92913c35b
)
2016-12-15 12:38:31 -08:00
Mingliang Liu
2dbd86bcf4
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
...
(cherry picked from commit c6a3923245
)
2016-12-12 15:06:39 -08:00
Mingliang Liu
89eaa94c59
HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
...
(cherry picked from commit 15dd1f3381
)
2016-12-05 12:06:35 -08:00
Mingliang Liu
c37e814fae
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
...
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
2016-12-02 13:48:38 -08:00
Mingliang Liu
a4e0539636
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
...
(cherry picked from commit 2ff84a0040
)
2016-12-02 13:39:23 -08:00
Mingliang Liu
39b74b6f9e
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
...
(cherry picked from commit 3fd844b99f
)
2016-11-30 13:04:44 -08:00
Mingliang Liu
1b831d8e01
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
...
(cherry picked from commit d60a60be8a
)
2016-11-28 16:35:18 -08:00
Steve Loughran
9701ab48e8
HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay.
2016-11-25 13:34:11 +00:00
Steve Loughran
f3fff5c2e1
HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran
2016-11-23 21:37:04 +00:00
Mingliang Liu
cae4a8c05d
HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran
2016-11-22 14:05:58 -08:00
Mingliang Liu
9af7d1e298
HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
2016-11-22 13:19:32 -08:00
Mingliang Liu
4e0fcff8ab
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
...
(cherry picked from commit 5af572b644
)
2016-11-15 11:36:08 -08:00
Mingliang Liu
d064d5dbb2
HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth
...
(cherry picked from commit 89354f0475
)
2016-11-10 08:59:07 -08:00
Jason Lowe
8bb264c70d
YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri
...
(cherry picked from commit 3f93ac0733
)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
2016-11-08 22:16:53 +00:00
Akira Ajisaka
3b279fe426
HADOOP-13798. TestHadoopArchives times out.
...
(cherry picked from commit b970446b2c
)
2016-11-07 19:54:50 +09:00
Steve Loughran
c4ccafdaff
HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi.
2016-10-31 20:52:49 +00:00
Masatake Iwasaki
7e918637de
HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
...
(cherry picked from commit 0bdd263d82
)
2016-10-27 15:54:03 +09:00
Chris Nauroth
082d69ee66
HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth.
...
(cherry picked from commit 1f8490a5ba
)
2016-10-26 08:57:29 -07:00
Chris Nauroth
67e01f7218
HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran.
...
(cherry picked from commit 9cad3e2350
)
2016-10-26 08:27:33 -07:00
Akira Ajisaka
e29d0a2fd8
YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton.
...
(cherry picked from commit d3bb69a667
)
2016-10-26 15:34:40 +09:00
Yongjun Zhang
ce72314657
HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang.
...
(cherry picked from commit 0f0c15f7a5
)
2016-10-25 12:29:28 -07:00
Yongjun Zhang
7bc170ba26
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
...
(cherry picked from commit 8650cc84f2
)
Conflicts
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
2016-10-25 11:32:26 -07:00
Chris Nauroth
05d772e297
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
...
(cherry picked from commit 309a43925c
)
2016-10-25 09:03:11 -07:00
Chris Nauroth
5c2f67bdae
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
...
(cherry picked from commit 3372e94030
)
2016-10-24 21:54:16 -07:00
Chris Nauroth
5b7cbb5a3c
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
...
(cherry picked from commit d8fa1cfa67
)
2016-10-24 21:22:46 -07:00
Steve Loughran
385c1daa46
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-21 19:11:31 +01:00
Steve Loughran
f0af3dee25
Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran."
...
This reverts commit 42f8a1d6eb
.
2016-10-20 22:48:26 +01:00
Anu Engineer
42f8a1d6eb
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-20 12:47:57 -07:00
Steve Loughran
c640ba5fc9
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
2016-10-20 14:50:30 +01:00
Steve Loughran
bc176961e6
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
2016-10-18 19:33:38 +01:00
Allen Wittenauer
a5a56c3564
HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw)
...
(cherry picked from commit 422c73a865
)
2016-10-17 16:35:32 -07:00
Andrew Wang
4ed7cf3b36
HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei.
...
(cherry picked from commit 987ee51141
)
2016-10-17 13:32:39 -07:00
Jing Zhao
ad69baf6a9
HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin.
...
(cherry picked from commit 0a85d07983
)
2016-10-13 13:26:02 -07:00
Chris Nauroth
2de2d0df1d
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
...
(cherry picked from commit 69620f9559
)
2016-10-07 11:41:31 -07:00
Steve Loughran
a6bb21eec4
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
2016-10-07 12:47:14 +01:00
Steve Loughran
6ed335a018
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
2016-10-05 15:01:15 +01:00
Chris Nauroth
213f0ad7f0
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
...
(cherry picked from commit 88b9444a81
)
2016-10-04 10:37:04 -07:00
Steve Loughran
896df3f55a
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
2016-09-29 17:00:34 +01:00
Chris Nauroth
1f1e47e411
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
...
(cherry picked from commit 47f80922dc
)
2016-09-28 16:21:00 -07:00
Brahma Reddy Battula
5737d04c5f
HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore
...
(cherry picked from commit e17a4970be
)
2016-09-27 10:47:07 +05:30
Steve Loughran
247cdbb86a
HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri.
2016-09-23 10:00:32 +01:00
Chris Nauroth
e19f910245
HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan.
...
(cherry picked from commit 98bdb51397
)
2016-09-19 15:16:56 -07:00
Steve Loughran
3b9268cfba
HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran
2016-09-14 15:44:45 +01:00
Steve Loughran
6a016faf59
HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran
2016-09-09 18:54:34 +01:00
Steve Loughran
5f8b6f0313
HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran
2016-09-07 12:24:15 +01:00
Chris Nauroth
e28930a38b
HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d152557cf7
)
2016-09-06 09:40:41 -07:00
Arun Suresh
b279f42d79
YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh)
...
(cherry picked from commit d6d9cff21b
)
2016-08-30 16:33:31 -07:00
Chris Nauroth
cda68de9b9
HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth.
2016-08-23 07:19:20 -07:00
Chris Nauroth
6dba38f08b
HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran.
...
(cherry picked from commit 763f0497bb
)
2016-08-19 12:30:47 -07:00
Chris Nauroth
804a40720b
HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss.
...
(cherry picked from commit ae4db25443
)
2016-08-18 10:45:03 -07:00
Junping Du
d464483bf7
YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi.
2016-08-18 07:23:29 -07:00
Steve Loughran
e39d81be80
HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie
2016-08-18 14:36:41 +01:00
Chris Nauroth
986162f977
HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran.
...
(cherry picked from commit 822d661b8f
)
2016-08-17 14:55:07 -07:00
Chris Nauroth
555a57adf2
HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran.
...
(cherry picked from commit 3808876c73
)
2016-08-16 17:09:28 -07:00
Chris Nauroth
287688d6ca
HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka.
...
(cherry picked from commit 2ed58c40e5
)
2016-08-08 12:28:46 -07:00
Chris Nauroth
8ec3d702a3
HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth.
...
(cherry picked from commit 4ef1bc0bdb
)
2016-08-02 22:32:37 -07:00
Chris Nauroth
59eec8248a
HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg.
...
(cherry picked from commit b43de80031
)
Conflicts:
hadoop-project/pom.xml
2016-07-27 15:52:07 -07:00
Steve Loughran
b757eff30c
HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran
2016-07-25 15:49:54 +01:00
Steve Loughran
271e7bb00b
HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan
2016-07-25 14:44:03 +01:00
Steve Loughran
bdde7edba5
HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong
2016-07-23 17:59:50 +01:00
mattf-horton
ea10e1384f
HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley.
2016-07-21 15:58:05 -07:00
Steve Loughran
d3df810dce
HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel.
2016-07-20 14:01:30 +01:00
Steve Loughran
5490ed5f87
HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan)
2016-07-20 13:42:33 +01:00
Jitendra Pandey
328866905c
HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu.
2016-07-15 14:29:59 -07:00
Jian He
e94e6be772
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:37:52 -07:00
Andrew Wang
38b1eafdbc
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
...
(cherry picked from commit f292624bd8
)
2016-07-11 18:06:35 -07:00
Steve Loughran
89121745c0
HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse.
2016-07-11 17:21:00 +01:00
Akira Ajisaka
f4b63a391b
Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux."
...
This reverts commit 1d175f0f3f
.
2016-07-09 21:00:13 -07:00
Jitendra Pandey
63b64a20c2
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 15:00:34 -07:00
Jitendra Pandey
06b89f0d14
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:35:50 -07:00
Steve Loughran
70a45e3122
HADOOP-13239 Deprecate s3:// in branch-2
2016-06-29 13:16:26 +01:00
Yongjun Zhang
8a121dc3b8
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
...
(cherry picked from commit 8113855b3a
)
2016-06-28 23:29:48 -07:00
Jing Zhao
856bc4e28b
HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu.
2016-06-22 15:15:47 -07:00
Steve Loughran
fb9167fdef
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:42:30 +01:00
Chris Nauroth
e6ebe9ab4e
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
...
(cherry picked from commit b2c596cdda
)
2016-06-21 11:32:11 -07:00
Jitendra Pandey
8f93bbcb11
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:28:55 -07:00
Chris Nauroth
e4b63ed7b4
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
...
(cherry picked from commit 127d2c7281
)
2016-06-16 11:18:40 -07:00
Ravi Prakash
b216dbfb55
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
...
(cherry picked from commit 4aefe119a0
)
2016-06-16 11:14:38 -07:00
Wangda Tan
863bfa4d6a
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:12:00 -07:00
Chris Nauroth
1925942294
Revert "HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane."
...
This reverts commit a8f03ef7ea
.
2016-06-09 15:17:18 -07:00
Chris Nauroth
a8f03ef7ea
HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane.
...
(cherry picked from commit 9581fb715c
)
Conflicts:
hadoop-tools/hadoop-tools-dist/pom.xml
hadoop-tools/pom.xml
2016-06-09 14:49:05 -07:00
Steve Loughran
6350e4b1fb
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 20:59:54 +01:00
Steve Loughran
df29f77846
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 16:36:27 +01:00
Steve Loughran
0cbbc2f0db
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:41:54 +02:00
Chris Nauroth
043a0c2e6c
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:56:07 -07:00
Chris Nauroth
e9942c5b34
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
...
(cherry picked from commit 16b1cc7af9
)
2016-06-01 14:49:31 -07:00
Steve Loughran
b29bf5d335
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:17:50 +01:00
Varun Vasudev
4f36c3d214
YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 20:55:39 +05:30
Yongjun Zhang
82a88c7e67
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
...
(cherry picked from commit cfb860dee7
)
2016-05-26 23:32:46 -07:00
Steve Loughran
15f997cca6
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 14:19:56 +01:00
Steve Loughran
eb160561e0
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:52:30 +01:00
Steve Loughran
e29d7fb585
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:47 +01:00
Chris Nauroth
60e5397a9b
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
...
(cherry picked from commit 93258459fa
)
2016-05-19 22:00:32 -07:00
Arun Suresh
1b4600abcf
YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
...
(cherry picked from commit 341888a0aa
)
2016-05-18 22:35:55 -07:00
Jing Zhao
dd7f5a3b88
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
...
(cherry picked from commit 03788d3015
)
2016-05-17 15:53:39 -07:00
Steve Loughran
dd0a8201bc
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:19:08 +01:00
Steve Loughran
dfc062a092
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:01:50 +01:00
Chris Nauroth
bb41547fb4
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
...
(cherry picked from commit b9685e85d5
)
2016-05-16 11:53:25 -07:00
Steve Loughran
f14208d362
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:46:37 +01:00
Andrew Wang
f6f20616c6
HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki.
...
(cherry picked from commit fa440a39b7
)
2016-05-12 23:58:24 -07:00
Steve Loughran
1370dfc775
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:23:31 +01:00
Steve Loughran
389dd91dd4
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:36:23 +01:00
Steve Loughran
257b2f5beb
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:57:05 +01:00
Chris Nauroth
ba48834431
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
2016-05-10 15:14:36 -07:00
Steve Loughran
f47ea0cc3f
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:40:13 +01:00
Jian He
007d6d1d57
YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan
...
(cherry picked from commit 996a210ab0
)
2016-05-09 14:50:00 -07:00
Jason Lowe
ee86cef2fe
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
...
(cherry picked from commit d0da13229c
)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00