2158496f6b
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
...
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
2017-02-21 18:28:49 +00:00
8035749c26
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
2017-02-20 16:21:46 +00:00
3a2e30fa9f
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
...
(cherry picked from commit b1c1f05b1d
2017-02-18 18:17:11 +00:00
658702efff
HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
2017-02-16 16:51:03 -08:00
bdad8b7b97
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
2017-02-16 16:41:31 -08:00
f432999025
HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane
2017-02-16 15:14:25 -08:00
cd3e59a3dc
HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
2017-02-15 16:34:30 +08:00
b9f8491252
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
2017-02-15 01:45:56 +09:00
71c23c9fc9
HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei)
2017-02-13 13:33:13 -08:00
01be4503c3
HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
2017-02-13 13:18:20 +08:00
839b690ed5
HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
2017-02-11 13:59:03 -08:00
2b7a7bbe0f
HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus.
2017-02-10 10:09:31 -08:00
5b151290ae
HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
2017-02-10 14:53:18 +08:00
a8a594b4c8
HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
2017-02-09 18:00:22 +08:00
98c35bb1a4
HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows.
2017-01-21 17:31:20 +09:00
ccf2d66244
HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei)
2017-01-21 10:52:40 +08:00
e015b56319
HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei)
2017-01-20 14:48:04 +08:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
ed33ce11dd
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fc
2017-01-19 11:25:40 +00:00
e648b6e138
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
2017-01-11 17:25:15 +00:00
287d3d6804
YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
2017-01-09 08:40:39 -08:00
4c8f9e1302
HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
2017-01-05 15:04:47 -08:00
0a55bd841e
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
2017-01-05 10:31:05 -08:00
5ed63e3e9d
YARN-4899. Queue metrics of SLS capacity scheduler only activated after app submit to the queue. (Jonathan Hung via wangda)
2017-01-04 19:17:11 -08:00
01d31fe938
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
2017-01-04 14:10:36 +09:00
c6a5b689db
HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
2016-12-29 16:24:06 -08:00
e92a77099b
HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
2016-12-22 20:15:18 -08:00
ae401539ea
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
2016-12-22 14:32:24 +09:00
3bcfe3a9de
YARN-5706. Fail to launch SLSRunner due to NPE. (Kai Sasaki via wangda)
2016-12-20 11:03:29 -08:00
f92913c35b
HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
2016-12-15 12:35:08 -08:00
ef34bf2bb9
HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane
2016-12-13 10:34:32 -08:00
c6a3923245
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
2016-12-12 14:55:34 -08:00
15dd1f3381
HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
2016-12-05 12:04:07 -08:00
4113ec5fa5
HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane
2016-12-02 15:54:57 -08:00
51211a7d7a
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
2016-12-02 14:01:42 -08:00
2ff84a0040
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
2016-12-02 13:36:04 -08:00
3fd844b99f
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
2016-11-30 13:01:19 -08:00
d60a60be8a
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
2016-11-28 16:30:29 -08:00
209e805430
HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki.
2016-11-28 14:34:57 +09:00
9e173d2bac
HADOOP-13816. Ambiguous plugin version warning from maven build. Contributed by Kai Sasaki.
2016-11-28 13:10:31 +09:00
0de0c32ddd
HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran
2016-11-23 21:37:39 +00:00
af791b774b
HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran
2016-11-22 14:00:35 -08:00
beb70fed4f
HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
...
This closes #131
2016-11-22 13:12:23 -08:00
c0b1a44f6c
HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean Mackrory.
2016-11-17 22:48:35 -06:00
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
5c61ad2488
HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane
2016-11-11 11:15:07 -08:00
89354f0475
HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth
2016-11-10 08:58:37 -08:00
3f93ac0733
YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri
2016-11-08 22:01:26 +00:00
acd509dc57
HADOOP-13795. Skip testGlobStatusThrowsExceptionForUnreadableDir in TestFSMainOperationsSwift. Contributed by John Zhuge.
2016-11-07 09:21:01 -08:00
b970446b2c
HADOOP-13798. TestHadoopArchives times out.
2016-11-07 19:53:43 +09:00
de0132748d
MAPREDUCE-6796. Remove unused properties from JTConfig.java. Contributed by Haibo Chen.
2016-11-05 03:49:19 +09:00
0aafc122d4
YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
2016-11-04 07:31:54 -07:00
20c4d8efab
HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. Contributed by John Zhuge.
2016-11-03 11:16:07 -07:00
a1761a841e
HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi.
2016-10-31 20:54:46 +00:00
5877f20f9c
HADOOP-10075. Update jetty dependency to version 9 (rkanter)
2016-10-27 16:09:00 -07:00
0bdd263d82
HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
2016-10-27 15:46:59 +09:00
1f8490a5ba
HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth.
2016-10-26 08:55:42 -07:00
9cad3e2350
HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran.
2016-10-26 08:27:26 -07:00
d3bb69a667
YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton.
2016-10-26 15:32:07 +09:00
0f0c15f7a5
HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang.
2016-10-25 12:25:40 -07:00
3a60573039
Revert "Fix HDFS-11040"
...
This reverts commit 54c1815790
2016-10-25 12:25:02 -07:00
54c1815790
Fix HDFS-11040
2016-10-25 12:19:34 -07:00
309a43925c
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
2016-10-25 09:03:03 -07:00
3372e94030
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
2016-10-24 21:54:06 -07:00
d8fa1cfa67
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
2016-10-24 21:22:34 -07:00
a1a0281e12
HADOOP-13626. Remove distcp dependency on FileStatus serialization
2016-10-24 12:46:54 -07:00
3fbf4cd5da
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-20 12:33:58 -07:00
9ae270af02
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
2016-10-20 14:50:59 +01:00
8650cc84f2
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-19 17:37:54 -07:00
efdf810cf9
HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge.
2016-10-18 18:18:43 -07:00
6c348c5691
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
2016-10-18 21:16:02 +01:00
0bc6d37f3c
Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang."
...
This reverts commit 412c4c9a34
2016-10-17 22:47:37 -07:00
987ee51141
HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei.
2016-10-17 13:25:58 -07:00
412c4c9a34
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-17 11:04:42 -07:00
d9f73f1b7c
HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
2016-10-13 22:33:55 -07:00
0a85d07983
HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin.
2016-10-13 13:24:54 -07:00
c31b5e61b1
HADOOP-13701. AbstractContractRootDirectoryTest can fail when handling delete "/". Contributed by Genmao Yu
2016-10-08 17:35:59 +06:00
a57bba470b
Merge branch 'trunk' into HADOOP-12756
2016-10-09 10:29:40 +08:00
69620f9559
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
2016-10-07 11:41:19 -07:00
ebd4f39a39
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
2016-10-07 12:51:40 +01:00
2cc841f16e
HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka.
2016-10-06 16:31:00 +01:00
d6be1e75d8
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
2016-10-05 15:02:11 +01:00
6cfe649375
HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. Contributed by Grant Henke.
...
This closes #133
2016-10-05 20:44:39 +09:00
88b9444a81
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
2016-10-04 10:36:58 -07:00
ee0c722dc8
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
2016-09-29 17:01:00 +01:00
47f80922dc
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
2016-09-28 16:19:32 -07:00
d1443988f8
Revert "HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back"
...
This reverts commit 5707f88d85
2016-09-29 09:18:27 +08:00
e17a4970be
HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore
2016-09-27 10:45:12 +05:30
5707f88d85
HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back.
...
HADOOP-12756 branch: Incorporate Aliyun OSS file system implementation. Contributors:
Mingfei Shi (mingfei.shi@intel.com )
Genmao Yu (genmao.ygm@alibaba-inc.com )
2016-09-26 20:42:22 +08:00
e5ef51e717
HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri.
2016-09-23 10:01:30 +01:00
26d5df390c
HADOOP-13634. Some configuration in doc has been outdated. Contributed by Genmao Yu
2016-09-23 08:44:28 +08:00
a49b3be38e
Merge branch 'trunk' into HADOOP-12756
2016-09-21 14:03:21 +08:00
22af6f8db3
HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
2016-09-21 14:02:44 +08:00
08b37603d9
HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
2016-09-20 15:12:02 +08:00
9cd4760257
HADOOP-13609. Refine credential provider related codes for AliyunOss integration. Contributed by Genmao Yu
2016-09-20 11:39:02 +08:00
005f4528c7
Merge branch 'trunk' into HADOOP-12756
2016-09-20 08:09:29 +08:00
98bdb51397
HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan.
2016-09-19 15:16:47 -07:00
a1940464a4
HADOOP-13610. Clean up AliyunOss integration tests. Contributed by Genmao Yu
2016-09-18 19:10:48 +08:00
dbb28eb1f2
Merge branch 'trunk' into HADOOP-12756
2016-09-18 19:03:39 +08:00
96142efa2d
HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu.
2016-09-17 22:07:46 -07:00
53a12fa721
HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran
2016-09-14 15:45:04 +01:00
60f66a9306
Merge branch 'trunk' into HADOOP-12756
2016-09-14 16:12:40 +08:00
58ed4fa544
HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS
...
This commit includes the following changes:
HADOOP-13356. Add a function to handle command_subcommand_OPTS
HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
HADOOP-13357. Modify common to use hadoop_subcommand_opts
HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
HADOOP-13564. modify mapred to use hadoop_subcommand_opts
HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
HADOOP-13360. Documentation for HADOOP_subcommand_OPTS
This closes apache/hadoop#126
2016-09-12 11:10:00 -07:00
9faccd1046
HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic
2016-09-12 08:26:08 -07:00
4b6d795f28
HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran
2016-09-09 18:55:32 +01:00
7fdfcd8a6c
HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran
2016-09-07 12:25:23 +01:00
e671a0f52b
HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
2016-09-07 11:17:43 +08:00
d33e928fbe
HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
cdb77110e7
HADOOP-13498. The number of multi-part upload part should not bigger than 10000. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
6bb741b9f8
HADOOP-13499. Support session credentials for authenticating with Aliyun. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
bd2d97adee
HADOOP-13483. File create should throw error rather than overwrite directories. Contributed by Genmao Yu.
2016-09-07 11:15:47 +08:00
4d84c814fc
HADOOP-13491. Fix several warnings from findbugs. Contributed by Genmao Yu.
2016-09-07 11:15:47 +08:00
a5d5342228
HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
2016-09-07 11:15:47 +08:00
d152557cf7
HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth.
2016-09-06 09:36:21 -07:00
d6d9cff21b
YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh)
2016-08-30 15:52:29 -07:00
8b57be108f
HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth.
...
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
(cherry picked from commit a65e159c59
2016-08-29 15:30:49 -07:00
c37346d0e3
HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton.
2016-08-23 14:12:52 -07:00
6f9c346e57
HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth.
2016-08-23 07:18:49 -07:00
763f0497bb
HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran.
2016-08-19 10:48:10 -07:00
ae4db25443
HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss.
2016-08-18 10:44:36 -07:00
0da69c324d
YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi.
...
(cherry picked from commit d464483bf7
2016-08-18 07:27:23 -07:00
040c185d62
HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie
2016-08-18 14:36:55 +01:00
822d661b8f
HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran.
2016-08-17 14:54:54 -07:00
3808876c73
HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran.
2016-08-16 17:05:52 -07:00
2ed58c40e5
HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka.
2016-08-08 12:28:40 -07:00
4ef1bc0bdb
HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth.
2016-08-02 22:32:21 -07:00
b43de80031
HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg.
2016-07-27 15:50:38 -07:00
54fe17a607
HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák.
2016-07-27 19:41:09 +09:00
86ae218893
HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran
2016-07-25 15:50:23 +01:00
be9e46b42d
HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan
2016-07-25 14:45:03 +01:00
7052ca8804
HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong
2016-07-23 18:00:08 +01:00
c60a68ce1d
HADOOP-13392. [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Contributed by Vishwajeet Dusane and Chris Douglas.
2016-07-22 09:16:38 -07:00
dca6335dbc
HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS documentation
2016-07-21 23:21:47 -07:00
12aa184479
HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley.
2016-07-21 14:43:57 -07:00
37362c2f92
HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan)
2016-07-20 13:42:51 +01:00
da456ffd62
Preparing for 3.0.0-alpha2 development
2016-07-15 19:04:17 -07:00
c2f9cd584c
HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu.
2016-07-15 14:28:53 -07:00
819224dcf9
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:36:20 -07:00
f292624bd8
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
2016-07-11 18:06:28 -07:00
6e597600f7
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 14:58:19 -07:00
aa42c7a6dd
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:34:03 -07:00
96fa0f848b
HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu.
2016-06-29 16:04:50 +01:00
8d202f1258
HADOOP-13235. Use Date and Time API in KafkaSink.
2016-06-29 07:40:12 -07:00
8113855b3a
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
2016-06-28 23:15:13 -07:00
422c73a865
HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw)
2016-06-28 07:21:04 -07:00
4ee3543625
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:45:25 +01:00
b2c596cdda
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
2016-06-21 11:28:52 -07:00
b7c4cf7129
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:25:30 -07:00
51d16e7b38
HADOOP-13242. Authenticate to Azure Data Lake using client ID and keys. Contributed by Atul Sikaria.
2016-06-16 23:35:20 -07:00
127d2c7281
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
2016-06-16 11:18:02 -07:00
4aefe119a0
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
2016-06-16 11:13:35 -07:00
c9e71382a5
HADOOP-12875. [Azure Data Lake] Support for contract test and unit test cases. Contributed by Vishwajeet Dusane.
2016-06-16 09:46:05 -07:00
c77a1095dc
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:06:38 -07:00
9581fb715c
HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane.
2016-06-09 14:33:31 -07:00
58c3486850
HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris Douglas.
2016-06-09 13:49:52 -07:00
31ffaf76f2
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 21:00:47 +01:00
656c460c0e
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 17:28:49 +01:00
a3f78d8fa8
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:42:36 +02:00
c58a59f708
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:55:33 -07:00
16b1cc7af9
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
2016-06-01 14:49:22 -07:00
5870611030
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:18:20 +01:00
42f90ab885
YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 21:24:16 +05:30
cfb860dee7
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
2016-05-26 23:30:31 -07:00
39ec1515a2
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 16:39:31 +01:00
757050ff35
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:52:15 +01:00
c918286b17
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:59 +01:00
93258459fa
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
2016-05-19 22:00:21 -07:00
03788d3015
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
2016-05-17 15:46:30 -07:00
08ea07f1b8
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:19:29 +01:00
c69a649257
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:00:18 +01:00
730bc746f9
HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)
...
This commit contains the following JIRA issues:
HADOOP-12931. bin/hadoop work for dynamic subcommands
HADOOP-12932. bin/yarn work for dynamic subcommands
HADOOP-12933. bin/hdfs work for dynamic subcommands
HADOOP-12934. bin/mapred work for dynamic subcommands
HADOOP-12935. API documentation for dynamic subcommands
HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
HADOOP-13086. enable daemonization of dynamic commands
HADOOP-13087. env var doc update for dynamic commands
HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
HADOOP-13089. hadoop distcp adds client opts twice when dynamic
HADOOP-13094. hadoop-common unit tests for dynamic commands
HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
HADOOP-13107. clean up how rumen is executed
HADOOP-13108. dynamic subcommands need a way to manipulate arguments
HADOOP-13110. add a streaming subcommand to mapred
HADOOP-13111. convert hadoop gridmix to be dynamic
HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
HADOOP-13117. clarify daemonization and security vars for dynamic commands
HADOOP-13120. add a --debug message when dynamic commands have been used
HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
HADOOP-13129. fix typo in dynamic subcommand docs
HADOOP-13151. Underscores should be escaped in dynamic subcommands document
HADOOP-13153. fix typo in debug statement for dynamic subcommands
2016-05-16 17:54:45 -07:00
b9685e85d5
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
2016-05-16 11:53:17 -07:00
dd3a8bed0a
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:47:12 +01:00
fa440a39b7
HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki.
2016-05-12 23:58:18 -07:00
3c5c57af28
HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.
2016-05-12 18:27:28 -07:00
ca5613af91
Revert "Update project version to 3.0.0-alpha1-SNAPSHOT."
...
This reverts commit 6b53802cba
2016-05-12 15:32:45 -07:00
27c4e90efc
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:24:20 +01:00
6b53802cba
Update project version to 3.0.0-alpha1-SNAPSHOT.
2016-05-12 11:05:05 -07:00
3f2816ab5a
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:35:57 +01:00
def2a6d385
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:57:35 +01:00
27242f211e
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
2016-05-10 15:21:06 -07:00
025219b12f
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:37:22 +01:00
996a210ab0
YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan
2016-05-09 14:49:41 -07:00
1c5bbf6499
HADOOP-12866. add a subcommand for gridmix (Kai Sasaki via aw)
2016-05-06 14:07:02 -07:00
d0da13229c
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
2016-05-05 14:07:54 +00:00
af942585a1
HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu.
2016-05-04 10:23:04 -07:00
ef0870ad03
HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA.
2016-04-29 20:25:52 +09:00
341888a0aa
YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
2016-04-26 20:12:12 -07:00
959a28dd12
HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang)
2016-04-26 16:08:03 -07:00
02c51c27d9
HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi.
2016-04-25 22:33:09 +09:00
63e5412f1a
HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou.
2016-04-22 15:14:40 -07:00
19f0f9608e
HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel)
2016-04-22 11:25:03 +01:00
a749ba0cea
HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang)
2016-04-21 12:17:17 -07:00
404f57f328
HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi.
2016-04-14 10:35:22 -07:00
df18b6e984
HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel)
2016-04-14 12:44:55 +01:00
2a5da97f81
Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
...
This reverts commit 1cbcd4a491
2016-04-11 15:51:01 +00:00
b9e3eff62a
HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel
2016-04-09 11:25:31 +01:00
843ee8d59d
HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran.
2016-04-08 13:36:58 -07:00
ce5b4812f0
HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel)
2016-04-08 17:10:00 +01:00
9c32f8785e
HDFS-10266. Remove unused properties dfs.client.file-block-storage-locations.num-threads and dfs.client.file-block-storage-locations.timeout.millis. Contributed by Ray Chiang.
2016-04-08 15:32:13 +09:00
Steve Loughran
Mingliang Liu
Kai Zheng
Akira Ajisaka
Lei Xu
Lei Xu
Arpit Agarwal
Andrew Wang
Wangda Tan
Chris Nauroth
Wei-Chiu Chuang
Chris Douglas
Jason Lowe
Xiao Chen
Arun Suresh
Robert Kanter
Masatake Iwasaki
Yongjun Zhang
Anu Engineer
Jing Zhao
Brahma Reddy Battula
Allen Wittenauer
Ravi Prakash
Mingfei
Xiaoyu Yao
Junping Du
mattf-horton
Jitendra Pandey
Jian He
Varun Vasudev