d1b7439b48
HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge.
2017-03-24 13:37:04 -07:00
52b00600df
HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy.
2017-03-24 11:44:46 -07:00
c55195588f
HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge
2017-03-23 12:17:27 -07:00
03955612b7
HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria
2017-03-23 12:13:23 -07:00
2e30aa72e0
HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu
2017-03-23 11:33:29 -07:00
a5a4867f3b
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject().
...
Contributed by Yuanbo Liu
(cherry picked from commit 20878d052c
2017-03-23 12:54:54 +00:00
cc938e99ec
YARN-6326. Shouldn't use AppAttemptIds to fetch applications while AM Simulator tracks app in SLS (yufeigu via rkanter)
2017-03-21 15:21:11 -07:00
2841666f1f
HADOOP-14204 S3A multipart commit failing,
...
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
Contributed by Steve Loughran
(cherry picked from commit 595501c9dd
2017-03-21 13:15:15 +00:00
8e15e24059
HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
...
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
2017-03-20 16:31:24 -07:00
6c399a88e9
HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran.
2017-03-20 11:29:23 -07:00
86035c1644
HADOOP-14192. Aliyun OSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
2017-03-17 11:01:00 -07:00
fa67a96d7b
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
2017-03-14 14:38:21 -07:00
e6cda5819b
Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu"
...
This reverts commit b8c69557b7
2017-03-14 13:03:42 -07:00
b8c69557b7
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
2017-03-13 14:15:02 -07:00
5a40bafdae
HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge.
...
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
2017-03-13 08:11:25 -07:00
092ec39fb9
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials.
...
Contributed by Yuanbo Liu
2017-03-10 17:43:22 +00:00
881ec4d97b
HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu
2017-03-10 00:16:09 -08:00
c5ee7fded4
HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge.
...
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
2017-03-09 18:30:17 -08:00
98142d2f72
Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
...
This reverts commit 53a12fa721
2017-03-08 09:54:22 -08:00
28daaf0eb2
HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
2017-03-07 14:55:52 -08:00
1598fd3b79
YARN-6275. Fail to show real-time tracking charts in SLS (yufeigu via rkanter)
2017-03-07 13:47:52 -08:00
f597f4c43e
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri.
2017-03-07 15:14:55 +09:00
686823529b
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-06 17:16:36 -08:00
52d7d5aa1a
Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
...
This reverts commit 6b7cd62b8c
2017-03-06 17:10:11 -08:00
c571cda5c7
HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
2017-03-06 17:00:13 -08:00
6b7cd62b8c
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-03 17:09:12 -08:00
490abfb10f
HADOOP-14057. Fix package.html to compile with Java 9.
2017-03-04 00:25:22 +09:00
a97833e0ed
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
...
Contributed by Steve Loughran
(cherry picked from commit 51fefbe38b
2017-03-02 21:44:30 +00:00
989bd56b9f
HADOOP-12556. KafkaSink jar files are created but not copied to target dist (Babak Behzad via aw)
2017-02-28 16:15:40 -08:00
eac6b4c35c
YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda)
2017-02-28 10:35:50 -08:00
258342e76c
HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR.
2017-02-28 13:47:44 +09:00
d235dcdf0b
HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang)
2017-02-27 20:42:13 -08:00
dab00da19f
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
...
Contributed by Steve Loughran.
(cherry picked from commit 29fe5af017
2017-02-25 15:35:19 +00:00
b32ffa2753
YARN-4779. Fix AM container allocation logic in SLS. Contributed by Wangda Tan.
2017-02-24 21:39:25 +05:30
e60c6543d5
HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran
2017-02-24 13:24:59 +00:00
9c22a91662
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
...
(cherry picked from commit ff87ca8441
2017-02-24 10:41:36 +00:00
82607fce39
HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
2017-02-23 13:48:44 -08:00
a4d4a23785
HADOOP-13321. Deprecate FileSystem APIs that promote inefficient call patterns. Contributed by Chris Nauroth and Mingliang Liu
2017-02-23 12:55:40 -08:00
0013090fb4
HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu
2017-02-22 13:34:20 -08:00
4f4250fbcc
HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran.
2017-02-22 11:43:48 +00:00
924def7854
HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane
2017-02-21 13:44:42 -08:00
2158496f6b
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
...
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
2017-02-21 18:28:49 +00:00
8035749c26
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
2017-02-20 16:21:46 +00:00
3a2e30fa9f
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
...
(cherry picked from commit b1c1f05b1d
2017-02-18 18:17:11 +00:00
658702efff
HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
2017-02-16 16:51:03 -08:00
bdad8b7b97
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
2017-02-16 16:41:31 -08:00
f432999025
HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane
2017-02-16 15:14:25 -08:00
cd3e59a3dc
HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
2017-02-15 16:34:30 +08:00
b9f8491252
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
2017-02-15 01:45:56 +09:00
71c23c9fc9
HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei)
2017-02-13 13:33:13 -08:00
01be4503c3
HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
2017-02-13 13:18:20 +08:00
839b690ed5
HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
2017-02-11 13:59:03 -08:00
2b7a7bbe0f
HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus.
2017-02-10 10:09:31 -08:00
5b151290ae
HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
2017-02-10 14:53:18 +08:00
a8a594b4c8
HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
2017-02-09 18:00:22 +08:00
98c35bb1a4
HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows.
2017-01-21 17:31:20 +09:00
ccf2d66244
HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei)
2017-01-21 10:52:40 +08:00
e015b56319
HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei)
2017-01-20 14:48:04 +08:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
ed33ce11dd
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fc
2017-01-19 11:25:40 +00:00
e648b6e138
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
2017-01-11 17:25:15 +00:00
287d3d6804
YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
2017-01-09 08:40:39 -08:00
4c8f9e1302
HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
2017-01-05 15:04:47 -08:00
0a55bd841e
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
2017-01-05 10:31:05 -08:00
5ed63e3e9d
YARN-4899. Queue metrics of SLS capacity scheduler only activated after app submit to the queue. (Jonathan Hung via wangda)
2017-01-04 19:17:11 -08:00
01d31fe938
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
2017-01-04 14:10:36 +09:00
c6a5b689db
HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
2016-12-29 16:24:06 -08:00
e92a77099b
HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
2016-12-22 20:15:18 -08:00
ae401539ea
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
2016-12-22 14:32:24 +09:00
3bcfe3a9de
YARN-5706. Fail to launch SLSRunner due to NPE. (Kai Sasaki via wangda)
2016-12-20 11:03:29 -08:00
f92913c35b
HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
2016-12-15 12:35:08 -08:00
ef34bf2bb9
HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane
2016-12-13 10:34:32 -08:00
c6a3923245
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
2016-12-12 14:55:34 -08:00
15dd1f3381
HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
2016-12-05 12:04:07 -08:00
4113ec5fa5
HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane
2016-12-02 15:54:57 -08:00
51211a7d7a
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
2016-12-02 14:01:42 -08:00
2ff84a0040
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
2016-12-02 13:36:04 -08:00
3fd844b99f
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
2016-11-30 13:01:19 -08:00
d60a60be8a
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
2016-11-28 16:30:29 -08:00
209e805430
HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki.
2016-11-28 14:34:57 +09:00
9e173d2bac
HADOOP-13816. Ambiguous plugin version warning from maven build. Contributed by Kai Sasaki.
2016-11-28 13:10:31 +09:00
0de0c32ddd
HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran
2016-11-23 21:37:39 +00:00
af791b774b
HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran
2016-11-22 14:00:35 -08:00
beb70fed4f
HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran
...
This closes #131
2016-11-22 13:12:23 -08:00
c0b1a44f6c
HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean Mackrory.
2016-11-17 22:48:35 -06:00
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
5c61ad2488
HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane
2016-11-11 11:15:07 -08:00
89354f0475
HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth
2016-11-10 08:58:37 -08:00
3f93ac0733
YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri
2016-11-08 22:01:26 +00:00
acd509dc57
HADOOP-13795. Skip testGlobStatusThrowsExceptionForUnreadableDir in TestFSMainOperationsSwift. Contributed by John Zhuge.
2016-11-07 09:21:01 -08:00
b970446b2c
HADOOP-13798. TestHadoopArchives times out.
2016-11-07 19:53:43 +09:00
de0132748d
MAPREDUCE-6796. Remove unused properties from JTConfig.java. Contributed by Haibo Chen.
2016-11-05 03:49:19 +09:00
0aafc122d4
YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
2016-11-04 07:31:54 -07:00
20c4d8efab
HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. Contributed by John Zhuge.
2016-11-03 11:16:07 -07:00
a1761a841e
HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi.
2016-10-31 20:54:46 +00:00
5877f20f9c
HADOOP-10075. Update jetty dependency to version 9 (rkanter)
2016-10-27 16:09:00 -07:00
0bdd263d82
HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
2016-10-27 15:46:59 +09:00
1f8490a5ba
HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth.
2016-10-26 08:55:42 -07:00
9cad3e2350
HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran.
2016-10-26 08:27:26 -07:00
d3bb69a667
YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton.
2016-10-26 15:32:07 +09:00
0f0c15f7a5
HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang.
2016-10-25 12:25:40 -07:00
3a60573039
Revert "Fix HDFS-11040"
...
This reverts commit 54c1815790
2016-10-25 12:25:02 -07:00
54c1815790
Fix HDFS-11040
2016-10-25 12:19:34 -07:00
309a43925c
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
2016-10-25 09:03:03 -07:00
3372e94030
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
2016-10-24 21:54:06 -07:00
d8fa1cfa67
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
2016-10-24 21:22:34 -07:00
a1a0281e12
HADOOP-13626. Remove distcp dependency on FileStatus serialization
2016-10-24 12:46:54 -07:00
3fbf4cd5da
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-20 12:33:58 -07:00
9ae270af02
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
2016-10-20 14:50:59 +01:00
8650cc84f2
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-19 17:37:54 -07:00
efdf810cf9
HADOOP-7352. FileSystem#listStatus should throw IOE upon access error. Contributed by John Zhuge.
2016-10-18 18:18:43 -07:00
6c348c5691
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
2016-10-18 21:16:02 +01:00
0bc6d37f3c
Revert "HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang."
...
This reverts commit 412c4c9a34
2016-10-17 22:47:37 -07:00
987ee51141
HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei.
2016-10-17 13:25:58 -07:00
412c4c9a34
HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang.
2016-10-17 11:04:42 -07:00
d9f73f1b7c
HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
2016-10-13 22:33:55 -07:00
0a85d07983
HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin.
2016-10-13 13:24:54 -07:00
c31b5e61b1
HADOOP-13701. AbstractContractRootDirectoryTest can fail when handling delete "/". Contributed by Genmao Yu
2016-10-08 17:35:59 +06:00
a57bba470b
Merge branch 'trunk' into HADOOP-12756
2016-10-09 10:29:40 +08:00
69620f9559
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
2016-10-07 11:41:19 -07:00
ebd4f39a39
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
2016-10-07 12:51:40 +01:00
2cc841f16e
HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka.
2016-10-06 16:31:00 +01:00
d6be1e75d8
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
2016-10-05 15:02:11 +01:00
6cfe649375
HADOOP-13681. Reduce Kafka dependencies in hadoop-kafka module. Contributed by Grant Henke.
...
This closes #133
2016-10-05 20:44:39 +09:00
88b9444a81
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
2016-10-04 10:36:58 -07:00
ee0c722dc8
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
2016-09-29 17:01:00 +01:00
47f80922dc
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
2016-09-28 16:19:32 -07:00
d1443988f8
Revert "HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back"
...
This reverts commit 5707f88d85
2016-09-29 09:18:27 +08:00
e17a4970be
HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore
2016-09-27 10:45:12 +05:30
5707f88d85
HADOOP-13584. hdoop-aliyun: merge HADOOP-12756 branch back.
...
HADOOP-12756 branch: Incorporate Aliyun OSS file system implementation. Contributors:
Mingfei Shi (mingfei.shi@intel.com )
Genmao Yu (genmao.ygm@alibaba-inc.com )
2016-09-26 20:42:22 +08:00
e5ef51e717
HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri.
2016-09-23 10:01:30 +01:00
26d5df390c
HADOOP-13634. Some configuration in doc has been outdated. Contributed by Genmao Yu
2016-09-23 08:44:28 +08:00
a49b3be38e
Merge branch 'trunk' into HADOOP-12756
2016-09-21 14:03:21 +08:00
22af6f8db3
HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
2016-09-21 14:02:44 +08:00
08b37603d9
HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
2016-09-20 15:12:02 +08:00
9cd4760257
HADOOP-13609. Refine credential provider related codes for AliyunOss integration. Contributed by Genmao Yu
2016-09-20 11:39:02 +08:00
005f4528c7
Merge branch 'trunk' into HADOOP-12756
2016-09-20 08:09:29 +08:00
98bdb51397
HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan.
2016-09-19 15:16:47 -07:00
a1940464a4
HADOOP-13610. Clean up AliyunOss integration tests. Contributed by Genmao Yu
2016-09-18 19:10:48 +08:00
dbb28eb1f2
Merge branch 'trunk' into HADOOP-12756
2016-09-18 19:03:39 +08:00
96142efa2d
HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu.
2016-09-17 22:07:46 -07:00
53a12fa721
HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran
2016-09-14 15:45:04 +01:00
60f66a9306
Merge branch 'trunk' into HADOOP-12756
2016-09-14 16:12:40 +08:00
58ed4fa544
HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS
...
This commit includes the following changes:
HADOOP-13356. Add a function to handle command_subcommand_OPTS
HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
HADOOP-13357. Modify common to use hadoop_subcommand_opts
HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
HADOOP-13564. modify mapred to use hadoop_subcommand_opts
HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
HADOOP-13360. Documentation for HADOOP_subcommand_OPTS
This closes apache/hadoop#126
2016-09-12 11:10:00 -07:00
9faccd1046
HADOOP-13587. distcp.map.bandwidth.mb is overwritten even when -bandwidth flag isn't set. Contributed by Zoran Dimitrijevic
2016-09-12 08:26:08 -07:00
4b6d795f28
HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran
2016-09-09 18:55:32 +01:00
7fdfcd8a6c
HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran
2016-09-07 12:25:23 +01:00
e671a0f52b
HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
2016-09-07 11:17:43 +08:00
d33e928fbe
HADOOP-13529. Do some code refactoring. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
cdb77110e7
HADOOP-13498. The number of multi-part upload part should not bigger than 10000. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
6bb741b9f8
HADOOP-13499. Support session credentials for authenticating with Aliyun. Contributed by Genmao Yu.
2016-09-07 11:15:48 +08:00
bd2d97adee
HADOOP-13483. File create should throw error rather than overwrite directories. Contributed by Genmao Yu.
2016-09-07 11:15:47 +08:00
4d84c814fc
HADOOP-13491. Fix several warnings from findbugs. Contributed by Genmao Yu.
2016-09-07 11:15:47 +08:00
a5d5342228
HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
2016-09-07 11:15:47 +08:00
d152557cf7
HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth.
2016-09-06 09:36:21 -07:00
d6d9cff21b
YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh)
2016-08-30 15:52:29 -07:00
8b57be108f
HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth.
...
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
(cherry picked from commit a65e159c59
2016-08-29 15:30:49 -07:00
c37346d0e3
HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton.
2016-08-23 14:12:52 -07:00
6f9c346e57
HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth.
2016-08-23 07:18:49 -07:00
763f0497bb
HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran.
2016-08-19 10:48:10 -07:00
ae4db25443
HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss.
2016-08-18 10:44:36 -07:00
0da69c324d
YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi.
...
(cherry picked from commit d464483bf7
2016-08-18 07:27:23 -07:00
040c185d62
HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie
2016-08-18 14:36:55 +01:00
822d661b8f
HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran.
2016-08-17 14:54:54 -07:00
3808876c73
HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran.
2016-08-16 17:05:52 -07:00
2ed58c40e5
HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka.
2016-08-08 12:28:40 -07:00
4ef1bc0bdb
HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth.
2016-08-02 22:32:21 -07:00
b43de80031
HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg.
2016-07-27 15:50:38 -07:00
54fe17a607
HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák.
2016-07-27 19:41:09 +09:00
86ae218893
HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran
2016-07-25 15:50:23 +01:00
be9e46b42d
HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan
2016-07-25 14:45:03 +01:00
7052ca8804
HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong
2016-07-23 18:00:08 +01:00
c60a68ce1d
HADOOP-13392. [Azure Data Lake] OAuth2 configuration should be default set to true when AdlFileSystem is configured. Contributed by Vishwajeet Dusane and Chris Douglas.
2016-07-22 09:16:38 -07:00
dca6335dbc
HADOOP-13393. Omit unsupported fs.defaultFS setting in ADLS documentation
2016-07-21 23:21:47 -07:00
12aa184479
HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley.
2016-07-21 14:43:57 -07:00
37362c2f92
HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan)
2016-07-20 13:42:51 +01:00
da456ffd62
Preparing for 3.0.0-alpha2 development
2016-07-15 19:04:17 -07:00
c2f9cd584c
HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu.
2016-07-15 14:28:53 -07:00
819224dcf9
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:36:20 -07:00
f292624bd8
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
2016-07-11 18:06:28 -07:00
6e597600f7
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 14:58:19 -07:00
aa42c7a6dd
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:34:03 -07:00
96fa0f848b
HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu.
2016-06-29 16:04:50 +01:00
8d202f1258
HADOOP-13235. Use Date and Time API in KafkaSink.
2016-06-29 07:40:12 -07:00
8113855b3a
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
2016-06-28 23:15:13 -07:00
422c73a865
HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw)
2016-06-28 07:21:04 -07:00
4ee3543625
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:45:25 +01:00
b2c596cdda
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
2016-06-21 11:28:52 -07:00
b7c4cf7129
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:25:30 -07:00
51d16e7b38
HADOOP-13242. Authenticate to Azure Data Lake using client ID and keys. Contributed by Atul Sikaria.
2016-06-16 23:35:20 -07:00
127d2c7281
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
2016-06-16 11:18:02 -07:00
4aefe119a0
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
2016-06-16 11:13:35 -07:00
c9e71382a5
HADOOP-12875. [Azure Data Lake] Support for contract test and unit test cases. Contributed by Vishwajeet Dusane.
2016-06-16 09:46:05 -07:00
c77a1095dc
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:06:38 -07:00
9581fb715c
HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane.
2016-06-09 14:33:31 -07:00
58c3486850
HADOOP-13175. Remove hadoop-ant from hadoop-tools. Contributed by Chris Douglas.
2016-06-09 13:49:52 -07:00
31ffaf76f2
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 21:00:47 +01:00
656c460c0e
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 17:28:49 +01:00
a3f78d8fa8
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:42:36 +02:00
c58a59f708
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:55:33 -07:00
16b1cc7af9
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
2016-06-01 14:49:22 -07:00
5870611030
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:18:20 +01:00
42f90ab885
YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 21:24:16 +05:30
cfb860dee7
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
2016-05-26 23:30:31 -07:00
39ec1515a2
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 16:39:31 +01:00
757050ff35
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:52:15 +01:00
c918286b17
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:59 +01:00
93258459fa
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
2016-05-19 22:00:21 -07:00
03788d3015
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
2016-05-17 15:46:30 -07:00
08ea07f1b8
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:19:29 +01:00
c69a649257
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:00:18 +01:00
730bc746f9
HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)
...
This commit contains the following JIRA issues:
HADOOP-12931. bin/hadoop work for dynamic subcommands
HADOOP-12932. bin/yarn work for dynamic subcommands
HADOOP-12933. bin/hdfs work for dynamic subcommands
HADOOP-12934. bin/mapred work for dynamic subcommands
HADOOP-12935. API documentation for dynamic subcommands
HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
HADOOP-13086. enable daemonization of dynamic commands
HADOOP-13087. env var doc update for dynamic commands
HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
HADOOP-13089. hadoop distcp adds client opts twice when dynamic
HADOOP-13094. hadoop-common unit tests for dynamic commands
HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
HADOOP-13107. clean up how rumen is executed
HADOOP-13108. dynamic subcommands need a way to manipulate arguments
HADOOP-13110. add a streaming subcommand to mapred
HADOOP-13111. convert hadoop gridmix to be dynamic
HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
HADOOP-13117. clarify daemonization and security vars for dynamic commands
HADOOP-13120. add a --debug message when dynamic commands have been used
HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
HADOOP-13129. fix typo in dynamic subcommand docs
HADOOP-13151. Underscores should be escaped in dynamic subcommands document
HADOOP-13153. fix typo in debug statement for dynamic subcommands
2016-05-16 17:54:45 -07:00
b9685e85d5
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
2016-05-16 11:53:17 -07:00
dd3a8bed0a
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:47:12 +01:00
fa440a39b7
HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki.
2016-05-12 23:58:18 -07:00
3c5c57af28
HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.
2016-05-12 18:27:28 -07:00
ca5613af91
Revert "Update project version to 3.0.0-alpha1-SNAPSHOT."
...
This reverts commit 6b53802cba
2016-05-12 15:32:45 -07:00
27c4e90efc
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:24:20 +01:00
6b53802cba
Update project version to 3.0.0-alpha1-SNAPSHOT.
2016-05-12 11:05:05 -07:00
3f2816ab5a
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:35:57 +01:00
def2a6d385
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:57:35 +01:00
27242f211e
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
2016-05-10 15:21:06 -07:00
025219b12f
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:37:22 +01:00
996a210ab0
YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan
2016-05-09 14:49:41 -07:00
1c5bbf6499
HADOOP-12866. add a subcommand for gridmix (Kai Sasaki via aw)
2016-05-06 14:07:02 -07:00
d0da13229c
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
2016-05-05 14:07:54 +00:00
af942585a1
HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu.
2016-05-04 10:23:04 -07:00
ef0870ad03
HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA.
2016-04-29 20:25:52 +09:00
341888a0aa
YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
2016-04-26 20:12:12 -07:00
959a28dd12
HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang)
2016-04-26 16:08:03 -07:00
02c51c27d9
HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi.
2016-04-25 22:33:09 +09:00
63e5412f1a
HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou.
2016-04-22 15:14:40 -07:00
19f0f9608e
HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel)
2016-04-22 11:25:03 +01:00
a749ba0cea
HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang)
2016-04-21 12:17:17 -07:00
404f57f328
HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi.
2016-04-14 10:35:22 -07:00
df18b6e984
HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel)
2016-04-14 12:44:55 +01:00
2a5da97f81
Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
...
This reverts commit 1cbcd4a491
2016-04-11 15:51:01 +00:00
b9e3eff62a
HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel
2016-04-09 11:25:31 +01:00
843ee8d59d
HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran.
2016-04-08 13:36:58 -07:00
ce5b4812f0
HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel)
2016-04-08 17:10:00 +01:00
9c32f8785e
HDFS-10266. Remove unused properties dfs.client.file-block-storage-locations.num-threads and dfs.client.file-block-storage-locations.timeout.millis. Contributed by Ray Chiang.
2016-04-08 15:32:13 +09:00
1cbcd4a491
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
2016-04-05 13:40:19 +00:00
89c9347511
HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass
2016-04-04 17:02:04 +01:00
0ecdd4cffa
HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.
2016-04-03 16:40:19 +01:00
256c82fe29
HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh.
2016-04-01 14:18:10 +05:30
0a74610d1c
HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw)
2016-03-31 07:51:05 -07:00
0050fa5f1c
MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote.
2016-03-29 18:17:52 +09:00
115be193df
Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote."
...
This reverts commit 2c268cc936
2016-03-28 16:05:35 +09:00
18c7e58283
HDFS-9640. Remove hsftp from DistCp in trunk. Contributed by Wei-Chiu Chuang.
2016-03-28 15:32:38 +09:00
2c268cc936
MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote.
2016-03-25 15:17:27 +09:00
738155063e
HADOOP-12857. rework hadoop-tools (aw)
2016-03-23 13:46:38 -07:00
c50aad0f85
HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade.
2016-03-04 22:57:43 -08:00
cbd31328a6
HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims)
2016-03-04 14:11:36 +09:00
33a412e8a4
HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims)
2016-03-03 18:57:23 +09:00
f98dff329b
HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty.
2016-03-01 13:18:42 -08:00
d251e55415
HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay.
2016-02-29 21:59:52 -08:00
809ebc0b14
HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims)
2016-02-29 18:20:14 +09:00
7634d404b7
HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay.
2016-02-28 11:22:55 -08:00
3369a4f691
HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty.
2016-02-23 21:37:50 -08:00
27b77751c1
HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay.
2016-02-22 17:30:13 -08:00
d4f5fc23b2
HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki.
2016-02-22 13:39:38 -08:00
ba1c9d484a
HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang)
2016-02-19 10:17:37 -08:00
8ab7658025
HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay.
2016-02-18 14:06:38 -08:00
76fab26c5c
HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay.
2016-02-17 12:19:59 -08:00
91a96eaa53
HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty.
2016-02-12 15:50:10 -08:00
29ae258013
HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel)
2016-02-06 15:05:16 +00:00
700a176fa7
HADOOP-12757. Findbug compilation fails for 'Kafka Library support'. (aajisaka)
2016-02-02 23:15:39 +09:00
eddd823cd6
HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)
2016-01-29 12:11:55 -08:00
41da9a0773
MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth.
2016-01-28 14:46:01 -08:00
4efdf3a979
YARN-4612. Fix rumen and scheduler load simulator handle killed tasks
...
properly. Contributed by Ming Ma.
2016-01-26 18:17:12 -08:00
736eb17a79
HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
2016-01-25 13:47:29 +09:00
0bae506c22
HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki.
2016-01-22 21:08:31 +09:00
1708a4cd23
YARN-4611. Fix scheduler load simulator to support multi-layer network
...
location. Contributed by Ming Ma.
2016-01-20 15:41:50 -08:00
1acc509b45
HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano
2016-01-19 13:57:08 -08:00
89d1fd5dac
HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda)
2016-01-19 21:27:38 +08:00
a44ce3f14f
YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)
2016-01-19 09:30:04 +08:00
150f5ae034
Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)"
...
This reverts commit 3fe5728563
2016-01-19 09:27:36 +08:00
8bc93db2e7
HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth.
2016-01-18 09:08:58 -08:00
3fe5728563
YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)
...
(cherry picked from commit 805a9ed85e
2016-01-18 17:06:05 +08:00
adf260a728
Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)"
...
This reverts commit 805a9ed85e
2016-01-18 16:50:45 +08:00
805a9ed85e
YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)
2016-01-18 11:04:25 +08:00
a9c69ebeb7
HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang)
2016-01-15 10:03:09 -08:00
b1ed28fa77
HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe)
2016-01-14 11:02:34 -08:00
8315582c4f
HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe)
2016-01-13 16:28:06 -08:00
126705f67e
HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei)
2016-01-12 12:19:53 -08:00
c2e2e13455
HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore.
2016-01-11 17:08:26 -08:00
95f32015ad
HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng.
...
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
2016-01-11 09:46:56 -08:00
0e76f1fcea
HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth.
2016-01-09 22:18:11 -08:00
f0fa6d869b
HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty.
2016-01-08 20:18:59 -08:00
978bbdfeb2
HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade.
2016-01-06 11:15:59 -08:00
2d16f40dab
HADOOP-12689. S3 filesystem operations stopped working correctly
2016-01-05 23:26:03 -08:00
a9594c61bb
YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda)
...
(cherry picked from commit 6eefae1b33
2015-12-30 15:39:27 -08:00
8310b2e9ff
YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda)
2015-12-30 15:30:12 -08:00
79c41b1d83
YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda)
2015-12-16 13:18:19 -08:00
6d84cc16b3
MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter)
2015-11-25 17:12:40 -08:00
bab03f3ee6
HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang.
2015-11-26 01:21:20 +09:00
52948bb20b
YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha)
2015-11-24 13:47:17 +05:30
dc46c46b91
HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux.
2015-11-22 18:10:08 -08:00
0d27917042
HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena.
2015-11-22 17:25:22 -08:00
176ff5ce90
HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su.
2015-11-22 15:54:41 -08:00
f94d892704
HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad.
2015-11-13 21:52:57 +09:00
3ae82863f0
HADOOP-12553. [JDK8] Fix javadoc error caused by illegal tag. (aajisaka)
2015-11-08 19:17:57 +09:00
bff7c90a56
HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei)
2015-11-05 18:35:15 -08:00
19a0c2660c
MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with SerialUtils.hh (Simone Leo via aw)
2015-11-05 16:57:22 -08:00
fc7cd46faf
MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes (Joe Mudd via aw)
2015-11-05 16:53:32 -08:00
0fb1867fd6
HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown. Contributed by Gaurav Kanade.
2015-11-04 10:28:44 -08:00
73b9c7b82b
HADOOP-10787. Rename/remove non-HADOOP_*, etc from the shell scripts. Contributed by Allen Wittenauer.
2015-11-04 15:56:17 +05:30
9e7dcab185
HADOOP-12508. delete fails with exception when lease is held on blob. Contributed by Gaurav Kanade.
2015-11-02 10:21:39 -08:00
3ce0a6502e
HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. Contributed by Dushyanth.
2015-11-02 10:17:41 -08:00
b24fe06483
Addendum to MAPREDUCE-6451
2015-10-30 16:05:23 -05:00
2868ca0328
MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla.
2015-10-30 14:56:41 -05:00
18727c63da
HADOOP-12133 Add schemas to Maven Assembly XMLs
...
Signed-off-by: Allen Wittenauer <aw@apache.org>
2015-10-30 11:36:52 -07:00
2d10cb8e00
HADOOP-12519. hadoop-azure tests should avoid creating a metrics configuration fiale in the module root directory. Contributed by Chris Nauroth.
2015-10-28 09:44:10 -07:00
1f7ecb0c84
HADOOP-11685. StorageException complaining " no lease ID" during HBase distributed log splitting. Contributed by Duo Xu.
2015-10-27 22:56:22 -07:00
73822de7c3
HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Contributed by Chris Nauroth.
2015-10-27 22:48:56 -07:00
c28e16b40c
YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt. Contributed by Bibin A Chundatt.
2015-10-27 20:56:00 +05:30
bcb2386e39
HADOOP-12515. Mockito dependency is missing in hadoop-kafka module. Contributed by Kai Zheng.
2015-10-27 20:17:16 +09:00
cb282d5b89
HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race Condition Scenarios. Contributed by Gaurav Kanade.
2015-10-22 14:29:57 -07:00
47641fcbc9
HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. Contributed by Gaurav Kanade.
2015-10-22 12:21:32 -07:00
0c4af0f998
MAPREDUCE-6495. Docs for archive-logs tool (rkanter)
2015-10-20 17:34:34 -07:00
da8441d0fe
HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu.
2015-10-13 15:24:24 -07:00
7e2c971fed
Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu."
...
This reverts commit fa5a54cac7
2015-10-10 03:24:59 -07:00
fa5a54cac7
HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu.
2015-10-09 22:38:06 -07:00
1bca1bb517
MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME (rkanter)
2015-10-06 16:49:50 -07:00
5f6edb30c2
HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and writes. Contributed by Dushyanth.
2015-10-05 20:11:41 -07:00
b59e434116
HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via aw)
2015-10-05 14:02:07 -07:00
ecbfd68974
HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). Contributed by Rahul Palamuttam and Kanaka Kumar Avvaru.
2015-10-01 18:25:31 +02:00
5db371f52f
MAPREDUCE-6494. Permission issue when running archive-logs tool as different users (rkanter)
2015-09-30 17:33:53 -07:00
3abbdc929b
HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu.
2015-09-28 17:31:51 -05:00
66dad854c0
MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian.
2015-09-28 13:12:43 +05:30
7fe521b1dd
HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu.
2015-09-25 22:33:53 -07:00
d3c49e7662
MAPREDUCE-6480. archive-logs tool may miss applications (rkanter)
2015-09-25 15:02:42 -07:00
89cab1ba5f
YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan
2015-09-23 13:29:38 -07:00
cc2b473990
MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused. Contributed by Neelesh Srinivas Salian.
2015-09-22 21:38:06 +05:30
dfd807afab
HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang.
2015-09-22 12:54:29 +09:00
7269906254
HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel).
2015-09-13 14:25:26 +01:00
119cc75e7e
MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha)
2015-09-09 17:45:19 -07:00
355eaaa33d
HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter error message. Contributed by Wei Yan.
2015-09-03 22:48:53 +09:00
6ab2d19f5c
HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu)
2015-08-29 09:59:30 -07:00
feaf034994
YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks)
2015-08-24 11:25:07 +05:30
0bc15cb6e6
HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang)
2015-08-20 08:02:54 -07:00
3da0bedaec
HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Contributed by Chris Nauroth.
2015-08-05 14:56:41 -07:00
d5403747b5
HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu)
2015-08-04 18:51:52 -07:00
efa97243ec
HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu.
2015-07-22 11:16:49 -07:00
94c6a4aa85
HADOOP-12017. Hadoop archives command should use configurable replication factor when closing (Contributed by Bibin A Chundatt)
2015-07-22 10:25:49 +05:30
68d1f4bfe8
HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw)
2015-07-21 11:19:29 -07:00
c39ca541f4
YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda)
2015-07-21 09:57:23 -07:00
87f29c6b8a
HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. (Contributed by Vinayakumar B)
2015-07-21 13:12:46 +05:30
05130e94c5
HADOOP-12235 hadoop-openstack junit & mockito dependencies should be "provided". (Ted Yu via stevel)
2015-07-20 11:22:39 +01:00
a431ed9075
HADOOP-12211. Collect disk usage on the node. Contributed by Robert Grandl
2015-07-13 15:36:11 -07:00
John Zhuge
Andrew Wang
Chris Douglas
Mingliang Liu
Steve Loughran
Robert Kanter
Arpit Agarwal
John Zhuge
Akira Ajisaka
Allen Wittenauer
Wangda Tan
Yongjun Zhang
Sunil G
Kai Zheng
Lei Xu
Lei Xu
Chris Nauroth
Wei-Chiu Chuang
Jason Lowe
Xiao Chen
Arun Suresh
Masatake Iwasaki
Anu Engineer
Jing Zhao
Brahma Reddy Battula
Ravi Prakash
Mingfei
Xiaoyu Yao
Junping Du
mattf-horton
Jitendra Pandey
Jian He
Varun Vasudev
Harsh J
Xuan
Colin Patrick Mccabe
Zhe Zhang
Ravi Prakash
Karthik Kambatla
Haohui Mai
Kihwal Lee
Gábor Lipták
Robert Kanter
Tsuyoshi Ozawa
Rohith Sharma K S
Vinayakumar B