Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,334 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

552 Commits

Author SHA1 Message Date
Akira Ajisaka 58a1753373 HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe938)
(cherry picked from commit 3e2d26a97f)
 2017-01-04 14:20:04 +09:00
Akira Ajisaka 8e4479dda3 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
(cherry picked from commit 522bc98026)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestHedgingRequestRMFailoverProxyProvider.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java
 2016-12-23 04:15:39 +09:00
Mingliang Liu 01b50b36b6 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:11:11 -08:00
Mingliang Liu 3232f5ffe6 HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:55:17 -08:00
Mingliang Liu 4a1b7d6dee HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:42:58 -08:00
Mingliang Liu afad13f8d9 HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99f)
 2016-11-30 13:06:22 -08:00
Mingliang Liu 59f6067568 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8a)
 2016-11-28 16:42:41 -08:00
Steve Loughran db37ad0ebf HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:33:32 +00:00
Steve Loughran 01e3b0dae8 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:23 +00:00
Mingliang Liu ed9dcdae57 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:24:42 -08:00
Akira Ajisaka 21f210bada HADOOP-13798. TestHadoopArchives times out. 
(cherry picked from commit b970446b2c)
(cherry picked from commit 3b279fe426)
 2016-11-07 19:55:10 +09:00
Steve Loughran a83f10c90d HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:53:22 +00:00
Masatake Iwasaki 6d9b6fac86 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82)
 2016-10-27 15:55:35 +09:00
Chris Nauroth 4df1780991 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
(cherry picked from commit 082d69ee66)
 2016-10-26 08:57:46 -07:00
Chris Nauroth c940c68c79 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
(cherry picked from commit 67e01f7218)
 2016-10-26 08:27:39 -07:00
Chris Nauroth 3d5f41544a HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
(cherry picked from commit 05d772e297)
 2016-10-25 09:03:19 -07:00
Chris Nauroth 02f0472d1e HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e94030)
(cherry picked from commit 5c2f67bdae)
 2016-10-24 21:54:25 -07:00
Chris Nauroth 78970e0db3 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
(cherry picked from commit 5b7cbb5a3c)
 2016-10-24 21:23:00 -07:00
Steve Loughran 4b56954fea HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:12:03 +01:00
Steve Loughran da7e2f08f1 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 671d219c9c.
 2016-10-20 22:49:11 +01:00
Anu Engineer 671d219c9c HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:50:49 -07:00
Steve Loughran 769babf470 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:44 +01:00
Steve Loughran 1ecbf323ba HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:34:07 +01:00
Andrew Wang 15ff590c37 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 
(cherry picked from commit 987ee51141)
(cherry picked from commit 4ed7cf3b36)
 2016-10-17 13:32:52 -07:00
Jing Zhao d3c4465147 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d07983)
(cherry picked from commit ad69baf6a9)
 2016-10-13 13:28:05 -07:00
Chris Nauroth 67d8301e55 HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
(cherry picked from commit e28930a38b)
 2016-10-10 14:01:19 -07:00
Chris Nauroth 027d76fa8c HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 
(cherry picked from commit cda68de9b9)
 2016-10-10 14:01:13 -07:00
Chris Nauroth d29dc39791 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
(cherry picked from commit 986162f977)
 2016-10-10 14:01:07 -07:00
Chris Nauroth c6e69f6fb5 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
(cherry picked from commit 2de2d0df1d)
 2016-10-07 11:41:40 -07:00
Steve Loughran 96cc056cb7 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 13:22:20 +01:00
Steve Loughran d24933ad3a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-10-07 13:12:04 +01:00
Steve Loughran f66863f848 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:52 +01:00
Chris Nauroth 4ea1e73e5b HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
(cherry picked from commit 213f0ad7f0)
 2016-10-04 10:37:10 -07:00
Chris Nauroth b7b06a5f34 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
(cherry picked from commit 1f1e47e411)
 2016-09-28 16:21:10 -07:00
Brahma Reddy Battula 686cbf45d4 HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970be)
 2016-09-27 10:49:04 +05:30
Steve Loughran a2ce4d0503 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:54 +01:00
Chris Nauroth 911ae15f69 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb51397)
(cherry picked from commit e19f910245)
 2016-09-19 15:17:50 -07:00
Steve Loughran 6a7ce4ee52 HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:44:20 +01:00
Steve Loughran 5706754f77 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:56 +01:00
Steve Loughran c45f1ec858 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:19:05 +01:00
Arun Suresh 979b29a03c YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 
(cherry picked from commit d6d9cff21b)
(cherry picked from commit b279f42d79)
 2016-08-31 20:06:49 -07:00
Chris Nauroth dd6c149c4a HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
(cherry picked from commit 6dba38f08b)
 2016-08-19 12:55:57 -07:00
Chris Nauroth 1c4e039ba7 HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 
(cherry picked from commit ae4db25443)
(cherry picked from commit 804a40720b)
 2016-08-18 10:45:20 -07:00
Steve Loughran c09797ce9f HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:35:26 +01:00
Chris Nauroth ac779f2554 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
(cherry picked from commit 555a57adf2)
 2016-08-16 17:11:12 -07:00
Chris Nauroth 5c0db4e522 HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth. 
(cherry picked from commit 4ef1bc0bdb)
(cherry picked from commit 8ec3d702a3)
 2016-08-02 22:32:44 -07:00
Steve Loughran 71c8bdd50c HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:48:49 +01:00
Steve Loughran 5f9bffbb66 HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:43:45 +01:00
Steve Loughran 7c46303de3 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 17:58:46 +01:00
mattf-horton c96cb3fd48 HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 16:01:05 -07:00
Steve Loughran c177823ebe HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:17 +01:00
Steve Loughran 0f1e02a298 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:19 +01:00
Jitendra Pandey a363277be5 HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:32:20 -07:00
Jian He 23eb3c7ceb YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:38:35 -07:00
Andrew Wang 4e7e48cdc1 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8)
(cherry picked from commit 38b1eafdbc)
 2016-07-11 18:06:45 -07:00
Steve Loughran 310c08454f HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:14 +01:00
Akira Ajisaka a8cfcce542 Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux." 
This reverts commit 1d175f0f3f.
 2016-07-09 21:01:07 -07:00
Jitendra Pandey 9c7ce389c7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:04:47 -07:00
Jitendra Pandey bc7fd76a1f HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:37:32 -07:00
Steve Loughran f4f0c5074d HADOOP-13239 Deprecate s3:// in branch-2 2016-06-29 13:16:10 +01:00
Yongjun Zhang 4a57c03f30 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3a)
 2016-06-28 23:34:44 -07:00
Jing Zhao d17265f467 HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 
(cherry picked from commit 856bc4e28b)
 2016-06-22 15:22:54 -07:00
Steve Loughran 1eb45180a6 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:41:52 +01:00
Chris Nauroth e30b7112ed HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 
(cherry picked from commit b2c596cdda)
(cherry picked from commit e6ebe9ab4e)
 2016-06-21 11:34:46 -07:00
Jitendra Pandey 2f95d22b17 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:31:54 -07:00
Ravi Prakash 605443c913 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 20:45:59 +01:00
Chris Nauroth eddefbd01b HADOOP-13241. document s3a better. Contributed by Steve Loughran. 
(cherry picked from commit 127d2c7281)
(cherry picked from commit e4b63ed7b4)
 2016-06-16 11:19:00 -07:00
Wangda Tan d838c6443d YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:21:41 -07:00
Steve Loughran faf9a9f4e0 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 20:58:30 +01:00
Steve Loughran 7e09601a90 HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 16:36:51 +01:00
Wangda Tan 19e578870d YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. (wangda) 2016-06-07 12:41:50 -07:00
Steve Loughran f9478c95bd HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:40:49 +02:00
Chris Nauroth b8216c10d8 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:56:36 -07:00
Chris Nauroth 3ab0e8d500 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 
(cherry picked from commit 16b1cc7af9)
(cherry picked from commit e9942c5b34)
 2016-06-01 14:49:41 -07:00
Steve Loughran 809e359d2d HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:17:18 +01:00
Wangda Tan 2f3e1d965d Revert "YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan." 
This reverts commit 457884737f.
 2016-05-31 22:16:53 -07:00
Varun Vasudev 457884737f YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:57:56 +05:30
Yongjun Zhang 3fc4d4a369 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee7)
 2016-05-26 23:34:32 -07:00
Steve Loughran 5930e813c5 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-21 19:10:16 +01:00
Steve Loughran f2aef95bd3 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:20:19 +01:00
Steve Loughran ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c571.
 2016-05-20 14:00:26 +01:00
Steve Loughran f32b37ec89 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:51:56 +01:00
Steve Loughran edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Chris Nauroth dbff8f2932 HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459fa)
(cherry picked from commit 60e5397a9b)
 2016-05-19 22:00:42 -07:00
Jing Zhao 8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015)
(cherry picked from commit dd7f5a3b88)
 2016-05-17 15:55:31 -07:00
Steve Loughran 42d9beaa1d HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:18:51 +01:00
Steve Loughran b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth 28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d5)
(cherry picked from commit bb41547fb4)
 2016-05-16 11:53:33 -07:00
Steve Loughran a84850b4ca HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:15 +01:00
Andrew Wang b5fd6dd4a8 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7)
 2016-05-12 23:58:26 -07:00
Steve Loughran 8d74a58042 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:18 +01:00
Steve Loughran 9cbb942465 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:10 +01:00
Steve Loughran af5e5d6d47 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:56:48 +01:00
Chris Nauroth bbd463b3b2 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
(cherry picked from commit ba48834431)
 2016-05-10 15:14:42 -07:00
Steve Loughran 11b170adc8 HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:23 +01:00
Jason Lowe b452ecf4a0 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit ee86cef2fe)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-05-05 14:40:35 +00:00
Masatake Iwasaki 50d06acb09 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 
(cherry picked from commit ef0870ad03)
 2016-04-29 20:40:47 +09:00
Yongjun Zhang 7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd12)
(cherry picked from commit 7955b6f66e)
 2016-04-26 16:30:54 -07:00
Jing Zhao b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328)
(cherry picked from commit 864baf23aa)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka 6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d9)
(cherry picked from commit 58642d2244)
 2016-04-25 22:34:37 +09:00