Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,257 Commits 326 Branches 371 Tags
Commit Graph

689 Commits

Author SHA1 Message Date
Steve Loughran
6843659689 HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 2017-02-21 18:29:30 +00:00
Steve Loughran
274c02d2fa HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:00 +00:00
Steve Loughran
8a88e8ea57 HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1dc997906390d653dfafb4f0d7e193c4)
 2017-02-18 18:17:54 +00:00
Mingliang Liu
b23e93d465 HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 
(cherry picked from commit 658702efffdf52cf5ddf8e92f959f1157c95a348)
 2017-02-16 16:59:31 -08:00
Mingliang Liu
29e1007172 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 
(cherry picked from commit bdad8b7b97d7f48119f016d68f32982d680c8796)
 2017-02-16 16:42:56 -08:00
Akira Ajisaka
25ac54a52e HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252f5a23a91a1d695d748556a0fd803ae)
 2017-02-15 01:46:59 +09:00
Lei Xu
6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Arpit Agarwal
c88ec54588 HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus. 2017-02-10 10:09:39 -08:00
Akira Ajisaka
b204a1eb5f HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows. 
(cherry picked from commit 98c35bb1a4faf794060612bee7494d501701b698)
 2017-01-21 17:35:57 +09:00
Steve Loughran
b2704702f9 HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
 2017-01-19 11:25:20 +00:00
Steve Loughran
ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Wangda Tan
447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804a869723ae36605a3c2d2b3eae3941e)
 2017-01-09 08:42:38 -08:00
Arun Suresh
7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d401a2f49d0f9ca556343f9ba871e979)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Chris Nauroth
9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e130230457fc897ed7a2a09e14d078be90a)
 2017-01-05 15:05:01 -08:00
Akira Ajisaka
3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe9389ccdc153d7f4bf6574bf8e509867c1)
 2017-01-04 14:15:00 +09:00
Mingliang Liu
ec646b1a9d HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki 
(cherry picked from commit c6a5b689db573046819f0e9193cb042e6c7298f4)
 2016-12-29 16:24:38 -08:00
Mingliang Liu
edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 
(cherry picked from commit e92a77099b91620cee84513cc879089907468075)
 2016-12-22 20:33:51 -08:00
Akira Ajisaka
522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539eaf7745ec8690f9281726fb4cdcdbe94)
 2016-12-23 04:06:53 +09:00
Mingliang Liu
867389247f HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade 
(cherry picked from commit f92913c35bfda0d565606f9fb9a301ddd4105fd8)
 2016-12-15 12:38:31 -08:00
Mingliang Liu
2dbd86bcf4 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a39232456fa0c98b2b9b6dbeaec762294ca01e)
 2016-12-12 15:06:39 -08:00
Mingliang Liu
89eaa94c59 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth 
(cherry picked from commit 15dd1f3381069c5fdc6690e3ab1907a133ba14bf)
 2016-12-05 12:06:35 -08:00
Mingliang Liu
c37e814fae HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:48:38 -08:00
Mingliang Liu
a4e0539636 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a00405e977b1fd791cfb974244580dd5ae8)
 2016-12-02 13:39:23 -08:00
Mingliang Liu
39b74b6f9e HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99fdfae6be6e5e261f371d175aad14229)
 2016-11-30 13:04:44 -08:00
Mingliang Liu
1b831d8e01 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8aa450c44d3be69d26c88025e253ac0c)
 2016-11-28 16:35:18 -08:00
Steve Loughran
9701ab48e8 HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:34:11 +00:00
Steve Loughran
f3fff5c2e1 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:04 +00:00
Mingliang Liu
cae4a8c05d HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:05:58 -08:00
Mingliang Liu
9af7d1e298 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:19:32 -08:00
Mingliang Liu
4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b6443715b7a741296c1bd520a1840f9a7c)
 2016-11-15 11:36:08 -08:00
Mingliang Liu
d064d5dbb2 HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth 
(cherry picked from commit 89354f0475efa8e393697b1ddc227c94a76b5923)
 2016-11-10 08:59:07 -08:00
Jason Lowe
8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733058238a2c8f23960c986c71dca0e02)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00
Akira Ajisaka
3b279fe426 HADOOP-13798. TestHadoopArchives times out. 
(cherry picked from commit b970446b2c59f8897bb2c3a562fa192ed3452db5)
 2016-11-07 19:54:50 +09:00
Steve Loughran
c4ccafdaff HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:52:49 +00:00
Masatake Iwasaki
7e918637de HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82a4510f16df49238d57c9f78ac28ae7)
 2016-10-27 15:54:03 +09:00
Chris Nauroth
082d69ee66 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5bacd98d0d371447ada3b31f93ca40a4e)
 2016-10-26 08:57:29 -07:00
Chris Nauroth
67e01f7218 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e235026dbe4658705ca85d263d0edf14521)
 2016-10-26 08:27:33 -07:00
Akira Ajisaka
e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a66776e9f410150c4030ddb15981f58fb9)
 2016-10-26 15:34:40 +09:00
Yongjun Zhang
ce72314657 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 
(cherry picked from commit 0f0c15f7a5ea33ced781978bea971f3750883f41)
 2016-10-25 12:29:28 -07:00
Yongjun Zhang
7bc170ba26 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 
(cherry picked from commit 8650cc84f20e7d8c32dcdcd91c94372d476e2276)

Conflicts
	hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
 2016-10-25 11:32:26 -07:00
Chris Nauroth
05d772e297 HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c078ff51cdb6bd1273e6f91f43311cb)
 2016-10-25 09:03:11 -07:00
Chris Nauroth
5c2f67bdae HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e940303149d6258e0b72c54d72f080f0daa2)
 2016-10-24 21:54:16 -07:00
Chris Nauroth
5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa6722cbf7a4ec3d6b9c44b034da9aa351)
 2016-10-24 21:22:46 -07:00
Steve Loughran
385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Steve Loughran
f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb5add3180094b126c4af2b3ddbc7cae.
 2016-10-20 22:48:26 +01:00
Anu Engineer
42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran
c640ba5fc9 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:30 +01:00
Steve Loughran
bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Allen Wittenauer
a5a56c3564 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 
(cherry picked from commit 422c73a8657d8699920f7db13d4be200e16c4272)
 2016-10-17 16:35:32 -07:00
Andrew Wang
4ed7cf3b36 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 
(cherry picked from commit 987ee51141a15d3f4d1df4dc792a192b92b87b5f)
 2016-10-17 13:32:39 -07:00