Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,660 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

712 Commits

Author SHA1 Message Date
Sammi Chen c617dba497 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb)
(cherry picked from commit 5da3e83597)
(cherry picked from commit 85e00477b8)
 2018-09-25 19:55:13 +08:00
Steve Loughran 4e342f603f
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.
 2018-09-04 16:32:22 +01:00
Sammi Chen 418e957c64 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 2018-08-02 10:14:54 +08:00
Steve Loughran 11e7f2e584
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d)
(cherry picked from commit cc0f14c13c)
 2018-06-25 17:27:49 +01:00
Chris Douglas 4789b8e9c9 HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603e)
(cherry picked from commit 7e655b642d)
(cherry picked from commit d1dcc39222)
 2018-06-17 23:14:12 -07:00
Inigo Goiri 85a7da0899 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594f)
 2018-06-12 11:01:45 -07:00
Steve Loughran 0b5d0c3740 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be6795)
(cherry picked from commit baac7c2b28)
(cherry picked from commit df338f2e1a)
 2018-06-11 15:34:54 -07:00
Inigo Goiri 0fc8b43dc8 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 595b44e2dc)
 2018-05-28 16:48:19 -07:00
Steve Loughran e2d32bd9c5 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c9)
 2018-05-21 11:13:57 +01:00
Steve Loughran 67468651b1 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt

(cherry picked from commit 5b11b9fd41)
 2018-05-10 10:33:03 -07:00
Junping Du 2b48854cfd HADOOP-15385. Many tests are failing in hadoop-distcp project in branch-2. Contributed by Jason Lowe. 2018-04-25 10:11:41 +08:00
Sammi Chen 1d78825bf1 HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 2018-04-10 14:24:31 +08:00
Steve Loughran e5351608ca HADOOP-14651. Update okhttp version to 2.7.5. 
Contributed by Ray Chiang and John Zhuge.
 2018-04-04 15:16:20 +01:00
Chris Douglas cc0a791794 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 
(cherry picked from commit 081c350188)
(cherry picked from commit ac16e8f4d3)
 2018-03-28 12:04:20 -07:00
Sammi Chen 2285afb32e HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec)
 2018-03-19 15:09:23 +08:00
Chris Douglas 340cd5f1b1 HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
(cherry picked from commit a03c8ea61f)
 2018-03-12 20:08:46 -07:00
Steve Loughran c245050538 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran; backport to branch-2 by Masatake Iwasaki.
 2018-03-05 12:46:19 +00:00
Gergely Novák 278676508f
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e7767)
 2018-03-02 16:22:20 -08:00
Steve Loughran fb0f2cc2fb Revert "HADOOP-15090. Add ADL troubleshooting doc." 
This reverts commit 6f944452f7.
 2018-02-28 21:19:03 +00:00
Chris Douglas 762125b864 Backport HADOOP-13514 (surefire upgrade) to branch-2 2018-02-26 16:32:06 -08:00
Xuan Gong a0b55392d4 MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 
(cherry picked from commit 68ce193efc)
 2018-02-23 14:40:51 -08:00
Steve Loughran 07c7df4b26 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b7)
 2018-02-15 16:29:44 +00:00
Steve Loughran 6f944452f7 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a)
 2018-02-15 14:26:36 +00:00
Sammi Chen 2816bd1f43 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302)
(cherry picked from commit 91184299c5)
 2018-01-30 16:25:02 +08:00
Steve Loughran 312c571694 HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-27 20:31:21 -08:00
Jason Lowe 4d7bbee0a1 YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:56:36 -06:00
Jason Lowe fd09f72151 Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)" 
This reverts commit 896dc7c780.
 2018-01-17 09:47:43 -06:00
Sammi Chen 896dc7c780 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302)
(cherry picked from commit 55142849db)
(cherry picked from commit 082a707bae)
 2018-01-17 16:36:03 +08:00
Steve Loughran d2ceef0f20 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
(Backported to branch-2 via /HADOOP-15156)

(cherry picked from commit a35267b47a)
 2018-01-08 15:13:50 +00:00
Sammi Chen 1fee447bf2 HADOOP-15111. Aliyun OSS: backport HADOOP-14993 to branch-2. Contributed by Genmao Yu. 2017-12-15 14:28:39 +08:00
Sammi Chen 27807e4ccb HADOOP-15104. Aliyun OSS: change the default value of max error retry. Contributed by Jinhu Wu. 
(cherry picked from commit ce04340ec7)
 2017-12-15 13:53:48 +08:00
Surendra Singh Lilhore 8d69b7f2d4 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-13 10:29:19 +05:30
Sammi Chen 94390fcd52 HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit 6e24fe2998)
 2017-12-08 22:02:00 +08:00
Xiao Chen c9d48352c2 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-12-07 21:20:33 -08:00
Steve Loughran 5c37a0b841 HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0fa)
 2017-11-28 11:57:51 +00:00
Kai Zheng 30ab9b6aef HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2. Contributed by Sammi Chen 
The consolidated commits in this backport are as follows:
    HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
    HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
    HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
    HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
    HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
    HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
    HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
    HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
    HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
    HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
    HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
    HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
    HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
    HADOOP-13624.  Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
    HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
    HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
    HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
 2017-11-13 09:29:33 +08:00
Subru Krishnan 5991c218a0 HADOOP-15026. Rebase ResourceEstimator start/stop scripts for branch-2. (Rui Li via Subru). 2017-11-09 00:32:34 -08:00
Subru Krishnan 46a740a82e HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983)
 2017-11-08 18:09:02 -08:00
Arun Suresh 0200fbac5b HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efe)
 2017-11-02 17:18:03 -07:00
Akira Ajisaka 82fc80a987
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca40)
 2017-11-02 18:33:25 +09:00
Arun Suresh 71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Chris Douglas ee89241282 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d2)
(cherry picked from commit 77ad73501c)
 2017-10-27 22:01:41 -07:00
Steve Loughran 3fbcf76324 HADOOP-14553. Add (parallelized) integration tests to hadoop-azure. 
Contributed by Steve Loughran.

(backport JIRA HADOOP-14870)
 2017-10-27 12:55:12 +01:00
Subru Krishnan 9897538a96 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20)
 2017-10-26 19:14:24 -07:00
Akira Ajisaka 13f1a7e0ed
Revert "HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru)." 
This reverts commit da7f989d86.
 2017-10-26 15:33:53 +09:00
Subru Krishnan da7f989d86 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20)
 2017-10-25 17:05:35 -07:00
Steve Loughran 062d9c9bab HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc8475)
 2017-10-20 22:27:59 +01:00
Wangda Tan d894f910d0 YARN-6608. Backport all SLS improvements from trunk to branch-2. (Carlo Curino via wangda) 2017-10-17 19:45:09 -07:00
Steve Loughran 8902178d3e HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak

Cherry-picked from commit 9fcc3a1fc8
 2017-10-16 16:06:41 +01:00
Steve Loughran ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c
 2017-10-11 18:09:13 +01:00