Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,609 Commits 326 Branches 371 Tags
Commit Graph

710 Commits

Author SHA1 Message Date
Sammi Chen
418e957c64 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 2018-08-02 10:14:54 +08:00
Steve Loughran
11e7f2e584
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
(cherry picked from commit cc0f14c13c1699489777b71a5c21bad27f9020ad)
 2018-06-25 17:27:49 +01:00
Chris Douglas
4789b8e9c9 HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603ee93e082e7ba148df1400af5ac7c30c)
(cherry picked from commit 7e655b642d0ed5b85e0e8c212e004d067318ed4c)
(cherry picked from commit d1dcc39222b6d1d8ba10f38a3f2fb69e4d6548b3)
 2018-06-17 23:14:12 -07:00
Inigo Goiri
85a7da0899 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594ffc372e3f846b0c8ebc2ff9e36ffdb0)
 2018-06-12 11:01:45 -07:00
Steve Loughran
0b5d0c3740 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be679554eb6b323f3bc6e8de267d85dd2e06)
(cherry picked from commit baac7c2b285454d71d0371505fb7a3403a548176)
(cherry picked from commit df338f2e1a19ed9c0c5c13f7d4aad08f9836de9f)
 2018-06-11 15:34:54 -07:00
Inigo Goiri
0fc8b43dc8 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 595b44e2dc3baa3b349768db2dca4d93158a44bf)
 2018-05-28 16:48:19 -07:00
Steve Loughran
e2d32bd9c5 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c94d31d3f737226d954c566b5d88656b)
 2018-05-21 11:13:57 +01:00
Steve Loughran
67468651b1 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt

(cherry picked from commit 5b11b9fd413470e134ecdc7c50468f8c7b39fa50)
 2018-05-10 10:33:03 -07:00
Junping Du
2b48854cfd HADOOP-15385. Many tests are failing in hadoop-distcp project in branch-2. Contributed by Jason Lowe. 2018-04-25 10:11:41 +08:00
Sammi Chen
1d78825bf1 HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 2018-04-10 14:24:31 +08:00
Steve Loughran
e5351608ca HADOOP-14651. Update okhttp version to 2.7.5. 
Contributed by Ray Chiang and John Zhuge.
 2018-04-04 15:16:20 +01:00
Chris Douglas
cc0a791794 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 
(cherry picked from commit 081c3501885c543bb1f159929d456d1ba2e3650c)
(cherry picked from commit ac16e8f4d3957c64bc2afddb9e58bd2f511f6c6e)
 2018-03-28 12:04:20 -07:00
Sammi Chen
2285afb32e HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec5c60d96b0c216182891cdfd7832ac5)
 2018-03-19 15:09:23 +08:00
Chris Douglas
340cd5f1b1 HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc04a86be91a9b72073cdc30bec04d3b)
(cherry picked from commit 09940b1eb3b7ed764149f4a993c1857e9c6ad938)
(cherry picked from commit a03c8ea61f6e30a9d462571ace23858b6e0fd1c9)
 2018-03-12 20:08:46 -07:00
Steve Loughran
c245050538 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran; backport to branch-2 by Masatake Iwasaki.
 2018-03-05 12:46:19 +00:00
Gergely Novák
278676508f
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e77671fa0e74a652bc2221bbe01d1a579d3)
 2018-03-02 16:22:20 -08:00
Steve Loughran
fb0f2cc2fb Revert "HADOOP-15090. Add ADL troubleshooting doc." 
This reverts commit 6f944452f71eedbd607564198396857bfe1ddfd5.
 2018-02-28 21:19:03 +00:00
Chris Douglas
762125b864 Backport HADOOP-13514 (surefire upgrade) to branch-2 2018-02-26 16:32:06 -08:00
Xuan Gong
a0b55392d4 MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 
(cherry picked from commit 68ce193efcb595f75d7addf751559c806a5aa399)
 2018-02-23 14:40:51 -08:00
Steve Loughran
07c7df4b26 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:29:44 +00:00
Steve Loughran
6f944452f7 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:36 +00:00
Sammi Chen
2816bd1f43 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
(cherry picked from commit 91184299c54df18540f841cca0efe0131d05b882)
 2018-01-30 16:25:02 +08:00
Steve Loughran
312c571694 HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-27 20:31:21 -08:00
Jason Lowe
4d7bbee0a1 YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:56:36 -06:00
Jason Lowe
fd09f72151 Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)" 
This reverts commit 896dc7c7801adaa4460fa6c19a4f452a6a6112d8.
 2018-01-17 09:47:43 -06:00
Sammi Chen
896dc7c780 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
(cherry picked from commit 55142849db02a9191db0dd6f4e1401ff19ec242a)
(cherry picked from commit 082a707bae4bb97444a34c00eecd62975807388d)
 2018-01-17 16:36:03 +08:00
Steve Loughran
d2ceef0f20 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
(Backported to branch-2 via /HADOOP-15156)

(cherry picked from commit a35267b47abf62d2ab32bc6860d2080284323042)
 2018-01-08 15:13:50 +00:00
Sammi Chen
1fee447bf2 HADOOP-15111. Aliyun OSS: backport HADOOP-14993 to branch-2. Contributed by Genmao Yu. 2017-12-15 14:28:39 +08:00
Sammi Chen
27807e4ccb HADOOP-15104. Aliyun OSS: change the default value of max error retry. Contributed by Jinhu Wu. 
(cherry picked from commit ce04340ec73617daff74378056a95c5d0cc0a790)
 2017-12-15 13:53:48 +08:00
Surendra Singh Lilhore
8d69b7f2d4 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-13 10:29:19 +05:30
Sammi Chen
94390fcd52 HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit 6e24fe2998734606868f2c39bc49a808f2856bc0)
 2017-12-08 22:02:00 +08:00
Xiao Chen
c9d48352c2 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-12-07 21:20:33 -08:00
Steve Loughran
5c37a0b841 HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0faa35c726dcb37249d48786bfc8ca04c)
 2017-11-28 11:57:51 +00:00
Kai Zheng
30ab9b6aef HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2. Contributed by Sammi Chen 
The consolidated commits in this backport are as follows:
    HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
    HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
    HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
    HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
    HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
    HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
    HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
    HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
    HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
    HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
    HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
    HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
    HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
    HADOOP-13624.  Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
    HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
    HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
    HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
 2017-11-13 09:29:33 +08:00
Subru Krishnan
5991c218a0 HADOOP-15026. Rebase ResourceEstimator start/stop scripts for branch-2. (Rui Li via Subru). 2017-11-09 00:32:34 -08:00
Subru Krishnan
46a740a82e HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983aace73ad27934bd9f7f4d766e0b25f)
 2017-11-08 18:09:02 -08:00
Arun Suresh
0200fbac5b HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efebfac4a79f5cce2dd42cf00411d51e7)
 2017-11-02 17:18:03 -07:00
Akira Ajisaka
82fc80a987
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca409f5e57e540849a80e6448bb4924cc6)
 2017-11-02 18:33:25 +09:00
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Chris Douglas
ee89241282 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d21c30bf4d1ca263bb4b0717c48b2c0a)
(cherry picked from commit 77ad73501ce519851f2f80a90aca2d73e8b2e20d)
 2017-10-27 22:01:41 -07:00
Steve Loughran
3fbcf76324 HADOOP-14553. Add (parallelized) integration tests to hadoop-azure. 
Contributed by Steve Loughran.

(backport JIRA HADOOP-14870)
 2017-10-27 12:55:12 +01:00
Subru Krishnan
9897538a96 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20e6011ab360131d70582a6e4bf2ec1d)
 2017-10-26 19:14:24 -07:00
Akira Ajisaka
13f1a7e0ed
Revert "HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru)." 
This reverts commit da7f989d86d28ee6f4a7ac410fc8a4a163e11012.
 2017-10-26 15:33:53 +09:00
Subru Krishnan
da7f989d86 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20e6011ab360131d70582a6e4bf2ec1d)
 2017-10-25 17:05:35 -07:00
Steve Loughran
062d9c9bab HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc847560d53e7955ced9ce7ce2773c805793)
 2017-10-20 22:27:59 +01:00
Wangda Tan
d894f910d0 YARN-6608. Backport all SLS improvements from trunk to branch-2. (Carlo Curino via wangda) 2017-10-17 19:45:09 -07:00
Steve Loughran
8902178d3e HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak

Cherry-picked from commit 9fcc3a1fc8cab873034f5c308ceb2d5671a954e8
 2017-10-16 16:06:41 +01:00
Steve Loughran
ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c4d5554213d02bd0ad343ba82cf82aa
 2017-10-11 18:09:13 +01:00
Steve Loughran
34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-07 11:20:18 +01:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
 2017-10-05 16:03:13 +01:00