Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,789 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

826 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 8360d15abc HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e)
(cherry picked from commit a7dd244a49)

	modified:   hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
(cherry picked from commit 84726679f0)
 2018-10-19 13:56:23 -07:00
Wei-Chiu Chuang 8754c0b65e Revert "HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu." 
This reverts commit 437c5187cd.
 2018-10-19 13:33:39 -07:00
Ted Yu 437c5187cd HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e)
(cherry picked from commit a7dd244a49)
(cherry picked from commit 84726679f0)
 2018-10-19 13:23:02 -07:00
Steve Loughran 2abc4ab60a
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.

(cherry picked from commit f1fbc56371)
 2018-10-12 09:49:02 +01:00
Sammi Chen 85e00477b8 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb)
(cherry picked from commit 5da3e83597)
 2018-09-25 19:53:03 +08:00
Sammi Chen f9aedf32ec HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 
(cherry picked from commit 0857f116b7)
 2018-07-30 10:56:31 +08:00
bibinchundatt 14fd114cd5 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 
(cherry picked from commit 3d3158cea4)
 2018-07-25 19:01:40 +05:30
Chris Douglas d1dcc39222 HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603e)
(cherry picked from commit 7e655b642d)
 2018-06-17 23:13:25 -07:00
Inigo Goiri fa9d4a7493 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594f)
 2018-06-12 11:01:22 -07:00
Steve Loughran df338f2e1a HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be6795)
(cherry picked from commit baac7c2b28)
 2018-06-11 15:33:36 -07:00
Yongjun Zhang f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Inigo Goiri 595b44e2dc HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 2018-05-28 16:47:43 -07:00
Sammi Chen ce7ebbe2cc HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 
(cherry picked from commit 6542d17ea4)
(cherry picked from commit e96c7bf82d)
 2018-04-13 10:17:46 +08:00
Lei Xu 3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Steve Loughran a3945e4cf1 HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge. 
(cherry picked from commit 8aca46e53f)
 2018-04-02 16:38:19 +01:00
Sean Mackrory ca92d1810b HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 
(cherry picked from commit 9d7a9031a5)
 2018-03-29 14:05:42 -06:00
Sammi Chen 7985c5fdc9 HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec)
(cherry picked from commit 2285afb32e)
(cherry picked from commit 322520eb76)
 2018-03-19 15:44:15 +08:00
Chris Douglas a03c8ea61f HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
 2018-03-12 20:06:22 -07:00
Steve Loughran 1771af2320 HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.

(cherry picked from commit 7ef4d942dd)
 2018-03-09 10:49:11 +00:00
Steve Loughran cc0f14c13c HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d)
 2018-03-07 19:20:53 +00:00
Gergely Novák 8b9a7b2e48
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e7767)
 2018-03-02 16:22:06 -08:00
Steve Loughran 3c03672e87 HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran

(cherry picked from commit edc9f1451b)
(cherry picked from commit abfe1eaf29)
 2018-02-28 15:33:21 +00:00
Steve Loughran c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-21 11:11:28 +00:00
Steve Loughran b302f728f8 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b7)
 2018-02-15 16:29:30 +00:00
Steve Loughran 1888b94806 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a)
 2018-02-15 14:26:18 +00:00
Anu Engineer 2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb3)
 2018-02-07 11:16:53 -08:00
Lei Xu 7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Sammi Chen 91184299c5 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302)
 2018-01-30 15:21:27 +08:00
Steve Loughran 95a96b13e2 HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.
 2018-01-29 09:46:53 -08:00
Steve Loughran ec4f5f0f0d HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-26 15:23:08 -08:00
Jason Lowe 217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b)
 2018-01-25 17:51:51 -06:00
Steve Loughran 1de0712578 HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.

(cherry picked from commit f63d13f10d)
 2018-01-23 11:37:33 -08:00
Jason Lowe b1cc14e9cb Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)" 
This reverts commit 55142849db.
 2018-01-17 09:53:24 -06:00
Sammi Chen 55142849db HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302)
 2018-01-17 16:12:23 +08:00
Kai Zheng 539fa11325 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 
(cherry picked from commit ce04340ec7)
 2018-01-09 15:41:12 +08:00
Steve Loughran 72a49503fe HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt

(cherry picked from commit 52babbb4a0)
 2017-12-22 14:31:20 +00:00
Steve Loughran 11f577f3e6 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 
(cherry picked from commit 1ba491ff90)
 2017-12-22 11:41:45 +00:00
Sammi Chen abaabb5dec HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit c326fc89b0)
 2017-12-08 21:28:19 +08:00
Sammi Chen fb809e05dc HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu. 
(cherry picked from commit 18621af7ae)
 2017-12-08 20:40:14 +08:00
John Zhuge 9942952dd8 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 
(cherry picked from commit bf6a660232)
 2017-12-07 21:06:25 -08:00
Steve Loughran 8bf74f325a HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0fa)
 2017-11-28 11:57:12 +00:00
Andrew Wang 1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Daniel Templeton 38f5d4cbba YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton) 
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
(cherry picked from commit ba8136615a)
 2017-11-09 13:39:08 -08:00
Yufei Gu ddef7d05a4 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 
(cherry picked from commit 99880d0a16)
 2017-11-09 13:38:57 -08:00
Subru Krishnan 7441d14f1c HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983)
 2017-11-08 18:08:09 -08:00
Arun Suresh cbd81f305b HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efe)
 2017-11-02 17:17:27 -07:00
Akira Ajisaka 2ba71afcd5
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca40)
 2017-11-02 18:32:57 +09:00
Chris Douglas 77ad73501c HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d2)
 2017-10-27 22:00:44 -07:00
Subru Krishnan 82c9b3bbb5 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20)
 2017-10-25 16:01:13 -07:00
Wangda Tan 881263260c YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 
(cherry picked from commit 281d83604d)
 2017-10-20 16:02:04 -07:00