Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,710 Commits 326 Branches 371 Tags
Commit Graph

821 Commits

Author SHA1 Message Date
Sammi Chen
f9aedf32ec HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 
(cherry picked from commit 0857f116b754d83d3c540cd6f989087af24fef27)
 2018-07-30 10:56:31 +08:00
bibinchundatt
14fd114cd5 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 
(cherry picked from commit 3d3158cea4580eb2e3b2af635c3a7d30f4dbb873)
 2018-07-25 19:01:40 +05:30
Chris Douglas
d1dcc39222 HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603ee93e082e7ba148df1400af5ac7c30c)
(cherry picked from commit 7e655b642d0ed5b85e0e8c212e004d067318ed4c)
 2018-06-17 23:13:25 -07:00
Inigo Goiri
fa9d4a7493 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594ffc372e3f846b0c8ebc2ff9e36ffdb0)
 2018-06-12 11:01:22 -07:00
Steve Loughran
df338f2e1a HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be679554eb6b323f3bc6e8de267d85dd2e06)
(cherry picked from commit baac7c2b285454d71d0371505fb7a3403a548176)
 2018-06-11 15:33:36 -07:00
Yongjun Zhang
f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Inigo Goiri
595b44e2dc HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 2018-05-28 16:47:43 -07:00
Sammi Chen
ce7ebbe2cc HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 
(cherry picked from commit 6542d17ea460ec222137c4b275b13daf15d3fca3)
(cherry picked from commit e96c7bf82de1e9fd97df5fb6b763e211ebad5913)
 2018-04-13 10:17:46 +08:00
Lei Xu
3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Steve Loughran
a3945e4cf1 HADOOP-14651. Update okhttp version to 2.7.5. Contributed by Ray Chiang and John Zhuge. 
(cherry picked from commit 8aca46e53f5962217851dc2103211f99a4ef7258)
 2018-04-02 16:38:19 +01:00
Sean Mackrory
ca92d1810b HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 
(cherry picked from commit 9d7a9031a5978efc8d97566e35ebaace20db2353)
 2018-03-29 14:05:42 -06:00
Sammi Chen
7985c5fdc9 HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec5c60d96b0c216182891cdfd7832ac5)
(cherry picked from commit 2285afb32e71622b3dab5051247a1d099cfcbe85)
(cherry picked from commit 322520eb76cdcef25190495ccf98b3ca39907f58)
 2018-03-19 15:44:15 +08:00
Chris Douglas
a03c8ea61f HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc04a86be91a9b72073cdc30bec04d3b)
(cherry picked from commit 09940b1eb3b7ed764149f4a993c1857e9c6ad938)
 2018-03-12 20:06:22 -07:00
Steve Loughran
1771af2320 HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.

(cherry picked from commit 7ef4d942dd96232b0743a40ed25f77065254f94d)
 2018-03-09 10:49:11 +00:00
Steve Loughran
cc0f14c13c HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 19:20:53 +00:00
Gergely Novák
8b9a7b2e48
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e77671fa0e74a652bc2221bbe01d1a579d3)
 2018-03-02 16:22:06 -08:00
Steve Loughran
3c03672e87 HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran

(cherry picked from commit edc9f1451b4af79069cd4b1d608b0fa68648ee72)
(cherry picked from commit abfe1eaf29a31d96f63022a59e29392b773e6bc1)
 2018-02-28 15:33:21 +00:00
Steve Loughran
c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-21 11:11:28 +00:00
Steve Loughran
b302f728f8 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:29:30 +00:00
Steve Loughran
1888b94806 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:18 +00:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Sammi Chen
91184299c5 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
 2018-01-30 15:21:27 +08:00
Steve Loughran
95a96b13e2 HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.
 2018-01-29 09:46:53 -08:00
Steve Loughran
ec4f5f0f0d HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-26 15:23:08 -08:00
Jason Lowe
217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b960c72d18a984c7e5d145b407ca11a)
 2018-01-25 17:51:51 -06:00
Steve Loughran
1de0712578 HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.

(cherry picked from commit f63d13f10db708228482b53883863c873c89f69c)
 2018-01-23 11:37:33 -08:00
Jason Lowe
b1cc14e9cb Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)" 
This reverts commit 55142849db02a9191db0dd6f4e1401ff19ec242a.
 2018-01-17 09:53:24 -06:00
Sammi Chen
55142849db HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
 2018-01-17 16:12:23 +08:00
Kai Zheng
539fa11325 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 
(cherry picked from commit ce04340ec73617daff74378056a95c5d0cc0a790)
 2018-01-09 15:41:12 +08:00
Steve Loughran
72a49503fe HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt

(cherry picked from commit 52babbb4a0e3c89f2025bf6e9a1b51a96e8f8fb0)
 2017-12-22 14:31:20 +00:00
Steve Loughran
11f577f3e6 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 
(cherry picked from commit 1ba491ff907fc5d2618add980734a3534e2be098)
 2017-12-22 11:41:45 +00:00
Sammi Chen
abaabb5dec HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit c326fc89b06a8fe0978306378ba217748c7f2054)
 2017-12-08 21:28:19 +08:00
Sammi Chen
fb809e05dc HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu. 
(cherry picked from commit 18621af7ae8f8ed703245744f8f2a770d07bbfb9)
 2017-12-08 20:40:14 +08:00
John Zhuge
9942952dd8 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 
(cherry picked from commit bf6a660232b01642b07697a289c773ea5b97217c)
 2017-12-07 21:06:25 -08:00
Steve Loughran
8bf74f325a HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0faa35c726dcb37249d48786bfc8ca04c)
 2017-11-28 11:57:12 +00:00
Andrew Wang
1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Daniel Templeton
38f5d4cbba YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton) 
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
(cherry picked from commit ba8136615ab66c450884614557eddc6509d63b7c)
 2017-11-09 13:39:08 -08:00
Yufei Gu
ddef7d05a4 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 
(cherry picked from commit 99880d0a16727c770da053464da87960c5b02065)
 2017-11-09 13:38:57 -08:00
Subru Krishnan
7441d14f1c HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983aace73ad27934bd9f7f4d766e0b25f)
 2017-11-08 18:08:09 -08:00
Arun Suresh
cbd81f305b HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efebfac4a79f5cce2dd42cf00411d51e7)
 2017-11-02 17:17:27 -07:00
Akira Ajisaka
2ba71afcd5
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca409f5e57e540849a80e6448bb4924cc6)
 2017-11-02 18:32:57 +09:00
Chris Douglas
77ad73501c HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d21c30bf4d1ca263bb4b0717c48b2c0a)
 2017-10-27 22:00:44 -07:00
Subru Krishnan
82c9b3bbb5 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20e6011ab360131d70582a6e4bf2ec1d)
 2017-10-25 16:01:13 -07:00
Wangda Tan
881263260c YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 
(cherry picked from commit 281d83604df8341c210cee39bdc745ca793c5afa)
 2017-10-20 16:02:04 -07:00
Steve Loughran
2bacb429a1 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc847560d53e7955ced9ce7ce2773c805793)
 2017-10-20 22:29:22 +01:00
Haibo Chen
175c4472d5 YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen) 
(cherry picked from commit 46eb1033a86ca53e7b94202567aef8af2417bdf8)
 2017-10-17 16:04:49 -07:00
Haibo Chen
c021c8a19f YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 
(cherry picked from commit c071aad5da6f6601978e73d38ce5813958848bc4)
 2017-10-05 15:51:08 -07:00
Yufei Gu
f982ffe336 YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu) 
(cherry picked from commit 79e37dc6dba44e7d1d913593304db7a8516056aa)
 2017-10-03 15:12:09 -07:00
Akira Ajisaka
b658ae002f
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 
(cherry picked from commit c9b525bd943fe7e098e72fb9be5623862c7ab5ff)
 2017-10-04 07:02:31 +09:00