Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,390 Commits 326 Branches 371 Tags
Commit Graph

810 Commits

Author SHA1 Message Date
Sammi Chen
7985c5fdc9 HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec5c60d96b0c216182891cdfd7832ac5)
(cherry picked from commit 2285afb32e71622b3dab5051247a1d099cfcbe85)
(cherry picked from commit 322520eb76cdcef25190495ccf98b3ca39907f58)
 2018-03-19 15:44:15 +08:00
Chris Douglas
a03c8ea61f HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc04a86be91a9b72073cdc30bec04d3b)
(cherry picked from commit 09940b1eb3b7ed764149f4a993c1857e9c6ad938)
 2018-03-12 20:06:22 -07:00
Steve Loughran
1771af2320 HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.

(cherry picked from commit 7ef4d942dd96232b0743a40ed25f77065254f94d)
 2018-03-09 10:49:11 +00:00
Steve Loughran
cc0f14c13c HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 19:20:53 +00:00
Gergely Novák
8b9a7b2e48
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e77671fa0e74a652bc2221bbe01d1a579d3)
 2018-03-02 16:22:06 -08:00
Steve Loughran
3c03672e87 HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran

(cherry picked from commit edc9f1451b4af79069cd4b1d608b0fa68648ee72)
(cherry picked from commit abfe1eaf29a31d96f63022a59e29392b773e6bc1)
 2018-02-28 15:33:21 +00:00
Steve Loughran
c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-21 11:11:28 +00:00
Steve Loughran
b302f728f8 HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b79324b7f6231b879c0bfc608203b980)
 2018-02-15 16:29:30 +00:00
Steve Loughran
1888b94806 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a31307f19551f87be4e81d4fb626de1)
 2018-02-15 14:26:18 +00:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Sammi Chen
91184299c5 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
 2018-01-30 15:21:27 +08:00
Steve Loughran
95a96b13e2 HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.
 2018-01-29 09:46:53 -08:00
Steve Loughran
ec4f5f0f0d HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-26 15:23:08 -08:00
Jason Lowe
217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b960c72d18a984c7e5d145b407ca11a)
 2018-01-25 17:51:51 -06:00
Steve Loughran
1de0712578 HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.

(cherry picked from commit f63d13f10db708228482b53883863c873c89f69c)
 2018-01-23 11:37:33 -08:00
Jason Lowe
b1cc14e9cb Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)" 
This reverts commit 55142849db02a9191db0dd6f4e1401ff19ec242a.
 2018-01-17 09:53:24 -06:00
Sammi Chen
55142849db HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 
(cherry picked from commit 9195a6e302028ed3921d1016ac2fa5754f06ebf0)
 2018-01-17 16:12:23 +08:00
Kai Zheng
539fa11325 HADOOP-15104. AliyunOSS: change the default value of max error retry. Contributed by Jinhu Wu 
(cherry picked from commit ce04340ec73617daff74378056a95c5d0cc0a790)
 2018-01-09 15:41:12 +08:00
Steve Loughran
72a49503fe HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt

(cherry picked from commit 52babbb4a0e3c89f2025bf6e9a1b51a96e8f8fb0)
 2017-12-22 14:31:20 +00:00
Steve Loughran
11f577f3e6 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 
(cherry picked from commit 1ba491ff907fc5d2618add980734a3534e2be098)
 2017-12-22 11:41:45 +00:00
Sammi Chen
abaabb5dec HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit c326fc89b06a8fe0978306378ba217748c7f2054)
 2017-12-08 21:28:19 +08:00
Sammi Chen
fb809e05dc HADOOP-14993. AliyunOSS: Override listFiles and listLocatedStatus. Contributed Genmao Yu. 
(cherry picked from commit 18621af7ae8f8ed703245744f8f2a770d07bbfb9)
 2017-12-08 20:40:14 +08:00
John Zhuge
9942952dd8 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 
(cherry picked from commit bf6a660232b01642b07697a289c773ea5b97217c)
 2017-12-07 21:06:25 -08:00
Steve Loughran
8bf74f325a HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0faa35c726dcb37249d48786bfc8ca04c)
 2017-11-28 11:57:12 +00:00
Andrew Wang
1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Daniel Templeton
38f5d4cbba YARN-7413. Support resource type in SLS (Contributed by Yufei Gu via Daniel Templeton) 
Change-Id: Ic0a897c123c5d2f57aae757ca6bcf1dad7b90d2b
(cherry picked from commit ba8136615ab66c450884614557eddc6509d63b7c)
 2017-11-09 13:39:08 -08:00
Yufei Gu
ddef7d05a4 YARN-6505. Define the strings used in SLS JSON input file format. (Contributed by Gergely Novak) 
(cherry picked from commit 99880d0a16727c770da053464da87960c5b02065)
 2017-11-09 13:38:57 -08:00
Subru Krishnan
7441d14f1c HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983aace73ad27934bd9f7f4d766e0b25f)
 2017-11-08 18:08:09 -08:00
Arun Suresh
cbd81f305b HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efebfac4a79f5cce2dd42cf00411d51e7)
 2017-11-02 17:17:27 -07:00
Akira Ajisaka
2ba71afcd5
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca409f5e57e540849a80e6448bb4924cc6)
 2017-11-02 18:32:57 +09:00
Chris Douglas
77ad73501c HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d21c30bf4d1ca263bb4b0717c48b2c0a)
 2017-10-27 22:00:44 -07:00
Subru Krishnan
82c9b3bbb5 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20e6011ab360131d70582a6e4bf2ec1d)
 2017-10-25 16:01:13 -07:00
Wangda Tan
881263260c YARN-7318. Fix shell check warnings of SLS. (Gergely Novák via wangda) 
(cherry picked from commit 281d83604df8341c210cee39bdc745ca793c5afa)
 2017-10-20 16:02:04 -07:00
Steve Loughran
2bacb429a1 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc847560d53e7955ced9ce7ce2773c805793)
 2017-10-20 22:29:22 +01:00
Haibo Chen
175c4472d5 YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo Chen) 
(cherry picked from commit 46eb1033a86ca53e7b94202567aef8af2417bdf8)
 2017-10-17 16:04:49 -07:00
Haibo Chen
c021c8a19f YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 
(cherry picked from commit c071aad5da6f6601978e73d38ce5813958848bc4)
 2017-10-05 15:51:08 -07:00
Yufei Gu
f982ffe336 YARN-7211. AMSimulator in SLS does't work due to responseId refactoring.(Contributed by Botong Huang via Yufei Gu) 
(cherry picked from commit 79e37dc6dba44e7d1d913593304db7a8516056aa)
 2017-10-03 15:12:09 -07:00
Akira Ajisaka
b658ae002f
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 
(cherry picked from commit c9b525bd943fe7e098e72fb9be5623862c7ab5ff)
 2017-10-04 07:02:31 +09:00
Andrew Wang
5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Aaron Fabbri
60b966cb45
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 16:01:15 -07:00
Jason Lowe
0eb0298027 MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b133dc5700b66318ac6d80e463b049723c511f37)
 2017-09-22 15:30:20 -05:00
Steve Loughran
9f6b08f840
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure 
Contributed by Steve Loughran

(cherry picked from commit 2d2d97fa7d4224369b3c13bc4a45e8cc9e29afb1)
 2017-09-15 17:04:43 +01:00
Aaron Fabbri
62e8a5ceed
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 16:12:29 -07:00
Kai Zheng
b873b1395e HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:22:00 +08:00
John Zhuge
bb107f5d28 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 
(cherry picked from commit a4661850c1e0794baf493a468191e12681d68ab4)
 2017-09-08 12:57:58 -07:00
Steve Loughran
70c95d15e3
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A. 
Contributed by Aaron Fabbri

(cherry picked from commit 5bbca80428ffbe776650652de86a3bba885edb31)
 2017-09-08 12:08:36 +01:00
Steve Loughran
5757600b8d
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.

(cherry picked from commit 83449abb40633a36b22de538cf82fca240af067e)
 2017-09-07 19:58:03 +01:00
Steve Loughran
e5bb78d3d9
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov

(cherry picked from commit 13eda5000304099d1145631f9be13ce8a00b600d)
 2017-09-07 18:38:24 +01:00
John Zhuge
3dd5444b4d HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 
(cherry picked from commit 50506e90a843aaf1f771a32b872fdd7704b619f5)
 2017-09-06 11:33:43 -07:00