Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,750 Commits 326 Branches 371 Tags
Commit Graph

973 Commits

Author SHA1 Message Date
Arpit Agarwal
3e57adee0a HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f14ddff581b63f0902b7760e4d2bd56)
 2018-11-15 13:58:38 -08:00
Sammi Chen
5d532cfc6f HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce84dcf68bcd7de5a5ed1c7d904796e9)
(cherry picked from commit 64cb97fb4467513f73fde18f96f391ad34e3bb0a)
 2018-11-14 13:12:22 +08:00
Steve Loughran
cd641f3218
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran.

Missed final change to ITestAssumeRole
 2018-11-01 12:55:19 +00:00
Steve Loughran
3f35946c6e
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran

This is a subset of c07715e37895a2e406 to track changes in the AWS STS error messages.
If the later AWS SDK is ever backported to older branches, the rest of that patch will need
cherry-picking
 2018-11-01 12:52:48 +00:00
Sammi Chen
366541d834 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538e838f40581519080d7c8621c65e53b)
 2018-10-26 15:28:20 +08:00
Ted Yu
84726679f0 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
(cherry picked from commit a7dd244a4995cbf20feb054530fc5913ccf384ac)
 2018-10-19 13:22:31 -07:00
Steve Loughran
f1fbc56371
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-12 09:48:17 +01:00
Todd Lipcon
89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3cd9ffdc71946fd12f5c3d59718c4939)
 2018-10-09 13:50:00 -05:00
Sammi Chen
5da3e83597 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9)
 2018-09-25 19:50:39 +08:00
Steve Loughran
1c2c0ed209
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.

(cherry picked from commit 1eef0acfe5725d08d3a4eb76c5138076ff1857f1)
 2018-09-10 14:25:42 +01:00
Steve Loughran
a0766bf66a
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.

(cherry picked from commit 5a0babf76550f63dad4c17173c4da2bf335c6532)
 2018-08-30 15:23:08 +01:00
Steve Loughran
6f939d4294
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.

(cherry picked from commit e8d138ca7c1b695688515d816ac693437c87df62)
 2018-08-30 14:37:07 +01:00
Wangda Tan
5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Sammi Chen
42e34dae57 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 
(cherry picked from commit 0857f116b754d83d3c540cd6f989087af24fef27)
(cherry picked from commit f9aedf32ecdc6075c8760220cdfadecb9f1b6738)
 2018-07-30 11:00:30 +08:00
bibinchundatt
d2212c20c5 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 
(cherry picked from commit 3d3158cea4580eb2e3b2af635c3a7d30f4dbb873)
 2018-07-25 19:00:12 +05:30
Steve Loughran
749fff577e
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.
 2018-07-19 12:29:21 -07:00
Wangda Tan
44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4cf0f6ebcb40a61f622f1a885c4bda7)
 2018-07-16 11:04:08 -07:00
Steve Loughran
caf38532f3
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting 
connection in case of timeout. Contributed by Sean Mackrory.

(cherry picked from commit d503f65b6689b19278ec2a0cf9da5a8762539de8)
 2018-07-11 14:55:11 +01:00
Steve Loughran
d54241e9c9
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan. 
Contributed by Steve Loughran.

(cherry picked from commit ca8b80bf59c0570bb9172208d3a6c993a6854514)
 2018-07-10 10:50:40 +01:00
Sunil G
aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Chris Douglas
7e655b642d HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603ee93e082e7ba148df1400af5ac7c30c)
 2018-06-17 23:12:59 -07:00
Inigo Goiri
8be28376a9 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594ffc372e3f846b0c8ebc2ff9e36ffdb0)
 2018-06-12 11:00:22 -07:00
Steve Loughran
baac7c2b28 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be679554eb6b323f3bc6e8de267d85dd2e06)
 2018-06-11 15:33:03 -07:00
Inigo Goiri
6918d9e9c8 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 8fdc993a993728c65084d7dc3ac469059cb1f603)
 2018-05-28 16:46:21 -07:00
Steve Loughran
a89e65faa0 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c94d31d3f737226d954c566b5d88656b)
 2018-05-21 11:13:18 +01:00
Steve Loughran
170f1040d4 HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest. 
Contributed by Steve Loughran.
 2018-05-17 19:08:05 +01:00
Steve Loughran
a0cb1e0941 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name. 
Contributed by Sean Mackrory.

(cherry picked from commit b6708374692e6c4d786e2f3f1f45cc7aa1e4e88f)
 2018-05-15 16:15:13 +01:00
Steve Loughran
057a4a6bb7 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt
 2018-05-07 11:56:00 +01:00
Weiwei Yang
386c39cd84 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:44:35 +08:00
Yiqun Lin
213f733604 HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 
(cherry picked from commit 1312f9ae4c3c86501b8f3a2dda8c48e8652e3128)
 2018-04-18 16:06:39 +08:00
Xiao Chen
805e33b62c HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-16 11:58:10 -07:00
Sean Mackrory
2b46bd3f45 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-04-13 10:13:56 -07:00
Aaron Fabbri
86f543aa85
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:58:22 -07:00
Wangda Tan
25bf1d3ac6 Updated branch-3.1 to track 3.1.1 
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
 2018-04-09 10:58:50 -07:00
Sammi Chen
e96c7bf82d HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 
(cherry picked from commit 6542d17ea460ec222137c4b275b13daf15d3fca3)
 2018-03-30 20:26:35 +08:00
Sean Mackrory
b038e21440 HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 
(cherry picked from commit 9d7a9031a5978efc8d97566e35ebaace20db2353)
 2018-03-29 14:04:55 -06:00
Chris Douglas
ac16e8f4d3 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 
(cherry picked from commit 081c3501885c543bb1f159929d456d1ba2e3650c)
 2018-03-28 11:59:08 -07:00
Haibo Chen
e00e08365e YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen) 
(cherry picked from commit fcea5a4d798a517443afd0b441883ccc4e9136ba)
 2018-03-26 14:33:46 -07:00
Sammi Chen
5acc13f340 HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec5c60d96b0c216182891cdfd7832ac5)
(cherry picked from commit 2285afb32e71622b3dab5051247a1d099cfcbe85)
(cherry picked from commit 322520eb76cdcef25190495ccf98b3ca39907f58)
(cherry picked from commit 7985c5fdc9fe26e7771a86a5004bf90fcbd8af71)
 2018-03-20 14:03:17 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
0bd99621c5 Preparing for 3.1.0 development 2018-03-19 15:16:54 -07:00
Wangda Tan
f47659fb97 MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda) 
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
 2018-03-15 13:26:01 -07:00
Steve Loughran
1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories. 
Contributed by Steve Loughran.
 2018-03-15 18:05:14 +00:00
fang zhenyi
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Chris Douglas
45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Chris Douglas
45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran
dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional. 
Contributed by Steve Loughran.
 2018-03-12 14:01:42 +00:00
Steve Loughran
7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.
 2018-03-08 11:24:06 +00:00
Steve Loughran
3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down. 
Contributed by Virajith Jalaparti.
 2018-03-08 11:15:46 +00:00
Steve Loughran
e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 18:51:10 +00:00
Steve Loughran
2e1e049bd0 HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile. 
Contributed by Steve Loughran.
 2018-03-05 18:12:38 +00:00