Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,787 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

974 Commits

Author SHA1 Message Date
Weiwei Yang fdb95ef150 HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 
(cherry picked from commit be0708c6eb)
 2018-11-19 14:32:58 +08:00
Arpit Agarwal 3e57adee0a HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f)
 2018-11-15 13:58:38 -08:00
Sammi Chen 5d532cfc6f HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce)
(cherry picked from commit 64cb97fb44)
 2018-11-14 13:12:22 +08:00
Steve Loughran cd641f3218
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran.

Missed final change to ITestAssumeRole
 2018-11-01 12:55:19 +00:00
Steve Loughran 3f35946c6e
HADOOP-15781. S3A assumed role tests failing due to changed error text in AWS exceptions. 
Contributed by Steve Loughran

This is a subset of c07715e378 to track changes in the AWS STS error messages.
If the later AWS SDK is ever backported to older branches, the rest of that patch will need
cherry-picking
 2018-11-01 12:52:48 +00:00
Sammi Chen 366541d834 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538)
 2018-10-26 15:28:20 +08:00
Ted Yu 84726679f0 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e)
(cherry picked from commit a7dd244a49)
 2018-10-19 13:22:31 -07:00
Steve Loughran f1fbc56371
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.
 2018-10-12 09:48:17 +01:00
Todd Lipcon 89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3c)
 2018-10-09 13:50:00 -05:00
Sammi Chen 5da3e83597 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb)
 2018-09-25 19:50:39 +08:00
Steve Loughran 1c2c0ed209
HADOOP-15677. WASB: Add support for StreamCapabilities. 
Contributed by Thomas Marquardt.

(cherry picked from commit 1eef0acfe5)
 2018-09-10 14:25:42 +01:00
Steve Loughran a0766bf66a
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs. 
Contributed by Steve Loughran.

(cherry picked from commit 5a0babf765)
 2018-08-30 15:23:08 +01:00
Steve Loughran 6f939d4294
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out. 
Contributed by Andras Bokor.

(cherry picked from commit e8d138ca7c)
 2018-08-30 14:37:07 +01:00
Wangda Tan 5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Sammi Chen 42e34dae57 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 
(cherry picked from commit 0857f116b7)
(cherry picked from commit f9aedf32ec)
 2018-07-30 11:00:30 +08:00
bibinchundatt d2212c20c5 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 
(cherry picked from commit 3d3158cea4)
 2018-07-25 19:00:12 +05:30
Steve Loughran 749fff577e
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.
 2018-07-19 12:29:21 -07:00
Wangda Tan 44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4)
 2018-07-16 11:04:08 -07:00
Steve Loughran caf38532f3
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting 
connection in case of timeout. Contributed by Sean Mackrory.

(cherry picked from commit d503f65b66)
 2018-07-11 14:55:11 +01:00
Steve Loughran d54241e9c9
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan. 
Contributed by Steve Loughran.

(cherry picked from commit ca8b80bf59)
 2018-07-10 10:50:40 +01:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Chris Douglas 7e655b642d HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603e)
 2018-06-17 23:12:59 -07:00
Inigo Goiri 8be28376a9 YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit c35481594f)
 2018-06-12 11:00:22 -07:00
Steve Loughran baac7c2b28 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be6795)
 2018-06-11 15:33:03 -07:00
Inigo Goiri 6918d9e9c8 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 8fdc993a99)
 2018-05-28 16:46:21 -07:00
Steve Loughran a89e65faa0 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c9)
 2018-05-21 11:13:18 +01:00
Steve Loughran 170f1040d4 HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest. 
Contributed by Steve Loughran.
 2018-05-17 19:08:05 +01:00
Steve Loughran a0cb1e0941 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name. 
Contributed by Sean Mackrory.

(cherry picked from commit b670837469)
 2018-05-15 16:15:13 +01:00
Steve Loughran 057a4a6bb7 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt
 2018-05-07 11:56:00 +01:00
Weiwei Yang 386c39cd84 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:44:35 +08:00
Yiqun Lin 213f733604 HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules. 
(cherry picked from commit 1312f9ae4c)
 2018-04-18 16:06:39 +08:00
Xiao Chen 805e33b62c HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-16 11:58:10 -07:00
Sean Mackrory 2b46bd3f45 HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota. 2018-04-13 10:13:56 -07:00
Aaron Fabbri 86f543aa85
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran. 2018-04-10 23:58:22 -07:00
Wangda Tan 25bf1d3ac6 Updated branch-3.1 to track 3.1.1 
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
 2018-04-09 10:58:50 -07:00
Sammi Chen e96c7bf82d HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu. 
(cherry picked from commit 6542d17ea4)
 2018-03-30 20:26:35 +08:00
Sean Mackrory b038e21440 HADOOP-15342. Updating ADLS connector to use the current SDK version (2.2.7). Contributed by Atul Sikaria. 
(cherry picked from commit 9d7a9031a5)
 2018-03-29 14:04:55 -06:00
Chris Douglas ac16e8f4d3 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 
(cherry picked from commit 081c350188)
 2018-03-28 11:59:08 -07:00
Haibo Chen e00e08365e YARN-7794. SLSRunner is not loading timeline service jars, causing failure. (Yufei Gu via Haibo Chen) 
(cherry picked from commit fcea5a4d79)
 2018-03-26 14:33:46 -07:00
Sammi Chen 5acc13f340 HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu. 
(cherry picked from commit d67a5e2dec)
(cherry picked from commit 2285afb32e)
(cherry picked from commit 322520eb76)
(cherry picked from commit 7985c5fdc9)
 2018-03-20 14:03:17 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0bd99621c5 Preparing for 3.1.0 development 2018-03-19 15:16:54 -07:00
Wangda Tan f47659fb97 MAPREDUCE-7047. Make HAR tool support IndexedLogAggregtionController. (Xuan Gong via wangda) 
Change-Id: Ice5ae1c706f2476361997dcbb29f2c33c77d4f0c
 2018-03-15 13:26:01 -07:00
Steve Loughran 1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories. 
Contributed by Steve Loughran.
 2018-03-15 18:05:14 +00:00
fang zhenyi 5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Chris Douglas 45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Chris Douglas 45d1b0fdcc HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 2018-03-12 20:05:39 -07:00
Steve Loughran dd05871b8b HADOOP-15297. Make S3A etag => checksum feature optional. 
Contributed by Steve Loughran.
 2018-03-12 14:01:42 +00:00
Steve Loughran 7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms. 
Contributed by Steve Loughran.
 2018-03-08 11:24:06 +00:00
Steve Loughran 3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down. 
Contributed by Virajith Jalaparti.
 2018-03-08 11:15:46 +00:00
Steve Loughran e0307e53e2 HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d)
 2018-03-07 18:51:10 +00:00
Steve Loughran 2e1e049bd0 HADOOP-15288. TestSwiftFileSystemBlockLocation doesn't compile. 
Contributed by Steve Loughran.
 2018-03-05 18:12:38 +00:00
Steve Loughran 8110d6a0d5 HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions. 
Contributed by Aaron Fabbri.
 2018-03-05 14:06:20 +00:00
Gergely Novák e99a1e7767
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-02 16:15:04 -08:00
Ray Chiang 923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
fang zhenyi 29233c3c4a
HADOOP-15274. Move hadoop-openstack to slf4j 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-01 15:08:30 -08:00
Steve Loughran edc9f1451b HADOOP-15263. hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake. 
Contributed by Steve Loughran
 2018-02-28 14:42:25 +00:00
Weiwei Yang 7af4f34de5 YARN-7929. Support to set container execution type in SLS. (Jiandan Yang via Weiwei Yang) 2018-02-28 17:58:09 +08:00
Arpit Agarwal 2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Xuan Gong 68ce193efc MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák 2018-02-23 14:37:26 -08:00
Carlo Curino 84cea0011f YARN-7732. Support Generic AM Simulator from SynthGenerator. (Contributed by Young Chen via curino) 2018-02-20 17:00:34 -08:00
fang zhenyi 4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Steve Loughran 7ac88244c5 HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key. 
Contributed by Steve Loughran.
 2018-02-16 16:37:06 +00:00
Steve Loughran 481d79fedc HADOOP-13972. ADLS to support per-store configuration. 
Contributed by Sharad Sonker.

(cherry picked from commit 050f5287b7)
 2018-02-15 16:27:31 +00:00
Steve Loughran 9a013b255f HADOOP-15176. Enhance IAM Assumed Role support in S3A client. 
Contributed by Steve Loughran

(cherry picked from commit 96c047fbb9)
 2018-02-15 15:57:10 +00:00
Steve Loughran b27ab7dd81 HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document. 
Contributed by Steve Loughran.

(cherry picked from commit c761e658f6)
 2018-02-15 14:57:56 +00:00
Steve Loughran c9a373fb14 HADOOP-15090. Add ADL troubleshooting doc. 
Contributed by Steve Loughran.

(cherry picked from commit 58a2120e8a)
 2018-02-15 14:26:00 +00:00
Steve Loughran f20dc0d577 HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-14 16:20:14 +00:00
Steve Loughran 8cf88fcd1f HADOOP-15187. Remove ADL mock test dependency on REST call invoked from Java SDK. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit bd5ab59125)
 2018-02-12 15:13:00 +00:00
Wangda Tan 60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Anu Engineer 4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Sunil G 9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Arun Suresh 38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Steve Loughran 7fd287b4af HADOOP-15186. Allow Azure Data Lake SDK dependency version to be set on the command line. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit 95a96b13e2)
 2018-01-29 09:48:14 -08:00
Jason Lowe ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Yufei Gu 16be42d309 YARN-7798. Refactor SLS Reservation Creation. Contributed by Young Chen. 2018-01-25 13:06:01 -08:00
Steve Loughran f63d13f10d HADOOP-15185. Update adls connector to use the current version of ADLS SDK. 
Contributed by Atul Sikaria.
 2018-01-23 09:09:11 -08:00
Steve Loughran 1093a73689 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2018-01-18 13:13:58 +00:00
Steve Loughran f274fe33ea Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits." 
This reverts commit 35ad9b1dd2.
 2018-01-18 12:35:57 +00:00
Aaron Fabbri 268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran. 2018-01-17 00:05:24 -08:00
Sammi Chen 9195a6e302 HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu) 2018-01-17 15:55:59 +08:00
Steve Loughran a0c71dcc33 HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing 
after OutputCommitter patch.
Contributed by Steve Loughran
 2018-01-15 11:33:32 +00:00
Mukul Kumar Singh 2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-15 15:45:51 +09:00
Steve Loughran 1a09da7400 HADOOP-15163. Fix S3ACommitter documentation 
Contributed by Alessandro Andrioni.

(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
 2018-01-10 15:37:07 +00:00
Steve Loughran b62a5ece95 HADOOP-15161. s3a: Stream and common statistics missing from metrics 
Contributed by Sean Mackrory
 2018-01-09 18:46:52 +00:00
Steve Loughran 52babbb4a0 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
 2017-12-22 11:39:55 +00:00
Steve Loughran c8ff0cc304 HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls. 
Contributed by Steve Loughran
 2017-12-21 14:58:58 +00:00
Steve Loughran ef450df443 HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance. 
Contributed by Steve Loughran.
 2017-12-21 14:15:53 +00:00
Akira Ajisaka 5ab632baf5
HDFS-12949. Fix findbugs warning in ImageWriter.java. 2017-12-21 10:04:34 +09:00
Steve Loughran 1ba491ff90 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 2017-12-20 18:25:33 +00:00
Steve Loughran 35ad9b1dd2 HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits. 
Contributed by Aaron Fabbri
 2017-12-18 21:19:06 +00:00
Chris Douglas 4b3a785914 HDFS-12903. [READ] Fix closing streams in ImageWriter. Contributed by Virajith Jalaparti 2017-12-15 17:51:42 -08:00
Chris Douglas e515103a83 Revert "HDFS-12903. [READ] Fix closing streams in ImageWriter" 
This reverts commit c1bf2654b0.
 2017-12-15 17:51:42 -08:00
Virajith Jalaparti 8239e3afb3 HDFS-12712. [9806] Code style cleanup 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 80c3fec3a1 HDFS-12912. [READ] Fix configuration and implementation of LevelDB-based alias maps 2017-12-15 17:51:41 -08:00
Virajith Jalaparti c89b29bd42 HDFS-12893. [READ] Support replication of Provided blocks with non-default topologies. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 0f6aa9564c HDFS-12905. [READ] Handle decommissioning and under-maintenance Datanodes with Provided storage. 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 962b5e722b HDFS-12903. [READ] Fix closing streams in ImageWriter 2017-12-15 17:51:41 -08:00
Virajith Jalaparti 4531588a94 HDFS-11640. [READ] Datanodes should use a unique identifier when reading from external stores 2017-12-15 17:51:41 -08:00
Virajith Jalaparti fb996a32a9 HDFS-12894. [READ] Skip setting block count of ProvidedDatanodeStorageInfo on DN registration update 2017-12-15 17:51:40 -08:00