Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,252 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26252 Commits

Author SHA1 Message Date
Mehakmeet Singh fba46aa5bb
HADOOP-18499. S3A to support HTTPS web proxies (#5051) 
The option "fs.s3a.proxy.ssl.enabled" controls
whether the s3a connects to a proxy over HTTP (default) or HTTPS.
Set to "true" to use HTTPS.

Contributed by Mehakmeet Singh
 2022-10-26 11:45:20 +01:00
Wang Yu 37bff63c0f
Refactor CallerContext's constructor to eliminate duplicate code (#5070) 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Zander Xu <zanderxu@apache.org>
 2022-10-26 06:40:31 +08:00
FuzzingTeam f140506d67
HADOOP-18504. Fixed an unhandled NullPointerException in class KeyProvider (#5064) 
Contributed by FuzzingTeam
 2022-10-25 18:07:49 +01:00
Ashutosh Gupta 0a26d84df1
HADOOP-9946. NumAllSinks metrics shows lower value than NumActiveSinks (#5002) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:22:25 +08:00
Ashutosh Gupta e6edbf1b4b
YARN-11338. Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher (#4994) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 17:21:12 +09:00
Ashutosh Gupta 21b7790866
YARN-11336. Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp (#4992) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-25 16:52:31 +09:00
slfan1989 454157a384
YARN-11345. [Federation] Refactoring Yarn Router's Application Web Page. (#5030) 2022-10-24 09:35:16 -07:00
Gautham B A 833750f72a
HADOOP-18506. Update build instructions for Windows using VS2019 (#5066) 2022-10-24 09:28:29 -07:00
PJ Fanning aac87ffe76
MAPREDUCE-7411: use secure XML parsers in mapreduce modules (#4980) 
Lockdown of parsers in hadoop-mapreduce.

Follow-on to HADOOP-18469. Add secure XML parser factories to XMLUtils

Contributed by P J Fanning
 2022-10-21 14:02:11 +01:00
slfan1989 d93e6f0cbb
YARN-11295. [Federation] Router Support DelegationToken in MemoryStore mode. (#5032) 2022-10-20 13:14:38 -07:00
Willi Raschkowski c4aa41aa80
HADOOP-18500. Upgrade maven-shade-plugin to 3.3.0 (#5045) 
Contributed by Willi Raschkowski
 2022-10-20 18:47:33 +01:00
FuzzingTeam 7f69e09290
HADOOP-18471. Fixed ArrayIndexOutOfBoundsException in DefaultStringifier (#4957) 
Contributed by FuzzingTeam
 2022-10-20 18:12:17 +01:00
Sneha Vijayarajan a996d889ec
HADOOP-17767. ABFS: Update test scripts (#3124) 
Contributed by Sneha Vijayarajan
 2022-10-20 18:07:04 +01:00
slfan1989 9adf0ca089
YARN-11342. [Federation] Refactor getNewApplication, submitApplication Use FederationActionRetry. (#5005) 2022-10-20 09:22:24 -07:00
jianghuazhu c5c00f3d2c
HDFS-16803.Improve some annotations in hdfs module. (#5031) 2022-10-20 10:58:23 +08:00
slfan1989 48b6f9f335
YARN-11328. Refactoring part of the code of SQLFederationStateStore. (#4976) 2022-10-19 16:11:28 -07:00
Viraj Jasani 8aa04b0b24
HADOOP-18189 S3APrefetchingInputStream to support status probes when closed (#5036) 
Contributed by Viraj Jasani
 2022-10-19 14:38:11 +01:00
Daniel Carl Jones 6207ac47e0
HADOOP-18304. Improve user-facing S3A committers documentation (#4478) 
Contributed by: Daniel Carl Jones
 2022-10-19 12:56:47 +05:30
Steve Loughran d80db6c9e5
HADOOP-18476. Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer (#4966) 
This is to try and close the underlying filesystems when the FileContext APIs are used.
Without this, threads may be leaked
 2022-10-18 14:53:02 +01:00
slfan1989 ee886cacd7
YARN-11247. Remove unused classes introduced by YARN-9615. (#4720) 2022-10-18 09:40:39 -04:00
Hexiaoqiao babb050fa3
HADOOP-18497. Upgrade commons-text version to fix CVE-2022-42889. (#5037). Contributed by PJ Fanning. 
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-10-18 11:28:56 +08:00
Ankit Saurabh 2d91daab5e
HADOOP-18156. Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc (#4965) 
Contributed by Ankit Saurabh
 2022-10-17 18:10:47 +01:00
Ashutosh Gupta 9a8aff69ff
HDFS-6874. Add GETFILEBLOCKLOCATIONS operation to HttpFS (#4750) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
 2022-10-17 18:56:15 +08:00
ZanderXu 136291d2d5
HADOOP-18462. InstrumentedWriteLock should consider Reentrant case (#4919). Contributed by ZanderXu. 
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-17 12:44:25 +08:00
PJ Fanning 4ff6c9b8de
HADOOP-18493: upgrade jackson-databind to 2.12.7.1 (#5011). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-17 10:03:10 +05:30
Szilard Nemeth b0d5182c31 YARN-10680. Revisit try blocks without catch blocks but having finally blocks. Contributed by Susheel Gupta 2022-10-15 21:51:08 +02:00
ahmarsuhail 77e551a478
HADOOP-18481. AWS v2 SDK upgrade log to not about standard AWS Credential Providers. (#4973) 
The AWS SDKV2 upgrade log no longer warns about instantiation
of the v1 SDK credential providers which are commonly used in
s3a configurations:

* com.amazonaws.auth.EnvironmentVariableCredentialsProvider
* com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper
* com.amazonaws.auth.InstanceProfileCredentialsProvider

When the hadoop-aws module moves to the v2 SDK, references to these
credential providers will be rewritten to their v2 equivalents.

Follow-on to HADOOP-18382. "Upgrade AWS SDK to V2 - Prerequisites"

Contributed by Ahmar Suhail
 2022-10-14 10:48:09 +01:00
slfan1989 5b52123c9d
YARN-8041. [Router] Federation: Improve Router REST API Metrics. (#4938) 2022-10-13 16:54:36 -07:00
slfan1989 1962851356
YARN-11294. [Federation] Router Support DelegationToken store/update/remove Token With MemoryStateStore. (#4915) 2022-10-13 16:52:22 -07:00
slfan1989 647457e6ab
YARN-11327. [Federation] Refactoring Yarn Router's Node Web Page. (#5009) 2022-10-13 14:05:30 -07:00
PJ Fanning bfce21ee08
YARN-11330. use secure XML parsers (#4981) 
Move construction of XML parsers in YARN
modules to using the locked-down parser factory
of HADOOP-18469.

One exception: GpuDeviceInformationParser still supports DTD resolution;
all other features are disabled.

Contributed by P J Fanning
 2022-10-13 18:19:19 +01:00
monthonk 9439d8e4e4
HADOOP-18292. Fix s3 select tests when running against unsupported storage class (#4489) 
Follow-on from HADOOP-12020.

Contributed by Monthon Klongklaew
 2022-10-13 13:13:36 +01:00
slfan1989 3ff8f58f8c
HADOOP-18360. Update commons-csv from 1.0 to 1.9.0. (#4928). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:10:54 +05:30
slfan1989 1ff7e84caf
YARN-11334. Improve SubClusterState#fromString parameter and LogMessage. (#4988). Contributed by fanshilun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-10-13 12:06:24 +05:30
Ashutosh Gupta d6b1e1eeb6
HDFS-16777. datatables@1.10.17 sonatype-2020-0988 vulnerability (#5003) 2022-10-12 14:51:12 -07:00
Gautham B A 5694d7e25f
Add Dockerfile_windows_10 (#4936) 2022-10-12 09:30:54 -07:00
slfan1989 d78b0b39a6
YARN-11323. [Federation] Improve ResourceManager Handler FinishApps. (#4954) 2022-10-11 14:53:02 -07:00
slfan1989 9e16f1f883
YARN-11317. [Federation] Refactoring Yarn Router's About Web Page. (#4946) 2022-10-11 13:30:48 -07:00
slfan1989 82a88a8ae6
YARN-11315. [Federation] YARN Federation Router Supports Cross-Origin. (#4934) 2022-10-11 10:18:50 -07:00
Gautham B A 2122733c30
Add .yetus/excludes.txt (#4984) 2022-10-11 09:23:34 -07:00
belugabehr 03d600fa82
HADOOP-17779: Lock File System Creator Semaphore Uninterruptibly (#3158) 2022-10-11 11:56:32 +01:00
huhaiyang d14b88c698
HDFS-16774.Improve async delete replica on datanode (#4903) 
HDFS-16774. Improve async delete replica on datanode to reduce the probability of ReplicationNotFoundException

Co-authored-by: Haiyang Hu <haiyang.hu@shopee.com>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-11 16:22:56 +08:00
PJ Fanning 4fe079f85f
HDFS-16795. Use secure XML parsers (#4979) 
Contributed by P J Fanning
 2022-10-10 18:56:35 +01:00
Szilard Nemeth 0c515b0ef0 YARN-6766. Add helper method in FairSchedulerAppsBlock to print app info. Contributed by Riya Khandelwal 2022-10-10 15:30:33 +02:00
ZanderXu 62ff4e36cf
HDFS-16787. Remove the redundant lock in DataSetLockManager#removeLock (#4948). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 20:05:02 +08:00
ZanderXu b0b2cb4a16
HDFS-16783. Remove the redundant lock in deepCopyReplica and getFinalizedBlocks (#4942). Contributed by ZanderXu. 
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-10 19:52:22 +08:00
Mukund Thakur be70bbb4be
HADOOP-18460. checkIfVectoredIOStopped before populating the buffers (#4986) 
Contributed by Mukund Thakur
 2022-10-10 11:17:45 +01:00
Steve Loughran 540a660429
HADOOP-18480. Upgrade aws sdk to 1.12.316 (#4972) 
Contributed by Steve Loughran
 2022-10-10 10:23:50 +01:00
Ashutosh Gupta 9a7d0e7ed0
YARN-11260. Upgrade JUnit from 4 to 5 in hadoop-yarn-server-timelineservice (#4775) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-10-10 00:28:03 +09:00
ZanderXu b0bfd09c41
HDFS-16798. HDFS-16798. SerialNumberMap should decrease current counter if the item exist. (#4987). Contributed by ZanderXu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-10-09 18:58:09 +08:00