Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,297 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25297 Commits

Author SHA1 Message Date
snehavarma 4c039fafeb
HADOOP-17715 ABFS: Append blob tests with non HNS accounts fail (#3028) 2021-06-09 10:54:10 +05:30
Viraj Jasani ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-08 19:37:23 -07:00
Ayush Saxena a1a3184171
HDFS-15916. Addendum. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#3056) 2021-06-08 19:34:45 -07:00
Viraj Jasani 00d372b663
HADOOP-17725. Improve error message for token providers in ABFS (#3041) 
Contributed by Viraj Jasani.
 2021-06-08 22:03:03 +01:00
Steve Loughran 762a83e044
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.
 2021-06-08 21:56:40 +01:00
Ahmed Hussein a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058) 2021-06-08 11:09:31 -05:00
Prabhu Josephraj 9445abb500 YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar 2021-06-08 20:37:40 +05:30
zhuqi-lucas ec16b1d3b9 YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke. 2021-06-08 21:03:43 +08:00
litao c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
Gautham B A 1c0b2edde9
HADOOP-17727. Modularize docker images (#3043) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-06-07 20:11:08 -07:00
Akira Ajisaka 57a3613e5d
HDFS-16050. Some dynamometer tests fail. (#3079) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-07 14:37:30 +09:00
Viraj Jasani f4b24c68e7
HADOOP-17743. Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects (#3072) 2021-06-07 13:24:09 +09:00
Viraj Jasani 207c92753f
MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project (#3074) 2021-06-07 11:51:29 +09:00
Hideyuki Furue 9c7b8cf54e
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-06 07:05:29 +09:00
July 4a26a61ecd
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:36:09 -07:00
Szilard Nemeth 200eec8f2e YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko 2021-06-03 13:27:36 +02:00
Viraj Jasani 59fc4061cb
HADOOP-17152. Provide Hadoop's own Lists utility to reduce dependency on Guava (#3061) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-03 18:56:00 +09:00
sumangala-patki 76d92eb2a2
HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795) 
. Contributed by Sumangala Patki.
 2021-06-03 14:26:15 +05:30
Szilard Nemeth 2707f69251 YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak 2021-06-01 16:01:39 +02:00
Gergely Pollak e9339aa376 YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth 2021-06-01 15:57:22 +02:00
huhaiyang b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Wei-Chiu Chuang 9a0a808338
HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-31 20:34:48 -07:00
Ayush Saxena b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Steve Loughran bd7b97ba13
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (#3057) 2021-05-31 21:43:26 +01:00
Jinglun e90c41af34 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 19:28:40 +08:00
Jinglun 2b11e160e2 Revert "HDFS-15973." because the user.name and user.email message is not correct. 
This reverts commit b7f69467c1.
 2021-05-30 19:27:10 +08:00
李经纶 b7f69467c1 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 18:50:03 +08:00
Akira Ajisaka 9983ab8a99
HDFS-16046. TestBalancerProcedureScheduler and TestDistCpProcedure timeouts. (#3060) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-29 23:04:48 +09:00
zhengchenyu d5ad181684
MAPREDUCE-7287. Distcp will delete exists file , If we use "-delete and -update" options and distcp file. (#2852) 
Contributed by zhengchenyu
 2021-05-28 20:21:37 +01:00
Konstantin V Shvachko 8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Takanobu Asanuma 53ff2dfed3
Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) 
This reverts commit 0774116756.

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-27 13:14:17 +09:00
Konstantin V Shvachko 1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Gautham B A bcaeb1ac8c
Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898)"" (#3044) 2021-05-26 14:12:53 -07:00
Hui Fei 98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani 986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Akira Ajisaka 8a489ce78e
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-05-26 15:47:56 +09:00
Narges Shadab 3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
zhuxiangyi e18d806212
HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009) 2021-05-26 13:41:49 +08:00
Jim Brennan c4581827a9 YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 2021-05-25 18:07:28 +00:00
Szilard Nemeth b86a6eb871 YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori 2021-05-25 18:27:53 +02:00
Szilard Nemeth 2541efa496 YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori 2021-05-25 13:55:59 +02:00
Akira Ajisaka 1cbcde04f3
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 2021-05-25 18:28:09 +09:00
Steve Loughran 832a3c6a89
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.
 2021-05-25 10:25:41 +01:00
Viraj Jasani 996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project (#3033) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 18:10:20 +09:00
Viraj Jasani 028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
Viraj Jasani 2a206c20cc
HADOOP-17722. Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project (#3032) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:07:22 +09:00
zhuqi-lucas 59172ada90 YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju. 2021-05-24 23:12:07 +08:00
Mehakmeet Singh 5f400032b6
HADOOP-17705. S3A to add Config to set AWS region (#3020) 
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.

This is needed when using AWS Private Link, as
the region cannot be automatically determined.

Contributed by Mehakmeet Singh
 2021-05-24 13:08:45 +01:00
Mehakmeet Singh c665ab02ed
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics 
to be logged at info. (also: `warn` or `error` for even higher
log levels).


Contributed by: Mehakmeet Singh
 2021-05-24 13:02:11 +01:00