65bba8c3ed
YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu.
...
(cherry picked from commit 19e418c10d
2021-03-25 22:35:19 +00:00
67d52af225
HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li.
...
(cherry picked from commit 6288e15118
2021-03-25 14:44:02 +00:00
a07e3c41ca
HADOOP-13551. AWS metrics wire-up ( #2778 )
...
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.
This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.
Explicitly setting the endpoint will save the cost of these probe
HTTP requests.
Contributed by Steve Loughran.
Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b
2021-03-25 13:59:33 +00:00
4c3324ca1a
HADOOP-17305. Fix ITestCustomSigner to work with s3 compatible endpoints ( #2395 )
...
Contributed by Mukund Thakur
Change-Id: Ia5def405056691c349cf05530fd3172047d2058b
2021-03-25 13:59:33 +00:00
b3e968d21e
HDFS-13639. SlotReleaser is not fast enough ( #1885 )
...
(cherry picked from commit be374faf42
2021-03-25 13:25:17 +00:00
80d4636067
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein.
...
(cherry picked from commit c5929c00a3
2021-03-25 19:54:51 +09:00
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.
2021-03-25 12:25:22 +08:00
4db5643ff9
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently ( #2299 )
...
(cherry picked from commit de055ec8aa
2021-03-25 05:09:11 +08:00
9bbf1f87c9
HADOOP-17476. ITestAssumeRole.testAssumeRoleBadInnerAuth failure. ( #2777 )
...
Contributed by Steve Loughran.
Change-Id: Ie96ca99f5d91e5a6aaea4cae4c2e850de9fddb01
2021-03-24 16:49:41 +00:00
9a0c5d0ca2
HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina.
...
(cherry picked from commit 1fdfaebd98
2021-03-24 17:44:29 +08:00
813aafcee2
HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.
...
(cherry picked from commit 6e04b00df1
2021-03-24 17:44:29 +08:00
e4217be749
HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut.
...
(cherry picked from commit 5ab1db9e21
2021-03-24 17:44:29 +08:00
580634882e
HDFS-14546. Document block placement policies. Contributed by Amithsha.
...
(cherry picked from commit 17ffcab5f6
2021-03-24 17:44:29 +08:00
c96821ece8
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 ( #2365 )
...
(cherry picked from commit 735e85a1d2
2021-03-24 17:44:29 +08:00
9e1e89b1b9
HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel.
...
(cherry picked from commit b4d97a8dc7
2021-03-24 17:53:19 +09:00
2d83369f6f
HDFS-15743. Fix -Pdist build failure of hadoop-hdfs-native-client. ( #2569 )
...
(cherry picked from commit 9aefe72ed6
2021-03-24 14:04:47 +08:00
a8c0083b1b
HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.
...
(cherry picked from commit 7025f39944
2021-03-24 14:04:47 +08:00
94766fdb13
HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.
...
(cherry picked from commit ff84a57483
2021-03-24 14:04:47 +08:00
4f79df4da1
HDFS-15551. Tiny Improve for DeadNode detector ( #2265 )
...
Contributed by imbajin.
Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 89428f142f
2021-03-24 14:04:47 +08:00
3ecd02ac78
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
...
(cherry picked from commit f8769e0f4b
2021-03-24 14:04:47 +08:00
f07bde90c9
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
...
(cherry picked from commit 115623a6ee
2021-03-24 14:04:47 +08:00
3a355282c9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream ( #2736 ). Contributed by Narges Shadab.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe9
2021-03-24 14:04:47 +08:00
4abe6e6ac1
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal ( #2790 ). Contributed by Narges Shadab.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0
2021-03-24 14:04:47 +08:00
a216ee3b85
HDFS-15910. Improve security with explicit_bzero ( #2793 )
...
(cherry picked from commit 714427c414
2021-03-24 14:04:47 +08:00
e9b06d942c
HDFS-15249 ThrottledAsyncChecker is not thread-safe. ( #1922 )
...
(cherry picked from commit c12ddbd1de
2021-03-24 14:04:47 +08:00
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result ( #2796 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 22:51:21 -07:00
76c40a52d7
HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.
2021-03-24 02:13:52 +05:30
78bddd0d9f
YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T.
...
(cherry picked from commit 174f3a96b1
2021-03-23 18:23:50 +00:00
b26d75ce37
HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.
...
(cherry picked from commit e0ae232f66
2021-03-23 15:34:48 +00:00
968c95bfba
HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.
...
(cherry picked from commit ae13a5ccbe
2021-03-23 15:34:36 +00:00
91d229bf35
MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
...
(cherry picked from commit ede490d131
2021-03-22 18:44:41 +00:00
d8314fffc0
HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.
...
(cherry picked from commit c2a17659d1
2021-03-22 12:38:09 +00:00
793ab9acf3
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
...
(cherry picked from commit e3d1966f58
2021-03-22 12:35:11 +00:00
bc2bdee76c
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2788 )
...
(cherry picked from commit 343ce8a8c9
2021-03-22 20:30:53 +09:00
23082ac6c7
YARN-10706. Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 ( #2791 )
...
Contributed by Mingliang Liu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-21 21:14:12 -07:00
3aae563421
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. ( #2780 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1
2021-03-22 11:45:42 +09:00
0b43c65816
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
...
(cherry picked from commit 2377df38ad
2021-03-19 18:50:31 +00:00
9665a0be73
HDFS-15633. Avoid redundant RPC calls for getDiskStatus. ( #2386 ). Contributed by Ayush Saxena.
2021-03-19 23:14:18 +05:30
b87c0ea7eb
HADOOP-17594. DistCp: Expose the JobId for applications executing through run method ( #2786 ). Contributed by Ayush Saxena.
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-19 14:20:32 +05:30
27944772d3
HADOOP-17310. Touch command with -c option is broken. ( #2393 ). Contributed by Ayush Saxena.
2021-03-19 00:13:31 +05:30
0e37da334b
YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu.
...
(cherry picked from commit 181b9ddc51
2021-03-18 17:47:18 +00:00
8c82e3507d
YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu.
...
(cherry picked from commit 38495af325
2021-03-18 17:47:10 +00:00
cd417f17ae
YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu.
...
(cherry picked from commit 49f89f1d3d
2021-03-17 18:16:59 +00:00
17307ee88d
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
...
(cherry picked from commit 9c43b60348
2021-03-17 15:13:27 +09:00
aec9d2cff7
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
...
(cherry picked from commit 9ba60c33d8
2021-03-17 14:15:53 +09:00
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7
2021-03-17 12:01:15 +09:00
ad74038e02
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
...
(cherry picked from commit 299b8062f1
2021-03-15 20:17:02 +00:00
1022ed91ac
YARN-10495. make the rpath of container-executor configurable. Contributed by
...
angerszhu.
(cherry picked from commit 7d3c8ef606
2021-03-15 20:01:24 +00:00
f5810ea83c
YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T
...
(cherry picked from commit aa4c17b9d7
2021-03-15 19:14:19 +00:00
e2aaa10512
HADOOP-17532. Yarn Job execution get failed when LZ4 Compression Codec is used. Contributed Bhavik Patel.
2021-03-14 21:20:44 -07:00
Eric Badger
Xiaoyu Yao
Steve Loughran
Mukund Thakur
leosunli
Akira Ajisaka
He Xiaoqiao
LeonGao91
Takanobu Asanuma
Inigo Goiri
Ayush Saxena
Karthik Palanisamy
Masatake Iwasaki
sunlisheng
imbajin
Narges Shadab
Gautham B A
Toshihiro Suzuki
Viraj Jasani
Jim Brennan
litao
Mingliang Liu
kwangsun
touchida
Eric Payne
Chao Sun