68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer ( #3127 )
...
(cherry picked from commit d9fbb3c508
2021-06-23 11:24:00 +08:00
4825c7c28d
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects ( #3123 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa
2021-06-21 16:47:37 +09:00
b8a98e4f82
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79
2021-06-21 10:28:05 +09:00
7d202e8365
HDFS-16079. Improve the block state change log ( #3120 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:24:03 +05:30
ef3a2a3b3b
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result ( #3121 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:23:52 +05:30
92ade1f6f9
HDFS-16061. DFTestUtil.waitReplication can produce false positives ( #3095 ). Contributed by Ahmed Hussein.
...
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:49 +05:30
8d241482bd
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() ( #3119 ). Contributed by tomscut.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:38 +05:30
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet ( #3113 )
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
2021-06-18 14:41:02 +08:00
577b96ef42
HDFS-16074. Remove an expensive debug string concatenation ( #3107 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548
2021-06-17 11:39:34 +08:00
25138c98bf
HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml ( #3099 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129
2021-06-17 10:00:36 +09:00
3e03ecd680
HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w.
2021-06-16 15:54:36 +05:30
57ee4238eb
YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy.
...
(cherry picked from commit 9a6a11c452
2021-06-15 19:30:16 +00:00
546c66b812
HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS ( #3104 )
...
(cherry picked from commit 839fcf7682
2021-06-15 17:22:34 +09:00
840612b3b1
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
...
(cherry picked from commit 747883af4d
2021-06-14 14:37:27 +00:00
10000f162d
HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. ( #3090 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580
2021-06-14 10:21:30 +09:00
254d943126
HADOOP-17547 Magic committer to downgrade abort in cleanup if list uploads fails with access denied ( #3051 )
...
Contributed by Bogdan Stolojan
Change-Id: I32d6dc4f72087783a3ea12473d11690ac14fe3cb
2021-06-12 17:46:11 +01:00
52d055a7eb
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state ( #3084 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0
2021-06-11 17:33:06 +09:00
91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp ( #3042 )"
...
This reverts commit 8c0f948054
2021-06-11 00:35:41 -07:00
19c969981d
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. ( #3068 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9a17f97133
2021-06-10 14:33:41 +09:00
0d6939ffc4
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils ( #3081 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca27744770
2021-06-09 10:38:38 +08:00
83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. ( #2863 ). Contributed by Ayush Saxena. ( #3054 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289
2021-06-08 19:32:34 -07:00
8f0ba9ee1b
HADOOP-17725. Improve error message for token providers in ABFS ( #3041 )
...
Contributed by Viraj Jasani.
2021-06-08 22:05:01 +01:00
4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true ( #2977 )
...
Contributed by Steve Loughran.
Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
2021-06-08 22:05:00 +01:00
02249171b1
HDFS-16048. RBF: Print network topology on the router web ( #3062 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a
2021-06-08 16:59:59 +09:00
46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 ( #3021 )
2021-06-08 00:10:37 -07:00
37516726d7
HDFS-16050. Some dynamometer tests fail. ( #3079 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 57a3613e5d
2021-06-07 15:03:06 +09:00
8ca57fc9f5
Fix container-executor
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9c7b8cf54e
2021-06-06 07:29:22 +09:00
8c0f948054
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp ( #3042 )
...
Contributed by kaifeiYi (yikf).
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-06-04 13:37:24 -07:00
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles ( #3036 )
...
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e528
2021-06-01 15:30:50 +08:00
5cebd02bbe
HADOOP-17739. Use hadoop-thirdparty 1.1.1. ( #3064 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9a0a808338
2021-06-01 11:36:27 +08:00
bb04b60197
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 ( #3057 )
...
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
2021-05-31 21:49:39 +01:00
7feb41b73d
MAPREDUCE-7287. Distcp will delete exists file , If we use "-delete and -update" options and distcp file. ( #2852 )
...
Contributed by zhengchenyu
Change-Id: I61edf9a443c0c6cd5b5dd911901708530cf131ed
2021-05-28 20:27:00 +01:00
c5535caf6e
HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.
...
(cherry picked from commit 8ce30f51f9
2021-05-27 18:53:46 -07:00
d8689f1a08
Revert "HADOOP-17563. Update Bouncy Castle to 1.68. ( #2740 )" ( #3055 )
...
This reverts commit 077411675653ff2dfed3
2021-05-27 13:17:24 +09:00
5308d44be4
HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit 1abd03d68f
2021-05-26 15:09:47 -07:00
e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. ( #3053 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e
2021-05-26 15:48:51 +09:00
41025cd585
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap ( #3027 )
...
(cherry picked from commit 3fdeb7435a
2021-05-26 14:45:20 +09:00
48aa23eb3b
YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T.
...
(cherry picked from commit c4581827a9
2021-05-25 18:10:24 +00:00
464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector ( #2807 )
...
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.
The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.
If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.
Contributed by Steve Loughran.
Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
2021-05-25 12:55:38 +01:00
ad7071c93b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja.
...
(cherry picked from commit 1cbcde04f3
2021-05-25 18:29:20 +09:00
b82a0fa9e6
HADOOP-17705. S3A to add Config to set AWS region ( #3020 )
...
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.
This is needed when using AWS Private Link, as
the region cannot be automatically determined.
Contributed by Mehakmeet Singh
Change-Id: I4b52f85d7af0ddd56b2b0505ac0124d4fcc67ca0
2021-05-24 13:13:37 +01:00
a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() ( #2963 )
...
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:
org.apache.hadoop.fs.statistics.IOStatisticsLogging
Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).
Contributed by: Mehakmeet Singh
Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
2021-05-24 13:04:20 +01:00
745a2f5ec3
HADOOP-17723. [build] fix the Dockerfile for ARM ( #3037 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1576f81dfe
2021-05-24 18:20:49 +08:00
dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist ( #2767 )
...
(cherry picked from commit 2bbeae3240
2021-05-24 18:00:38 +08:00
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )"
...
This reverts commit 4ffe5eb1dd
2021-05-24 17:37:18 +08:00
ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
...
(cherry picked from commit da1ea2530f
2021-05-24 10:48:48 +08:00
6777acb08c
YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja
2021-05-22 14:36:51 +02:00
87e32fbb8d
HADOOP-17700. ExitUtil#halt info log should log HaltException ( #3015 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548
2021-05-22 19:10:28 +09:00
526dbe4716
HADOOP-17666. Update LICENSE for 3.3.1 ( #3011 )
...
* Inspected the jar files in the produced tarball and updated LICENSE-binary accordingly.
* add LICENSE from hadoop-thirdparty jars.
* remove any dependencies no longer in the tarball.
* Updated the license of thirdparty javascripts and C/C++ files.
Added LICENSE-asio.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/asio-1.10.2/COPYING
Added LICENSE-gmock.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/LICENSE
Added LICENSE-rapidxml.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/rapidxml-1.13/rapidxml/license.txt
Added LICENSE-uriparser2.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser/COPYING
Added LICENSE-tr2.txt, copied from the header of hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/tr2/optional.hpp
Added LICENSE-cJSON.txt, moved from the bottom of LICENSE.txt
* Generated license report for yarn-managed packages.
* Add LICENSE and NOTICES file of jaxb-api.
* Exclude LICENSE-binary-{yarn-applications-catalog-webapp|yarn-ui} from rat report.
These two files are autogenerated.
2021-05-21 18:15:48 -07:00
23ada04dac
HADOOP-17509. Parallelize building of dependencies ( #2669 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6c891c0dd4
2021-05-22 01:32:37 +08:00
Viraj Jasani
litao
Ahmed Hussein
Hui Fei
Wei-Chiu Chuang
Takanobu Asanuma
Ayush Saxena
Jim Brennan
Petre Bogdan Stolojan
Mingliang Liu
Akira Ajisaka
Steve Loughran
Hideyuki Furue
July
huhaiyang
zhengchenyu
Konstantin V Shvachko
Narges Shadab
Mehakmeet Singh
Vinayakumar B
Yiqun Lin
Szilard Nemeth
Gautham B A