ec3311975c
HADOOP-16290. Enable RpcMetrics units to be configurable ( #3198 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e1d00addb5
2021-07-20 14:56:28 +08:00
177569f1af
YARN-10858. [UI2] YARN-10826 breaks Queue view. ( #3213 )
...
(cherry picked from commit 17bf2fcbc5
2021-07-20 14:16:05 +09:00
95a70c0fdf
HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee.
...
(cherry picked from commit 47002719f2
2021-07-16 14:23:36 -05:00
14a3e74c5c
HADOOP-17801. No error message reported when bucket doesn't exist in S3AFS ( #3202 )
...
Contributed by: Mehakmeet Singh.
Change-Id: I26c2a85ef6bbfd1b8269a23fc44d9a55d7fa091c
2021-07-16 15:36:54 +01:00
cd15b0cb8a
HADOOP-17803. Remove WARN logging from LoggingAuditor when executing a request outside an audit span ( #3207 )
...
Followup to HADOOP-17511. "Add audit/telemetry logging to S3A connector"
Contributed by Mehakmeet Singh
2021-07-16 11:52:37 +01:00
3faae2e06d
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable ( #2414 )
...
(cherry picked from commit 4bb25c810b
2021-07-16 11:07:27 +08:00
0f2b89b791
HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein.
2021-07-15 20:47:02 +00:00
b3481062e0
YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne.
...
(cherry picked from commit 632f64cadb
2021-07-15 14:26:03 +00:00
450dae7383
HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das ( #2260 )
...
(cherry picked from commit 1dd03cc4b5
2021-07-13 18:23:27 -07:00
11825d30e8
HADOOP-17714 ABFS: testBlobBackCompatibility, testRandomRead & WasbAbfsCompatibility tests fail when triggered with default configs ( #3035 ) ( #3126 )
...
(cherry picked from commit 35e4c31fff
2021-07-12 11:53:46 +05:30
ab3809cf8d
HADOOP-17715 ABFS: Append blob tests with non HNS accounts fail ( #3028 ) ( #3125 )
...
(cherry picked from commit 4c039fafeb
2021-07-12 11:51:41 +05:30
a1f90883f0
HADOOP-12665. Document hadoop.security.token.service.use_ip. ( #3187 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d
2021-07-12 10:18:19 +09:00
aa6a9cac72
HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const ( #3106 )
...
* HADOOP-17596. ABFS: Change default Readahead Queue Depth from num(processors) to const (#2795 )
. Contributed by Sumangala Patki.
(cherry picked from commit 76d92eb2a2
2021-07-10 15:09:59 +05:30
7cb91db575
HDFS-16122. Fix DistCpContext#toString() ( #3191 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-10 13:56:36 +05:30
c18ad9a007
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada
2021-07-10 13:43:31 +09:00
6cc9549499
HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS ( #3190 )
...
(cherry picked from commit 6ac5d8a1ed
2021-07-09 22:50:18 +09:00
705884b87a
HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma
...
(cherry picked from commit b6c06c4b76
2021-07-09 10:19:47 +01:00
ab939814a3
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. ( #3188 )
...
(cherry picked from commit 6450c1ea9a
2021-07-09 09:17:10 +01:00
e3fb63f33f
HADOOP-17402. Add GCS config to the core-site ( #2638 )
...
Contributed by Rafal Wojdyla
2021-07-07 22:43:31 +01:00
e395711164
MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST)
...
(cherry picked from commit 7581413156
2021-07-07 20:57:32 +00:00
99cff01e38
HADOOP-17775. Remove JavaScript package from Docker environment. ( #3182 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-07 17:49:19 +09:00
24b780820c
HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout ( #3080 )
...
(cherry picked from commit a5db6831bc
2021-07-07 09:41:11 +08:00
e8f9af6f2a
HADOOP-17250 Lot of short reads can be merged with readahead. ( #3110 )
...
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.
This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.
Contributed By: Mukund Thakur
Change-Id: Ib920eedd0087caa150afa4d4c23e89df56b29e83
2021-07-05 11:23:32 +01:00
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer ( #3169 )
...
(cherry picked from commit da575a53cf
2021-07-05 13:42:51 +09:00
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout ( #3172 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e4
2021-07-05 08:15:10 +09:00
f1a14df9e6
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value ( #3144 )
...
Contributed by: Mehakmeet Singh
Change-Id: I3302654ca36474a5f399aa848f88bce4587022d8
2021-07-02 14:13:26 +01:00
bf6ed21104
HDFS-16090. Fine grained lock for datanodeNetworkCounts ( #3148 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack Chuang <stack@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-30 12:59:25 -07:00
628b08db63
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 ( #3152 )
...
(cherry picked from commit c005195b78
2021-06-30 09:35:36 +08:00
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor ( #3146 )
...
(cherry picked from commit 95454d821c
2021-06-29 18:15:56 +08:00
8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects ( #3150 )
...
(cherry picked from commit 93a1685073
2021-06-29 18:13:00 +08:00
47b3939009
YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne.
...
(cherry picked from commit dc6f456e95
2021-06-28 14:55:26 +00:00
75f8198aa8
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
...
(cherry picked from commit 7c7d02edbd
2021-06-25 20:36:41 +00:00
80859d714d
HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt ( #3109 )
...
Contributed by Zamil Majdy.
Change-Id: I680d9c425c920ff1a7cd4764d62e10e6ac78bee4
2021-06-25 20:47:11 +01:00
22e7567475
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery ( #3130 ). Contributed by Ahmed Hussein. ( #3138 )
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 581f43dce1
2021-06-25 22:48:56 +05:30
39e6f2d191
HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain." ( #3133 )
...
This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint
is set in fs.s3a.endpoint, S3A filesystem creation may fail on
non-EC2 deployments, depending on the local host environment setup.
* If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region
is null, the region is set to "us-east-1".
* If fs.s3a.endpoint.region is explicitly set to "" then the client
falls back to the SDK region resolution chain; this works on EC2
* Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+
* Also contains some minor restructuring of troubleshooting.md
Contributed by Steve Loughran.
Change-Id: Ife482cff513307cd52d59eec56beac0a33e031f5
2021-06-24 16:38:55 +01:00
4ca49fcd44
HDFS-16085. Move the getPermissionChecker out of the read lock ( #3134 ). Contributed by tomscut.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-24 16:00:47 +05:30
5fc61d2e56
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations ( #3117 )
...
(cherry picked from commit fdef2b4cca
2021-06-24 13:27:33 +09:00
b672edb64c
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. ( #3128 )
...
(cherry picked from commit 11d144d228
2021-06-23 10:50:59 +00:00
68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer ( #3127 )
...
(cherry picked from commit d9fbb3c508
2021-06-23 11:24:00 +08:00
4825c7c28d
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects ( #3123 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa
2021-06-21 16:47:37 +09:00
b8a98e4f82
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79
2021-06-21 10:28:05 +09:00
7d202e8365
HDFS-16079. Improve the block state change log ( #3120 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:24:03 +05:30
ef3a2a3b3b
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result ( #3121 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:23:52 +05:30
92ade1f6f9
HDFS-16061. DFTestUtil.waitReplication can produce false positives ( #3095 ). Contributed by Ahmed Hussein.
...
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:49 +05:30
8d241482bd
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() ( #3119 ). Contributed by tomscut.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:38 +05:30
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet ( #3113 )
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
2021-06-18 14:41:02 +08:00
577b96ef42
HDFS-16074. Remove an expensive debug string concatenation ( #3107 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548
2021-06-17 11:39:34 +08:00
25138c98bf
HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml ( #3099 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129
2021-06-17 10:00:36 +09:00
3e03ecd680
HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w.
2021-06-16 15:54:36 +05:30
57ee4238eb
YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy.
...
(cherry picked from commit 9a6a11c452
2021-06-15 19:30:16 +00:00
Viraj Jasani
Masatake Iwasaki
Kihwal Lee
Mehakmeet Singh
crossfire
Jim Brennan
Abhishek Das
snehavarma
Akira Ajisaka
sumangala-patki
litao
Artem Smotrakov
S O'Donnell
bshashikant
Rafal Wojdyla
Eric Payne
liangxs
Mukund Thakur
Wanqiang Ji
Zamil Majdy
Ahmed Hussein
Steve Loughran
Hui Fei
Wei-Chiu Chuang
Takanobu Asanuma
Ayush Saxena