Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,875 Commits 326 Branches 371 Tags
Commit Graph

21875 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Igor Dvorzhak
399580d302 HADOOP-18300. Upgrade Gson dependency to version 2.9.0 (#4454) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 77d1b194c77ec62dc31f89e15594bc1250d88de3)
(cherry picked from commit d41e0a9cc30c9dbf4ba311b3ed18a6942a160287)
 2022-06-23 00:19:56 +00:00
KevinWikant
4dad353597 HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6028f9604421a9ae10dda34bc1f9532)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
 2022-06-20 11:40:11 +09:00
xuzq
aba4418981 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a47311bad542964c42c1f776e4350446b9)
(cherry picked from commit ee3ee98ee5a4c34da7a4960b099686bdc1186a71)
 2022-06-10 19:33:36 +00:00
Viraj Jasani
557ab95dc2 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 17:11:57 +08:00
Wei-Chiu Chuang
4ebdc38634
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication. (#4358) 2022-06-04 06:56:07 +08:00
Stephen O'Donnell
58513d367e HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90ef89b633c9b5c13a79aa1ac11c92eb5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 7d6b133af3f7ee4b986bdd2b266b83693614b6ab)
 2022-06-01 21:05:21 +01:00
Stephen O'Donnell
52e405390e
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop (#4394) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-06-01 07:55:48 +08:00
Duo Zhang
57fd58bc3c HADOOP-16664 Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches 2022-05-29 17:58:31 +09:00
Duo Zhang
1d053b0c65 HADOOP-16663 Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches 2022-05-29 17:54:25 +09:00
Vinayakumar B
db4c32b03f HADOOP-16985. Handle release package related issues (#1957) 
(cherry picked from commit 4d24d99e854e1718270a75807f7779a623758247)
 2022-05-29 17:26:50 +09:00
Ashutosh Gupta
cc52627ad4
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785c012794a2dcf122dc8717b18beb0f0)
(cherry picked from commit d921cc71fdd2422fdf9f5417fd622580eb4438a2)
 2022-05-26 15:28:42 -07:00
Masatake Iwasaki
f9b5366be7 HADOOP-18251. Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py. (#4344) 
(cherry picked from commit 6b331dde31cec2676a93406c9ecf699fc0675c8c)
 2022-05-26 03:30:52 +00:00
Michael Stack
ad9b6e8d81
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4348) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 16:35:56 -07:00
Ashutosh Gupta
2cc21714bc HADOOP-18240. Upgrade Yetus to 0.14.0 (#4328) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 84b0455cf869b9acd4cdc293d4a1612cb883e7f2)
 2022-05-25 17:33:04 +09:00
Owen O'Malley
2113c84260 YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. (#4350) 
(cherry picked from commit f390edaec44cfa91b2b09549091f033f1749d8ac)
 2022-05-24 14:15:35 +09:00
Ashutosh Gupta
6322a774f3 YARN-10080. Support show app id on localizer thread pool (#4283) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6985f9aabebefa86037f7dbc478374392afef990)
(cherry picked from commit d58f9d438fc35288492e0e41b690fe08e94894f0)
 2022-05-19 12:33:23 +09:00
Szilard Nemeth
45801fba8b YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-18 14:23:56 +02:00
Szilard Nemeth
1ae834fe0b YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-18 14:04:51 +02:00
Zilong Zhu
e88563336c YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280) 
(cherry picked from commit 79cc0caf9fde8ad6a9170ed7572a73e01b37d9e0)
(cherry picked from commit 2dd5852ad0af558c9f96e83071f78f11203726df)
 2022-05-18 03:04:05 +09:00
Ashutosh Gupta
45e5b71a1d YARN-11092. Upgrade jquery ui to 1.13.1 (#4260) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 931abbd92706aa5377ba303b7b09f0adf1d8e1c7)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/pom.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/view/JQueryUI.java
 2022-05-16 23:44:52 +09:00
litao
48ca153100
HDFS-16579. Fix build failure for TestBlockManager on branch-3.2 (#4310) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-15 17:59:23 +09:00
Ashutosh Gupta
92c71b1377
HADOOP-17479. Fix the examples of hadoop config prefix (#4197) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 40a8b9a6a54181b58d1770dd779d637041a631ce)
 2022-05-08 08:10:08 +09:00
Ashutosh Gupta
4a532f77a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9789eb5b386729f52b7bb9e52fa5352)
 2022-05-08 08:06:39 +09:00
jteagles
6daa369bcf
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d4a91bd0c0d9d438ffe95fafee0212499b0abcfd)
 2022-05-03 03:18:25 +09:00
Ashutosh Gupta
7f08b96bcd
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a74acc755edcde38203416f9d5225343ea877c08)
 2022-05-03 03:11:00 +09:00
Ashutosh Gupta
ca9f18b7d6 MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". (#4223) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb13c1e4a8a945328781fa533b5bf22786b830f1)
 2022-04-25 14:31:39 +09:00
Renukaprasad C
5f259ef7fd
HDFS-16551. CryptoInputStream#close() should be syncronized (#4227) 2022-04-25 02:19:12 +08:00
litao
0965822ae1 HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc48023c0c71dd428df40a4352feede373b)
 2022-04-23 12:18:19 +09:00
Inigo Goiri
27621c9eef HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B. 
(cherry picked from commit f069d38c8d3c0bfa91b70a60e4e556ec708fc411)
 2022-04-22 16:00:40 +09:00
daimin
181efebdfe
HDFS-16546. Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection (#4187) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-22 15:34:28 +09:00
S O'Donnell
bacd0b9a92 Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 045d48530cbbd7baa2b67937e103b9400487f1a4.
 2022-04-20 20:57:23 +01:00
qinyuren
4fed01ef21 HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd173749f2af4f17946fb37c4c72e2de26764)
 2022-04-20 15:08:05 +09:00
Stephen O'Donnell
045d48530c HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee03639f41a022dd07d5fc04e3aa65a94b5f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2022-04-19 11:38:34 +01:00
qinyuren
d993d22038 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0d5f522f3b799ea72c6c887d5d2c42d)
 2022-04-19 13:40:16 +09:00
jianghuazhu
ed83a278c2
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e7120e7af250ebf0b4bafb40b412d3e11c)
(cherry picked from commit 4aa9698ee1e9a10a0f882229036bbb4db1ede939)
 2022-04-19 11:13:29 +08:00
daimin
248c548616
Fix decommission UnsupportedOperationException (#4172) 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 10:40:12 +08:00
Takanobu Asanuma
b8c6ba6046 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959ca0a68c52ed6a9c721704e57afbcc7)
 2022-04-14 11:52:12 +09:00
qinyuren
cb14e8de81 HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a112334e48087bd60674538af739922a)
 2022-04-13 11:53:58 +09:00
Viraj Jasani
2d2631afc0 HADOOP-18191. Log retry count while handling exceptions in RetryInvocationHandler (#4133) 
(cherry picked from commit b69ede7154d44538a4a66824c34f7ba143deef25)
 2022-04-11 15:24:24 +09:00
Hanisha Koneru
9cdf5ca105 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 
(cherry picked from commit e62d8f841275ee47a0ba911415aac9e39af291c6)
 2022-04-11 15:19:54 +09:00
Stephen O'Donnell
72e48ac84e HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf76ce661e0b10bb5d16cb419f9b9b10)
 2022-04-06 13:06:28 +01:00
wangzhaohui
545d3a6cb7 HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a73a18cf05aeb2c40429c34c4b9b0732)
 2022-04-06 17:31:42 +09:00
Masatake Iwasaki
2d770e49f1
HADOOP-18192. Fix multiple_bindings warning about slf4j-reload4j. (#4136) 2022-04-06 14:13:19 +09:00
litao
6b24b85e3f HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:04:04 -07:00
Kengo Seki
5c6547d794 MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (#4120) 
(cherry picked from commit dc4a680da8bcacf152cc8638d86dd171a7901245)
 2022-03-30 22:51:40 +09:00
Masatake Iwasaki
228ad1ac25 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a2449ec49f840d93e1c7d290c5218ef4ea)
 2022-03-28 08:09:58 +00:00
Masatake Iwasaki
5c16db8b17 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 10876333acd4194da3cb545c4f7a2b2cc88be959)
 2022-03-28 08:04:47 +00:00
GuoPhilipse
ff03212145 HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4a895b98ccd41dde1c9524a6bb0ea31)
 2022-03-27 22:33:17 +09:00
Akira Ajisaka
73775fe277
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. (#4102) 
(cherry picked from commit a0deda1a777d8967fb8c08ac976543cda895773d)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

Co-authored-by: Peter Bacsko <pbacsko@cloudera.com>
 2022-03-24 17:36:44 +09:00
qinyuren
66be0a713f HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50c3ff65676d946e96bbc7846ad3131a)
 2022-03-23 15:06:20 +09:00