Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,870 Commits 326 Branches 371 Tags
Commit Graph

19870 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gautham B A
5a18d9248b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:49:54 +09:00
Viraj Jasani
739f70b552
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:05:07 -07:00
Takanobu Asanuma
ddf9ca14ee Revert "HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787)" 
This reverts commit bd27853bfa0f2a0811bc194afe5b942fc6a2365c.
 2021-04-01 13:53:05 +09:00
He Xiaoqiao
16d50f4bbf
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:59 +08:00
Akira Ajisaka
b26eac6e8a
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:39 +09:00
Harunobu Daikoku
bd27853bfa HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:49:28 +09:00
Akira Ajisaka
7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml

(cherry picked from commit 0b278502e25e7a5d52bc9c2873058413f77dd7fa)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-27 00:40:24 +09:00
Akira Ajisaka
c18ecc5de6
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d77de469384edd19d67092ae140faf0)
 2021-03-27 00:36:50 +09:00
Akira Ajisaka
408cffe2d0
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de25507bc22ff5a44da4b2a112fbccc64500)
 2021-03-27 00:36:17 +09:00
Akira Ajisaka
3e2534c032
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956e6fa9b55758f3d2c1b92ae2597cdbb)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890d65c4215c5e19cdf45cc2a4f0a348e)
(cherry picked from commit 08652eeee76ea06465157c95740741dcc3e12479)
 2021-03-27 00:33:16 +09:00
Akira Ajisaka
ce08b9d2eb
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9ddec46d6b7d1336bb88136d8826972e7a)
(cherry picked from commit 43b8396b77a00de8182f2e5f65ed52195c32437e)
 2021-03-27 00:33:00 +09:00
Akira Ajisaka
db94908212
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534645a2109a037115fb955351edfbf64)

 Conflicts:
	dev-support/docker/Dockerfile

(cherry picked from commit fa5c9a18866fa7b52ff90f7412b9869487f600c8)
 2021-03-27 00:19:55 +09:00
Akira Ajisaka
a8ddc179df
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d46360671f2d52bbe5ff3683b661f6f8adfe7d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 819239f9718bd898ef05a6558eb4ed9451798876)
 2021-03-25 20:29:18 +09:00
He Xiaoqiao
b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani
a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00
Akira Ajisaka
332c2a637d
YARN-10036. Install yarnpkg and upgrade nodejs in Dockerfile (#1772) 
(cherry picked from commit bc825b9628f2fca9d250f91728d4aebb155f1331)
(cherry picked from commit 710a24b6cd7a9895a131471377f0af52f7c098d9)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-03-24 16:29:33 +09:00
Kevin Su
65b1945f22
HADOOP-16754. Fix docker failed to build yetus/hadoop. 
Contributed by Kevin Su.

Change-Id: Ie9cf27b7a7ba5923405f24d39b95b0248f3718b7
(cherry picked from commit 0f25cbbb251002a87ff48686eba3a427fc6d06f8)
 2021-03-24 16:25:07 +09:00
Allen Wittenauer
1bcde89983
HADOOP-14671. Upgrade Apache Yetus to 0.8.0 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4b8c2b1c1c9e070002ab3a2afb850d9d7b810d0a)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.2/CHANGELOG.3.0.2.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.2/RELEASENOTES.3.1.2.md
 2021-03-24 16:04:11 +09:00
Jim Brennan
34e507cb8c YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:57:23 +00:00
litao
75f2d47ff5 HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 
(cherry picked from commit d05d15620e320d28bfbeeee2a4f5224e69062528)
 2021-03-23 18:46:19 +09:00
Jim Brennan
c1b25422a5 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 
(cherry picked from commit ede490d13182f0fd3831afe6db97e01917add5b4)
 2021-03-22 19:28:23 +00:00
Ayush Saxena
c9c471e04e HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 13:10:02 +00:00
Mingliang Liu
f2de8cc5cf
YARN-10706. Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 (#2791) 
Contributed by Mingliang Liu.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-21 23:03:33 -07:00
kwangsun
7af7eb3e9f
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:44 +09:00
Ayush Saxena
30a1b17b22 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad2396f8b42d60031001a25d9d6fa32e)
 2021-03-19 19:50:19 +00:00
Ayush Saxena
8354ce680b HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:37 +05:30
Ayush Saxena
162923e799 HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-19 14:21:02 +05:30
Ayush Saxena
8ec256db04 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:14:11 +05:30
Viraj Jasani
503101c1f1 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:14:29 +09:00
Takanobu Asanuma
b898058e45 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:18:54 +09:00
touchida
b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 13:12:40 +09:00
Jim Brennan
a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 38f86cc8c72201b081a083cd32bc3a322c6d9398)
 2021-03-16 19:19:01 +00:00
Eric Payne
2439e07708 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7af122163789a731ced05f740562e45)
 2021-03-15 19:55:13 +00:00
He Xiaoqiao
0224dd87af HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:20:16 +00:00
Ayush Saxena
77aedb7d4c HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 Conflicts:
	LICENSE-binary
 2021-03-13 15:20:50 +05:30
Jim Brennan
a6c2fced53 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:47:43 +00:00
Viraj Jasani
6ecf829e79
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 12:28:02 -08:00
Hui Fei
7250f0e327 HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:45:57 +08:00
Akira Ajisaka
be253c2fd0
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
 2021-03-09 11:11:32 +09:00
tomscut
7b925bc486 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:18:20 -08:00
Akira Ajisaka
5dda43c061
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:41 +09:00
litao
c503130d8b HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:38:04 +09:00
Eric Badger
14dea40154 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e8218b77e7fb5b5e253cb2d20014073b49)
 2021-03-02 21:50:12 +00:00
Konstantin V Shvachko
78bd68a0b8 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:27:38 -08:00
S O'Donnell
53f46214e4 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java

(cherry picked from commit 707806092f64f709814e18cb0d1b4f58731a5716)
 2021-03-01 12:26:41 +00:00
Jim Brennan
4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Eric Badger
5862543c89 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec45ded55e8b7fdc29f8caf85547cca3)
 2021-02-26 19:36:55 +00:00
Surendra Singh Lilhore
379a7a5726 HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3959220c6f985f940e2e731f99d8e30)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
(cherry picked from commit 5315de3d437f78d81dc5fa9f43849de6198220e6)
 2021-02-26 10:49:59 +08:00
Jonathan Hung
eb6c08e423 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963ded94adf6f447ff53f2ba66b99120ca)
(cherry picked from commit 6863a5bb8ace591de3374102920bba916dbebfda)
 2021-02-25 15:35:01 -08:00
Jim Brennan
0a45b5460d [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit 3795f66364e39d9e7bafacd54d16bb60ab1d2188)
 2021-02-25 20:09:27 +00:00