Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,881 Commits 326 Branches 371 Tags
Commit Graph

19881 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
f3503303dc
HADOOP-17651. Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) (#2935) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
(cherry picked from commit 8ff61f9b076a7022a1dfc067b9f94434756b64a7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java

Change-Id: I465f6003b6f4c5df9c41c83eac3738bac56403e1
 2021-04-20 06:13:32 -07:00
Wei-Chiu Chuang
cd8ecd490c HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
(cherry picked from commit 77315abe474ad3a17507bfc67956b85327f487e6)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 289c7d91b4ac42e49533b22bb1d604bcd503753a)
 2021-04-20 21:06:38 +08:00
Eric Badger
9c3acb3018 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 
(cherry picked from commit bf0d058b698b15d1c51927b8fd8cf678ab95661a)
 2021-04-19 20:18:20 +00:00
Viraj Jasani
bef21ceb14 HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 Conflicts:
	LICENSE-binary
 2021-04-16 12:49:20 +05:30
Akira Ajisaka
1b5ef939d3
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:48 +09:00
Wei-Chiu Chuang
8a865273db HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
(cherry picked from commit 0faa626bd84a02688e0938de1320fc32ec550aea)
 2021-04-13 11:09:52 +08:00
Wei-Chiu Chuang
5a6867e2bd
HADOOP-17614. Bump netty to the latest 4.1.61. (#2871) 2021-04-11 22:55:01 -07:00
Eric Badger
c70ac91d3f YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:41:28 +00:00
Stephen O'Donnell
40ed9e1dc0 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:45:01 +01:00
Gautham B A
1f3697b3a4
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a7be214c185fe1fb8aad214b3210bb8)
 2021-04-07 19:58:37 +09:00
Harunobu Daikoku
c2cd0ab237 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2866) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ee9ddc8184043cbba8f0a5bd1ecbdfd6378c1c5b)
 2021-04-06 16:03:09 +09:00
Gautham B A
5a18d9248b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:49:54 +09:00
Viraj Jasani
739f70b552
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:05:07 -07:00
Takanobu Asanuma
ddf9ca14ee Revert "HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787)" 
This reverts commit bd27853bfa0f2a0811bc194afe5b942fc6a2365c.
 2021-04-01 13:53:05 +09:00
He Xiaoqiao
16d50f4bbf
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:59 +08:00
Akira Ajisaka
b26eac6e8a
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:39 +09:00
Harunobu Daikoku
bd27853bfa HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:49:28 +09:00
Akira Ajisaka
7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml

(cherry picked from commit 0b278502e25e7a5d52bc9c2873058413f77dd7fa)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-27 00:40:24 +09:00
Akira Ajisaka
c18ecc5de6
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d77de469384edd19d67092ae140faf0)
 2021-03-27 00:36:50 +09:00
Akira Ajisaka
408cffe2d0
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de25507bc22ff5a44da4b2a112fbccc64500)
 2021-03-27 00:36:17 +09:00
Akira Ajisaka
3e2534c032
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956e6fa9b55758f3d2c1b92ae2597cdbb)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890d65c4215c5e19cdf45cc2a4f0a348e)
(cherry picked from commit 08652eeee76ea06465157c95740741dcc3e12479)
 2021-03-27 00:33:16 +09:00
Akira Ajisaka
ce08b9d2eb
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9ddec46d6b7d1336bb88136d8826972e7a)
(cherry picked from commit 43b8396b77a00de8182f2e5f65ed52195c32437e)
 2021-03-27 00:33:00 +09:00
Akira Ajisaka
db94908212
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534645a2109a037115fb955351edfbf64)

 Conflicts:
	dev-support/docker/Dockerfile

(cherry picked from commit fa5c9a18866fa7b52ff90f7412b9869487f600c8)
 2021-03-27 00:19:55 +09:00
Akira Ajisaka
a8ddc179df
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d46360671f2d52bbe5ff3683b661f6f8adfe7d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 819239f9718bd898ef05a6558eb4ed9451798876)
 2021-03-25 20:29:18 +09:00
He Xiaoqiao
b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani
a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00
Akira Ajisaka
332c2a637d
YARN-10036. Install yarnpkg and upgrade nodejs in Dockerfile (#1772) 
(cherry picked from commit bc825b9628f2fca9d250f91728d4aebb155f1331)
(cherry picked from commit 710a24b6cd7a9895a131471377f0af52f7c098d9)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-03-24 16:29:33 +09:00
Kevin Su
65b1945f22
HADOOP-16754. Fix docker failed to build yetus/hadoop. 
Contributed by Kevin Su.

Change-Id: Ie9cf27b7a7ba5923405f24d39b95b0248f3718b7
(cherry picked from commit 0f25cbbb251002a87ff48686eba3a427fc6d06f8)
 2021-03-24 16:25:07 +09:00
Allen Wittenauer
1bcde89983
HADOOP-14671. Upgrade Apache Yetus to 0.8.0 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4b8c2b1c1c9e070002ab3a2afb850d9d7b810d0a)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.2/CHANGELOG.3.0.2.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.2/RELEASENOTES.3.1.2.md
 2021-03-24 16:04:11 +09:00
Jim Brennan
34e507cb8c YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:57:23 +00:00
litao
75f2d47ff5 HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 
(cherry picked from commit d05d15620e320d28bfbeeee2a4f5224e69062528)
 2021-03-23 18:46:19 +09:00
Jim Brennan
c1b25422a5 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 
(cherry picked from commit ede490d13182f0fd3831afe6db97e01917add5b4)
 2021-03-22 19:28:23 +00:00
Ayush Saxena
c9c471e04e HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 13:10:02 +00:00
Mingliang Liu
f2de8cc5cf
YARN-10706. Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 (#2791) 
Contributed by Mingliang Liu.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-21 23:03:33 -07:00
kwangsun
7af7eb3e9f
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:44 +09:00
Ayush Saxena
30a1b17b22 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad2396f8b42d60031001a25d9d6fa32e)
 2021-03-19 19:50:19 +00:00
Ayush Saxena
8354ce680b HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:37 +05:30
Ayush Saxena
162923e799 HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-19 14:21:02 +05:30
Ayush Saxena
8ec256db04 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:14:11 +05:30
Viraj Jasani
503101c1f1 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:14:29 +09:00
Takanobu Asanuma
b898058e45 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:18:54 +09:00
touchida
b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 13:12:40 +09:00
Jim Brennan
a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 38f86cc8c72201b081a083cd32bc3a322c6d9398)
 2021-03-16 19:19:01 +00:00
Eric Payne
2439e07708 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7af122163789a731ced05f740562e45)
 2021-03-15 19:55:13 +00:00
He Xiaoqiao
0224dd87af HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:20:16 +00:00
Ayush Saxena
77aedb7d4c HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 Conflicts:
	LICENSE-binary
 2021-03-13 15:20:50 +05:30
Jim Brennan
a6c2fced53 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:47:43 +00:00
Viraj Jasani
6ecf829e79
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 12:28:02 -08:00
Hui Fei
7250f0e327 HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:45:57 +08:00
Akira Ajisaka
be253c2fd0
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b770eda31e83ca6c6e5e96cb802b8b6a9)
 2021-03-09 11:11:32 +09:00