Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,640 Commits 326 Branches 371 Tags
Commit Graph

21640 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
996a065c3b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3b025e4c0c844e8bfe8d9007af2be3b)
 2021-05-25 18:29:39 +09:00
Szilard Nemeth
3951fb6dea YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:35:47 +02:00
Viraj Jasani
3c1f3b87fa HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:23 +09:00
Wei-Chiu Chuang
2823d77f19
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de63e1e483c207085207f5d73d7b8adfd)
(cherry picked from commit 0df47432847e1a8b9f3cd31d60b3003fc5a8bc9b)
 2021-05-18 11:36:56 +08:00
lujiefsi
bc97dd0d26
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 14:05:26 +09:00
litao
6df2dc8463
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f98967ced0cbcf32d3acf6dfd10712e66a3)
(cherry picked from commit 2f56e2b54d0080cd22d3d8461567aa0364f983d8)
 2021-05-17 12:29:38 +08:00
Wei-Chiu Chuang
2ba3890b15
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60113e4dd4ef2375e6f9413e519b1044)
(cherry picked from commit 1b8eb2654a8d0a73da720f5724289e25cf34d198)
 2021-05-13 20:31:20 +08:00
hchaverr
f6af5bff9e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:04:25 -07:00
Akira Ajisaka
a5bb211a2b YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc137239682e6189d1663b6b868757938)
 2021-05-07 00:25:43 +09:00
Hui Fei
ab4e90cc28 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:48:06 -07:00
Ayush Saxena
5d1127c8f3 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6c6ce9215ea998a2f544b2eabca4340)
 2021-04-27 23:19:22 +09:00
Wei-Chiu Chuang
650bcaf694
HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2956) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: I7de255cb9a167e3a5e25ae339b050e82cb4b7c30
(cherry picked from commit 64264d726d81c919d3f682228ff4bd7fe349fcae)
 2021-04-27 11:11:46 +09:00
Stephen O'Donnell
7a81e50bd2 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c291a6250b077da32a49dbb35f3b78bf7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit f6efb58b0735fa19a93fa9acae05666cdba8997a)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
 2021-04-26 12:19:01 +01:00
Wei-Chiu Chuang
ef197db8b9 Revert "HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952)" 
This reverts commit dc30b34079c2ebceb561c4196b1c9953830a7988.
 2021-04-26 16:43:00 +08:00
Wei-Chiu Chuang
dc30b34079 HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 66dda86fecee966c7b0625b19f700a43db9787a3)
(cherry picked from commit 41da92def46f84e99e2d8cab604b8f1e290f1c15)
 2021-04-26 14:06:59 +08:00
Eric Badger
951ce9b371 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 22:55:25 +00:00
Eric Badger
7ad844ee9e MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java
 2021-04-23 18:53:09 +00:00
Akira Ajisaka
5db7880783
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0b04c9694da86941e04a480d55913701f9fe514d)
 2021-04-23 19:54:01 +09:00
Viraj Jasani
5c2b728516
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2943) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 12:58:16 +09:00
Eric Badger
b36810ee3f YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 17:54:20 +00:00
Steve Loughran
3991139b0f HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937) 
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
 2021-04-20 15:46:23 +01:00
Wei-Chiu Chuang
8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
 2021-04-19 20:12:27 -07:00
Eric Badger
bf0d058b69 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 2021-04-19 20:06:36 +00:00
Wei-Chiu Chuang
289c7d91b4
HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
(cherry picked from commit 77315abe474ad3a17507bfc67956b85327f487e6)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-18 22:13:24 -07:00
Viraj Jasani
9208218704 HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 Conflicts:
	LICENSE-binary
 2021-04-16 12:45:13 +05:30
Masatake Iwasaki
08cc11cd28 HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f858ff654de0fc01cfb9afef2e705e33c)
 2021-04-15 13:35:49 +00:00
Wei-Chiu Chuang
f0331827e5
HDFS-15940. Fixing and refactoring tests specific to Block recovery. (#2902) 
* HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani
(cherry picked from commit 2b5fd341b969a6c920c27e06b158a5b608544f17)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

* HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874)

(cherry picked from commit 56bd968fb434c7d2ae7ac9e7659d0ca682340b2a)
(cherry picked from commit 4994b73eebafc7b718f59ed293fcbadc677f95b8)

Co-authored-by: Viraj Jasani <vjasani@apache.org>
 2021-04-15 11:20:23 +08:00
Akira Ajisaka
d9858a4663
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:26 +09:00
Wei-Chiu Chuang
e6df0fb84c HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e414abf8c94f7bd4719053efa3b538a)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit bfba6f1f3cd1966ff59655feb7e19503e06eeeae)
 2021-04-13 15:00:14 +08:00
Wei-Chiu Chuang
0faa626bd8
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
 2021-04-12 19:45:49 -07:00
Eric Badger
899cef53bd YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:33:42 +00:00
Stephen O'Donnell
3422691fb2 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:31:36 +01:00
Gautham B A
78ee59afb6
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a7be214c185fe1fb8aad214b3210bb8)
 2021-04-07 19:58:20 +09:00
Gautham B A
48a228893a
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 36014b8282579ac3d89b62a05586206fb6e0adcf)
 2021-04-06 19:10:49 +09:00
Wei-Chiu Chuang
df6aa293ea
HADOOP-17614. Bump netty to the latest 4.1.61 (#2864) 2021-04-05 22:44:47 -07:00
Harunobu Daikoku
ee9ddc8184
HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2866) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:22:11 +09:00
Gautham B A
52f4f9a063
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:47:40 +09:00
litao
d566e4e41d HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 11:00:33 +09:00
Viraj Jasani
5458ebf607
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:45:24 -07:00
Takanobu Asanuma
93517ba9bc Revert "HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787)" 
This reverts commit 510a43cbe4f5221c852d71ae3f7e7975b9497e86.
 2021-04-01 13:51:08 +09:00
He Xiaoqiao
eb10bd8ffd
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:29 +08:00
Akira Ajisaka
d93f2c8cb8
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:20 +09:00
Harunobu Daikoku
510a43cbe4 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:48:43 +09:00
Akira Ajisaka
819239f971
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d46360671f2d52bbe5ff3683b661f6f8adfe7d)

 Conflicts:
	hadoop-project/pom.xml
 2021-03-25 20:22:49 +09:00
He Xiaoqiao
d456106dce
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:26:25 +08:00
Viraj Jasani
97b8992fd8
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2797) 2021-03-24 10:13:59 -07:00
Takanobu Asanuma
46b2e9634b HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:54:07 +09:00
Takanobu Asanuma
9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
Jim Brennan
0fcb2f28ce YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 174f3a96b10a0ab0fd8aed1b0f904ca5f0c3f268)
 2021-03-23 18:37:32 +00:00
litao
d05d15620e
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 2021-03-23 18:45:24 +09:00