Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,904 Commits 326 Branches 371 Tags
Commit Graph

19904 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
ac30549b06 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:55:43 +02:00
9uapaw
3f14a9b21f YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
(cherry picked from commit f001595b47ae9f60cc2ca13676d9daf3baff9963)
(cherry picked from commit 1dc343726c0910141cc156b5e3f58d248a8a67c5)
 2021-07-28 17:15:21 +02:00
Abhishek Das
f7f753d6fd HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:26:49 -07:00
Konstantin V Shvachko
a357d4ce60 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:58:01 -07:00
Konstantin V Shvachko
49490af23c HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f4f236674ce929164cc460037730abb)
# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
 2021-05-26 15:12:03 -07:00
Akira Ajisaka
76e92af4c4
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3b025e4c0c844e8bfe8d9007af2be3b)
 2021-05-25 18:29:58 +09:00
Viraj Jasani
2407d81f5d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:57 +09:00
Wei-Chiu Chuang
d4f45100c7
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de63e1e483c207085207f5d73d7b8adfd)
(cherry picked from commit 0df47432847e1a8b9f3cd31d60b3003fc5a8bc9b)
(cherry picked from commit 2823d77f19efebff17a6c07e38a1f4be92c52404)
 2021-05-18 11:37:08 +08:00
lujiefsi
e0b84c45d1
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 15:44:00 +09:00
litao
9cead46c63
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f98967ced0cbcf32d3acf6dfd10712e66a3)
(cherry picked from commit 2f56e2b54d0080cd22d3d8461567aa0364f983d8)
(cherry picked from commit 6df2dc8463c09239a97663d9c3b57d5bf873ee1b)
 2021-05-17 12:29:49 +08:00
Wei-Chiu Chuang
a5d584c684
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60113e4dd4ef2375e6f9413e519b1044)
(cherry picked from commit 1b8eb2654a8d0a73da720f5724289e25cf34d198)
(cherry picked from commit 2ba3890b15adba38e716b5eb9b417f6d0be061b2)
 2021-05-13 20:31:29 +08:00
Peter Bacsko
4cd1596fe7 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-05-11 15:18:16 +02:00
hchaverr
39bf9e270e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:10:27 -07:00
Akira Ajisaka
badb6b3d0d YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc137239682e6189d1663b6b868757938)
 2021-05-07 00:26:46 +09:00
Hui Fei
c4c78016eb HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:49:52 -07:00
Ayush Saxena
d40f953c6e HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6c6ce9215ea998a2f544b2eabca4340)
 2021-04-27 23:20:26 +09:00
Wei-Chiu Chuang
c95bacfa2a
HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2956) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: I7de255cb9a167e3a5e25ae339b050e82cb4b7c30
(cherry picked from commit 64264d726d81c919d3f682228ff4bd7fe349fcae)
 2021-04-27 11:14:09 +09:00
Stephen O'Donnell
71a9885c97 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c291a6250b077da32a49dbb35f3b78bf7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit f6efb58b0735fa19a93fa9acae05666cdba8997a)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java

(cherry picked from commit 7a81e50bd2a48d372bd9788aac369078e580e107)
 2021-04-26 12:21:48 +01:00
Eric Badger
c7ed492800 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 23:04:07 +00:00
Eric Badger
be5fa9615d MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java

(cherry picked from commit 7ad844ee9ed7eaab4fbb4d38576be10d497460cf)
 2021-04-23 19:01:29 +00:00
Akira Ajisaka
63b43a2201
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0b04c9694da86941e04a480d55913701f9fe514d)
 2021-04-23 19:54:23 +09:00
Viraj Jasani
d1c46ff324
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2944) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 13:27:19 +09:00
Eric Badger
2af2c03560 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 18:01:41 +00:00
Wei-Chiu Chuang
f3503303dc
HADOOP-17651. Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) (#2935) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
(cherry picked from commit 8ff61f9b076a7022a1dfc067b9f94434756b64a7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java

Change-Id: I465f6003b6f4c5df9c41c83eac3738bac56403e1
 2021-04-20 06:13:32 -07:00
Wei-Chiu Chuang
cd8ecd490c HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
(cherry picked from commit 77315abe474ad3a17507bfc67956b85327f487e6)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 289c7d91b4ac42e49533b22bb1d604bcd503753a)
 2021-04-20 21:06:38 +08:00
Eric Badger
9c3acb3018 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 
(cherry picked from commit bf0d058b698b15d1c51927b8fd8cf678ab95661a)
 2021-04-19 20:18:20 +00:00
Viraj Jasani
bef21ceb14 HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 Conflicts:
	LICENSE-binary
 2021-04-16 12:49:20 +05:30
Akira Ajisaka
1b5ef939d3
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:48 +09:00
Wei-Chiu Chuang
8a865273db HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
(cherry picked from commit 0faa626bd84a02688e0938de1320fc32ec550aea)
 2021-04-13 11:09:52 +08:00
Wei-Chiu Chuang
5a6867e2bd
HADOOP-17614. Bump netty to the latest 4.1.61. (#2871) 2021-04-11 22:55:01 -07:00
Eric Badger
c70ac91d3f YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:41:28 +00:00
Stephen O'Donnell
40ed9e1dc0 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:45:01 +01:00
Gautham B A
1f3697b3a4
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a7be214c185fe1fb8aad214b3210bb8)
 2021-04-07 19:58:37 +09:00
Harunobu Daikoku
c2cd0ab237 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2866) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ee9ddc8184043cbba8f0a5bd1ecbdfd6378c1c5b)
 2021-04-06 16:03:09 +09:00
Gautham B A
5a18d9248b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74838fca3598587eb4770b797f436832)
 2021-04-05 11:49:54 +09:00
Viraj Jasani
739f70b552
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:05:07 -07:00
Takanobu Asanuma
ddf9ca14ee Revert "HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787)" 
This reverts commit bd27853bfa0f2a0811bc194afe5b942fc6a2365c.
 2021-04-01 13:53:05 +09:00
He Xiaoqiao
16d50f4bbf
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:59 +08:00
Akira Ajisaka
b26eac6e8a
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:39 +09:00
Harunobu Daikoku
bd27853bfa HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:49:28 +09:00
Akira Ajisaka
7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml

(cherry picked from commit 0b278502e25e7a5d52bc9c2873058413f77dd7fa)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-27 00:40:24 +09:00
Akira Ajisaka
c18ecc5de6
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d77de469384edd19d67092ae140faf0)
 2021-03-27 00:36:50 +09:00
Akira Ajisaka
408cffe2d0
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de25507bc22ff5a44da4b2a112fbccc64500)
 2021-03-27 00:36:17 +09:00
Akira Ajisaka
3e2534c032
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956e6fa9b55758f3d2c1b92ae2597cdbb)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890d65c4215c5e19cdf45cc2a4f0a348e)
(cherry picked from commit 08652eeee76ea06465157c95740741dcc3e12479)
 2021-03-27 00:33:16 +09:00
Akira Ajisaka
ce08b9d2eb
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9ddec46d6b7d1336bb88136d8826972e7a)
(cherry picked from commit 43b8396b77a00de8182f2e5f65ed52195c32437e)
 2021-03-27 00:33:00 +09:00
Akira Ajisaka
db94908212
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534645a2109a037115fb955351edfbf64)

 Conflicts:
	dev-support/docker/Dockerfile

(cherry picked from commit fa5c9a18866fa7b52ff90f7412b9869487f600c8)
 2021-03-27 00:19:55 +09:00
Akira Ajisaka
a8ddc179df
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d46360671f2d52bbe5ff3683b661f6f8adfe7d)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 819239f9718bd898ef05a6558eb4ed9451798876)
 2021-03-25 20:29:18 +09:00
He Xiaoqiao
b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani
a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00
Akira Ajisaka
332c2a637d
YARN-10036. Install yarnpkg and upgrade nodejs in Dockerfile (#1772) 
(cherry picked from commit bc825b9628f2fca9d250f91728d4aebb155f1331)
(cherry picked from commit 710a24b6cd7a9895a131471377f0af52f7c098d9)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-03-24 16:29:33 +09:00