Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,618 Commits 326 Branches 371 Tags
Commit Graph

21618 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Viraj Jasani
707d003eb6
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073d76bef5c6ad3819e7710317733bd7a)
(cherry picked from commit 8538881710df3dd33ab23670d50d1b0119930169)
 2021-06-29 18:14:02 +08:00
Jim Brennan
c3c86b18cb YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 
(cherry picked from commit dc6f456e953e685370277d3d6bf3515b5001bca3)
 2021-06-28 15:28:38 +00:00
Jim Brennan
9c4ac62d68 YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 
(cherry picked from commit 7c7d02edbd6c17ee8ae2c4bf75e87adace059b76)
 2021-06-25 20:50:11 +00:00
Jim Brennan
9731dfbf0d HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 17:57:15 +00:00
Ahmed Hussein
e0ec1849d7
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3130). Contributed by Ahmed Hussein. (#3139) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 581f43dce15d5753deb33f72ec275d0ea67b1403)
 2021-06-25 22:50:04 +05:30
Szilard Nemeth
c8b090d9a4 YARN-10828. Backport YARN-9789 to branch-3.2. Contributed by Tarun Parimi 2021-06-24 20:20:23 +02:00
Masatake Iwasaki
7d2186ffcf YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 
(cherry picked from commit 11d144d2284be29da1f49e163db0763636dcf058)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock
 2021-06-24 14:45:44 +09:00
dependabot[bot]
b8f0be3299 Bump bower from 1.7.7 to 1.8.8 (#1683) 
Bumps [bower](https://github.com/bower/bower) from 1.7.7 to 1.8.8.
- [Release notes](https://github.com/bower/bower/releases)
- [Changelog](https://github.com/bower/bower/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bower/bower/compare/v1.7.7...v1.8.8)

Signed-off-by: dependabot[bot] <support@github.com>
(cherry picked from commit e9c5bb8078833d72b3e831d511ef3ed4d16c27d3)
 2021-06-24 14:33:33 +09:00
Akira Ajisaka
99d8133968
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 
(cherry picked from commit ae4945fb2c89397f81e7d0566784e6548b6de685)

 Conflicts:
	dev-support/docker/Dockerfile
	dev-support/docker/Dockerfile_aarch64
 2021-06-24 13:17:47 +09:00
Akira Ajisaka
6827d49b10
YARN-10331. Upgrade node.js to 10.21.0. (#2106) 
(cherry picked from commit cd188ea9f0e807df1e2cc13f62be3e4c956b1e69)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2021-06-24 13:14:37 +09:00
Masatake Iwasaki
2eb6bf6bb1
YARN-10037. Upgrade build tools for YARN Web UI v2. 
(cherry picked from commit 7868da894ae148bff1d5e5159a2bc1aad44fd6aa)
 2021-06-24 13:12:44 +09:00
Masatake Iwasaki
3a224b7a93
YARN-10020. Fix build instruction of hadoop-yarn-ui. (#1750) 
(cherry picked from commit 92c8962f98857c7c442160561cf53d354167d6de)
 2021-06-24 13:09:44 +09:00
Ahmed Hussein
dc46cec277 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
litao
00bd7b2aa4 HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
Hui Fei
c14f3e4ed1
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3114) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 21:04:31 +08:00
Wei-Chiu Chuang
11034f2488
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e5489e6080f21c308673ecc73a54ba3752)
 2021-06-17 11:38:58 +08:00
Takanobu Asanuma
2f29784293 HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129fcf466d9647e0672ecf7dd72213e72)
 2021-06-17 10:01:28 +09:00
Ayush Saxena
77f61df352 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:55:06 +05:30
Jim Brennan
2636b630b4 YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit 9a6a11c4522f34fa4245983d8719675036879d7a)
 2021-06-15 19:51:18 +00:00
Takanobu Asanuma
b8fc956a4d HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682270bc6ba151875549d64d81917f425)
 2021-06-15 17:23:34 +09:00
Takanobu Asanuma
8e6b09c0c2 HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. (#3090) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580d73e81042f3b8b2b337aba86130653)
 2021-06-14 10:22:14 +09:00
Mingliang Liu
d92af85045
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 20a4cb0c67b413e7b9bc2b3213b5b592bfaa99d5.
 2021-06-11 00:38:36 -07:00
Suma Shivaprasad
e62cdf915f
YARN-9246 NPE when executing a command yarn node -status or -states without additional arguments. Contributed by Masahiro Tanaka 
(cherry picked from commit 2044967e7581f00c3f6378860426a69078faf694)
 2021-06-10 21:47:40 +09:00
Hideyuki Furue
f5568e15d0
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9c7b8cf54ea88833d54fc71a9612c448dc0eb78d)
 2021-06-06 08:01:39 +09:00
July
20a4cb0c67
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:59:26 -07:00
Konstantin V Shvachko
549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:55:35 -07:00
Takanobu Asanuma
1eef5c2861 Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) 
This reverts commit 077411675679900f94adba5329d0e33a4a528793.

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53ff2dfed3b784ff2120fadc85ba70226ecfd5ca)
 2021-05-27 13:20:08 +09:00
Konstantin V Shvachko
14fd4776ae HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f4f236674ce929164cc460037730abb)
 2021-05-26 15:10:52 -07:00
Akira Ajisaka
996a065c3b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3b025e4c0c844e8bfe8d9007af2be3b)
 2021-05-25 18:29:39 +09:00
Szilard Nemeth
3951fb6dea YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:35:47 +02:00
Viraj Jasani
3c1f3b87fa HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:23 +09:00
Wei-Chiu Chuang
2823d77f19
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de63e1e483c207085207f5d73d7b8adfd)
(cherry picked from commit 0df47432847e1a8b9f3cd31d60b3003fc5a8bc9b)
 2021-05-18 11:36:56 +08:00
lujiefsi
bc97dd0d26
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 14:05:26 +09:00
litao
6df2dc8463
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f98967ced0cbcf32d3acf6dfd10712e66a3)
(cherry picked from commit 2f56e2b54d0080cd22d3d8461567aa0364f983d8)
 2021-05-17 12:29:38 +08:00
Wei-Chiu Chuang
2ba3890b15
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60113e4dd4ef2375e6f9413e519b1044)
(cherry picked from commit 1b8eb2654a8d0a73da720f5724289e25cf34d198)
 2021-05-13 20:31:20 +08:00
hchaverr
f6af5bff9e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:04:25 -07:00
Akira Ajisaka
a5bb211a2b YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc137239682e6189d1663b6b868757938)
 2021-05-07 00:25:43 +09:00
Hui Fei
ab4e90cc28 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:48:06 -07:00
Ayush Saxena
5d1127c8f3 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6c6ce9215ea998a2f544b2eabca4340)
 2021-04-27 23:19:22 +09:00
Wei-Chiu Chuang
650bcaf694
HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2956) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: I7de255cb9a167e3a5e25ae339b050e82cb4b7c30
(cherry picked from commit 64264d726d81c919d3f682228ff4bd7fe349fcae)
 2021-04-27 11:11:46 +09:00
Stephen O'Donnell
7a81e50bd2 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c291a6250b077da32a49dbb35f3b78bf7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit f6efb58b0735fa19a93fa9acae05666cdba8997a)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
 2021-04-26 12:19:01 +01:00
Wei-Chiu Chuang
ef197db8b9 Revert "HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952)" 
This reverts commit dc30b34079c2ebceb561c4196b1c9953830a7988.
 2021-04-26 16:43:00 +08:00
Wei-Chiu Chuang
dc30b34079 HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 66dda86fecee966c7b0625b19f700a43db9787a3)
(cherry picked from commit 41da92def46f84e99e2d8cab604b8f1e290f1c15)
 2021-04-26 14:06:59 +08:00
Eric Badger
951ce9b371 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 22:55:25 +00:00
Eric Badger
7ad844ee9e MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java
 2021-04-23 18:53:09 +00:00
Akira Ajisaka
5db7880783
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0b04c9694da86941e04a480d55913701f9fe514d)
 2021-04-23 19:54:01 +09:00
Viraj Jasani
5c2b728516
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2943) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 12:58:16 +09:00
Eric Badger
b36810ee3f YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 17:54:20 +00:00
Steve Loughran
3991139b0f HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937) 
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
 2021-04-20 15:46:23 +01:00
Wei-Chiu Chuang
8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
 2021-04-19 20:12:27 -07:00