Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,905 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19905 Commits

Author SHA1 Message Date
Tamas Domok f7ac747b6a
YARN-10882. Fix branch-3.1 build: zstd library is missing from the Dockerfile (#3297) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-11 18:29:00 +02:00
Szilard Nemeth ac30549b06 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:55:43 +02:00
9uapaw 3f14a9b21f YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a)
(cherry picked from commit f001595b47)
(cherry picked from commit 1dc343726c)
 2021-07-28 17:15:21 +02:00
Abhishek Das f7f753d6fd HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b5)
 2021-07-13 18:26:49 -07:00
Konstantin V Shvachko a357d4ce60 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:58:01 -07:00
Konstantin V Shvachko 49490af23c HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)
# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestObserverNode.java
 2021-05-26 15:12:03 -07:00
Akira Ajisaka 76e92af4c4
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3)
 2021-05-25 18:29:58 +09:00
Viraj Jasani 2407d81f5d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548)
 2021-05-22 19:12:57 +09:00
Wei-Chiu Chuang d4f45100c7
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de6)
(cherry picked from commit 0df4743284)
(cherry picked from commit 2823d77f19)
 2021-05-18 11:37:08 +08:00
lujiefsi e0b84c45d1
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 15:44:00 +09:00
litao 9cead46c63
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f9896)
(cherry picked from commit 2f56e2b54d)
(cherry picked from commit 6df2dc8463)
 2021-05-17 12:29:49 +08:00
Wei-Chiu Chuang a5d584c684
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60)
(cherry picked from commit 1b8eb2654a)
(cherry picked from commit 2ba3890b15)
 2021-05-13 20:31:29 +08:00
Peter Bacsko 4cd1596fe7 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-05-11 15:18:16 +02:00
hchaverr 39bf9e270e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb059)
 2021-05-06 18:10:27 -07:00
Akira Ajisaka badb6b3d0d YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc1)
 2021-05-07 00:26:46 +09:00
Hui Fei c4c78016eb HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151)
 2021-04-29 12:49:52 -07:00
Ayush Saxena d40f953c6e HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6)
 2021-04-27 23:20:26 +09:00
Wei-Chiu Chuang c95bacfa2a
HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2956) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: I7de255cb9a167e3a5e25ae339b050e82cb4b7c30
(cherry picked from commit 64264d726d)
 2021-04-27 11:14:09 +09:00
Stephen O'Donnell 71a9885c97 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c29)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit f6efb58b07)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java

(cherry picked from commit 7a81e50bd2)
 2021-04-26 12:21:48 +01:00
Eric Badger c7ed492800 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 23:04:07 +00:00
Eric Badger be5fa9615d MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java

(cherry picked from commit 7ad844ee9e)
 2021-04-23 19:01:29 +00:00
Akira Ajisaka 63b43a2201
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0b04c9694d)
 2021-04-23 19:54:23 +09:00
Viraj Jasani d1c46ff324
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2944) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 13:27:19 +09:00
Eric Badger 2af2c03560 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7)
 2021-04-20 18:01:41 +00:00
Wei-Chiu Chuang f3503303dc
HADOOP-17651. Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) (#2935) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0)
(cherry picked from commit 6340ac857b)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282)
(cherry picked from commit 1110b03752)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99)
(cherry picked from commit 7f7535534d)
(cherry picked from commit 8ff61f9b07)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java

Change-Id: I465f6003b6f4c5df9c41c83eac3738bac56403e1
 2021-04-20 06:13:32 -07:00
Wei-Chiu Chuang cd8ecd490c HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a507)
(cherry picked from commit 77315abe47)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 289c7d91b4)
 2021-04-20 21:06:38 +08:00
Eric Badger 9c3acb3018 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 
(cherry picked from commit bf0d058b69)
 2021-04-19 20:18:20 +00:00
Viraj Jasani bef21ceb14 HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 Conflicts:
	LICENSE-binary
 2021-04-16 12:49:20 +05:30
Akira Ajisaka 1b5ef939d3
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be)
 2021-04-13 17:10:48 +09:00
Wei-Chiu Chuang 8a865273db HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e6892675)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
(cherry picked from commit 0faa626bd8)
 2021-04-13 11:09:52 +08:00
Wei-Chiu Chuang 5a6867e2bd
HADOOP-17614. Bump netty to the latest 4.1.61. (#2871) 2021-04-11 22:55:01 -07:00
Eric Badger c70ac91d3f YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:41:28 +00:00
Stephen O'Donnell 40ed9e1dc0 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7)
 2021-04-08 12:45:01 +01:00
Gautham B A 1f3697b3a4
HDFS-15948. Fix test4tests for libhdfspp (#2873) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 02b9506c5a)
 2021-04-07 19:58:37 +09:00
Harunobu Daikoku c2cd0ab237 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2866) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ee9ddc8184)
 2021-04-06 16:03:09 +09:00
Gautham B A 5a18d9248b
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74)
 2021-04-05 11:49:54 +09:00
Viraj Jasani 739f70b552
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:05:07 -07:00
Takanobu Asanuma ddf9ca14ee Revert "HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787)" 
This reverts commit bd27853bfa.
 2021-04-01 13:53:05 +09:00
He Xiaoqiao 16d50f4bbf
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:59 +08:00
Akira Ajisaka b26eac6e8a
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9)
 2021-03-30 17:48:39 +09:00
Harunobu Daikoku bd27853bfa HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b6)
 2021-03-29 11:49:28 +09:00
Akira Ajisaka 7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml

(cherry picked from commit 0b278502e2)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-27 00:40:24 +09:00
Akira Ajisaka c18ecc5de6
HADOOP-17582. Replace GitHub App Token with GitHub OAuth token (#2766) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b62b12d0d)
 2021-03-27 00:36:50 +09:00
Akira Ajisaka 408cffe2d0
HADOOP-17570. Apply YETUS-1102 to re-enable GitHub comments (#2745) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9b78de2550)
 2021-03-27 00:36:17 +09:00
Akira Ajisaka 3e2534c032
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890)
(cherry picked from commit 08652eeee7)
 2021-03-27 00:33:16 +09:00
Akira Ajisaka ce08b9d2eb
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9dde)
(cherry picked from commit 43b8396b77)
 2021-03-27 00:33:00 +09:00
Akira Ajisaka db94908212
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534)

 Conflicts:
	dev-support/docker/Dockerfile

(cherry picked from commit fa5c9a1886)
 2021-03-27 00:19:55 +09:00
Akira Ajisaka a8ddc179df
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a3)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d4636067)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 819239f971)
 2021-03-25 20:29:18 +09:00
He Xiaoqiao b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00