Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,712 Commits 326 Branches 371 Tags
Commit Graph

5219 Commits

Author SHA1 Message Date
Viraj Jasani
28d4affea0 HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 
(cherry picked from commit 783e4805e7ef46da6095e0375fe8dd5e9cd26e85)
 2021-10-06 02:19:44 +09:00
Viraj Jasani
ce30f796f6 HADOOP-17947. Provide alternative to Guava VisibleForTesting (#3505) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 5b1d594005293e8de7e58cd70cde88c9041107cf)
 2021-10-05 10:02:08 +09:00
Rintaro Ikeda
a777264c7f HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 
(cherry picked from commit 607c20c6122238ee579a3ad5da882128e7203fad)
 2021-09-17 13:35:54 +00:00
Steve Loughran
5094dede8e
HADOOP-17126. implement non-guava Precondition checkNotNull 
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
 2021-09-17 11:37:29 +01:00
Stephen O'Donnell
7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao
ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69bc578db1ce95198aa7fa7bc7199320.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein
318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Chris Nauroth
736367e578 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 19:19:44 +00:00
Viraj Jasani
25fe54b890
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99a157fa4ad174c4ff979414af2edb67a98eb9fe)
 2021-09-03 11:06:19 +09:00
jianghuazhu
dac74b0e11
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364e9258247029c22a222a665771ab4c0)
(cherry picked from commit 7c663043b2c1e207dd8c05e09e17811d68badfac)
 2021-08-27 12:50:22 +08:00
jianghuazhu
3b38d011a8
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit ad54f5195c8c01f333703c55cd70703109d75f29)
(cherry picked from commit 2b2f8f575bffb18df53a3cf70cc45ba6384798e2)
 2021-08-25 17:36:07 +08:00
Viraj Jasani
f26a547f2b HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 
(cherry picked from commit fc566ad9b0e1336381c5f6b321cdcff81768f47a)
 2021-08-24 22:46:09 +09:00
Tamas Domok
3fdb056b54
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-10 01:35:18 +02:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Bryan Beaudreault
dab829063d
HADOOP-17840: Backport HADOOP-17837 to branch-3.2 (#3275) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-08-07 09:49:46 +05:30
Viraj Jasani
2f5e527910 HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec11c3e1c2ed5ee9db801d40a2f7a2ed6)
 2021-08-06 09:54:34 +08:00
Akira Ajisaka
b838647b16
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#3264) 
* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
 2021-08-04 15:46:00 +09:00
hchaverr
4f602cb17b HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 15:59:45 -07:00
Viraj Jasani
1df581a20c HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db1b13a71d5a5c6ee0cd689fb8b27895)
 2021-07-23 12:57:36 +08:00
Abhishek Das
f9832031c2 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:25:03 -07:00
Akira Ajisaka
a9505cf5d1
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d137b6d45c20c481e7bbc6160d72f94)
 2021-07-12 10:18:44 +09:00
Artem Smotrakov
6c7f192f0f HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 14:25:08 +09:00
Akira Ajisaka
aae60521a3 HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 
(cherry picked from commit 6ac5d8a1ed6ec3f6de7af3b8a79869830a4377ee)
 2021-07-09 22:52:20 +09:00
liangxs
b7108cfcee HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc674a24a3251cf1b20f22a4fd4fac9f)
 2021-07-07 09:54:16 +08:00
Takanobu Asanuma
2f29784293 HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129fcf466d9647e0672ecf7dd72213e72)
 2021-06-17 10:01:28 +09:00
Mingliang Liu
d92af85045
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 20a4cb0c67b413e7b9bc2b3213b5b592bfaa99d5.
 2021-06-11 00:38:36 -07:00
July
20a4cb0c67
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:59:26 -07:00
Konstantin V Shvachko
549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:55:35 -07:00
Viraj Jasani
3c1f3b87fa HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:23 +09:00
hchaverr
f6af5bff9e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:04:25 -07:00
Eric Badger
7ad844ee9e MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java
 2021-04-23 18:53:09 +00:00
Wei-Chiu Chuang
8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
 2021-04-19 20:12:27 -07:00
Eric Badger
bf0d058b69 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 2021-04-19 20:06:36 +00:00
Wei-Chiu Chuang
289c7d91b4
HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
(cherry picked from commit 77315abe474ad3a17507bfc67956b85327f487e6)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-18 22:13:24 -07:00
Masatake Iwasaki
08cc11cd28 HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f858ff654de0fc01cfb9afef2e705e33c)
 2021-04-15 13:35:49 +00:00
Akira Ajisaka
d9858a4663
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:26 +09:00
Wei-Chiu Chuang
0faa626bd8
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
 2021-04-12 19:45:49 -07:00
litao
d566e4e41d HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 11:00:33 +09:00
He Xiaoqiao
eb10bd8ffd
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:29 +08:00
Takanobu Asanuma
46b2e9634b HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:54:07 +09:00
Takanobu Asanuma
9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
kwangsun
35855007f6
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:22 +09:00
Ayush Saxena
54bb30c955 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:52 +05:30
Jim Brennan
38f86cc8c7 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-16 19:17:18 +00:00
He Xiaoqiao
7e20910d3d HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:10:12 +00:00
Akira Ajisaka
0b278502e2
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml
 2021-03-11 15:37:56 +09:00
Akira Ajisaka
4fa57485e7
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:19 +09:00
S O'Donnell
707806092f HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
 2021-03-01 12:24:31 +00:00
Ahmed Hussein
e4c638eb21 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a061f7d89fb1353cd5ef6f550c049f36c)
 2021-02-26 20:38:13 +00:00
Renukaprasad C
9951ea9fa8 HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010ac193a8242d59b14f9fa9d17957fe5)
 2021-02-24 17:34:14 +08:00