Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,720 Commits 326 Branches 371 Tags
Commit Graph

5063 Commits

Author SHA1 Message Date
Xing Lin
c8f22fe737 HADOOP-16532. Fix TestViewFsTrash to use the correct homeDir. Contributed by Xing Lin. (#3514) 
(cherry picked from commit 97c0f968792e1a45a1569a3184af7b114fc8c022)
 2021-10-13 14:56:53 -07:00
Ahmed Hussein
e99af5cf81 HADOOP-17930. implement non-guava Precondition checkState (#3522) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit c36f9402dc082a8903cf6e7fdca128658b11c59d)
 2021-10-07 11:02:51 +09:00
Ahmed Hussein
4b6316a7fd HADOOP-17929. implement non-guava Precondition checkArgument (#3473) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 0c498f21dee7a5bbf91ad8afbfb372d08bacce6c)
 2021-10-07 11:02:45 +09:00
Rintaro Ikeda
a777264c7f HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 
(cherry picked from commit 607c20c6122238ee579a3ad5da882128e7203fad)
 2021-09-17 13:35:54 +00:00
Steve Loughran
5094dede8e
HADOOP-17126. implement non-guava Precondition checkNotNull 
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
 2021-09-17 11:37:29 +01:00
Stephen O'Donnell
7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao
ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69bc578db1ce95198aa7fa7bc7199320.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein
318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Chris Nauroth
736367e578 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 19:19:44 +00:00
Viraj Jasani
25fe54b890
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99a157fa4ad174c4ff979414af2edb67a98eb9fe)
 2021-09-03 11:06:19 +09:00
jianghuazhu
dac74b0e11
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364e9258247029c22a222a665771ab4c0)
(cherry picked from commit 7c663043b2c1e207dd8c05e09e17811d68badfac)
 2021-08-27 12:50:22 +08:00
jianghuazhu
3b38d011a8
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit ad54f5195c8c01f333703c55cd70703109d75f29)
(cherry picked from commit 2b2f8f575bffb18df53a3cf70cc45ba6384798e2)
 2021-08-25 17:36:07 +08:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Bryan Beaudreault
dab829063d
HADOOP-17840: Backport HADOOP-17837 to branch-3.2 (#3275) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-08-07 09:49:46 +05:30
Viraj Jasani
2f5e527910 HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec11c3e1c2ed5ee9db801d40a2f7a2ed6)
 2021-08-06 09:54:34 +08:00
Akira Ajisaka
b838647b16
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#3264) 
* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
 2021-08-04 15:46:00 +09:00
hchaverr
4f602cb17b HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 15:59:45 -07:00
Viraj Jasani
1df581a20c HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db1b13a71d5a5c6ee0cd689fb8b27895)
 2021-07-23 12:57:36 +08:00
Abhishek Das
f9832031c2 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:25:03 -07:00
Akira Ajisaka
a9505cf5d1
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d137b6d45c20c481e7bbc6160d72f94)
 2021-07-12 10:18:44 +09:00
Artem Smotrakov
6c7f192f0f HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 14:25:08 +09:00
liangxs
b7108cfcee HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc674a24a3251cf1b20f22a4fd4fac9f)
 2021-07-07 09:54:16 +08:00
Takanobu Asanuma
2f29784293 HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129fcf466d9647e0672ecf7dd72213e72)
 2021-06-17 10:01:28 +09:00
Mingliang Liu
d92af85045
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 20a4cb0c67b413e7b9bc2b3213b5b592bfaa99d5.
 2021-06-11 00:38:36 -07:00
July
20a4cb0c67
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:59:26 -07:00
Konstantin V Shvachko
549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:55:35 -07:00
Viraj Jasani
3c1f3b87fa HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:23 +09:00
hchaverr
f6af5bff9e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:04:25 -07:00
Eric Badger
7ad844ee9e MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java
 2021-04-23 18:53:09 +00:00
Wei-Chiu Chuang
8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
 2021-04-19 20:12:27 -07:00
Eric Badger
bf0d058b69 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 2021-04-19 20:06:36 +00:00
Masatake Iwasaki
08cc11cd28 HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f858ff654de0fc01cfb9afef2e705e33c)
 2021-04-15 13:35:49 +00:00
Akira Ajisaka
d9858a4663
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:26 +09:00
Wei-Chiu Chuang
0faa626bd8
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
 2021-04-12 19:45:49 -07:00
litao
d566e4e41d HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 11:00:33 +09:00
He Xiaoqiao
eb10bd8ffd
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:29 +08:00
Takanobu Asanuma
46b2e9634b HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:54:07 +09:00
Takanobu Asanuma
9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
kwangsun
35855007f6
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:22 +09:00
Ayush Saxena
54bb30c955 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:52 +05:30
Jim Brennan
38f86cc8c7 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-16 19:17:18 +00:00
He Xiaoqiao
7e20910d3d HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:10:12 +00:00
Akira Ajisaka
4fa57485e7
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:19 +09:00
S O'Donnell
707806092f HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
 2021-03-01 12:24:31 +00:00
Ahmed Hussein
e4c638eb21 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a061f7d89fb1353cd5ef6f550c049f36c)
 2021-02-26 20:38:13 +00:00
Kengo Seki
5abf19a773
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a6b9d600b43909a0770a6b0afea899a)
 2021-02-20 19:15:16 +09:00
He Xiaoqiao
c7cdcbfe2b
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:10:08 +08:00
Akira Ajisaka
52004affa8
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:57:48 +09:00
Anton Kutuzov
16e908e757 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:12:46 -08:00
He Xiaoqiao
03151aa78b
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:44:15 +08:00