Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,679 Commits 326 Branches 371 Tags
Commit Graph

5207 Commits

Author SHA1 Message Date
Stephen O'Donnell
3c715a272a
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 14:25:57 +08:00
Brahma Reddy Battula
182167a9fb Preparing for 3.2.3 release. 2021-08-08 19:17:14 +05:30
Bryan Beaudreault
dab829063d
HADOOP-17840: Backport HADOOP-17837 to branch-3.2 (#3275) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-08-07 09:49:46 +05:30
Viraj Jasani
2f5e527910 HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec11c3e1c2ed5ee9db801d40a2f7a2ed6)
 2021-08-06 09:54:34 +08:00
Akira Ajisaka
b838647b16
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#3264) 
* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
 2021-08-04 15:46:00 +09:00
hchaverr
4f602cb17b HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 15:59:45 -07:00
Viraj Jasani
1df581a20c HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db1b13a71d5a5c6ee0cd689fb8b27895)
 2021-07-23 12:57:36 +08:00
Abhishek Das
f9832031c2 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:25:03 -07:00
Akira Ajisaka
a9505cf5d1
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d137b6d45c20c481e7bbc6160d72f94)
 2021-07-12 10:18:44 +09:00
Artem Smotrakov
6c7f192f0f HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 14:25:08 +09:00
Akira Ajisaka
aae60521a3 HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 
(cherry picked from commit 6ac5d8a1ed6ec3f6de7af3b8a79869830a4377ee)
 2021-07-09 22:52:20 +09:00
liangxs
b7108cfcee HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc674a24a3251cf1b20f22a4fd4fac9f)
 2021-07-07 09:54:16 +08:00
Takanobu Asanuma
2f29784293 HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129fcf466d9647e0672ecf7dd72213e72)
 2021-06-17 10:01:28 +09:00
Mingliang Liu
d92af85045
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 20a4cb0c67b413e7b9bc2b3213b5b592bfaa99d5.
 2021-06-11 00:38:36 -07:00
July
20a4cb0c67
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:59:26 -07:00
Konstantin V Shvachko
549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:55:35 -07:00
Viraj Jasani
3c1f3b87fa HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:23 +09:00
hchaverr
f6af5bff9e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:04:25 -07:00
Eric Badger
7ad844ee9e MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java
 2021-04-23 18:53:09 +00:00
Wei-Chiu Chuang
8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
 2021-04-19 20:12:27 -07:00
Eric Badger
bf0d058b69 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 2021-04-19 20:06:36 +00:00
Wei-Chiu Chuang
289c7d91b4
HADOOP-17608. Fix TestKMS failure (#2880) (#2929) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a5075e6b9c15fde22e43ac9bcf2a6c22f9)
(cherry picked from commit 77315abe474ad3a17507bfc67956b85327f487e6)

 Conflicts:
	hadoop-common-project/hadoop-kms/src/test/java/org/apache/hadoop/crypto/key/kms/server/TestKMS.java

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-18 22:13:24 -07:00
Masatake Iwasaki
08cc11cd28 HADOOP-17569. Building native code fails on Fedora 33. (#2886) 
(cherry picked from commit 2717203f858ff654de0fc01cfb9afef2e705e33c)
 2021-04-15 13:35:49 +00:00
Akira Ajisaka
d9858a4663
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:26 +09:00
Wei-Chiu Chuang
0faa626bd8
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
 2021-04-12 19:45:49 -07:00
litao
d566e4e41d HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 11:00:33 +09:00
He Xiaoqiao
eb10bd8ffd
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:29 +08:00
Takanobu Asanuma
46b2e9634b HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 
(cherry picked from commit b4d97a8dc75e4dc823d092f5bac37d6096dc6dc4)
 2021-03-24 17:54:07 +09:00
Takanobu Asanuma
9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
kwangsun
35855007f6
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:22 +09:00
Ayush Saxena
54bb30c955 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:13:52 +05:30
Jim Brennan
38f86cc8c7 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-16 19:17:18 +00:00
He Xiaoqiao
7e20910d3d HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:10:12 +00:00
Akira Ajisaka
0b278502e2
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml
 2021-03-11 15:37:56 +09:00
Akira Ajisaka
4fa57485e7
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:19 +09:00
S O'Donnell
707806092f HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
 2021-03-01 12:24:31 +00:00
Ahmed Hussein
e4c638eb21 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a061f7d89fb1353cd5ef6f550c049f36c)
 2021-02-26 20:38:13 +00:00
Renukaprasad C
9951ea9fa8 HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010ac193a8242d59b14f9fa9d17957fe5)
 2021-02-24 17:34:14 +08:00
Kengo Seki
5abf19a773
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a6b9d600b43909a0770a6b0afea899a)
 2021-02-20 19:15:16 +09:00
He Xiaoqiao
c7cdcbfe2b
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:10:08 +08:00
Akira Ajisaka
52004affa8
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:57:48 +09:00
Anton Kutuzov
16e908e757 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:12:46 -08:00
He Xiaoqiao
03151aa78b
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:44:15 +08:00
He Xiaoqiao
a17793534c
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:46 +08:00
Wei-Chiu Chuang
8af0fcf628 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
(cherry picked from commit 94c126cc9e9aadaf9aa28ebd55a8712aca54e709)
 2021-01-04 20:56:32 -08:00
He Xiaoqiao
1c0ac5ed5c
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:31:01 +08:00
He Xiaoqiao
8d7600ae6b HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
(cherry picked from commit 3a860e876e799ef44fca34e3cb19a89dbd4f6744)
 2020-12-15 14:02:01 -08:00
Jim Brennan
f5a98b6000 HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 20:42:59 +00:00
Ahmed Hussein
d9bf607066 HADOOP-17389. KMS should log full UGI principal. (#2476) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
(cherry picked from commit 8c234fc0d4472b65ff555a79acb9a12f2dae3b27)
 2020-12-07 17:29:05 +00:00
Takanobu Asanuma
3f14715215
HADOOP-15775. [JDK9] Add missing javax.activation-api dependency. Contributed by Akira Ajisaka. 2020-12-04 22:28:19 +08:00