Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,911 Commits 326 Branches 371 Tags
Commit Graph

4907 Commits

Author SHA1 Message Date
Steve Loughran
10a7270f43
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I767e11bc3b2133aa4aa36d837162673930e1e7e1
 2022-02-21 17:29:08 +00:00
Abhishek Das
f7f753d6fd HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:26:49 -07:00
Konstantin V Shvachko
a357d4ce60 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:58:01 -07:00
Viraj Jasani
2407d81f5d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548feb9ec9a957e120045983dd21d9964)
 2021-05-22 19:12:57 +09:00
hchaverr
39bf9e270e HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb0590f85bb42d2471992bf5d524628fdd6)
 2021-05-06 18:10:27 -07:00
Eric Badger
be5fa9615d MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java

(cherry picked from commit 7ad844ee9ed7eaab4fbb4d38576be10d497460cf)
 2021-04-23 19:01:29 +00:00
Wei-Chiu Chuang
f3503303dc
HADOOP-17651. Backport to branch-3.1 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) (#2935) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0dc0dd8242018153fd652a3206e73b5)
(cherry picked from commit 6340ac857b7ff3f73bbcf95b59b98aac134f33af)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282013d7776667415a429e7ca35efba66)
(cherry picked from commit 1110b03752b45bc4695baaa6d9655e18de67303a)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99aae5e29c17b03fddb0a805da8fcae8)
(cherry picked from commit 7f7535534d5541e10b6b14dee3aa38a00058f201)
(cherry picked from commit 8ff61f9b076a7022a1dfc067b9f94434756b64a7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java

Change-Id: I465f6003b6f4c5df9c41c83eac3738bac56403e1
 2021-04-20 06:13:32 -07:00
Eric Badger
9c3acb3018 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 
(cherry picked from commit bf0d058b698b15d1c51927b8fd8cf678ab95661a)
 2021-04-19 20:18:20 +00:00
Akira Ajisaka
1b5ef939d3
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be3ae1f42bde9c22ab5ba96cf60df3c6)
 2021-04-13 17:10:48 +09:00
Wei-Chiu Chuang
8a865273db HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
(cherry picked from commit 0faa626bd84a02688e0938de1320fc32ec550aea)
 2021-04-13 11:09:52 +08:00
He Xiaoqiao
16d50f4bbf
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:59 +08:00
Allen Wittenauer
1bcde89983
HADOOP-14671. Upgrade Apache Yetus to 0.8.0 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4b8c2b1c1c9e070002ab3a2afb850d9d7b810d0a)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.2/CHANGELOG.3.0.2.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.2/RELEASENOTES.3.1.2.md
 2021-03-24 16:04:11 +09:00
kwangsun
7af7eb3e9f
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1a7f15d6be35051414199d1d3fdc5ef)
 2021-03-22 11:46:44 +09:00
Ayush Saxena
8ec256db04 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:14:11 +05:30
Jim Brennan
a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 38f86cc8c72201b081a083cd32bc3a322c6d9398)
 2021-03-16 19:19:01 +00:00
He Xiaoqiao
0224dd87af HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0695fedd8269a2a30a74da43e1d0ae7)
 2021-03-14 15:20:16 +00:00
Akira Ajisaka
5dda43c061
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa58610c04bdc1ac7570476accdc25b7)
 2021-03-04 14:57:41 +09:00
S O'Donnell
53f46214e4 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java

(cherry picked from commit 707806092f64f709814e18cb0d1b4f58731a5716)
 2021-03-01 12:26:41 +00:00
Jim Brennan
4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Kengo Seki
7bcb1a83a3
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a6b9d600b43909a0770a6b0afea899a)
 2021-02-20 19:16:07 +09:00
He Xiaoqiao
3a835753ee
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:00:41 +08:00
Akira Ajisaka
fb5b97305c
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:58:06 +09:00
Anton Kutuzov
397ff302b3 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:16:21 -08:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Wei-Chiu Chuang
13768fe6f0 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
(cherry picked from commit 94c126cc9e9aadaf9aa28ebd55a8712aca54e709)
(cherry picked from commit 8af0fcf628b0a8ab1b9b44762eb4a9341f9d7b6a)
 2021-01-04 21:00:29 -08:00
He Xiaoqiao
5d0eb4c6f0
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:35:36 +08:00
He Xiaoqiao
1fe15113cf HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
(cherry picked from commit 3a860e876e799ef44fca34e3cb19a89dbd4f6744)
(cherry picked from commit 8d7600ae6b7fadba1b4695666cf9287be021d481)
 2020-12-15 14:02:15 -08:00
Jim Brennan
392f69471d HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 21:07:03 +00:00
Hui Fei
2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
Ayush Saxena
cfb96240dc
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 
(cherry picked from commit b52fd05d42d9a76f6936a5d86c23fcd66244fe3d)
 2020-12-04 11:19:21 +09:00
Takanobu Asanuma
43f7ab2cf0
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 
(cherry picked from commit 559cb11551d7045015aa60afe7e7998fe0b2ef13)
 2020-12-04 11:19:14 +09:00
Jim Brennan
c43eca6cdc HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 5bfb97bc7d5ff5e083aff90e5a380a850d9e4530)
 2020-12-03 19:47:46 +00:00
Andrea Scarpino
c8178a9caa
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:18:12 +09:00
Eric Payne
c7845dc574 HADOOP-17346. Fair call queue is defeated by abusive service principals. Contributed by Ahmed Hussein (ahussein). 2020-11-23 21:34:52 +00:00
Jim Brennan
da87653b66 HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 6b3d65b46ed6535c7754501287186c7128b32ed4)
 2020-11-19 21:59:57 +00:00
Ahmed Hussein
56e327b7ca HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 1ea3f74246294c280a2ccb0ff3e90b5721c5f0e2)
 2020-11-16 22:08:14 +00:00
Jim Brennan
118d8f5aa1 HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 4bdba35b9ffd57e249d2467d618ed63f1950b438)
 2020-11-13 21:52:10 +00:00
Ahmed Hussein
487eaa6dea HADOOP-17358. Improve excessive reloading of Configurations (#2436) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0fcaf73a3989000dbc60fa214a317da1)
(cherry picked from commit 23fe3bdab3410b8ed370fa6af9f2ed63817006ab)
(cherry picked from commit ac3e10af63c4a70e45b26acd50b01a93fa8fd605)
 2020-11-12 10:35:57 -08:00
Stephen Jung
9f7553bab4 HADOOP-17096. Fix ZStandardCompressor input buffer offset (#2104). Contributed by Stephen Jung (Stripe). 
(cherry picked from commit 45434c93e865386de8c5f4036e795a3362557d56)
(cherry picked from commit 0712505b592b7cb6510c0d57bffa71125b7fdf0e)
 2020-11-10 11:41:34 -08:00
Steve Loughran
d903278607 HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. (#2427). Contributed by Steve Loughran. 2020-11-07 17:05:59 +05:30
hchaverr
3443d75946 HDFS-15623. Respect configured values of rpc.engine (#2403) Contributed by Hector Chaverri. 
(cherry picked from commit 6eacaffeea21e7e9286497ee17f89fd939d2eead)
 2020-11-06 15:02:47 -08:00
Eric Badger
4cec4bebd9 HADOOP-17342. Creating a token identifier should not do kerberos name 
resolution. Contributed by Jim Brennan.

(cherry picked from commit af389d989770c4218c24e640f572bc7492f7d4b3)
 2020-11-05 22:11:52 +00:00
Wei-Chiu Chuang
d4633fcff9 Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291)" 
This reverts commit 166bee93af6fb68ee2b3fab4c355413cae9fc1d6.
 2020-11-04 16:18:54 -08:00
Akira Ajisaka
166bee93af HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. (#2291) 
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04cb8e5dab8410ef64daa0fe799e389fd)
(cherry picked from commit dd1634ec3b676d6c3da18af7da46734584ddd9f6)
(cherry picked from commit 1c7043b1ff89ba9c995cbf546a99bbbcccbe56b1)
 2020-11-03 11:23:19 -08:00
Ayush Saxena
66e4a8b1d7 HADOOP-17119. Jetty upgrade to 9.4.x causes MR app fail with IOException. Contributed by Bilwa S T. 
(cherry picked from commit 8a58a12626f0332a4a828f312e2007d8c23564ee)
(cherry picked from commit d2ecaf217fcde5f900262ef156df9cee5de950d7)
 2020-10-23 17:53:42 -07:00
Takanobu Asanuma
c808a21550 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c742806c7b9edf00a92c98b1e9cfcb0e1)
 2020-10-20 19:13:59 +09:00
Konstantin V Shvachko
2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3cc13b0b92b9f42da1731c4ce35c9ded)
 2020-10-12 17:52:37 -07:00
Steve Loughran
5f0004e012 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I9261cadedb6b64993351611e140c4e3e9cf50d3a
 2020-08-28 17:55:18 +01:00
sguggilam
e1ac832d5d
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-27 00:06:05 -07:00
Mingliang Liu
9f94c9e60d
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit 12fb9e0600f665aca3e7ebe0be9b95ff232d520f.
 2020-08-26 11:24:03 -07:00