Commit Graph

5542 Commits

Author SHA1 Message Date
Ayush Saxena f5c1557288
HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. (#2820). Contributed by Ayush Saxena.
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-27 03:01:41 +05:30
Akira Ajisaka af1f9f43ea
HADOOP-17133. Implement HttpServer2 metrics (#2145) 2021-03-25 12:09:43 -07:00
touchida 95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Takanobu Asanuma b4d97a8dc7 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 2021-03-24 17:51:35 +09:00
Ayush Saxena 03cfc85279
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732). Contributed by Ayush Saxena.
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-24 02:36:26 +05:30
Jack Jiang d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Xiaoyu Yao 63eb289462
HADOOP-17598. Fix java doc issue introduced by HADOOP-17578. (#2802). Contributed by Xiaoyu Yao.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 10:23:45 +05:30
kwangsun c8d327a4f1
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-22 11:43:32 +09:00
Xiaoyu Yao b503de2328
HADOOP-17578. Improve UGI debug log to help troubleshooting TokenCach… (#2762) 2021-03-17 10:57:11 -07:00
Jim Brennan 299b8062f1 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 2021-03-15 20:13:17 +00:00
He Xiaoqiao b1dc6c40a0
HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 2021-03-14 18:09:50 +08:00
Masatake Iwasaki 0f6ba5e031
HADOOP-17514. Remove trace subcommand from hadoop CLI. (#2680) 2021-03-12 10:26:46 +09:00
Steve Loughran bcd9c67082
HADOOP-16721. Improve S3A rename resilience (#2742)
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.

When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.

Contributed by Steve Loughran.
2021-03-11 12:47:39 +00:00
Akira Ajisaka 23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753)
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
Chao Sun 176bd88890
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. (#2522)
Contributed by Chao Sun.

(Cherry-picked via PR #2575)
2021-03-09 20:01:29 +00:00
Haoze Wu ef7ab535c5
HADOOP-17552. Change ipc.client.rpc-timeout.ms from 0 to 120000 by default to avoid potential hang. (#2727) 2021-03-06 22:26:16 +09:00
Akira Ajisaka 9fd2198daa
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 2021-03-04 14:55:37 +09:00
S O'Donnell c3b3b36dee HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 2021-03-01 11:36:41 +00:00
Ahmed Hussein e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Renukaprasad C 924fa87010
HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 2021-02-24 01:29:25 -08:00
Mike 7b7c0019f4
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701)
Contributed by Mike Pryakhin.
2021-02-23 17:03:27 +00:00
Akira Ajisaka 9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)"
This reverts commit 4cf3531583.
2021-02-19 11:09:10 +09:00
Akira Ajisaka 4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
Steve Loughran 78905d7e3f
HADOOP-16906. Abortable (#2684)
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.
2021-02-11 17:37:20 +00:00
Steve Loughran 798df6d699
HADOOP-13327 Output Stream Specification. (#2587)
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
Akira Ajisaka 52217fce3d
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546)
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-02-05 16:14:10 +09:00
YaYun-Wang 394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377)
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
belugabehr 21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Siyao Meng 1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
He Xiaoqiao 06e836cd57
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 19:45:40 +08:00
Steve Loughran 0bb52a42e5
HADOOP-17483. Magic committer is enabled by default. (#2656)
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
  default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
  want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.

Contributed by Steve Loughran.
2021-01-27 19:04:22 +00:00
Steve Loughran 28cc912a5c
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic (#2649)
Follow-on to HADOOP-16830/HADOOP-17271.

Contributed by Steve Loughran.
2021-01-27 16:39:29 +00:00
Steve Loughran 80c7404b51
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark (#2530)
This needs SPARK-33739 in the matching spark branch in order to work

Contributed by Steve Loughran.
2021-01-26 19:30:51 +00:00
Akira Ajisaka d82009599a
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628)
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-01-24 22:56:47 +09:00
Anton Kutuzov 91d4ba57c5 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 2021-01-22 17:55:37 -08:00
Ayush Saxena 9b4f09a6bb HDFS-15758. Fix typos in MutableMetric. Contributed by Haibin Huang. 2021-01-18 20:35:37 +05:30
He Xiaoqiao 97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao 7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Steve Loughran 724edb0354
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16830 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.

Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
2021-01-12 17:30:32 +00:00
Steve Loughran 05c9c2ed02 Revert "HADOOP-17451. IOStatistics test failures in S3A code. (#2594)"
This reverts commit d3014e01f3.
(fixing commit text before it is frozen)
2021-01-12 17:29:59 +00:00
Steve Loughran d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. (#2594)
Caused by HADOOP-16380 and HADOOP-17271.

Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.

Contributed by Steve Loughran.
2021-01-12 17:25:14 +00:00
Mehakmeet Singh 0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API (#2604)
Contributed by Mehakmeet Singh.
2021-01-12 15:48:09 +00:00
stack d4fd675a95 Revert "HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470)"
This reverts commit e306f59421.
2021-01-11 08:54:55 -08:00
He Xiaoqiao ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein 77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
Borislav Iordanov e306f59421
HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode (#2470)
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
Signed-off-by: stack <stack@apache.org>
2021-01-08 09:10:21 -08:00
lfengnan 87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578)
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
dgzdot b1abb10ea2
HADOOP-17430. Restore ability to set Text to empty byte array (#2545)
Contributed by gaozhan.ding
2021-01-05 21:09:41 +00:00
Wei-Chiu Chuang 2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 2021-01-04 20:54:44 -08:00
Ayush Saxena 77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30