Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,848 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24848 Commits

Author SHA1 Message Date
Jonathan Hung ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko a89c624002 YARN-10609. Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue). Contributed by Qi Zhu. 2021-02-24 20:41:35 +01:00
S O'Donnell 9c583f36d5 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 2021-02-24 10:29:36 +00:00
Renukaprasad C 924fa87010
HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 2021-02-24 01:29:25 -08:00
LeonGao 940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704) 2021-02-23 22:29:48 -08:00
Masatake Iwasaki e9a3c2950e
HADOOP-17538. Add kms-default.xml and httpfs-default.xml to site index. (#2716) 2021-02-24 07:46:02 +09:00
Mike 7b7c0019f4
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem (#2701) 
Contributed by Mike Pryakhin.
 2021-02-23 17:03:27 +00:00
Ayush Saxena 647d23b611 HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:08:38 +05:30
Takanobu Asanuma a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714) 2021-02-23 15:12:06 +09:00
Eric Badger 47420ae3ed YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.
 2021-02-22 18:48:14 +00:00
Akira Ajisaka 8bca105655
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-02-22 23:15:04 +09:00
Peter Bacsko b722c52f4e YARN-10513. CS Flexible Auto Queue Creation RM UIv2 modifications. Contributed by Andras Gyori. 2021-02-22 12:55:45 +01:00
sumangala-patki 7f64030314
HADOOP-17537. ABFS: Correct assertion reversed in HADOOP-13327 
Contributed Sumangala Patki.
 2021-02-22 11:45:58 +00:00
Wang Yuxuan 1e32ab7d2f HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang f8059d0bdf Revert "first commit (#2559)" 
This reverts commit 1b94569d89.
 2021-02-22 18:13:06 +08:00
Wang Yuxuan 1b94569d89
first commit (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 02:10:39 -08:00
Akira Ajisaka e119063a07
HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-02-22 17:33:02 +09:00
jianghuazhu 98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-02-21 23:22:32 -08:00
Ayush Saxena ff84a57483 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 2021-02-20 19:48:00 +05:30
Ayush Saxena 2ce5752fa8 HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun. 2021-02-20 19:34:29 +05:30
Akira Ajisaka b9b49ed956
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (#1738) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
 2021-02-20 17:18:23 +09:00
Eric Badger 4891e68c2b YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.
 2021-02-19 22:29:10 +00:00
Peter Bacsko d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko 4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Takanobu Asanuma 25af8901a9 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 2021-02-19 14:44:58 +09:00
Akira Ajisaka 9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Hui Fei 2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:07:22 +08:00
Ayush Saxena e391844e8e HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. 2021-02-18 01:39:04 +05:30
Akira Ajisaka 0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Akira Ajisaka 4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Ahmed Hussein eac7aefaaf
HADOOP-17109. add guava BaseEncoding to illegalClasses (#2703) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-02-17 10:15:41 +09:00
Szilard Nemeth 9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Anoop Sam John 1bb4101b59
HADOOP-17038 Support disabling buffered reads in ABFS positional reads. (#2646) 
- Contributed by @anoopsjohn
 2021-02-16 22:27:52 +05:30
Jim Brennan 3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Akira Ajisaka 2b3c5b1733
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-02-16 14:50:50 +09:00
Zehao Chen 07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
Szilard Nemeth bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth 852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
litao c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676) 2021-02-14 00:12:24 -05:00
Masatake Iwasaki f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Steve Loughran 78905d7e3f
HADOOP-16906. Abortable (#2684) 
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.

Contributed by Jungtaek Lim and Steve Loughran.
 2021-02-11 17:37:20 +00:00
Szilard Nemeth 98ca6afd17 YARN-10618. RM UI2 Application page shows the AM preempted containers instead of the nonAM ones. Contributed by Benjamin Teke 2021-02-11 09:14:37 +01:00
Akira Ajisaka 933fbeffee
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-02-11 08:07:08 +09:00
Szilard Nemeth cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Steve Loughran 798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
Szilard Nemeth a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Eric Badger 462561654b MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.
 2021-02-09 19:01:38 +00:00
Szilard Nemeth fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
LeonGao 9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 2021-02-09 10:49:28 +00:00