7a45ef4164
MAPREDUCE-7435. Manifest Committer OOM on abfs ( #5519 )
...
This modifies the manifest committer so that the list of files
to rename is passed between stages as a file of
writeable entries on the local filesystem.
The map of directories to create is still passed in memory;
this map is built across all tasks, so even if many tasks
created files, if they all write into the same set of directories
the memory needed is O(directories) with the
task count not a factor.
The _SUCCESS file reports on heap size through gauges.
This should give a warning if there are problems.
Contributed by Steve Loughran
2023-06-09 17:00:59 +01:00
1dbaba8e70
HADOOP-18740. S3A prefetch cache blocks should be accessed by RW locks ( #5675 )
...
Contributed by Viraj Jasani
2023-06-07 14:05:52 +01:00
1d0c9ab433
Revert "HADOOP-18207. Introduce hadoop-logging module ( #5503 )"
...
This reverts commit 03a499821c
2023-06-05 09:34:40 +05:30
e0a339223a
HADOOP-18709. Add curator based ZooKeeper communication support over SSL/TLS into the common library. Contributed by Ferenc Erdelyi
2023-06-04 14:40:41 -04:00
03a499821c
HADOOP-18207. Introduce hadoop-logging module ( #5503 )
...
Reviewed-by: Duo Zhang <zhangduo@apache.org>
2023-06-02 18:07:34 -07:00
160b9fc3c9
HADOOP-18755. openFile builder new optLong() methods break hbase-filesystem ( #5704 )
...
This is a followup to
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem
Contributed by Steve Loughran
2023-06-01 14:31:08 +01:00
4627242c44
HADOOP-18652. Path.suffix raises NullPointerException ( #5653 ). Contributed by Patrick Grandjean.
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-19 05:16:55 +05:30
f6770dee47
HDFS-16979. RBF: Add proxyuser port in hdfsauditlog ( #5552 ). Contributed by liuguanghua.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-19 05:02:16 +05:30
a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport ( #5662 )
...
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()
This brings trunk into sync with the branch-3.3 changes
2023-05-16 18:16:24 +01:00
bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops ( #5621 )
...
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter
Contributed by Viraj Jasani
2023-05-16 10:40:16 +01:00
ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods ( #5650 )
...
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
2023-05-12 15:22:37 +01:00
905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. ( #5520 ). Contributed by WangYuanben.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-12 04:12:02 +05:30
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem ( #5611 )
...
This:
1. Adds optLong, optDouble, mustLong and mustDouble
methods to the FSBuilder interface to let callers explicitly
passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
now only set long values instead, so as to avoid problems
related to overloaded methods resulting in a ".0" being appended
to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
the new methods
4. And the s3a code is resilient to parse errors in is numeric options
-it will downgrade to the default.
This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.
For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.
Contributed by Steve Loughran
2023-05-11 17:57:25 +01:00
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. ( #5095 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-10 01:42:12 +05:30
c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 ( #5618 )
2023-05-05 13:08:58 -07:00
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common ( #5553 )
...
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them. Applications which use the leasing APIs should migrate to the new
interface where possible.
Contributed by Stephen Wu
2023-05-03 11:05:55 +01:00
fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. ( #5612 ). Contributed by Shuyan Zhang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-03 00:09:36 +08:00
bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile ( #5580 )
...
Contributed by Viraj Jasani
2023-05-02 15:21:46 +01:00
73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase ( #5455 )
2023-05-02 15:52:57 +02:00
d75c6d9d57
HADOOP-18715. Add debug log for getting details of tokenKindMap ( #5608 ). Contributed by Pralabh Kumar.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-29 17:28:49 +05:30
6aac6cb212
HADOOP-18660. Filesystem Spelling Mistake ( #5475 ). Contributed by Sebastian Baunsgaard.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-25 21:44:04 +05:30
2f66f0b83a
HADOOP-18694. Client.Connection#updateAddress needs to ensure that address is resolved before updating ( #5542 ). Contributed by dzcxzl.
...
Reviewed-by: Steve Vaughan <email@stevevaughan.me>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
2023-04-25 03:52:49 +05:30
5b23224970
HADOOP-18714. Wrong StringUtils.join() called in AbstractContractRootDirectoryTest ( #5578 )
2023-04-24 09:17:12 +02:00
742e07d9c3
HADOOP-18710. Add RPC metrics for response time ( #5545 ). Contributed by liuguanghua.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-22 01:06:08 +05:30
9e24ed2196
HADOOP-18691. Add a CallerContext getter on the Schedulable interface ( #5540 )
2023-04-20 10:11:25 -07:00
d07356e60e
HADOOP-18597. Simplify single node instructions for creating directories for Map Reduce. ( #5305 )
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-20 16:12:44 +05:30
5119d0c72f
HDFS-16982 Use the right Quantiles Array for Inverse Quantiles snapshot ( #5556 )
2023-04-18 10:47:37 -07:00
0e3aafe6c0
HADOOP-18399. S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator ( #5054 )
...
Contributed by Viraj Jasani
2023-04-18 16:37:48 +01:00
405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up ( #5558 )
...
Post-release updates of the branches
* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
2023-04-18 10:12:07 +01:00
2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 ( #5536 )
2023-04-13 09:07:42 -07:00
3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… ( #5495 )
2023-04-10 08:56:00 -07:00
b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics ( #5497 )
2023-03-29 10:43:13 -07:00
ee01c64c6c
HADOOP-18676. jettison dependency override in hadoop-common lib ( #5513 )
2023-03-27 09:59:02 +02:00
b82bcbd8ad
Revert "HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib ( #5507 )"
...
This reverts commit 72b0122706
2023-03-25 12:04:28 +05:30
72b0122706
HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib ( #5507 )
...
* HADOOP-18587. Fixing jettison vulnerability of hadoop-common lib
* no need for excluding, let it come
Change-Id: Ia6e4ad351158dd4b0510dec34bbde531a60e7654
2023-03-24 16:31:45 +01:00
e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. ( #5477 ). Contributed by Ayush Saxena.
...
Reviewed-by: Steve Loughran <stevel@apache.org
2023-03-23 08:30:08 +05:30
67e02a92e0
HADOOP-18666. A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server ( #5480 )
2023-03-22 10:54:41 +09:00
9a8287c36f
HADOOP-18669. Remove Log4Json Layout ( #5493 )
2023-03-21 10:07:06 +08:00
405bfa2800
HADOOP-18654. Remove unused custom appender TaskLogAppender ( #5457 )
2023-03-16 00:45:37 +08:00
aff840c59c
HADOOP-18653. LogLevel servlet to determine log impl before using setLevel ( #5456 )
...
The log level can only be set on Log4J log implementations;
probes are used to downgrade to a warning when other
logging back ends are used
Contributed by Viraj Jasani
2023-03-13 12:30:12 +00:00
09469bf47d
HADOOP-18661. Fix bin/hadoop usage script terminology. ( #5473 )
...
Followup to HADOOP-13209: s/slaves/r/workers in
the usage message you get when you type "bin/hadoop"
Contributed by Steve Loughran
2023-03-13 12:24:36 +00:00
28d2753d2f
HADOOP-18645. Provide keytab file key name with ServiceStateException ( #5433 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2023-03-01 09:34:12 +08:00
0ca5686034
HDFS-16917 Add transfer rate quantile metrics for DataNode reads ( #5397 )
...
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
2023-02-27 18:26:32 +00:00
4cc33e5e37
HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. ( #5346 )
2023-02-22 21:58:44 +00:00
fb31393b65
HADOOP-18535. Implement token storage solution based on MySQL
...
Fixes #1240
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2023-02-22 10:38:50 -08:00
11a220c6e7
HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion ( #5412 )
...
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.
Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.
Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.
Contributed by Steve Loughran
2023-02-22 11:48:12 +00:00
02fd87a4d8
HADOOP-18627. Add stronger wording in 'secure mode' introduction ( #5406 )
...
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.
Contributed by Arnout Engelen
2023-02-17 16:30:41 +00:00
7e19bc31b6
HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers ( #4215 )
2023-02-16 18:13:25 +00:00
c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. ( #5244 )
2023-02-15 14:38:41 -08:00
f4f2793f3b
HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads ( #5274 )
...
Contributed by Ankit Saurabh
2023-02-15 18:28:42 +00:00
Steve Loughran
Viraj Jasani
Ayush Saxena
Szilard Nemeth
Patrick GRANDJEAN
LiuGuH
WangYuanben
slfan1989
Gautham B A
Tak Lon (Stephen) Wu
zhangshuyan
Szilard Nemeth
Pralabh Kumar
Sebastian Baunsgaard
cxzl25
Doroszlai, Attila
Christos Bisias
Nikita Eshkeev
rdingankar
Melissa You
Andras Katona
Yubi Lee
Simbarashe Dzinamarira
hchaverr
Arnout Engelen
Bryan Beaudreault
Ankit Saurabh