cf43a7eaae
HADOOP-17397. ABFS: SAS Test updates for version and permission update ( #2492 )
...
Contributed by Sneha Vijayarajan.
2020-11-26 10:21:01 +00:00
009ce4f02a
HADOOP-17396. ABFS: testRenameFileOverExistingFile fails ( #2491 )
...
Contributed by Sneha Vijayarajan.
2020-11-26 10:11:25 +00:00
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:32:10 +09:00
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs ( #2488 )" because JIRA ID is missing.
...
This reverts commit ce5b3d7975
2020-11-26 11:30:53 +09:00
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:27:30 +09:00
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check ( #2472 )
2020-11-25 11:01:04 -08:00
ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. ( #2396 )
...
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".
This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.
The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.
If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.
Contributed by Steve Loughran.
2020-11-25 14:31:02 +00:00
3193d8c793
HADOOP-17311. ABFS: Logs should redact SAS signature ( #2422 )
...
Contributed by bilaharith.
2020-11-25 14:22:10 +00:00
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair ( #2485 )
...
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-11-25 13:00:09 +09:00
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions ( #2461 )
2020-11-24 11:09:30 -08:00
f813f14b5b
MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
2020-11-24 17:02:35 +01:00
5fee95076b
HADOOP-17323. S3A getFileStatus("/") to skip IO ( #2479 )
...
Contributed by Mukund Thakur.
2020-11-24 11:06:56 +00:00
c4ba0ab7df
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi.
2020-11-24 15:22:07 +09:00
9b4faf2b51
HADOOP-17332. S3A MarkerTool -min and -max are inverted. ( #2425 )
...
This patch
* fixes the inversion
* adds a precondition check
* if the commands are supplied inverted, swaps them with a warning.
This is to stop breaking any tests written to cope with the existing
behavior.
Contributed by Steve Loughran
2020-11-23 20:49:42 +00:00
07b7d07388
HADOOP-17325. WASB Test Failures
...
Contributed by Ayush Saxena and Steve Loughran
Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
2020-11-23 17:22:13 +00:00
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. ( #2469 )
...
Contributed by Zhihua Deng.
2020-11-23 15:40:22 +00:00
fb79be932c
HADOOP-17343. Upgrade AWS SDK to 1.11.901 ( #2468 )
...
Contributed by Steve Loughran.
2020-11-23 14:08:12 +00:00
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2020-11-23 11:26:52 +08:00
d73029463c
HDFS-15690. Add lz4-java as test dependency ( #2481 )
2020-11-21 17:49:56 -08:00
740399a602
HADOOP-17390. Skip license check on lz4 code files ( #2478 )
...
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license
Contributed by Zhihua Deng.
2020-11-20 14:12:20 +00:00
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
2020-11-20 18:03:45 +05:30
fb92aa4012
MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann
2020-11-20 13:13:51 +01:00
f3c629c27e
HADOOP-17388. AbstractS3ATokenIdentifier to issue date in UTC. ( #2477 )
...
Followup to HADOOP-17379.
Contributed by Jungtaek Lim.
2020-11-20 10:38:42 +00:00
8fa699b53f
HDFS-15635. ViewFileSystemOverloadScheme support specifying mount table loader imp through conf ( #2389 ). Contributed by Junfan Zhang.
2020-11-19 20:21:16 -08:00
07050339e0
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize ( #2449 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-11-19 14:37:14 -06:00
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-11-18 12:03:25 -08:00
0d3155a687
YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak
2020-11-18 17:02:47 +01:00
5ff70a59c4
YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko
2020-11-18 16:51:42 +01:00
ce7827c82a
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. ( #2399 )
...
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs
Contributed by Steve Loughran.
Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
2020-11-18 13:34:51 +00:00
e3c08f285a
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. ( #2310 )
...
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.
In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.
Contributed by Steve Loughran.
Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
2020-11-18 12:18:11 +00:00
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. ( #2467 )
2020-11-18 16:11:09 +09:00
2045a9df13
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. ( #2463 )
2020-11-18 14:15:11 +09:00
a7b923c80c
HADOOP-17379. AbstractS3ATokenIdentifier to set issue date == now. ( #2466 )
...
Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned.
Contributed by Jungtaek Lim.
2020-11-17 14:43:29 +00:00
b57f04cd5b
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. ( #2465 )
2020-11-17 10:56:56 +09:00
0b2510ee1f
YARN-10485. TimelineConnector swallows InterruptedException ( #2450 ). Contributed by Ahmed Hussein
2020-11-16 14:55:40 -06:00
dd85a90da6
HADOOP-17376. ITestS3AContractRename failing against stricter tests. ( #2462 )
...
Contributed by Attila Doroszlai.
2020-11-16 11:24:00 +00:00
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file ( #2444 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-11-13 14:22:35 -06:00
f56cd88d7d
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
2020-11-13 11:02:26 +09:00
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… ( #2255 )
...
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
2020-11-12 14:57:26 -08:00
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals ( #2431 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-12 13:13:12 -06:00
fc961b63d1
Revert "HADOOP-17269. [JDK 11] Upgrade SpotBugs to 4.1.3 to fix false-positive warnings. ( #2374 )" ( #2460 )
...
This reverts commit 2de7240d4f
2020-11-12 15:52:36 +09:00
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient ( #2447 )
...
Contributed by Attila Doroszlai.
2020-11-11 21:20:09 +00:00
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations ( #2436 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-11 12:39:03 -08:00
188ebb5a9b
HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade ( #2458 )
2020-11-11 11:37:23 -06:00
567600fd80
YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.
2020-11-11 17:10:15 +01:00
cd0490e8c6
Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak."
...
This reverts commit b0ab222a6c
2020-11-11 17:06:12 +01:00
b0ab222a6c
YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.
2020-11-11 17:03:16 +01:00
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. ( #2138 )
2020-11-10 16:34:17 -08:00
375900049c
HDFS-15608.Reset the DistCp#CLEANUP variable definition. ( #2351 ). Contributed by JiangHua Zhu.
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
2020-11-10 13:02:29 -08:00
fbd2220167
HDFS-15607. Addendum: Create trash dir when allowing snapshottable dir ( #2448 )
2020-11-10 12:13:15 -08:00
Sneha Vijayarajan
Akira Ajisaka
Siyao Meng
Steve Loughran
bilaharith
Ankit Kumar
Ahmed Hussein
Szilard Nemeth
Mukund Thakur
dengzh
He Xiaoqiao
Liang-Chi Hsieh
Peter Bacsko
Jungtaek Lim
zhang_jf
Masatake Iwasaki
Doroszlai, Attila
Takanobu Asanuma
Issac Buenrostro
Chao Sun
jianghuazhu