1e59bf7394
HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure ( #2473 ).
...
Contributed by Steve Loughran
The addition of deprecated S3A configuration options in HADOOP-17318
triggered a reload of default (xml resource) configurations, which breaks
tests which fail if there's a per-bucket setting inconsistent with test
setup.
Creating an S3AFS instance before creating the Configuration() instance
for test runs gets that reload out the way before test setup takes
place.
Along with the fix, extra changes in the failing test suite to fail
fast when marker policy isn't as expected, and to log FS state better.
Rather than create and discard an instance, add a new static method
to S3AFS and invoke it in test setup. This forces the load
Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
2020-11-26 17:28:01 +00:00
1eeb9d9d67
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. ( #2399 )
...
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs
Contributed by Steve Loughran.
Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
2020-11-26 17:22:56 +00:00
c48c774d6c
HADOOP-17397. ABFS: SAS Test updates for version and permission update ( #2492 )
...
Contributed by Sneha Vijayarajan.
Change-Id: I89c1061b1efb1e3bef019dd22f221d03bf015929
2020-11-26 10:21:37 +00:00
39fa2c93c4
HADOOP-17396. ABFS: testRenameFileOverExistingFile fails ( #2491 )
...
Contributed by Sneha Vijayarajan.
Change-Id: I57a866b95ff18229caee8a6028874074a29cb5bd
2020-11-26 10:13:55 +00:00
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287
2020-11-26 11:34:39 +09:00
1ef34d0819
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. ( #2396 )
...
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".
This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.
The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.
If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.
Contributed by Steve Loughran.
Change-Id: I57161b026f28349e339dc8b9d74f6567a62ce196
2020-11-25 14:55:29 +00:00
b8454a4b10
HADOOP-17311. ABFS: Logs should redact SAS signature ( #2422 )
...
Contributed by bilaharith.
Change-Id: Iff0ed4303ac5ce41b62bfda8150ee983dafa40be
2020-11-25 14:33:29 +00:00
8ed565382f
MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan.
2020-11-25 11:36:58 +01:00
9dd74141a6
HADOOP-17323. S3A getFileStatus("/") to skip IO ( #2479 )
...
Contributed by Mukund Thakur.
Change-Id: I1709ad72b829999b6dd324f0755b51bc38918d30
2020-11-24 11:34:19 +00:00
bb4c6db9b3
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi.
...
(cherry picked from commit c4ba0ab7df
2020-11-24 15:22:51 +09:00
38cc47d308
HADOOP-17332. S3A MarkerTool -min and -max are inverted. ( #2425 )
...
This patch
* fixes the inversion
* adds a precondition check
* if the commands are supplied inverted, swaps them with a warning.
This is to stop breaking any tests written to cope with the existing
behavior.
Contributed by Steve Loughran
Change-Id: I15c40863f0db0675c7d60db477cb3bf1693cae49
2020-11-23 21:49:33 +00:00
8459f1d955
HADOOP-17346. Fair call queue is defeated by abusive service principals. Contributed by Ahmed Hussein (ahussein).
2020-11-23 20:37:33 +00:00
7ca539bc1b
HADOOP-17325. WASB Test Failures
...
Contributed by Ayush Saxena and Steve Loughran
Change-Id: I4bb76815bc1d11d1804dc67bafde68b6a995b974
2020-11-23 17:25:58 +00:00
abc87aef18
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. ( #2469 )
...
Contributed by Zhihua Deng.
Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
2020-11-23 16:33:41 +00:00
e4bc64cce0
HADOOP-17343. Upgrade AWS SDK to 1.11.901 ( #2468 )
...
Contributed by Steve Loughran.
2020-11-23 14:09:14 +00:00
ced08fd87f
MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann
2020-11-20 13:14:49 +01:00
401cadbac5
HADOOP-17388. AbstractS3ATokenIdentifier to issue date in UTC. ( #2477 )
...
Followup to HADOOP-17379.
Contributed by Jungtaek Lim.
Change-Id: I7b2fce36028d297c1e095499691a08caba92d9fd
2020-11-20 10:56:57 +00:00
e24a6b550e
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize ( #2449 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-11-19 21:26:47 +00:00
4687c25389
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. ( #2310 )
...
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.
In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.
Contributed by Steve Loughran.
Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
2020-11-18 12:30:43 +00:00
4bb9d593da
HADOOP-17261. s3a rename() needs s3:deleteObjectVersion permission ( #2303 )
...
Contributed by Steve Loughran.
Change-Id: I8e89a402a24bd9fb958e0fa93d1a28191093851d
2020-11-18 12:20:12 +00:00
71c106b412
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. ( #2463 )
...
(cherry picked from commit 2045a9df13
2020-11-18 14:16:45 +09:00
22039a14ff
HADOOP-17379. AbstractS3ATokenIdentifier to set issue date == now. ( #2466 )
...
Unless you explicitly set it, the issue date of a delegation token identifier is 0, which confuses spark renewal (SPARK-33440). This patch makes sure that all S3A DT identifiers have the current time as issue date, fixing the problem as far as S3A tokens are concerned.
Contributed by Jungtaek Lim.
Change-Id: Ic80ac7895612a1aa669459c73a78a9c17ecf0c0d
2020-11-17 14:56:58 +00:00
be108159b1
HDFS-15685. [JDK 14] TestConfiguredFailoverProxyProvider#testResolveDomainNameUsingDNS fails. ( #2465 )
...
(cherry picked from commit b57f04cd5b
2020-11-17 11:01:23 +09:00
df4edb99f7
HADOOP-17360. Log the remote address for authentication success ( #2441 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 1ea3f74246
2020-11-16 21:48:37 +00:00
bc634c077b
YARN-10485. TimelineConnector swallows InterruptedException ( #2450 ). Contributed by Ahmed Hussein
...
(cherry picked from commit 0b2510ee1f
2020-11-16 21:09:36 +00:00
bf2ff35a04
HADOOP-17376. ITestS3AContractRename failing against stricter tests. ( #2462 )
...
Contributed by Attila Doroszlai.
Change-Id: Ie15624ec07b1c5e34ca7fde0a72a54431d79e746
2020-11-16 11:26:06 +00:00
75ca0c0f23
HADOOP-17362. reduce RPC calls doing ls on HAR file ( #2444 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit ebe1d1fbf7
2020-11-13 21:14:47 +00:00
d78bf82722
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
...
(cherry picked from commit f56cd88d7d
2020-11-13 11:05:20 +09:00
23fe3bdab3
HADOOP-17358. Improve excessive reloading of Configurations ( #2436 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 71071e5c0f
2020-11-12 10:35:28 -08:00
47131cdf7c
HADOOP-17365. Contract test for renaming over existing file is too lenient ( #2447 )
...
Contributed by Attila Doroszlai.
Change-Id: I21c29256b52449b7fea335704b3afa02e39c6a39
2020-11-11 21:21:11 +00:00
8e4b1cd192
HADOOP-17373. hadoop-client-integration-tests doesn't work when building with skipShade ( #2458 )
2020-11-11 09:49:20 -08:00
64344fb302
HADOOP-17324. Don't relocate org.bouncycastle in shaded client jars ( #2411 )
...
Contributed by Chao Sun.
2020-11-11 09:48:57 -08:00
0712505b59
HADOOP-17096. Fix ZStandardCompressor input buffer offset ( #2104 ). Contributed by Stephen Jung (Stripe).
...
(cherry picked from commit 45434c93e8
2020-11-10 11:41:21 -08:00
9a3f2bb545
YARN-10480. replace href tags with ng-href ( #2426 )
...
(cherry picked from commit 61f8c5767e
2020-11-10 11:00:00 -08:00
12bfd9194a
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
...
(cherry picked from commit 82de07c794
2020-11-10 09:14:12 -08:00
0361837c9e
YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko.
2020-11-10 13:16:52 +01:00
aa3807ed1d
HADOOP-17352. Update PATCH_NAMING_RULE in the personality file. ( #2433 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 34e2abd811
2020-11-10 14:00:08 +09:00
7cb5325dda
HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. ( #2427 ). Contributed by Steve Loughran.
2020-11-07 17:05:23 +05:30
043cca01b1
HDFS-15623. Respect configured values of rpc.engine ( #2403 ) Contributed by Hector Chaverri.
...
(cherry picked from commit 6eacaffeea
2020-11-06 14:31:31 -08:00
1e22929885
YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan)
...
(cherry picked from commit 55339c2bdd
2020-11-05 22:08:51 +00:00
c6fee0a2c8
HADOOP-17342. Creating a token identifier should not do kerberos name
...
resolution. Contributed by Jim Brennan.
(cherry picked from commit af389d9897
2020-11-05 21:56:46 +00:00
41d58d190d
Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 ( #2387 )"
...
This reverts commit e21b81276e
2020-11-05 17:31:39 +00:00
cfa0986d00
Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. ( #2291 )"
...
This reverts commit dd1634ec3b
2020-11-04 16:18:23 -08:00
cb11fd8793
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:44:08 +05:30
e48dd9daea
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit d07dc7afb4
2020-11-03 14:52:05 -08:00
dd1634ec3b
HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. ( #2291 )
...
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7f5caca04c
2020-11-03 11:22:48 -08:00
2473e8b711
YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan).
2020-11-02 17:16:28 +00:00
cdaef111d5
YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne
...
(cherry picked from commit 8ee6bc2518
2020-10-29 19:52:43 +00:00
91a3d298b9
HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He.
...
(cherry picked from commit f17e067d52
2020-10-29 07:20:46 +05:30
f95c0824b0
YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen
...
(cherry picked from commit bab5bf9743
2020-10-28 10:33:41 -07:00
Steve Loughran
Sneha Vijayarajan
Akira Ajisaka
bilaharith
Peter Bacsko
Mukund Thakur
Eric Payne
dengzh
Jungtaek Lim
Jim Brennan
Ahmed Hussein
Doroszlai, Attila
Takanobu Asanuma
Chao Sun
Stephen Jung
Gabriel Medeiros Coelho
Yizhong Zhang
hchaverr
Eric Badger
Wei-Chiu Chuang
Ayush Saxena
Konstantin V Shvachko
Sunil G
Jonathan Hung