75220b7738
fixes issues after rebase
2023-05-17 13:35:28 +01:00
1f137d3283
HADOOP-18565. Complete outstanding items for the AWS SDK V2 upgrade. ( #5421 )
...
Changes include
* use bundled transfer manager
* adds transfer listener to upload
* adds support for custom signers
* don't set default endpoint
* removes v1 sdk bundle, only use core package
* implements region caching
+ many more
Note: spotbugs is warning about inconsistent
synchronization in accessing a new s3a FS field.
This will be fixed in a follow-up patch.
Contributed by Ahmar Suhail
2023-05-17 10:20:38 +01:00
fb6d12a9a5
adds in unit check for bytes
2023-05-17 10:20:38 +01:00
ce02671de4
update GET range referrer header logic to V2
2023-05-17 10:20:35 +01:00
644a32f9d2
updates new providers in TestS3AAWSCredentialsProvider to V2
2023-05-17 10:18:09 +01:00
0338fd89f6
Add unit tests for SelectEventStreamPublisher
2023-05-17 10:18:09 +01:00
38dff867b9
Add unit tests for BlockingEnumeration
2023-05-17 10:18:09 +01:00
0c2fd28252
Move TransferManager initialization to ClientFactory
2023-05-17 10:18:09 +01:00
1ab7e6827d
Reinstate old constants
2023-05-17 10:18:09 +01:00
aa1cf19ffa
Move MultiObjectDeleteException to impl
2023-05-17 10:18:08 +01:00
68510c8ab1
HADOOP-18073. Address review comments. ( #31 )
...
addresses review comments + yetus errors
Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk>
2023-05-17 10:18:05 +01:00
f2bf08cae1
HADOOP-18073. Upgrade AWS SDK to v2 in S3A [work in progress]
...
See aws_sdk_v2_changelog.md for details.
Co-authored-by: Ahmar Suhail <ahmarsu@amazon.co.uk>
Co-authored-by: Alessandro Passaro <alexpax@amazon.co.uk>
2023-05-17 10:17:16 +01:00
a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport ( #5662 )
...
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()
This brings trunk into sync with the branch-3.3 changes
2023-05-16 18:16:24 +01:00
bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops ( #5621 )
...
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter
Contributed by Viraj Jasani
2023-05-16 10:40:16 +01:00
11af08d67a
YARN-11489. Fix memory leak of DelegationTokenRenewer futures in DelegationTokenRenewerPoolTracker. ( #5629 ). Contributed by Chun Chen.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-14 21:38:04 +08:00
251439d769
HDFS-16985. Fix data missing issue when delete local block file. ( #5564 ). Contributed by Chengwei Wang.
...
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-14 21:33:38 +08:00
e0938b4c2a
YARN-11495. Fix typos in hadoop-yarn-server-web-proxy. ( #5652 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-13 11:41:38 +05:30
2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). ( #5625 )
2023-05-12 18:12:05 -07:00
5d0cc455f5
HDFS-17008. Fix RBF JDK 11 javadoc warnings ( #5648 )
2023-05-12 17:27:13 -07:00
0c77629849
HDFS-17001. Support getStatus API in WebHDFS ( #5628 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-13 03:49:13 +05:30
ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods ( #5650 )
...
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
2023-05-12 15:22:37 +01:00
0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade ( #5649 )
2023-05-12 14:20:26 +02:00
905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. ( #5520 ). Contributed by WangYuanben.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-12 04:12:02 +05:30
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem ( #5611 )
...
This:
1. Adds optLong, optDouble, mustLong and mustDouble
methods to the FSBuilder interface to let callers explicitly
passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
now only set long values instead, so as to avoid problems
related to overloaded methods resulting in a ".0" being appended
to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
the new methods
4. And the s3a code is resilient to parse errors in is numeric options
-it will downgrade to the default.
This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.
For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.
Contributed by Steve Loughran
2023-05-11 17:57:25 +01:00
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points ( #5554 ). Contributed by Viraj Jasani.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-11 08:45:34 +05:30
5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API ( #5602 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-10 23:10:19 +05:30
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. ( #5631 )
2023-05-10 09:37:47 -07:00
be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist ( #5565 )
2023-05-10 15:16:33 +08:00
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. ( #5363 )
2023-05-09 16:17:23 -07:00
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. ( #5095 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-10 01:42:12 +05:30
03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). ( #5622 ). Contributed by Shuyan Zhang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-09 10:05:28 +08:00
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. ( #5403 )
2023-05-08 10:13:09 -07:00
a80e3dba3b
HADOOP-18734. Create qbt.sh symlink on Windows ( #5626 )
2023-05-08 09:55:15 -07:00
4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. ( #5200 ). Contributed by ZanderXu.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-06 08:50:21 +05:30
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. ( #5616 )
2023-05-05 13:43:40 -07:00
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. ( #5609 )
2023-05-05 13:41:38 -07:00
c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 ( #5618 )
2023-05-05 13:08:58 -07:00
372631c566
HADOOP-18706. Improve S3ABlockOutputStream recovery ( #5563 )
...
Contributed by Chris Bevard
2023-05-05 11:57:42 +01:00
c7699d3dcd
YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta
2023-05-04 22:16:18 -04:00
ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity ( #5615 )
2023-05-04 09:45:40 -07:00
27776ac45e
HADOOP-18727. Fix WriteOperations.listMultipartUploads function description ( #5613 )
...
Contributed by Dongjoon Hyun
2023-05-04 13:03:48 +01:00
bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum ( #5614 )
2023-05-04 12:27:25 +02:00
0d06fd77de
HADOOP-18134. Setup Jenkins nightly CI for Windows 10 ( #5062 )
...
This PR gets Yetus to run on Windows 10
against the Hadoop codebase. It introduces
the following changes to allow us to setup
the nightly CI on Jenkins for Hadoop on
Windows 10.
* Hadoop personality changes for Yetus.
Additional arguments have been passed,
which are necessary to build and test
Hadoop on Windows 10.
* Docker image for building Hadoop on
Windows 10.
Installs the necessary tools that are
needed to run Yetus.
* dev-support/jenkins.sh file.
Passing of some flags are handled here
which are needed for the nightly CI.
2023-05-03 22:44:54 +05:30
476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. ( #5587 )
2023-05-03 09:21:56 -07:00
c1d10f3872
YARN-9049. Add application submit data to state store. ( #5606 )
2023-05-03 09:19:54 -07:00
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common ( #5553 )
...
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them. Applications which use the leasing APIs should migrate to the new
interface where possible.
Contributed by Stephen Wu
2023-05-03 11:05:55 +01:00
668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. ( #5596 )
2023-05-02 13:46:06 -07:00
fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. ( #5612 ). Contributed by Shuyan Zhang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-03 00:09:36 +08:00
bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile ( #5580 )
...
Contributed by Viraj Jasani
2023-05-02 15:21:46 +01:00
73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase ( #5455 )
2023-05-02 15:52:57 +02:00
Ahmar Suhail
ahmarsuhail
Alessandro Passaro
Steve Loughran
Viraj Jasani
Chun Chen
smarthan
slfan1989
zhtttylz
susheel-gupta
WangYuanben
cxzl25
zhangshuyan
Gautham B A
ZanderXu
Chris
Szilard Nemeth
Dongjoon Hyun
Peter Szucs
Tak Lon (Stephen) Wu
Szilard Nemeth