5ddaf2e133
YARN-11493. [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router. ( #5651 )
2023-05-17 11:09:10 -07:00
8e17385141
HDFS-17009. RBF: state store putAll should also return failed records ( #5664 )
2023-05-17 09:33:34 -07:00
482897a0f6
Fix typos in HDFS documents. ( #5665 )
2023-05-17 09:28:01 -07:00
a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport ( #5662 )
...
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()
This brings trunk into sync with the branch-3.3 changes
2023-05-16 18:16:24 +01:00
bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops ( #5621 )
...
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter
Contributed by Viraj Jasani
2023-05-16 10:40:16 +01:00
11af08d67a
YARN-11489. Fix memory leak of DelegationTokenRenewer futures in DelegationTokenRenewerPoolTracker. ( #5629 ). Contributed by Chun Chen.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-14 21:38:04 +08:00
251439d769
HDFS-16985. Fix data missing issue when delete local block file. ( #5564 ). Contributed by Chengwei Wang.
...
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-14 21:33:38 +08:00
e0938b4c2a
YARN-11495. Fix typos in hadoop-yarn-server-web-proxy. ( #5652 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-13 11:41:38 +05:30
2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). ( #5625 )
2023-05-12 18:12:05 -07:00
5d0cc455f5
HDFS-17008. Fix RBF JDK 11 javadoc warnings ( #5648 )
2023-05-12 17:27:13 -07:00
0c77629849
HDFS-17001. Support getStatus API in WebHDFS ( #5628 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-13 03:49:13 +05:30
ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods ( #5650 )
...
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
2023-05-12 15:22:37 +01:00
0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade ( #5649 )
2023-05-12 14:20:26 +02:00
905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. ( #5520 ). Contributed by WangYuanben.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-12 04:12:02 +05:30
e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem ( #5611 )
...
This:
1. Adds optLong, optDouble, mustLong and mustDouble
methods to the FSBuilder interface to let callers explicitly
passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
now only set long values instead, so as to avoid problems
related to overloaded methods resulting in a ".0" being appended
to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
the new methods
4. And the s3a code is resilient to parse errors in is numeric options
-it will downgrade to the default.
This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.
For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.
Contributed by Steve Loughran
2023-05-11 17:57:25 +01:00
fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points ( #5554 ). Contributed by Viraj Jasani.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-11 08:45:34 +05:30
5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API ( #5602 ). Contributed by Hualong Zhang.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-10 23:10:19 +05:30
690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. ( #5631 )
2023-05-10 09:37:47 -07:00
be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist ( #5565 )
2023-05-10 15:16:33 +08:00
d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. ( #5363 )
2023-05-09 16:17:23 -07:00
a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. ( #5095 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-10 01:42:12 +05:30
03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). ( #5622 ). Contributed by Shuyan Zhang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-09 10:05:28 +08:00
bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. ( #5403 )
2023-05-08 10:13:09 -07:00
a80e3dba3b
HADOOP-18734. Create qbt.sh symlink on Windows ( #5626 )
2023-05-08 09:55:15 -07:00
4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. ( #5200 ). Contributed by ZanderXu.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-06 08:50:21 +05:30
cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. ( #5616 )
2023-05-05 13:43:40 -07:00
eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. ( #5609 )
2023-05-05 13:41:38 -07:00
c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 ( #5618 )
2023-05-05 13:08:58 -07:00
372631c566
HADOOP-18706. Improve S3ABlockOutputStream recovery ( #5563 )
...
Contributed by Chris Bevard
2023-05-05 11:57:42 +01:00
c7699d3dcd
YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta
2023-05-04 22:16:18 -04:00
ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity ( #5615 )
2023-05-04 09:45:40 -07:00
27776ac45e
HADOOP-18727. Fix WriteOperations.listMultipartUploads function description ( #5613 )
...
Contributed by Dongjoon Hyun
2023-05-04 13:03:48 +01:00
bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum ( #5614 )
2023-05-04 12:27:25 +02:00
0d06fd77de
HADOOP-18134. Setup Jenkins nightly CI for Windows 10 ( #5062 )
...
This PR gets Yetus to run on Windows 10
against the Hadoop codebase. It introduces
the following changes to allow us to setup
the nightly CI on Jenkins for Hadoop on
Windows 10.
* Hadoop personality changes for Yetus.
Additional arguments have been passed,
which are necessary to build and test
Hadoop on Windows 10.
* Docker image for building Hadoop on
Windows 10.
Installs the necessary tools that are
needed to run Yetus.
* dev-support/jenkins.sh file.
Passing of some flags are handled here
which are needed for the nightly CI.
2023-05-03 22:44:54 +05:30
476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. ( #5587 )
2023-05-03 09:21:56 -07:00
c1d10f3872
YARN-9049. Add application submit data to state store. ( #5606 )
2023-05-03 09:19:54 -07:00
0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common ( #5553 )
...
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them. Applications which use the leasing APIs should migrate to the new
interface where possible.
Contributed by Stephen Wu
2023-05-03 11:05:55 +01:00
668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. ( #5596 )
2023-05-02 13:46:06 -07:00
fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. ( #5612 ). Contributed by Shuyan Zhang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2023-05-03 00:09:36 +08:00
bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile ( #5580 )
...
Contributed by Viraj Jasani
2023-05-02 15:21:46 +01:00
73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase ( #5455 )
2023-05-02 15:52:57 +02:00
87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. ( #5598 )
2023-05-01 16:01:50 -07:00
5147106b59
HADOOP-18725. Avoid cross-platform build for irrelevant Dockerfile changes ( #5610 )
2023-05-01 09:35:50 -07:00
1079890ae3
HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus ( #4665 ). Contributed by Jiale Qi.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-05-01 21:22:26 +05:30
d75c6d9d57
HADOOP-18715. Add debug log for getting details of tokenKindMap ( #5608 ). Contributed by Pralabh Kumar.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-29 17:28:49 +05:30
1a2cd965a7
HDFS-16897. Fix abundant Broken pipe exception in BlockSender ( #5329 ). Contributed by fanluo.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-29 17:24:56 +05:30
70c0aa342e
YARN-11482. Fix bug of DRF comparision DominantResourceFairnessComparator2 in fair scheduler. ( #5607 ). Contributed by Xiaoqiao He.
...
Reviewed-by: Shilun Fan <slfan1989@apache.org>
2023-04-29 11:18:42 +08:00
5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. ( #5570 )
...
Co-authored-by: slfan1989 <louj1988@@>
2023-04-28 14:11:13 -07:00
0e63152218
HDFS-16995. Remove unused parameters at NameNodeHttpServer#initWebHdfs ( #5601 ). Contributed by Zhaohui Wang.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2023-04-28 05:25:01 +05:30
d9576bb9ee
HDFS-16972. Delete a snapshot may deleteCurrentFile. ( #5532 )
2023-04-27 09:17:47 -07:00
slfan1989
Viraj Jasani
liang3zy22
Steve Loughran
Chun Chen
smarthan
zhtttylz
susheel-gupta
WangYuanben
cxzl25
zhangshuyan
Gautham B A
ZanderXu
Chris
Szilard Nemeth
Dongjoon Hyun
Peter Szucs
Tak Lon (Stephen) Wu
Szilard Nemeth
SevenAddSix
Pralabh Kumar
fanluoo
Hexiaoqiao
wangzhaohui
Tsz-Wo Nicholas Sze