Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,688 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26688 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze 350dcaf616
HDFS-17010. Add a subtree test to TestSnapshotDiffReport. (#5656) 2023-05-18 15:53:26 +08:00
wangzhaohui 03163f9de2
HDFS-17011. Fix the metric of "HttpPort" at DataNodeInfo (#5657). Contributed by Zhaohui Wang. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-18 12:12:11 +05:30
slfan1989 5ddaf2e133
YARN-11493. [Federation] ConfiguredRMFailoverProxyProvider Supports Randomly Select an Router. (#5651) 2023-05-17 11:09:10 -07:00
Viraj Jasani 8e17385141
HDFS-17009. RBF: state store putAll should also return failed records (#5664) 2023-05-17 09:33:34 -07:00
liang3zy22 482897a0f6
Fix typos in HDFS documents. (#5665) 2023-05-17 09:28:01 -07:00
Steve Loughran a90c722143
HADOOP-18724. [FOLLOW-UP] cherrypick changes from branch-3.3 backport (#5662) 
* move FileContext.copy() onto optLong()
* move FileUtil onto optLong()

This brings trunk into sync with the branch-3.3 changes
 2023-05-16 18:16:24 +01:00
Viraj Jasani bef40e9427
HADOOP-18688. S3A audit header to include count of items in delete ops (#5621) 
The auditor-generated http referrer URL now includes the count of keys
to delete in the "ks" query parameter

Contributed by Viraj Jasani
 2023-05-16 10:40:16 +01:00
Chun Chen 11af08d67a
YARN-11489. Fix memory leak of DelegationTokenRenewer futures in DelegationTokenRenewerPoolTracker. (#5629). Contributed by Chun Chen. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:38:04 +08:00
smarthan 251439d769
HDFS-16985. Fix data missing issue when delete local block file. (#5564). Contributed by Chengwei Wang. 
Reviewed-by: Shuyan Zhang <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-14 21:33:38 +08:00
slfan1989 e0938b4c2a
YARN-11495. Fix typos in hadoop-yarn-server-web-proxy. (#5652). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 11:41:38 +05:30
slfan1989 2f87f716fa
YARN-3660. BackPort [GPG] Federation Global Policy Generator (service hook only). (#5625) 2023-05-12 18:12:05 -07:00
Viraj Jasani 5d0cc455f5
HDFS-17008. Fix RBF JDK 11 javadoc warnings (#5648) 2023-05-12 17:27:13 -07:00
zhtttylz 0c77629849
HDFS-17001. Support getStatus API in WebHDFS (#5628). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-13 03:49:13 +05:30
Steve Loughran ad1e3a0f5b
HADOOP-18724. (followup) remove deprecation on optLong/optDouble methods (#5650) 
Somehow @Deprecated crept in to the declaration of the
new FSBuilder optLong/optDouble methods.
 2023-05-12 15:22:37 +01:00
susheel-gupta 0f3406ac34
YARN-11312: [UI2] Refresh buttons don't work after EmberJS upgrade (#5649) 2023-05-12 14:20:26 +02:00
WangYuanben 905bfa84a8
HDFS-16965. Add switch to decide whether to enable native codec. (#5520). Contributed by WangYuanben. 
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-12 04:12:02 +05:30
Steve Loughran e76c09ac3b
HADOOP-18724. Open file fails with NumberFormatException for S3AFileSystem (#5611) 
This:

1. Adds optLong, optDouble, mustLong and mustDouble
   methods to the FSBuilder interface to let callers explicitly
   passin long and double arguments.
2. The opt() and must() builder calls which take float/double values
   now only set long values instead, so as to avoid problems
   related to overloaded methods resulting in a ".0" being appended
   to a long value.
3. All of the relevant opt/must calls in the hadoop codebase move to
   the new methods
4. And the s3a code is resilient to parse errors in is numeric options
   -it will downgrade to the default.

This is nominally incompatible, but the floating-point builder methods
were never used: nothing currently expects floating point numbers.

For anyone who wants to safely set numeric builder options across all compatible
releases, convert the number to a string and then use the opt(String, String)
and must(String, String) methods.

Contributed by Steve Loughran
 2023-05-11 17:57:25 +01:00
Viraj Jasani fe61d8f073
HDFS-16978. RBF: Admin command to support bulk add of mount points (#5554). Contributed by Viraj Jasani. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-11 08:45:34 +05:30
zhtttylz 5084e881ef
HDFS-16990. HttpFS Add Support getFileLinkStatus API (#5602). Contributed by Hualong Zhang. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 23:10:19 +05:30
slfan1989 690db3c34b
YARN-11479. [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData. (#5631) 2023-05-10 09:37:47 -07:00
cxzl25 be50d221f5
YARN-11467. RM failover may fail when the nodes.exclude-path file does not exist (#5565) 2023-05-10 15:16:33 +08:00
slfan1989 d95b5c679d
YARN-11424. [Federation] Router Supports DeregisterSubCluster. (#5363) 2023-05-09 16:17:23 -07:00
slfan1989 a2dda0ce03
HADOOP-18359. Update commons-cli from 1.2 to 1.5. (#5095). Contributed by Shilun Fan. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-10 01:42:12 +05:30
zhangshuyan 03bf8f982a
HDFS-16999. Fix wrong use of processFirstBlockReport(). (#5622). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-09 10:05:28 +08:00
slfan1989 bdeca45294
YARN-11340. [Federation] Improve SQLFederationStateStore DataSource Config. (#5403) 2023-05-08 10:13:09 -07:00
Gautham B A a80e3dba3b
HADOOP-18734. Create qbt.sh symlink on Windows (#5626) 2023-05-08 09:55:15 -07:00
ZanderXu 4ee92efb73
HDFS-16865. The source path is always / after RBF proxied the complete, addBlock and getAdditionalDatanode RPC. (#5200). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-06 08:50:21 +05:30
slfan1989 cda9863d54
YARN-11477. [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData. (#5616) 2023-05-05 13:43:40 -07:00
slfan1989 eab4c33d09
YARN-11470. FederationStateStoreFacade Cache Support Guava Cache. (#5609) 2023-05-05 13:41:38 -07:00
Gautham B A c974710d8e
HADOOP-18729. Fix mvnsite on Windows 10 (#5618) 2023-05-05 13:08:58 -07:00
Chris 372631c566
HADOOP-18706. Improve S3ABlockOutputStream recovery (#5563) 
Contributed by Chris Bevard
 2023-05-05 11:57:42 +01:00
Szilard Nemeth c7699d3dcd YARN-11079. Make an AbstractParentQueue to store common ParentQueue and ManagedParentQueue functionality. Contributed by Susheel Gupta 2023-05-04 22:16:18 -04:00
Viraj Jasani ceb8878d4f
HDFS-16998. RBF: Add ops metrics for getSlowDatanodeReport in RouterClientActivity (#5615) 2023-05-04 09:45:40 -07:00
Dongjoon Hyun 27776ac45e
HADOOP-18727. Fix WriteOperations.listMultipartUploads function description (#5613) 
Contributed by Dongjoon Hyun
 2023-05-04 13:03:48 +01:00
Peter Szucs bd607951c0
YARN-11463. Node Labels root directory creation doesn't have a retry logic - addendum (#5614) 2023-05-04 12:27:25 +02:00
Gautham B A 0d06fd77de
HADOOP-18134. Setup Jenkins nightly CI for Windows 10 (#5062) 
This PR gets Yetus to run on Windows 10
against the Hadoop codebase. It introduces
the following changes to allow us to setup
the nightly CI on Jenkins for Hadoop on
Windows 10.
* Hadoop personality changes for Yetus.
  Additional arguments have been passed,
  which are necessary to build and test
  Hadoop on Windows 10.
* Docker image for building Hadoop on
  Windows 10.
  Installs the necessary tools that are
  needed to run Yetus.
* dev-support/jenkins.sh file.
  Passing of some flags are handled here
  which are needed for the nightly CI.
 2023-05-03 22:44:54 +05:30
slfan1989 476f60a806
YARN-10144. Federation: Add missing FederationClientInterceptor APIs. (#5587) 2023-05-03 09:21:56 -07:00
slfan1989 c1d10f3872
YARN-9049. Add application submit data to state store. (#5606) 2023-05-03 09:19:54 -07:00
Tak Lon (Stephen) Wu 0e46388474
HADOOP-18671. Add recoverLease(), setSafeMode(), isFileClosed() as interfaces to hadoop-common (#5553) 
The HDFS lease APIs have been replicated as interfaces in hadoop-common so other filesystems can
also implement them.  Applications which use the leasing APIs should migrate to the new
interface where possible.

Contributed by Stephen Wu
 2023-05-03 11:05:55 +01:00
slfan1989 668c0a0930
YARN-11379. [Federation] Support mapAttributesToNodes, getGroupsForUser API's for Federation. (#5596) 2023-05-02 13:46:06 -07:00
zhangshuyan fddc9769a5
HADOOP-18726. Set the locale to avoid printing useless logs. (#5612). Contributed by Shuyan Zhang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-05-03 00:09:36 +08:00
Viraj Jasani bfcf5dd03b
HADOOP-18697. S3A prefetch: failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile (#5580) 
Contributed by Viraj Jasani
 2023-05-02 15:21:46 +01:00
Szilard Nemeth 73ca64a3ba
YARN-11450. Improvements for TestYarnConfigurationFields and TestConfigurationFieldsBase (#5455) 2023-05-02 15:52:57 +02:00
slfan1989 87e17b2713
YARN-11437. [hotfix][Federation] SQLFederationStateStore Support Version. (#5598) 2023-05-01 16:01:50 -07:00
Gautham B A 5147106b59
HADOOP-18725. Avoid cross-platform build for irrelevant Dockerfile changes (#5610) 2023-05-01 09:35:50 -07:00
SevenAddSix 1079890ae3
HDFS-16707. RBF: Expose RouterRpcFairnessPolicyController related request record metrics for each nameservice to Prometheus (#4665). Contributed by Jiale Qi. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-05-01 21:22:26 +05:30
Pralabh Kumar d75c6d9d57
HADOOP-18715. Add debug log for getting details of tokenKindMap (#5608). Contributed by Pralabh Kumar. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:28:49 +05:30
fanluoo 1a2cd965a7
HDFS-16897. Fix abundant Broken pipe exception in BlockSender (#5329). Contributed by fanluo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-29 17:24:56 +05:30
Hexiaoqiao 70c0aa342e
YARN-11482. Fix bug of DRF comparision DominantResourceFairnessComparator2 in fair scheduler. (#5607). Contributed by Xiaoqiao He. 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
 2023-04-29 11:18:42 +08:00
slfan1989 5ed7e912dc
YARN-11469. Refactor FederationStateStoreFacade Cache Code. (#5570) 
Co-authored-by: slfan1989 <louj1988@@>
 2023-04-28 14:11:13 -07:00