Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,557 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26557 Commits

Author SHA1 Message Date
Viraj Jasani 0e3aafe6c0
HADOOP-18399. S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator (#5054) 
Contributed by Viraj Jasani
 2023-04-18 16:37:48 +01:00
Steve Loughran 405ed1dde6
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:07 +01:00
Steve Loughran 6ea10cf41b
HADOOP-18696. ITestS3ABucketExistence arn test failures. (#5557) 
Explicitly sets the fs.s3a.endpoint.region to eu-west-1 so
the ARN-referenced fs creation fails with unknown store
rather than IllegalArgumentException.

Steve Loughran
 2023-04-17 10:18:33 +01:00
yl09099 2c4d6bf33d
YARN-11465. Improved YarnClient Log Format (#5550) 
Co-authored-by: yl09099 <shaq376260428@163.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-17 09:27:52 +08:00
Dongjoon Hyun 0d1b4a3556
HADOOP-18590. Publish SBOM artifacts (#5555). Contributed by Dongjoon Hyun. 2023-04-15 21:35:43 +05:30
slfan1989 0bcdea7912
YARN-11239. Optimize FederationClientInterceptor audit log. (#5127) 2023-04-14 13:09:18 -07:00
zhangshuyan 0185afafea
HDFS-16974. Consider volumes average load of each DataNode when choosing target. (#5541). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-04-14 10:33:30 +08:00
dependabot[bot] f1936d29f1
HADOOP-18693. Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project (#5427) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-13 10:25:17 -07:00
Melissa You 2b60d0c1f4
[HDFS-16971] Add read metrics for remote reads in FileSystem Statistics #5534 (#5536) 2023-04-13 09:07:42 -07:00
slfan1989 06f9bdffa6
YARN-10846. Add dispatcher metrics to NM. (#4687) 2023-04-12 09:53:20 -07:00
slfan1989 dd6d0ac510
YARN-11462. Fix Typo of hadoop-yarn-common. (#5539) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-12 11:08:23 +08:00
Steve Loughran 7c3d94a032
HADOOP-18637. S3A to support upload of files greater than 2 GB using DiskBlocks (#5543) 
Contributed By: HarshitGupta and Steve Loughran
 2023-04-12 05:17:45 +05:30
slfan1989 bffa49a64f
YARN-11377. [Federation] Support addToClusterNodeLabels、removeFromClusterNodeLabels、replaceLabelsOnNode API's for Federation. (#5525) 2023-04-11 09:47:58 -07:00
Sadanand Shenoy 74ddf69f80
HDFS-16911. Distcp with snapshot diff to support Ozone filesystem. (#5364) 2023-04-10 14:03:16 -07:00
rdingankar 3e2ae1da00
HDFS-16949 Introduce inverse quantiles for metrics where higher numer… (#5495) 2023-04-10 08:56:00 -07:00
mjwiq e45451f9c7
HADOOP-18687. hadoop-auth: remove unnecessary dependency on json-smart (#5524) 
Contributed by Michiel de Jong
 2023-04-06 16:00:33 +01:00
zhtttylz 523ff81624
HDFS-16952. Support getLinkTarget API in WebHDFS (#5517) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-06 19:44:47 +08:00
Simbarashe Dzinamarira 47c22e388e
HDFS-16943. RBF: Implements MySQL based StateStoreDriver. (#5469) 2023-04-05 16:44:29 -07:00
Viraj Jasani 422bf3b24c
HDFS-16973. RBF: MountTableResolver cache size lookup should take read lock (#5533) 2023-04-05 14:06:38 -07:00
slfan1989 69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
HarshitGupta11 dfb2ca0a64
HADOOP-18684. S3A filesystem to support binding to to other URI schemes (#5521) 
Contributed by Harshit Gupta
 2023-04-05 12:42:11 +01:00
Viraj Jasani 937caf7de9
HDFS-16967. RBF: File based state stores should allow concurrent access to the records (#5523) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-04-04 22:39:53 +09:00
Chris Nauroth 14c5810d5e HADOOP-18680: Insufficient heap during full test runs in Docker container. 
Closes #5522

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-03 22:53:29 +00:00
zhtttylz 811441d5bc
HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc (#5491) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-01 18:18:20 +08:00
slfan1989 eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00
sreeb-msft 389b3ea6e3
HADOOP-18012. ABFS: Enable config controlled ETag check for Rename idempotency (#5488) 
To support recovery of network failures during rename, the abfs client
fetches the etag of the source file, and when recovering from a
failure, uses this tag to determine whether the rename succeeded
before the failure happened.

* This works for files, but not directories
* It adds the overhead of a HEAD request before each rename.
* The option can be disabled by setting "fs.azure.enable.rename.resilience"
  to false

Contributed by Sree Bhattacharyya
 2023-03-31 19:15:15 +01:00
Galsza 016362a28b
HADOOP-18548. Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems (#5355) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-03-30 07:12:02 +08:00
Viraj Jasani b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
slfan1989 5bc8f25327
YARN-11446. [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. (#5476) 2023-03-28 09:33:19 -07:00
slfan1989 aa602381c5
YARN-11426. Improve YARN NodeLabel Memory Display. (#5335) 
YARN-11426. Improve YARN NodeLabel Memory Display.
Co-authored-by: slfan1989 <louj1988@@>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-03-28 22:48:46 +08:00
zhangshuyan 700147b4ac
HDFS-16964. Improve processing of excess redundancy after failover. (#5510). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-28 16:14:59 +08:00
Jinhu Wu b5e8269d9b
HADOOP-18458: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS (#4912) 2023-03-28 14:27:01 +08:00
slfan1989 926993cb73
YARN-11376. [Federation] Support updateNodeResource、refreshNodesResources API's for Federation. (#5496) 2023-03-27 09:27:21 -07:00
Anmol Asrani 762d3ddb43
HADOOP-18146: ABFS: Added changes for expect hundred continue header (#4039) 
This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
 2023-03-27 12:43:34 +01:00
Andras Katona ee01c64c6c
HADOOP-18676. jettison dependency override in hadoop-common lib (#5513) 2023-03-27 09:59:02 +02:00
Ayush Saxena b82bcbd8ad
Revert "HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507)" 
This reverts commit 72b0122706.
 2023-03-25 12:04:28 +05:30
Andras Katona 72b0122706
HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507) 
* HADOOP-18587. Fixing jettison vulnerability of hadoop-common lib

* no need for excluding, let it come

Change-Id: Ia6e4ad351158dd4b0510dec34bbde531a60e7654
 2023-03-24 16:31:45 +01:00
Tamas Domok 69748aae32
YARN-11461. fix NPE in determineMissingParents (auto queue creation / CS). (#5506) 
Change-Id: Iaaaf43a545588eaff8a0a20f6f3c27258a45f390
 2023-03-24 09:38:53 +01:00
Kidd5368 5cf62d1498
HDFS-16948. Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. (#5474). Contributed by Kidd53685368. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 17:00:23 +08:00
zhaixiaojuan@loongson.cn 028cde0006
HADOOP-18644. Add bswap support for LoongArch64. (#5453). Contributed by zhaixiaojuan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 11:08:59 +08:00
Ayush Saxena e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Yubi Lee 67e02a92e0
HADOOP-18666. A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server (#5480) 2023-03-22 10:54:41 +09:00
Viraj Jasani 0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem (#5492) 2023-03-21 10:23:16 +08:00
Viraj Jasani 9a8287c36f
HADOOP-18669. Remove Log4Json Layout (#5493) 2023-03-21 10:07:06 +08:00
Viraj Jasani f8d0949f7d
HDFS-16953. RBF: Mount table store APIs should update cache only if state store record is successfully updated (#5482) 2023-03-18 14:43:25 -07:00
Viraj Jasani b6a9d7b442
HADOOP-18631. (ADDENDUM) Use LogCapturer to match audit log pattern and remove hdfs async audit log configs (#5451) 2023-03-18 06:33:50 +08:00
slfan1989 fa723ae839
YARN-11445. [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. (#5472) 2023-03-17 11:59:45 -07:00
Pranav Saxena 759ddebb13
HADOOP-18647. x-ms-client-request-id to identify the retry of an API. (#5437) 
The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous
operation

Contributed by Pranav Saxena
 2023-03-15 20:03:22 +00:00
Masatake Iwasaki 7c42d0f7da
HADOOP-17746. Compatibility table in directory_markers.md doesn't render right. (#3116) 
Contributed by Masatake Iwasaki
 2023-03-15 17:10:42 +00:00
Viraj Jasani 15935fa865
HDFS-16947. RBF NamenodeHeartbeatService to report error for not being able to register namenode in state store (#5470) 2023-03-16 00:59:55 +08:00