Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,688 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26688 Commits

Author SHA1 Message Date
slfan1989 69b90b5698
YARN-11436. [Federation] MemoryFederationStateStore Support Version. (#5518) 2023-04-05 10:35:24 -07:00
HarshitGupta11 dfb2ca0a64
HADOOP-18684. S3A filesystem to support binding to to other URI schemes (#5521) 
Contributed by Harshit Gupta
 2023-04-05 12:42:11 +01:00
Viraj Jasani 937caf7de9
HDFS-16967. RBF: File based state stores should allow concurrent access to the records (#5523) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Simbarashe Dzinamarira <sdzinamarira@linkedin.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-04-04 22:39:53 +09:00
Chris Nauroth 14c5810d5e HADOOP-18680: Insufficient heap during full test runs in Docker container. 
Closes #5522

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-03 22:53:29 +00:00
zhtttylz 811441d5bc
HDFS-16951. Add description of GETSERVERDEFAULTS to WebHDFS doc (#5491) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-04-01 18:18:20 +08:00
slfan1989 eb1d3ebe2f
YARN-11442. Refactor FederationInterceptorREST Code. (#5420) 2023-03-31 15:29:18 -07:00
sreeb-msft 389b3ea6e3
HADOOP-18012. ABFS: Enable config controlled ETag check for Rename idempotency (#5488) 
To support recovery of network failures during rename, the abfs client
fetches the etag of the source file, and when recovering from a
failure, uses this tag to determine whether the rename succeeded
before the failure happened.

* This works for files, but not directories
* It adds the overhead of a HEAD request before each rename.
* The option can be disabled by setting "fs.azure.enable.rename.resilience"
  to false

Contributed by Sree Bhattacharyya
 2023-03-31 19:15:15 +01:00
Galsza 016362a28b
HADOOP-18548. Hadoop Archive tool (HAR) should acquire delegation tokens from source and destination file systems (#5355) 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-03-30 07:12:02 +08:00
Viraj Jasani b4bcbb9515
HDFS-16959. RBF: State store cache loading metrics (#5497) 2023-03-29 10:43:13 -07:00
slfan1989 5bc8f25327
YARN-11446. [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. (#5476) 2023-03-28 09:33:19 -07:00
slfan1989 aa602381c5
YARN-11426. Improve YARN NodeLabel Memory Display. (#5335) 
YARN-11426. Improve YARN NodeLabel Memory Display.
Co-authored-by: slfan1989 <louj1988@@>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-03-28 22:48:46 +08:00
zhangshuyan 700147b4ac
HDFS-16964. Improve processing of excess redundancy after failover. (#5510). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-28 16:14:59 +08:00
Jinhu Wu b5e8269d9b
HADOOP-18458: AliyunOSSBlockOutputStream to support heap/off-heap buffer before uploading data to OSS (#4912) 2023-03-28 14:27:01 +08:00
slfan1989 926993cb73
YARN-11376. [Federation] Support updateNodeResource、refreshNodesResources API's for Federation. (#5496) 2023-03-27 09:27:21 -07:00
Anmol Asrani 762d3ddb43
HADOOP-18146: ABFS: Added changes for expect hundred continue header (#4039) 
This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
 2023-03-27 12:43:34 +01:00
Andras Katona ee01c64c6c
HADOOP-18676. jettison dependency override in hadoop-common lib (#5513) 2023-03-27 09:59:02 +02:00
Ayush Saxena b82bcbd8ad
Revert "HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507)" 
This reverts commit 72b0122706.
 2023-03-25 12:04:28 +05:30
Andras Katona 72b0122706
HADOOP-18676. Fixing jettison vulnerability of hadoop-common lib (#5507) 
* HADOOP-18587. Fixing jettison vulnerability of hadoop-common lib

* no need for excluding, let it come

Change-Id: Ia6e4ad351158dd4b0510dec34bbde531a60e7654
 2023-03-24 16:31:45 +01:00
Tamas Domok 69748aae32
YARN-11461. fix NPE in determineMissingParents (auto queue creation / CS). (#5506) 
Change-Id: Iaaaf43a545588eaff8a0a20f6f3c27258a45f390
 2023-03-24 09:38:53 +01:00
Kidd5368 5cf62d1498
HDFS-16948. Update log of BlockManager#chooseExcessRedundancyStriped when EC internal block is moved by balancer. (#5474). Contributed by Kidd53685368. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 17:00:23 +08:00
zhaixiaojuan@loongson.cn 028cde0006
HADOOP-18644. Add bswap support for LoongArch64. (#5453). Contributed by zhaixiaojuan. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-23 11:08:59 +08:00
Ayush Saxena e3cb9573e1
HADOOP-18662. ListFiles with recursive fails with FNF. (#5477). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org
 2023-03-23 08:30:08 +05:30
Yubi Lee 67e02a92e0
HADOOP-18666. A whitelist of endpoints to skip Kerberos authentication doesn't work for ResourceManager and Job History Server (#5480) 2023-03-22 10:54:41 +09:00
Viraj Jasani 0dbe1d3284
HADOOP-18668. Path capability probe for truncate is only honored by RawLocalFileSystem (#5492) 2023-03-21 10:23:16 +08:00
Viraj Jasani 9a8287c36f
HADOOP-18669. Remove Log4Json Layout (#5493) 2023-03-21 10:07:06 +08:00
Viraj Jasani f8d0949f7d
HDFS-16953. RBF: Mount table store APIs should update cache only if state store record is successfully updated (#5482) 2023-03-18 14:43:25 -07:00
Viraj Jasani b6a9d7b442
HADOOP-18631. (ADDENDUM) Use LogCapturer to match audit log pattern and remove hdfs async audit log configs (#5451) 2023-03-18 06:33:50 +08:00
slfan1989 fa723ae839
YARN-11445. [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. (#5472) 2023-03-17 11:59:45 -07:00
Pranav Saxena 759ddebb13
HADOOP-18647. x-ms-client-request-id to identify the retry of an API. (#5437) 
The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous
operation

Contributed by Pranav Saxena
 2023-03-15 20:03:22 +00:00
Masatake Iwasaki 7c42d0f7da
HADOOP-17746. Compatibility table in directory_markers.md doesn't render right. (#3116) 
Contributed by Masatake Iwasaki
 2023-03-15 17:10:42 +00:00
Viraj Jasani 15935fa865
HDFS-16947. RBF NamenodeHeartbeatService to report error for not being able to register namenode in state store (#5470) 2023-03-16 00:59:55 +08:00
Viraj Jasani cf4a678ce9
HADOOP-18649. CLA and CRLA appenders to be replaced with RFA (#5448) 2023-03-16 00:46:17 +08:00
Viraj Jasani 405bfa2800
HADOOP-18654. Remove unused custom appender TaskLogAppender (#5457) 2023-03-16 00:45:37 +08:00
Stephen O'Donnell eee2ea075d
HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/ 2023-03-15 10:03:00 +05:30
Viraj Jasani aff840c59c
HADOOP-18653. LogLevel servlet to determine log impl before using setLevel (#5456) 
The log level can only be set on Log4J log implementations;
probes are used to downgrade to a warning when other
logging back ends are used

Contributed by Viraj Jasani
 2023-03-13 12:30:12 +00:00
Steve Loughran 09469bf47d
HADOOP-18661. Fix bin/hadoop usage script terminology. (#5473) 
Followup to HADOOP-13209: s/slaves/r/workers in
the usage message you get when you type "bin/hadoop"

Contributed by Steve Loughran
 2023-03-13 12:24:36 +00:00
PJ Fanning 476340c699
HADOOP-18658. snakeyaml dependency: upgrade to v2.0 (#5467). Contributed by PJ Fanning. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-03-13 10:08:04 +05:30
Stephen O'Donnell ca6f5afb6d
HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460) 2023-03-11 16:40:07 +00:00
nao 734f7abfb8
HADOOP-18646. Upgrade Netty to 4.1.89.Final to fix CVE-2022-41881 (#5435) 
This fixes CVE-2022-41881.

This also upgrades io.opencensus dependencies to 0.12.3
 
Contributed by Aleksandr Nikolaev
 2023-03-10 15:27:22 +00:00
slfan1989 b406060c6b
YARN-8972. [Router] Add support to prevent DoS attack over ApplicationSubmissionContext size. (#5382) 2023-03-08 13:29:30 -08:00
rohit-kb 487368c4b9
HADOOP-18655. Upgrade kerby to 2.0.3 due to CVE-2023-25613 (#5458) 
Upgrade kerby to 2.0.3 due to the CVE https://nvd.nist.gov/vuln/detail/CVE-2023-25613


Contributed by Rohit Kumar Badeau
 2023-03-08 15:31:03 +00:00
Pranav Saxena 358bf80c94
HADOOP-18606. ABFS: Add reason in x-ms-client-request-id on a retried API call. (#5299) 
Contributed by Pranav Saxena
 2023-03-07 17:02:13 +00:00
slfan1989 927401886a
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434) 
Contributed by Shilun Fan
 2023-03-06 15:26:53 +00:00
zhangshuyan 2cb0c35fc1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-06 20:10:31 +08:00
Varun Saxena 2a0dc2ab2f YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary
 2023-03-05 21:25:16 +05:30
ZanderXu 6bd2444815
HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>
 2023-03-01 16:18:38 -08:00
Viraj Jasani e1ca466bdb
HADOOP-18648. Avoid loading kms log4j properties dynamically by KMSWebServer (#5441) 2023-03-02 08:02:07 +08:00
Tom 162288bc0a
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) 
HDFS-16896 clear ignoredNodes list when we clear deadnode list on refetchLocations. 
ignoredNodes list is only used on hedged read codepath

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-01 19:47:04 +00:00
Viraj Jasani 2ab7eb4caa
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:10 +00:00
Szilard Nemeth 8f6be3678d MAPREDUCE-7434. Fix ShuffleHandler tests. Contributed by Tamas Domok 2023-03-01 16:10:05 +01:00