86ac1ad9e5
YARN-10978. Fix ApplicationClassLoader to Correctly Expand Glob for Windows Path ( #3558 )
2022-12-06 16:39:49 +05:30
dadd3d9138
YARN-11386. Fix issue with classpath resolution ( #5183 )
...
* This PR ensures that all the special notations such as
<CPS> are resolved before getting added to classpath.
2022-12-06 16:32:26 +05:30
b666075a41
HADOOP-18560. AvroFSInput opens a stream twice and discards the second one without closing ( #5186 )
...
This is needed for branches with the hadoop-common changes of
HADOOP-16202. Enhanced openFile()
2022-12-06 09:58:51 +00:00
84b33b897c
HADOOP-18470. index.md update for 3.3.5 release
2022-12-05 16:13:24 +00:00
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces ( #5123 )
2022-12-05 16:15:47 +08:00
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. ( #5050 )
2022-12-05 16:34:51 +09:00
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. ( #5179 )
2022-12-02 10:56:17 -08:00
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. ( #5104 )
2022-12-01 13:20:21 -08:00
5440c75c4a
YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs
2022-12-01 15:11:58 +01:00
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory ( #4209 )
2022-11-30 07:44:21 -08:00
7786600744
HADOOP-18457. ABFS: Support account level throttling ( #5034 )
...
This allows abfs request throttling to be shared across all
abfs connections talking to containers belonging to the same abfs storage
account -as that is the level at which IO throttling is applied.
The option is enabled/disabled in the configuration option
"fs.azure.account.throttling.enabled";
The default is "true"
Contributed by Anmol Asrani
2022-11-30 13:05:31 +00:00
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy ( #5128 )
...
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
2022-11-30 09:43:15 +08:00
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. ( #5155 )
2022-11-29 22:12:35 +00:00
0ef572abed
HADOOP-18530. ChecksumFileSystem::readVectored might return byte buffers not positioned at 0 ( #5168 )
...
Contributed by Harshit Gupta
2022-11-29 14:51:22 +00:00
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration ( #5143 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-11-29 10:51:21 +09:00
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. ( #5142 )
2022-11-29 00:49:10 +00:00
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. ( #5145 )
2022-11-29 00:47:01 +00:00
f93167e678
YARN-11380. Fix hadoop-yarn-api module Java Doc Errors. ( #5152 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-28 18:54:32 +05:30
1a7acc403b
HADOOP-18498. ABFS: Remove unwanted ? prefix from SAS Tokens ( #5136 )
...
This commit parses SAS Tokens and removes the unwanted prefix of '?' from them, if present.
At present, SAS Tokens are provided to the driver through customer implementations of the SASTokenProvider interface. The SAS token providers should not assume that the token will be the first query parameter in the URIs that communicate with the backend. However, it was observed that certain public interfaces provided by Storage to generate SAS can include the '?' as the first character of the SAS Token, which would ideally be the case when it is the first query parameter. Thus, tokens that contain this prefix will lead to an error in the driver due to a clash of query parameters.
To avoid failures for use of such SAS tokens, after receiving the SAS Token from the provider, the code checks for whether any ? prefix is present or not. If yes, it is removed before further usage of the token. This way, users would not have to manually remove the prefix before passing it on as a configuration.
Contributed by Sree Bhattacharya
2022-11-28 11:38:13 +00:00
e09e81abe4
HADOOP-18496: remove unused okhttp.version ( #5140 ). Contributed by PJ Fanning.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-27 18:59:40 +05:30
1ddc9091f6
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. ( #5153 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-26 21:01:07 +05:30
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md ( #4922 )
2022-11-25 09:26:28 +08:00
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request ( #5086 )
2022-11-25 09:23:33 +08:00
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand ( #5137 )
2022-11-24 09:17:27 +08:00
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap ( #5125 )
...
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:07:27 +08:00
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB ( #5129 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:00:27 +08:00
2c1158e858
HADOOP-18531. Fix assertion failure in ITestS3APrefetchingInputStream ( #5149 )
...
This patch MUST be applied to all branches containing HADOOP-18378
so as to ensure reliable test runs.
Contributed by Ashutosh Gupta
2022-11-23 17:47:39 +00:00
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate ( #5063 )
2022-11-23 09:50:57 +08:00
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. ( #5130 )
2022-11-22 14:38:24 -08:00
3c37a01654
YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta
2022-11-22 13:37:55 +01:00
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state ( #4201 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 10:14:07 -08:00
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory ( #5099 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 08:26:16 -08:00
069bd973d8
HADOOP-18532. Update command usage in FileSystemShell.md ( #5141 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 15:55:46 +09:00
696d042054
HADOOP-8728. Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. ( #5010 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 14:54:50 +09:00
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common ( #4717 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 14:40:02 +09:00
dcde414570
MAPREDUCE-7422. Upgrade Junit 4 to 5 in hadoop-mapreduce-examples ( #5029 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 14:36:35 +09:00
c71a68ca80
HDFS-16844: Adds resilancy when StateStore gets exceptions. ( #5138 )
...
Allows the StateStore to stay up when there are errors reading the data.
2022-11-18 17:24:08 +00:00
1ea5db52dd
HADOOP-18324. Interrupting RPC Client calls can lead to thread exhaustion. ( #4527 )
...
* Exactly 1 sending thread per an RPC connection.
* If the calling thread is interrupted before the socket write, it will be skipped instead of sending it anyways.
* If the calling thread is interrupted during the socket write, the write will finish.
* RPC requests will be written to the socket in the order received.
* Sending thread is only started by the receiving thread.
* The sending thread periodically checks the shouldCloseConnection flag.
2022-11-18 16:24:45 +00:00
7d39abd799
HADOOP-18429. fix infinite loop in MutableGaugeFloat#incr(float) ( #4823 )
2022-11-17 17:50:39 +08:00
eccd2d0492
YARN-11359. [Federation] Routing admin invocations transparently to multiple RMs. ( #5057 )
2022-11-16 18:00:38 -08:00
142df247ed
YARN-11369. Commons.compress throws an IllegalArgumentException with large uids after 1.21. Contributed by Benjamin Teke
2022-11-16 13:07:05 +01:00
cd929457c9
HDFS-16836: StandbyCheckpointer shouldn't trigger rollback fs image after RU is finalized ( #5135 )
...
Co-authored-by: Lei Yang <leyang@linkedin.com>
2022-11-15 23:06:37 +00:00
69e50c7b44
HADOOP-18528. Disable abfs prefetching by default ( #5134 )
...
Disables block prefetching on ABFS InputStreams, by setting
fs.azure.enable.readahead to false in core-default.xml and
the matching java constant.
This prevents
HADOOP-18521. ABFS ReadBufferManager buffer sharing across concurrent HTTP requests.
Once a fix for that is committed, this change can be reverted.
Contributed by Mehakmeet Singh.
2022-11-15 14:28:41 +00:00
a48e8c9beb
MAPREDUCE-5608. Replace and deprecate mapred.tasktracker.indexcache.mb ( #5014 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-14 11:07:40 +09:00
04b31d7ecf
MAPREDUCE-7390. Remove WhiteBox in mapreduce module. ( #4462 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-14 10:45:20 +09:00
d3c1c453f0
HDFS-16785. Avoid to hold write lock to improve performance when add volume. ( #4945 ). Contributed by ZanderXu.
...
Signed-off-by: Tao Li <tomscut@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-11-13 22:01:35 +08:00
22c9f28f4d
YARN-10005. Code improvements in MutableCSConfigurationProvider. Contributed by Peter Szucs
2022-11-12 18:35:49 +01:00
d340c4a7a1
HADOOP-18496. Upgrade okhttp3 and dependencies due to kotlin CVEs ( #5035 )
...
Updates okhttp3 and okio so their transitive dependency on Kotlin
stdlib is free from recent CVEs.
okhttp3:okhttp => 4.10.0
okio:okio => 3.2.0
kotlin stdlib => 1.6.20
kotlin CVEs fixed:
CVE-2022-24329
CVE-2020-29582
Contributed by PJ Fanning.
2022-11-12 14:14:19 +00:00
5bb11cecea
HADOOP-15327. Upgrade MR ShuffleHandler to use Netty4 #3259 . Contributed by Szilard Nemeth.
2022-11-11 09:05:01 +01:00
552ee44eba
HDFS-16834: Removes request stateID consistency constraint between clients in different connection pools. ( #5121 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Zander Xu <zanderxu@apache.org>
2022-11-11 15:26:31 +08:00
Akshat Bordia
Gautham B A
Steve Loughran
ZanderXu
dingshun3016
slfan1989
Szilard Nemeth
litao
Anmol Asrani
Kidd5368
Owen O'Malley
HarshitGupta11
caozhiqiang
Simbarashe Dzinamarira
sreeb-msft
PJ Fanning
huhaiyang
Ashutosh Gupta
zhengchenyu
GuoPhilipse
Hu Xinqiu
Lei Yang
Mehakmeet Singh
ZanderXu
Szilard Nemeth