63b9a6a2b6
YARN-11350. [Federation] Router Support DelegationToken With ZK. ( #5131 )
2022-12-14 09:09:38 -08:00
4de8791deb
HADOOP-18569. NFS Gateway may release buffer too early ( #5212 )
...
(cherry picked from commit df4812df65
2022-12-14 15:55:44 +01:00
1cecf8ab70
HADOOP-18183. s3a audit logs to publish range start/end of GET requests. ( #5110 )
...
The start and end of the range is set in a new audit param "rg",
e.g "?rg=100-200"
Contributed by Ankit Saurabh
2022-12-14 14:01:28 +00:00
85ec7969a7
MAPREDUCE-7428. Fix failures related to Junit 4 to Junit 5 upgrade in org.apache.hadoop.mapreduce.v2.app.webapp ( #5209 )
...
Contributed by: Ashutosh Gupta
2022-12-14 12:54:08 +00:00
fdcbc8b072
HDFS-16868. Fix audit log duplicate issue when an ACE occurs in FSNamesystem. ( #5206 ). Contributed by Beibei Zhao.
...
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-12-13 12:24:51 +08:00
a71aaef9a9
YARN-11385. Fix hadoop-yarn-server-common module Java Doc Errors. ( #5182 )
2022-12-10 15:03:49 -08:00
a46b20d25f
HADOOP-18329. Support for IBM Semeru JVM > 11.0.15.0 Vendor Name Changes ( #4537 )
...
The static boolean PlatformName.IBM_JAVA now identifies
Java 11+ IBM Semeru runtimes as IBM JVM releases.
Contributed by Jack Buggins.
2022-12-10 14:27:05 +00:00
0a7dfcc332
HADOOP-18546. Followup: ITestReadBufferManager fix ( #5198 )
...
This is a followup to the original HADOOP-18546
patch; cherry-picks of that should include this
or follow up with it.
Removes risk of race conditions in assertions
of ITestReadBufferManager on the state of the in-progress
and completed queues by removing assertions brittle
to race conditions in scheduling/network IO
* Waits for all the executor pool shutdown to complete before
making any assertions
* Assertions that there are no in progress reads MUST be
cut as there may be some and they won't be cancelled.
* Assertions that the completed list is without buffers
of a closed stream are brittle because if there was
an in progress stream which completed after stream.close()
then it will end up in the list.
Contributed by Steve Loughran
2022-12-09 13:47:11 +00:00
e76616f690
HDFS-16860 Upgrade moment.min.js to 2.29.4 ( #5194 )
2022-12-09 11:18:44 +05:30
2fa540dca1
HDFS-16858. Dynamically adjust max slow disks to exclude. ( #5180 )
...
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Reviewed-by: slfan1989 <55643692+slfan1989@users.noreply.github.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-12-09 08:10:04 +08:00
ee7d1787cd
YARN-11390. TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> ( #5190 )
...
Reviewed-by: Peter Szucs
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
2022-12-08 09:52:19 -08:00
0a4528cd7f
HADOOP-18563. Misleading AWS SDK S3 timeout configuration comment ( #5197 )
...
Contributed by Oleksandr Shevchenko
2022-12-08 15:07:59 +00:00
c67c2b7569
HADOOP-18546. ABFS. disable purging list of in progress reads in abfs stream close() ( #5176 )
...
This addresses HADOOP-18521, "ABFS ReadBufferManager buffer sharing
across concurrent HTTP requests" by not trying to cancel
in progress reads.
It supercedes HADOOP-18528, which disables the prefetching.
If that patch is applied *after* this one, prefetching
will be disabled.
As well as changing the default value in the code,
core-default.xml is updated to set
fs.azure.enable.readahead = true
As a result, if Configuration.get("fs.azure.enable.readahead")
returns a non-null value, then it can be inferred that
it was set in or core-default.xml (the fix is present)
or in core-site.xml (someone asked for it).
Contributed by Pranav Saxena.
2022-12-07 20:15:45 +00:00
2e88096266
HADOOP-18538. Upgrade kafka to 2.8.2 ( #5164 )
...
Signed-off-by: Brahma Reddy Battula <brahma@apache.org>
2022-12-06 22:27:46 +05:30
f71fd885be
YARN-11373. [Federation] Support refreshQueues refreshNodes API's for Federation. ( #5146 )
2022-12-06 08:17:05 -08:00
86ac1ad9e5
YARN-10978. Fix ApplicationClassLoader to Correctly Expand Glob for Windows Path ( #3558 )
2022-12-06 16:39:49 +05:30
dadd3d9138
YARN-11386. Fix issue with classpath resolution ( #5183 )
...
* This PR ensures that all the special notations such as
<CPS> are resolved before getting added to classpath.
2022-12-06 16:32:26 +05:30
b666075a41
HADOOP-18560. AvroFSInput opens a stream twice and discards the second one without closing ( #5186 )
...
This is needed for branches with the hadoop-common changes of
HADOOP-16202. Enhanced openFile()
2022-12-06 09:58:51 +00:00
84b33b897c
HADOOP-18470. index.md update for 3.3.5 release
2022-12-05 16:13:24 +00:00
8a9bdb1edc
HDFS-16837. [RBF SBN] ClientGSIContext should merge RouterFederatedStates to get the max state id for each namespaces ( #5123 )
2022-12-05 16:15:47 +08:00
02afb9ebe1
HDFS-16809. EC striped block is not sufficient when doing in maintenance. ( #5050 )
2022-12-05 16:34:51 +09:00
60e0fe8709
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. ( #5179 )
2022-12-02 10:56:17 -08:00
4af4997e11
YARN-11158. Support (Create/Renew/Cancel) DelegationToken API's for Federation. ( #5104 )
2022-12-01 13:20:21 -08:00
5440c75c4a
YARN-10946. AbstractCSQueue: Create separate class for constructing Queue API objects. Contributed by Peter Szucs
2022-12-01 15:11:58 +01:00
2067fcb646
HDFS-16550. Allow JN edit cache size to be set as a fraction of heap memory ( #4209 )
2022-11-30 07:44:21 -08:00
7786600744
HADOOP-18457. ABFS: Support account level throttling ( #5034 )
...
This allows abfs request throttling to be shared across all
abfs connections talking to containers belonging to the same abfs storage
account -as that is the level at which IO throttling is applied.
The option is enabled/disabled in the configuration option
"fs.azure.account.throttling.enabled";
The default is "true"
Contributed by Anmol Asrani
2022-11-30 13:05:31 +00:00
72749a4ff8
HDFS-16839 It should consider EC reconstruction work when we determine if a node is busy ( #5128 )
...
Co-authored-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Tao Li <tomscut@apache.org>
2022-11-30 09:43:15 +08:00
03471a736c
HDFS-16851: RBF: Add a utility to dump the StateStore. ( #5155 )
2022-11-29 22:12:35 +00:00
0ef572abed
HADOOP-18530. ChecksumFileSystem::readVectored might return byte buffers not positioned at 0 ( #5168 )
...
Contributed by Harshit Gupta
2022-11-29 14:51:22 +00:00
35c65005d0
HDFS-16846. EC: Only EC blocks should be effected by max-streams-hard-limit configuration ( #5143 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-11-29 10:51:21 +09:00
909aeca86c
HDFS-16845: Adds configuration flag to allow clients to use router observer reads without using the ObserverReadProxyProvider. ( #5142 )
2022-11-29 00:49:10 +00:00
ec2856d79c
HDFS-16847: RBF: Prevents StateStoreFileSystemImpl from committing tmp file after encountering an IOException. ( #5145 )
2022-11-29 00:47:01 +00:00
f93167e678
YARN-11380. Fix hadoop-yarn-api module Java Doc Errors. ( #5152 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-28 18:54:32 +05:30
1a7acc403b
HADOOP-18498. ABFS: Remove unwanted ? prefix from SAS Tokens ( #5136 )
...
This commit parses SAS Tokens and removes the unwanted prefix of '?' from them, if present.
At present, SAS Tokens are provided to the driver through customer implementations of the SASTokenProvider interface. The SAS token providers should not assume that the token will be the first query parameter in the URIs that communicate with the backend. However, it was observed that certain public interfaces provided by Storage to generate SAS can include the '?' as the first character of the SAS Token, which would ideally be the case when it is the first query parameter. Thus, tokens that contain this prefix will lead to an error in the driver due to a clash of query parameters.
To avoid failures for use of such SAS tokens, after receiving the SAS Token from the provider, the code checks for whether any ? prefix is present or not. If yes, it is removed before further usage of the token. This way, users would not have to manually remove the prefix before passing it on as a configuration.
Contributed by Sree Bhattacharya
2022-11-28 11:38:13 +00:00
e09e81abe4
HADOOP-18496: remove unused okhttp.version ( #5140 ). Contributed by PJ Fanning.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-27 18:59:40 +05:30
1ddc9091f6
YARN-11381. Fix hadoop-yarn-common module Java Doc Errors. ( #5153 ). Contributed by Shilun Fan.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-11-26 21:01:07 +05:30
87429f443a
HDFS-16779. Add ErasureCodingPolicy information to the response description for GETFILESTATUS in WebHDFS.md ( #4922 )
2022-11-25 09:26:28 +08:00
e0974298ce
HDFS-16826. [RBF SBN] ConnectionManager should advance the client stateId for each request ( #5086 )
2022-11-25 09:23:33 +08:00
ef84d21867
HDFS-16841. Enhance the function of DebugAdmin#VerifyECCommand ( #5137 )
2022-11-24 09:17:27 +08:00
bcc3d2a20e
HDFS-16838. Fix NPE in testAddRplicaProcessorForAddingReplicaInMap ( #5125 )
...
Reviewed-by: Xing Lin <xinglin@linkedin.com>
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:07:27 +08:00
dfa9edacce
HDFS-16840. Enhance the usage description about oiv in HDFSCommands.md and OfflineImageViewerPB ( #5129 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: ZanderXu <zanderxu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-11-24 09:00:27 +08:00
2c1158e858
HADOOP-18531. Fix assertion failure in ITestS3APrefetchingInputStream ( #5149 )
...
This patch MUST be applied to all branches containing HADOOP-18378
so as to ensure reliable test runs.
Contributed by Ashutosh Gupta
2022-11-23 17:47:39 +00:00
ac958777af
HDFS-16813. Remove parameter validation logic such as dfs.namenode.decommission.blocks.per.interval in DatanodeAdminManager#activate ( #5063 )
2022-11-23 09:50:57 +08:00
7cb22eb72d
YARN-11371. [Federation] Refactor FederationInterceptorREST#createNewApplication\submitApplication Use FederationActionRetry. ( #5130 )
2022-11-22 14:38:24 -08:00
3c37a01654
YARN-8262. get_executable in container-executor should provide meaningful error codes. Contributed by Susheel Gupta
2022-11-22 13:37:55 +01:00
8f971b0e54
HDFS-16547. [SBN read] Namenode in safe mode should not be transfer to observer state ( #4201 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 10:14:07 -08:00
dc2fba45fe
HDFS-16832. [SBN READ] Follow-on to HDFS-16732. Fix NPE when check the block location of empty directory ( #5099 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
Reviewed-by: Zengqiang Xu <xuzq_zander@163.com>
2022-11-21 08:26:16 -08:00
069bd973d8
HADOOP-18532. Update command usage in FileSystemShell.md ( #5141 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 15:55:46 +09:00
696d042054
HADOOP-8728. Display (fs -text) shouldn't hard-depend on Writable serialized sequence files. ( #5010 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 14:54:50 +09:00
2e993fdf4e
YARN-6946. Upgrade JUnit from 4 to 5 in hadoop-yarn-common ( #4717 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-11-21 14:40:02 +09:00
slfan1989
Doroszlai, Attila
Steve Loughran
Ashutosh Gupta
curie71
Jack Richard Buggins
Anurag P
dingshun3016
K0K0V0K
Oleksandr Shevchenko
Pranav Saxena
Murali Krishna
Akshat Bordia
Gautham B A
ZanderXu
Szilard Nemeth
litao
Anmol Asrani
Kidd5368
Owen O'Malley
HarshitGupta11
caozhiqiang
Simbarashe Dzinamarira
sreeb-msft
PJ Fanning
huhaiyang
zhengchenyu
GuoPhilipse