Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,207 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25207 Commits

Author SHA1 Message Date
cxzl25 4d052a2456
HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager (#4598). Contributed by dzcxzl. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org
 2023-04-26 02:12:40 +05:30
Sebastian Baunsgaard 919c3f615b
HADOOP-18660. Filesystem Spelling Mistake (#5475). 
Contributed by Sebastian Baunsgaard.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-25 19:59:54 +01:00
Steve Loughran 0b56be3ca4
MAPREDUCE-7437. MR Fetcher class to use an AtomicInteger to generate IDs. (#5579) 
...as until now it wasn't thread safe

Contributed by Steve Loughran
 2023-04-25 19:56:18 +01:00
Ayush Saxena d7d36b9d2a
HADOOP-18689. Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project (#5502) (#5586) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-25 21:26:59 +05:30
Steve Loughran 21cf507db3
HADOOP-17450. Add Public IOStatistics API -missed backport (#5590) 
This cherrypicks SemaphoredDelegatingExecutor HADOOP-17450 changes
from trunk somehow they didn't get into the main IOStatistics backport
to branch-3.3
 2023-04-25 15:02:56 +01:00
Tamas Domok 1b59e3123b
HADOOP-18705. ABFS should exclude incompatible credential providers. (#5560) 
Contributed by Tamas Domok.
 2023-04-24 15:48:02 +01:00
Doroszlai, Attila 13d3cfd311
HADOOP-18714. Wrong StringUtils.join() called in AbstractContractRootDirectoryTest (#5588) 
(cherry picked from commit 5b23224970)
 2023-04-24 15:49:20 +02:00
wangzhaohui 1ea5fbd192
HDFS-16988. Improve NameServices info at JournalNode web UI (#5584). Contributed by Zhaohui Wang. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:14:18 +05:30
Nikita Eshkeev 7a32e7cc38
HADOOP-18597. Simplify single node instructions for creating directories for Map Reduce. (#5305) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-24 01:14:09 +05:30
Christos Bisias 57ff8bdb67 HADOOP-18691. Add a CallerContext getter on the Schedulable interface (#5540) 2023-04-20 10:13:33 -07:00
Steve Loughran a505940a2f
HADOOP-18470. Hadoop 3.3.5 release wrap-up (#5558) 
Post-release updates of the branches

* Add jdiff xml files from 3.3.5 release.
* Declare 3.3.5 as the latest stable release.
* Copy release notes.
 2023-04-18 10:12:41 +01:00
Steve Loughran f5464831a0
HADOOP-18696. ITestS3ABucketExistence arn test failures. (#5557) 
Explicitly sets the fs.s3a.endpoint.region to eu-west-1 so
the ARN-referenced fs creation fails with unknown store
rather than IllegalArgumentException.

Steve Loughran
 2023-04-17 10:21:01 +01:00
dependabot[bot] 8e15216045 HADOOP-18693. Bump derby from 10.10.2.0 to 10.14.2.0 in /hadoop-project (#5427) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-04-16 06:51:17 -07:00
Dongjoon Hyun 854eddb0cc
HADOOP-18590. Publish SBOM artifacts (#5555). Contributed by Dongjoon Hyun. 2023-04-15 21:40:25 +05:30
sreeb-msft f324efd247
HADOOP-18012. ABFS: Enable config controlled ETag check for Rename idempotency (#5488) 
To support recovery of network failures during rename, the abfs client
fetches the etag of the source file, and when recovering from a
failure, uses this tag to determine whether the rename succeeded
before the failure happened.

* This works for files, but not directories
* It adds the overhead of a HEAD request before each rename.
* The option can be disabled by setting "fs.azure.enable.rename.resilience"
  to false

Contributed by Sree Bhattacharyya
 2023-04-05 15:07:39 +01:00
HarshitGupta11 42ed2b9075
HADOOP-18684. S3A filesystem to support binding to to other URI schemes (#5521) 
Contributed by Harshit Gupta
 2023-04-05 14:57:27 +01:00
Chris Nauroth 3400e8257e HADOOP-18680: Insufficient heap during full test runs in Docker container on branch-3.3. 
Closes #5511

Signed-off-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-04-03 22:58:53 +00:00
Pranav Saxena 054afa1180
HADOOP-18647. x-ms-client-request-id to identify the retry of an API. (#5437) 
The x-ms-client-request-id now includes a field to indicate a call is a retry of a previous
operation

Contributed by Pranav Saxena
 2023-03-30 14:26:12 +01:00
Viraj Jasani 20d3b9cc46
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) (#5374) 2023-03-30 07:13:10 +08:00
Anmol Asrani 6306f5b2bc
HADOOP-18146: ABFS: Added changes for expect hundred continue header #4039 
This change lets the client react pre-emptively to server load without getting to 503 and the exponential backoff
which follows. This stops performance suffering so much as capacity limits are approached for an account.

Contributed by Anmol Asranii
 2023-03-28 16:32:01 +01:00
Pranav Saxena 2b156c2b32
HADOOP-18606. ABFS: Add reason in x-ms-client-request-id on a retried API call. (#5299) 
Contributed by Pranav Saxena
 2023-03-28 12:00:57 +01:00
Masatake Iwasaki dd9ef9e0e7
HADOOP-17746. Compatibility table in directory_markers.md doesn't render right. (#3116) 
Contributed by Masatake Iwasaki
 2023-03-15 17:11:30 +00:00
Stephen O'Donnell c39696ab1d HDFS-16942. Addendum. Send error to datanode if FBR is rejected due to bad lease (#5478). Contributed by Stephen O'Donnell/ 
(cherry picked from commit eee2ea075d)
 2023-03-15 10:17:55 +00:00
Steve Loughran 0dd4e500b0
HADOOP-18661. Fix bin/hadoop usage script terminology. (#5473) 
Followup to HADOOP-13209: s/slaves/r/workers in
the usage message you get when you type "bin/hadoop"

Contributed by Steve Loughran
 2023-03-13 12:24:10 +00:00
Stephen O'Donnell d7b89d08e1 HDFS-16942. Send error to datanode if FBR is rejected due to bad lease (#5460) 
(cherry picked from commit ca6f5afb6d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockReportLease.java
 2023-03-11 17:04:33 +00:00
zhangshuyan 8cc57f5eb1
HDFS-16939. Fix the thread safety bug in LowRedundancyBlocks. (#5450 #5471). Contributed by Shuyan Zhang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-03-11 16:09:06 +08:00
slfan1989 4a23389f5e
YARN-11394. Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. (#5288) 
Contributed by Shilun Fan
 2023-03-10 15:16:28 +00:00
Steve Loughran b75ced1e5d
HADOOP-17836. Improve logging on ABFS error reporting (#3281) 
Contributed by Steve Loughran.
 2023-03-08 15:31:16 +00:00
slfan1989 e12bc4e1d8
HDFS-16934. TestDFSAdmin.testAllDatanodesReconfig regression (#5434) 
Contributed by Shilun Fan
 2023-03-06 15:45:42 +00:00
Varun Saxena 157af0cb22 YARN-11383. Workflow priority mappings is case sensitive (#5171) 
Contributed by Aparajita Choudhary

(cherry picked from commit 2a0dc2ab2f)
 2023-03-05 23:02:17 +05:30
Tom 703158c9c6
HDFS-16896 clear ignoredNodes list when we clear deadnode list on ref… (#5322) (#5444) 
Cherry picked from: 162288bc0a

Co-authored-by: Tom McCormick <tmccormi@linkedin.com>
 2023-03-02 00:44:30 +00:00
ZanderXu a7c1fad0c9 HDFS-16923. [SBN read] getlisting RPC to observer will throw NPE if path does not exist (#5400) 
Signed-off-by: Erik Krogen <xkrogen@apache.org>

(cherry picked from commit 6bd2444815)
 2023-03-01 16:20:50 -08:00
Viraj Jasani 91ce13ea01
HDFS-16935. Fix TestFsDatasetImpl#testReportBadBlocks (#5432) 
Contributed by Viraj Jasani
 2023-03-01 18:53:52 +00:00
Steve Loughran bca38f84af
HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429) 
POM and LICENSE fixup of transient dependencies
* Exclude hadoop-cloud-storage imports which come in with hadoop-common
* Add explicit import of hadoop's org.codehaus.jettison declaration
  to hadoop-aliyun
* Tune aliyun jars imports
* Cut duplicate and inconsistent hbase-server declarations from
  hadoop-project
* Update LICENSE-binary for the current set of libraries in the
  hadoop 3.3.5 release.

Contributed by Steve Loughran
 2023-02-28 14:05:13 +00:00
rdingankar 94b3c6dd90 HDFS-16917 Add transfer rate quantile metrics for DataNode reads (#5397) 
Co-authored-by: Ravindra Dingankar <rdingankar@linkedin.com>
 2023-02-27 15:49:26 -08:00
Owen O'Malley c25ac781ca HDFS-16756: Fixup for 3.3 backport. 2023-02-27 15:47:10 -08:00
Simbarashe Dzinamarira 5fe19a0f01 HDFS-16901: RBF: Propagates real user's username via the caller context, when a proxy user is being used. (#5346) 2023-02-24 13:32:23 -08:00
ZanderXu 9a89deca1d HDFS-16756. RBF proxies the client's user by the login user to enable CacheEntry (#4853). Contributed by ZanderXu. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-24 13:31:53 -08:00
Steve Loughran 35e04ff52a
HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file 
+ add a link to mukund's apachecon talk

Change-Id: I3d04b385ff1312aabf2a81d034f54f124d544a54
 2023-02-23 13:26:54 +00:00
Steve Loughran 92d6826d10
Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" 
Causes HADOOP-18641. cyclonedx maven plugin breaks on recent maven releases

This reverts branch-3.3 commit b6fd322307.
 2023-02-23 11:25:50 +00:00
hchaverr eab7215354
HADOOP-18535. Implement token storage solution based on MySQL 
Fixes #1240

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2023-02-22 14:02:13 -08:00
Ayush Saxena 84e999b35c
HADOOP-18582. Addendum: Skip unnecessary cleanup logic in DistCp. (#5409) 
Followup to the original HADOOP-18582.

Temporary path cleanup is re-enabled for -append jobs
as these will create temporary files when creating or overwriting files.

Contributed by Ayush Saxena
 2023-02-22 19:32:05 +00:00
Steve Loughran ee71318d72
HADOOP-18636 LocalDirAllocator cannot recover from directory tree deletion (#5412) 
Even though DiskChecker.mkdirsWithExistsCheck() will create the directory tree,
it is only called *after* the enumeration of directories with available
space has completed.

Directories which don't exist are reported as having 0 space, therefore
the mkdirs code is never reached.

Adding a simple mkdirs() -without bothering to check the outcome-
ensures that if a dir has been deleted then it will be reconstructed
if possible. If it can't it will still have 0 bytes of space
reported and so be excluded from the allocation.

Contributed by Steve Loughran
 2023-02-22 11:50:17 +00:00
Mehakmeet Singh a3b0135ea6
HADOOP-18633. fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip (#5422) 
Contributed by: Mehakmeet Singh
 2023-02-22 14:31:46 +05:30
Viraj Jasani e47ff5f3c0
HDFS-16925. Namenode audit log to only include IP address of client (#5413) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-21 08:33:54 +08:00
Arnout Engelen 477f17be97
HADOOP-18627. Add stronger wording in 'secure mode' introduction (#5406) 
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.

Contributed by Arnout Engelen
 2023-02-17 16:31:21 +00:00
Steve Loughran b9b999f7b1
YARN-11441. Revert YARN-10495. 
This reverts commit 1022ed91ac.
 2023-02-17 15:07:41 +00:00
Bryan Beaudreault aa6c51364a HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers (#4215) 2023-02-16 11:38:20 -08:00
Viraj Jasani 8c9c68c19e
HADOOP-18628. IPC Server Connection should log host name before returning VersionMismatch error (#5385) 
Contributed by Viraj Jasani
 2023-02-15 18:23:44 +00:00
Steve Loughran cd2401d2cc
HADOOP-18470. More in the 3.3.5 index.html about security (#5383) 
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners

Contributed by Steve Loughran
 2023-02-14 17:25:20 +00:00