Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,620 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25620 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
Steve Loughran 88a550bc3a
HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.
 2021-04-25 18:33:55 +01:00
jianghuazhu c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
Eric Badger 6857a05d6a YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
 2021-04-23 22:02:04 +00:00
Jim Brennan 1cbe35946b YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 2021-04-23 21:35:09 +00:00
Szilard Nemeth ebf6b14c67 YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 
This is required to highlight that an implementation at framework level is needed for the stop/start functionality to work.
 2021-04-23 22:49:26 +02:00
Steve Loughran 027c8fb257
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.
 2021-04-23 18:44:41 +01:00
Ayush Saxena 6800b21e3b
HADOOP-17620. DistCp: Use Iterator for listing target directory as well. (#2861). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-04-23 22:48:15 +05:30
Peter Bacsko 14a84c47b0 YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja. 2021-04-23 17:38:47 +02:00
Szilard Nemeth 3b6b21e459 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 2021-04-23 16:17:29 +02:00
Szilard Nemeth 919daec36b YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak 2021-04-23 16:12:12 +02:00
Szilard Nemeth f76a2a7606 YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko 2021-04-23 16:07:58 +02:00
Akira Ajisaka 0b04c9694d
HADOOP-17655. Upgrade Jetty to 9.4.40. (#2947) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-23 19:52:41 +09:00
Mehakmeet Singh 6085f09db5
HADOOP-17471. ABFS to collect IOStatistics (#2731) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-23 10:28:31 +01:00
zhuxiangyi dff95c5eca
HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 10:19:11 +09:00
Vinayakumar B c4c0683dff
HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B. 2021-04-22 01:02:03 +05:30
Steve Loughran 5221322b96
HADOOP-17535. ABFS: ITestAzureBlobFileSystemCheckAccess test failure if no oauth key. (#2920) 
Contributed by Steve Loughran.
 2021-04-21 16:06:06 +01:00
Viraj Jasani acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
Eric Badger 6cb90005a7 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 2021-04-20 17:34:49 +00:00
Steve Loughran 2dd1e04010
HADOOP-17641. ITestWasbUriAndConfiguration failing. (#2937) 
This moves the mock account name --which is required to never exist-- from
"mockAccount"  to an account name containing a static UUID.

Contributed by Steve Loughran.
 2021-04-20 15:32:01 +01:00
litao 85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
Masatake Iwasaki e66d7d91be
HADOOP-17635. Update the ubuntu version in the build instruction. (#2931) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-19 16:44:16 +09:00
Viraj Jasani 9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-17 20:47:44 +09:00
litao af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
Eric Badger aed13f0f42 Revert "HDFS-15971. Make mkstemp cross platform (#2898)" 
This reverts commit b088f46b68.
 2021-04-16 17:15:33 +00:00
zhangshuyan0 14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Viraj Jasani f182798695
HADOOP-17633. Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs (#2895). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-04-16 12:36:01 +05:30
Akira Ajisaka f0241ec216
HDFS-15977. Call explicit_bzero only if it is available. (#2914) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-04-16 13:26:45 +09:00
lfengnan 17be99f9f9
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903) 2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Masatake Iwasaki 2717203f85
HADOOP-17569. Building native code fails on Fedora 33. (#2886) 2021-04-15 21:33:18 +09:00
Viraj Jasani 9179638017
HADOOP-17524. Remove EventCounter and Log counters from JVM Metrics (#2909) 
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-15 18:04:46 +09:00
Miklos Gergely ba3bc53f4e
YARN-10736. Fix GetApplicationsRequest JavaDoc. Contributed by Miklos Gergely. (#2906) 2021-04-14 17:16:30 +02:00
Ayush Saxena 7d6f0ca0bd Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." 
This reverts commit e565b05c80.

The removed method needs to be used in HDFS-15423.
 2021-04-13 23:28:42 +05:30
Gautham B A b088f46b68
HDFS-15971. Make mkstemp cross platform (#2898) 2021-04-13 08:54:45 -07:00
Akira Ajisaka 156ecc89be
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. (#2890) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-13 17:08:49 +09:00
Inigo Goiri 82462739f8
Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) 
This reverts commit cb3ed32fe0.
 2021-04-12 20:44:31 -07:00
billierinaldi c1fde4fe94
HADOOP-16948. Support infinite lease dirs. (#1925) 
* HADOOP-16948. Support single writer dirs.

* HADOOP-16948. Fix findbugs and checkstyle problems.

* HADOOP-16948. Fix remaining checkstyle problems.

* HADOOP-16948. Add DurationInfo, retry policy for acquiring lease, and javadocs

* HADOOP-16948. Convert ABFS client to use an executor for lease ops

* HADOOP-16948. Fix ABFS lease test for non-HNS

* HADOOP-16948. Fix checkstyle and javadoc

* HADOOP-16948. Address review comments

* HADOOP-16948. Use daemon threads for ABFS lease ops

* HADOOP-16948. Make lease duration configurable

* HADOOP-16948. Add error messages to test assertions

* HADOOP-16948. Remove extra isSingleWriterKey call

* HADOOP-16948. Use only infinite lease duration due to cost of renewal ops

* HADOOP-16948. Remove acquire/renew/release lease methods

* HADOOP-16948. Rename single writer dirs to infinite lease dirs

* HADOOP-16948. Fix checkstyle

* HADOOP-16948. Wait for acquire lease future

* HADOOP-16948. Add unit test for acquire lease failure
 2021-04-12 19:47:59 -04:00
lfengnan cb3ed32fe0
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605) 2021-04-12 12:42:33 -07:00
Akira Ajisaka 2bd810a507
HADOOP-17608. Fix TestKMS failure (#2880) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-04-12 12:53:58 +09:00
dependabot[bot] 6a9a5ae5fd
Bump y18n (#2843) 
Bumps [y18n](https://github.com/yargs/y18n) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases)
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md)
- [Commits](https://github.com/yargs/y18n/commits)

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-11 01:30:49 +09:00
Gautham B A 3148791da4
HDFS-15962. Make strcasecmp cross platform (#2883) 2021-04-09 09:01:33 -07:00
Akira Ajisaka 1448756505
MAPREDUCE-7334. TestJobEndNotifier fails. (#2877) 2021-04-08 23:53:51 -07:00
sumangala-patki 6f640abbaf
HADOOP-17576. ABFS: Disable throttling update for auth failures (#2761) 
Contributed by Sumangala Patki
 2021-04-09 09:31:23 +05:30
lichaojacobs 663ca14a76
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x (#2775) 2021-04-09 11:58:53 +09:00
Viraj Jasani 56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Eric Badger 213d3deb26 YARN-10503. Support queue capacity in terms of absolute resources with custom 
resourceType. Contributed by Qi Zhu.
 2021-04-09 00:34:15 +00:00
Gautham B A bf66116407
HDFS-15955. Make explicit_bzero cross platform (#2875) 2021-04-08 09:44:47 -07:00
Ayush Saxena c6539e3289
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-08 20:49:08 +05:30
Stephen O'Donnell 4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00