Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,437 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24437 Commits

Author SHA1 Message Date
Ayush Saxena 41b39ff292
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 
(cherry picked from commit 960c9ebaea)

Conflicts:
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java
 2021-05-13 12:14:09 +08:00
bilaharith 1c6c3920b6
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2842) 
Contributed by bilaharith
 2021-05-12 18:15:27 +01:00
dependabot[bot] b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63)
 2021-05-12 10:58:39 +09:00
Peter Bacsko 051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Viraj Jasani 3176ab028b
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b944084b32)
 2021-05-11 12:39:42 +09:00
Borislav Iordanov f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Ayush Saxena 43db94c7a3 HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:11:09 +05:30
sumangala-patki b20bc668d5
HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter (#2729) (#2781) 
Contributed by Sumangala Patki.

(cherry picked from commit fe633d4739)
 2021-05-10 11:50:01 +05:30
Ayush Saxena 2a95ba29bb HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li. 2021-05-09 04:00:30 +05:30
Akira Ajisaka 1326cad66a
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditsLogOpCodes (branch-3.3) (#2989). Contributed by Akira Ajisaka. 2021-05-08 10:48:56 +08:00
Szilard Nemeth 3d715c2e4c
YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 
(cherry picked from commit eacbe07b56)
 2021-05-07 14:32:17 +09:00
Szilard Nemeth 3303aa5947
YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 
(cherry picked from commit 8620984b8d)
 2021-05-07 14:26:58 +09:00
Mike 0f12f3e125
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists. 
Contributed by Mikhail Pryakhin.

(cherry picked from commit 017d24e970)
 2021-05-07 14:20:29 +09:00
Takanobu Asanuma df50e210dd
HADOOP-17375. Fix the error of TestDynamometerInfra. (#2471) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 207210263a)
 2021-05-07 14:09:03 +09:00
hchaverr cedebf1c27 HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 
(cherry picked from commit f40e3eb059)
 2021-05-06 17:50:32 -07:00
Akira Ajisaka 217655269a YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc1)
 2021-05-06 23:47:24 +09:00
Viraj Jasani be34c1222a HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ (#2973) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b93e448f9a)
 2021-05-06 04:53:06 +09:00
Takanobu Asanuma 65bf544118 HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina. 
(cherry picked from commit b89d875f7b)
 2021-05-04 17:44:34 +01:00
Steve Loughran e944cc0338
HADOOP-16742. NullPointerException in S3A MultiObjectDeleteSupport 
Contributed by Tor Arvid Lund.

Change-Id: Iadfe9b2f355cf373031075bfbe681705a2c65bdc
 2021-05-04 16:15:13 +01:00
Istvan Fajth 00ba886a22 HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException (#2965) 
(cherry picked from commit 0d78d73973)
 2021-05-04 11:36:15 +01:00
Karthik Palanisamy 00c3f3c42f HDFS-15865. Interrupt DataStreamer thread if no ack (#2728) 
(cherry picked from commit bd3da73a0f)
 2021-05-04 11:11:36 +01:00
kishendas 98aa4fc32c
HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported (#2949) 
Co-authored-by: Kishen Das <kishen@cloudera.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit e571025f5b)
 2021-05-04 16:35:00 +08:00
Viraj Jasani 658bb49e66
HADOOP-17676. Restrict imports from org.apache.curator.shaded (#2969) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 62bcc79a2c)
 2021-05-04 01:11:30 +09:00
Akira Ajisaka 25244c4eec
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970) 
(cherry picked from commit 7c561e2a02)
 2021-05-04 00:26:34 +09:00
lfengnan 43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c)
 2021-05-02 19:19:55 +09:00
Wei-Chiu Chuang 670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029)
 2021-05-02 11:12:37 +09:00
Hui Fei 9aa6106689 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151)
 2021-04-29 12:10:04 -07:00
Eric Badger da2ebfa8a9 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 
(cherry picked from commit 803ac4b1a0)
 2021-04-29 17:52:03 +00:00
litao 0e9042fed5 HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit b968fa0957)
 2021-04-28 18:52:05 +09:00
Hui Fei 842e05ba53 HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 (#2495) 
(cherry picked from commit 68442b48c8)
 2021-04-28 18:51:50 +09:00
Ayush Saxena f809b863fe HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena. 
(cherry picked from commit 6e416a83d1)
 2021-04-28 18:51:43 +09:00
Ayush Saxena 3c7433f2a1 HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 
(cherry picked from commit 8d49229c37)
 2021-04-28 18:51:37 +09:00
Ayush Saxena 187b4a63a4 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6)
 2021-04-27 23:18:12 +09:00
litao f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 (#2940) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-27 22:39:25 +09:00
bilaharith 6649e5888b
HADOOP-17536. ABFS: Supporting customer provided encryption key (#2707) 
Contributed by bilahari t h

Change-Id: I86216e755b81e9d14f5e87844d9fd58e8940560c
 2021-04-27 13:16:33 +01:00
Wei-Chiu Chuang 99e4e9bad6
HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2956) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: I7de255cb9a167e3a5e25ae339b050e82cb4b7c30
(cherry picked from commit 64264d726d)
 2021-04-27 10:43:18 +09:00
Wei-Chiu Chuang df29aaaf80
YARN-10752. Shaded guava not found when compiling with profile hbase2.0. (#2957) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ic9943d18d4086b075ceed7924dcfa566653a16cb
 2021-04-27 00:41:42 +09:00
Stephen O'Donnell f6efb58b07 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c29)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-04-26 11:23:56 +01:00
Wei-Chiu Chuang 1425512032 Revert "HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952)" 
This reverts commit d525e1e33d.
 2021-04-26 16:42:44 +08:00
Wei-Chiu Chuang d525e1e33d HADOOP-17661. mvn versions:set fails to parse pom.xml. (#2952) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 66dda86fec)
 2021-04-26 14:07:35 +08:00
Takanobu Asanuma a5f038b3f6 HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 
(cherry picked from commit 01bad0e92a)
 2021-04-26 13:31:51 +09:00
Brahma Reddy Battula 99dfd3b2d0 HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2621d3f15b)
 2021-04-26 11:40:16 +08:00
Steve Loughran 8308aab658
HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

Change-Id: I2746aabcfeb0fbb138a80b02c4d5bbf2a8cf75da
 2021-04-25 18:43:25 +01:00
Steve Loughran fb71e6c91e
HADOOP-17597. Optionally downgrade on S3A Syncable calls (#2801) 
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.

Adds a new option fs.s3a.downgrade.syncable.exceptions

When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter

With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
  JIRA.

Contributed by Steve Loughran.

Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
 2021-04-24 18:32:39 +01:00
Gabor Bota 1b2bc77923
HADOOP-17454. [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0 (#2593) 
Also fixes HADOOP-16995. ITestS3AConfiguration proxy tests failures when bucket probes == 0
The improvement should include the fix, because the test would fail by default otherwise.

Change-Id: I9a7e4b5e6d4391ebba096c15e84461c038a2ec59
 2021-04-24 18:28:21 +01:00
Mukund Thakur 33f9ceb3cc HADOOP-17136. ITestS3ADirectoryPerformance.testListOperations failing (#2153) 
A regression caused by HADOOP-17022: the reduction in LIST calls broken an assertion.

Contributed by Mukund Thakur

Change-Id: Ib9725165906931634567fd1f62a81e3a6ea5620c
 2021-04-24 18:28:14 +01:00
Mehakmeet Singh 389d3034c6
HADOOP-17471. ABFS to collect IOStatistics (#2731) (#2950) 
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.

This includes the min/max/mean durations of all REST API calls.

Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
 2021-04-24 17:59:26 +01:00
Steve Loughran 77fddcfcb1
HADOOP-17535. ABFS: ITestAzureBlobFileSystemCheckAccess test failure if no oauth key. (#2920) 
Contributed by Steve Loughran.

Change-Id: I165f5ed3a8486404403827b5c0338cf7f80c2bb1
 2021-04-24 17:24:15 +01:00
Eric Badger 003deeeecf YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 22:14:57 +00:00
Jim Brennan 3b1cde2f42 YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 
(cherry picked from commit 1cbe35946b)
 2021-04-23 21:38:02 +00:00