Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,037 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

18037 Commits

Author SHA1 Message Date
Yiqun Lin b46ec61909 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2020-12-07 18:54:32 +08:00
Adam Antal 3101299f02 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760df)
 2020-10-05 11:09:48 +02:00
Steve Loughran 105fd92e82 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: I72c9ee96e11880ed8ecbb9d3427ac02ff95249e8
 2020-09-02 16:20:08 +01:00
Toshihiro Suzuki fe89969a9d HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit e5a02a7499)
 2020-06-05 12:51:40 +01:00
S O'Donnell 0d3332037f HADOOP-14698. Make copyFromLocals -t option available for put as well. Contributed by Andras Bokor. 
(cherry picked from commit 1f674e26ba)
 2020-06-05 12:49:32 +01:00
S O'Donnell b7b3164ce7 Revert "HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein." 
This reverts commit ed59412434.
 2020-06-05 12:48:07 +01:00
Steve Loughran cb94911252 Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-15 13:51:45 +01:00
Steve Loughran 31c1a24426 Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf49.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-15 13:51:44 +01:00
Weiwei Yang a55f96c053 HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71e)
 2020-03-09 14:44:10 -07:00
John Zhuge def9a04087
HADOOP-14206. TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature. Contributed by Jim Brennan. 
(cherry picked from commit 999096d82e)
 2020-03-08 00:02:33 -08:00
Inigo Goiri da4d0d2982 YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d)
 2020-02-27 13:20:56 -08:00
Inigo Goiri ed59412434 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f6)
 2020-01-31 11:26:32 -08:00
Inigo Goiri 380fefc40e Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 0b75c5238b.
 2019-12-20 11:28:17 -08:00
Inigo Goiri 0b75c5238b HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6)
 2019-12-19 09:48:02 -08:00
Mingliang Liu 109a2f513f
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:55:29 -08:00
Mingliang Liu cb63c3cc0d
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:53:04 -08:00
Lukas Majercak 049d061001 HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log. 
MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.

(cherry picked from commit c73334a924)
 2019-11-15 13:27:36 -08:00
Duo Zhang 322cef7762 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:57:26 -07:00
Eric Badger 5196b035f7 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 18:17:47 +00:00
Eric Badger fdd5ce1720 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:18:16 +00:00
Xiaoyu Yao 0b80308b78
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao 121c6cb59e
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 5369a3bd94.
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao 5369a3bd94
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:44:29 -07:00
Eric E Payne 97ee0af90e YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec)
 2019-08-28 21:16:38 +00:00
Jonathan Hung c98f7cef9f YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca1)
(cherry picked from commit e4249c3202)
(cherry picked from commit 6baa0d1e4d)
 2019-08-26 16:04:57 -07:00
Shashikant Banerjee 5b064381aa HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ab7509efe7)
 2019-08-19 10:07:16 -07:00
Surendra Singh Lilhore 0cb3738567 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
(cherry picked from commit 6b01effd01)
 2019-08-18 18:18:02 -07:00
Christopher Gregorian ccb41000f1 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628)
(cherry picked from e4b650f91e)
 2019-08-14 13:37:21 -07:00
Eric Badger db88224e8f YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
 2019-08-13 17:19:57 +00:00
Chao Sun 4c06b2f96c HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
(cherry picked from 90447baeb0)
 2019-08-08 10:47:33 -07:00
Erik Krogen 5657e45fb2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
(cherry picked from f6ce2f4a50)
 2019-08-07 09:34:01 -07:00
Yiqun Lin ea5330643c HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:05:09 +08:00
Haibo Chen 6109ecf16d YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d539)
(cherry picked from commit 8d357343c4)
(cherry picked from commit f943bff254)
 2019-08-06 15:08:56 -07:00
Erik Krogen b0c5151959 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
(cherry picked from 69e3ac5cbb)
 2019-08-06 09:49:56 -07:00
Weiwei Yang 373e32480b YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23)
(cherry picked from commit 70efe253f3)
 2019-08-05 10:02:33 -07:00
Weiwei Yang fd977a138a YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b67)
(cherry picked from commit 7dd8eafe33)
 2019-08-05 10:02:33 -07:00
Wei Yan ea345de738 YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat. 
(cherry picked from commit 333ef303ff)
 2019-08-05 10:02:33 -07:00
Wangda Tan d18ec19ed8 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
(cherry picked from commit 46baafedf1)
 2019-08-05 10:02:33 -07:00
Eric Badger 01f76b4f45 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3)
 2019-08-02 21:01:15 +00:00
Erik Krogen 055f824bcb HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263)
 2019-08-02 11:52:26 -07:00
Christopher Gregorian ec00431eaa HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
 2019-07-30 14:46:11 -07:00
Eric E Payne c04ccf296d YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 2019-07-30 20:00:32 +00:00
Inigo Goiri 998593e7f2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f)
(cherry picked from commit fe40fbbd4d)
 2019-07-29 14:39:20 -07:00
Erik Krogen f7e517ee24 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb63006)
(cherry-picked from 5ec34bf43a)
(cherry-picked from e2aab0efef9d504e4695a9c1df2835ed60f6c0c7)
 2019-07-26 11:43:34 -07:00
Wei-Chiu Chuang b5303012a0 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-25 10:07:06 -07:00
Erik Krogen 32d80d2201 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen 11fee2d4e1 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen 16a4043f01 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen 2598282293 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen 45cd309a40 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00