Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,783 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

23783 Commits

Author SHA1 Message Date
Takanobu Asanuma c44d8b8bdf HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:33:39 +09:00
Szilard Nemeth 9ce913a7bf FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e)
 2020-05-22 14:48:53 +02:00
Masatake Iwasaki 4d30c395f7 HADOOP-17040. Fix intermittent failure of ITestBlockingThreadPoolExecutorService. (#2020) 
(cherry picked from commit 9685314633)
 2020-05-22 18:53:53 +09:00
Akira Ajisaka 6b54f259e7
HADOOP-17049. javax.activation-api and jakarta.activation-api define overlapping classes (#2027) 
* Removed javax.activation-api from dependency

(cherry picked from commit 52b21de1d8)
 2020-05-22 11:20:16 +09:00
Eric Yang d7cf19d7c0 YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts. 
Contributed by Bilwa S T via eyang
 2020-05-20 09:31:40 -07:00
Chen Liang e452163a06
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 
(cherry picked from commit 7bb902bc0d)
 2020-05-19 09:52:25 +09:00
He Xiaoqiao eb045ea056 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 09:59:12 -07:00
Ayush Saxena 3915d1afc7 HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He. 
(cherry picked from commit 178336f8a8)
 2020-05-18 09:58:59 -07:00
Wei-Chiu Chuang 240cba7e6d HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
(cherry picked from commit 2abcf7762a)
 2020-05-18 09:41:11 -07:00
Wei-Chiu Chuang 289f8acc64 Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016)" 
This reverts commit ad9a6a0ee3.
 2020-05-18 09:41:03 -07:00
Wei-Chiu Chuang 53d22fdb88 Revert "HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He." 
This reverts commit acae31aa28.
 2020-05-18 09:39:57 -07:00
Wei-Chiu Chuang 032ccba67c HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun. 
(cherry picked from commit 50caba1a92)
 2020-05-18 08:42:20 -07:00
He Xiaoqiao acae31aa28 HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a3f44dacc1)
 2020-05-18 07:10:27 -07:00
pustota2009 ad9a6a0ee3 HDFS-15202 Boost short circuit cache (rebase PR-1884) (#2016) 
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.

(cherry picked from commit 86e6aa8eec)
 2020-05-18 07:05:03 -07:00
Akira Ajisaka 77587ffb1e
HADOOP-17042. Hadoop distcp throws 'ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found'. Contributed by Aki Tanaka. 
(cherry picked from commit 27601fc79e)
 2020-05-18 15:37:22 +09:00
Steve Loughran f2f727359b Revert "HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default)." 
This reverts commit 44350fdf49.

It is related to the rollback of HADOOP-8143.

Change-Id: If48e3dd670c920ada702dc36461ff398fe9d35cc
 2020-05-14 19:20:34 +01:00
Steve Loughran 7f2cf334a8
Revert "HADOOP-8143. Change distcp to have -pb on by default." 
This reverts commit dd65eea74b.

Change-Id: I74180cf59d5bbad8c9f66cb331535addcbea863e
 2020-05-14 19:14:39 +01:00
Surendra Singh Lilhore 7cef7109de YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:17:13 +05:30
Ayush Saxena 74887ab2ad YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:09 +05:30
Akira Ajisaka ee58c112d0
Revert "SPNEGO TLS verification" 
This reverts commit 0f27c04c23.
 2020-05-13 17:14:51 +09:00
Jonathan Hung 564d3211f2 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c122)
 2020-05-12 10:49:10 -07:00
S O'Donnell 433aaeefa4 HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun. 2020-05-12 15:25:05 +01:00
Ayush Saxena ef41229f70 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:43:37 +05:30
Szilard Nemeth 95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Babneet Singh d1e5e393c3 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
 2020-05-11 10:45:57 -07:00
Ayush Saxena 61a4cd5539 HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun. 2020-05-11 16:52:23 +05:30
Akira Ajisaka 282427f6d1
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003) 
(cherry picked from commit 328eae9a14)
 2020-05-11 14:46:43 +09:00
Ayush Saxena 6ccb1cd5f1 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:44:02 +05:30
Akira Ajisaka 763a79916d
HDFS-15343. TestConfiguredFailoverProxyProvider is failing. (#2001) 
(cherry picked from commit c784ba370e)
 2020-05-08 17:19:48 +09:00
Ayush Saxena 817814105a HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:06 +05:30
Masatake Iwasaki 6a0aeea60f HADOOP-17025. Fix invalid metastore configuration in S3GuardTool tests. (#1994) 
(cherry picked from commit 99840aaba6)
 2020-05-07 12:02:15 +09:00
Uma Maheswara Rao G edf52d29f1 HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-06 15:13:33 -07:00
Ahmed Hussein 7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:28:04 -05:00
Ayush Saxena 4e0d99c4d6 HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun. 2020-05-06 17:02:40 +05:30
Wei-Chiu Chuang 78c97907cb HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. (#1998) 
Reviewed-by: Arpit Agarwal <arp@apache.org>
(cherry picked from commit adecdb8b53)
 2020-05-05 21:25:56 -07:00
Konstantin V Shvachko 8809b43fb9 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:45:18 -07:00
Ahmed Hussein 1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:24:01 -05:00
Ayush Saxena df16146e7f Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:17:14 +05:30
Akira Ajisaka e32e1384d9
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter (#1992) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-02 13:59:58 -07:00
Eric Yang 0f27c04c23
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:51:12 +09:00
Ayush Saxena ccebc9d9d0 HADOOP-16952. Add .diff to gitignore. Contributed by Ayush Saxena. 2020-05-01 14:43:34 +05:30
Akira Ajisaka dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
S O'Donnell a34174fea3 HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun. 
(cherry picked from commit 9ca6298a9a)
 2020-04-29 16:01:48 +01:00
Szilard Nemeth 8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth 75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Aryan Gupta 08dbe32122 HDFS-15301. statfs function in hdfs-fuse not working. Contributed by Aryan Gupta.(#1980) 
(cherry picked from commit 816042e62b)
 2020-04-29 11:16:56 +05:30
Szilard Nemeth e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Akira Ajisaka d504574d82
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:18:25 +09:00
Akira Ajisaka 62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba)
 2020-04-28 05:45:37 +09:00
Eric E Payne 741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles). 
(cherry picked from commit e2322e1117)
 2020-04-27 19:34:38 +00:00