Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,301 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24301 Commits

Author SHA1 Message Date
bilaharith 51598d8b1b
HADOOP-17183. ABFS: Enabling checkaccess on ABFS (#2331) 
Contributed by Bilahari TH
 2020-10-01 21:29:05 +01:00
Adam Antal bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
maobaolong d68d2a5c1e
HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils (#2341) 2020-09-30 14:22:55 -07:00
He Xiaoqiao a490d87eb7 HADOOP-17280. Service-user cost shouldn't be accumulated to totalDecayedCallCost and totalRawCallCost. Contributed by Jinglun. 2020-09-30 12:12:52 +08:00
Xiaoyu Yao 4c5ad57818
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334) 2020-09-29 20:15:27 -07:00
huangtianhua a89ca56a1b
HDFS-15600 Fix TestRouterQuota#testStorageTypeQuota (#2347) 2020-09-29 19:00:55 -07:00
Szilard Nemeth a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Karen Coppage 75d10f8499
HADOOP-17267. Add debug-level logs in Filesystem.close() (#2321) 
When a filesystem is closed, the FileSystem log will, at debug level,
log the method calling close/closeAll.

At trace level: the full calling stack.

Contributed by Karen Coppage.
 2020-09-29 16:06:50 +01:00
Wanqiang Ji 143bdd4188
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338) 2020-09-29 23:06:25 +09:00
Ayush Saxena d686a655bc HDFS-15603. RBF: Fix getLocationsForPath twice in create operation. Contributed by wangzhaohui. 2020-09-29 00:03:11 +05:30
Ayush Saxena fc8a6dd8a9 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 2020-09-28 23:55:31 +05:30
Fei Hui 984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345) 2020-09-28 09:54:01 -07:00
Inigo Goiri eae0035a2d HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao. 2020-09-28 09:40:39 -07:00
Takanobu Asanuma d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Vinayakumar B 82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Aryan Gupta bbbfa7d415
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 2020-09-27 03:18:01 +09:00
Uma Maheswara Rao G 899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00
Ye Ni 00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332) 2020-09-25 09:47:54 -07:00
Shubham Gupta e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
tkuramoto33 3fc007a37b
HADOOP-17282. libzstd-dev should be used instead of libzstd1-dev on Ubuntu 18.04 or higher. (#2336) 2020-09-25 14:17:50 +09:00
Uma Maheswara Rao G 3ccc962b99
HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-24 07:07:48 -07:00
Aryan Gupta 486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-09-24 21:15:55 +09:00
huangtianhua ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
bshashikant 368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326) 2020-09-24 14:00:41 +05:30
Sneha Vijayarajan c3a90dd918
HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. 
Contributed by Sneha Vijayarajan

Testing:

namespace.enabled=false
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 246
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 33
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-09-23 15:59:00 +00:00
Adam Antal 7ec16ac7f6 YARN-10443. Document options of logs CLI. Contributed by Ankit Kumar. 
Fixes #2325
 2020-09-23 17:42:56 +02:00
tkuramoto33 18f9d77c46
YARN-6754. Fair scheduler docs should explain meaning of weight=0 for a queue. (#2300) 2020-09-23 14:50:28 +09:00
Ye Ni ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted (#2274) 2020-09-22 13:23:20 -07:00
Steve Loughran 7fae4133e0
HADOOP-17261. s3a rename() needs s3:deleteObjectVersion permission (#2303) 
Contributed by Steve Loughran.
 2020-09-22 17:22:04 +01:00
Hui Fei 474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
Kihwal Lee dfc2682213 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 2020-09-22 10:53:04 -05:00
Xiaoyu Yao 6b5d9e2334
HADOOP-17259. Allow SSLFactory fallback to input config if ssl-*.xml … (#2301) 2020-09-21 12:41:06 -07:00
crossfire c8c1cc43d3
HADOOP-17088.Failed to load XInclude files with relative path. (#2097) 
Contributed by Yushi Hayasaka.
 2020-09-21 19:10:51 +01:00
lfengnan 3e8b1e7426
HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266). Contributed by Fengnan Li. 2020-09-21 23:24:13 +05:30
Mukund Thakur 83c7c2b4c4
HADOOP-17023. Tune S3AFileSystem.listStatus() (#2257) 
S3AFileSystem.listStatus() is optimized for invocations
where the path supplied is a non-empty directory.
The number of S3 requests is significantly reduced, saving
time, money, and reducing the risk of S3 throttling.

Contributed by Mukund Thakur.
 2020-09-21 17:20:16 +01:00
Ayush Saxena 7a6265ac42 HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 2020-09-21 08:50:08 +05:30
zz 95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
Ayush Saxena 8eba922bb3 HDFS-15579. RBF: The constructor of PathLocation may got some misunderstanding. Contributed by Janus Chow. 2020-09-20 08:54:25 +05:30
Masatake Iwasaki 364b958085
HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-09-20 00:47:02 +09:00
Sneha Vijayarajan e31a636e92
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-09-19 01:28:44 +00:00
ThomasMarquardt 0dc54d0247
HADOOP-17203: Revert HADOOP-17183. ABFS: Enabling checkaccess on ABFS 
This reverts commit a2610e21ed.
 2020-09-18 17:52:11 -07:00
David Tucker fc2435cb5c
HADOOP-15136. Correct typos in filesystem.md (#2314) 
Contributed by David Tucker
 2020-09-18 18:28:14 +01:00
Ayush Saxena 2377df38ad HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G 7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:18:10 +05:30
Szilard Nemeth eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Xiaoyu Yao 6adf8462ba
HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances. (#2259) 2020-09-17 10:39:19 -07:00
S O'Donnell 20a0e6278d HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-17 14:11:42 +01:00
bshashikant 425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296) 2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-16 22:43:00 -07:00
Wanqiang Ji ce86183691
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 2020-09-16 16:23:57 +09:00