4b5b54c73f
HADOOP-17089: WASB: Update azure-storage-java SDK
...
Contributed by Thomas Marquardt
DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.
See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.
TESTS: A new test was added to validate the fix. All tests are passing:
wasb:
mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 248, Failures: 0, Errors: 0, Skipped: 11
Tests run: 651, Failures: 0, Errors: 0, Skipped: 65
abfs:
mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 64, Failures: 0, Errors: 0, Skipped: 0
Tests run: 437, Failures: 0, Errors: 0, Skipped: 33
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
2020-06-25 02:32:42 +00:00
84110d850e
HDFS-15383. RBF: Add support for router delegation token without watch ( #2047 )
...
Improving router's performance for delegation tokens related operations. It achieves the goal by removing watchers from router on tokens since based on our experience. The huge number of watches inside Zookeeper is degrading Zookeeper's performance pretty hard. The current limit is about 1.2-1.5 million.
2020-06-23 13:12:29 -07:00
03f855e3e7
YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko
2020-06-23 12:13:04 +02:00
fa14e4bc00
HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow.
2020-06-23 16:59:51 +08:00
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
2020-06-23 01:42:25 -07:00
201d734af3
HDFS-15428. Javadocs fails for hadoop-federation-balance. Contributed by Xieming Li.
2020-06-22 19:43:19 +09:00
ce1008fe61
YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios
...
Contributed by Szilard Nemeth. Reviewed by Gergely Pollak.
2020-06-22 12:49:14 +05:30
17ffcab5f6
HDFS-14546. Document block placement policies. Contributed by Amithsha.
2020-06-21 17:27:28 +05:30
b2facc84a1
YARN-9460. QueueACLsManager and ReservationsACLManager should not use instanceof checks. Contributed by Bilwa S T.
2020-06-20 20:09:31 +05:30
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
2020-06-20 00:32:02 -07:00
8fd0fdf889
MAPREDUCE-7281. Fix NoClassDefFoundError on 'mapred minicluster'. ( #2077 )
2020-06-20 07:37:55 +09:00
100ec8e870
HADOOP-17009: Embrace Immutability of Java Collections
2020-06-19 10:23:43 -07:00
3472c3efc0
HADOOP-17065. Add Network Counters to ABFS ( #2056 )
...
Contributed by Mehakmeet Singh.
2020-06-19 14:03:49 +01:00
469841446f
YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko
2020-06-19 14:50:24 +02:00
9821b94c94
HADOOP-16888. [JDK11] Support JDK11 in the precommit job ( #2012 )
...
* Install JDK 11 in the docker image
* Refactor Dockerfile
* Add compile test with JDK11 via Yetus multijdk feature
2020-06-19 13:29:44 +09:00
d50e93ce7b
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-18 06:45:15 -07:00
edf716a5c3
Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell."
...
This reverts commit 730a39d138
2020-06-18 06:45:15 -07:00
123777823e
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
2020-06-18 12:26:22 +01:00
9cbd76cc77
HDFS-15346. FedBalance tool implementation. Contributed by Jinglun.
2020-06-18 13:33:25 +08:00
caf3995ac2
HADOOP-17076: ABFS: Delegation SAS Generator Updates
...
Contributed by Thomas Marquardt.
DETAILS:
1) The authentication version in the service has been updated from Dec19 to Feb20, so need to update the client.
2) Add support and test cases for getXattr and setXAttr.
3) Update DelegationSASGenerator and related to use Duration instead of int for time periods.
4) Cleanup DelegationSASGenerator switch/case statement that maps operations to permissions.
5) Cleanup SASGenerator classes to use String.equals instead of ==.
TESTS:
Added tests for getXAttr and setXAttr.
All tests are passing against my account in eastus2euap:
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 76, Failures: 0, Errors: 0, Skipped: 0
Tests run: 441, Failures: 0, Errors: 0, Skipped: 33
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
2020-06-18 02:07:08 +00:00
89689c52c3
YARN-10308. Update javadoc and variable names for YARN service.
...
Contributed by Bilwa S T via eyang
2020-06-17 09:04:26 -07:00
2bfb22840a
HADOOP-17020. Improve RawFileSystem Performance ( #2063 )
...
Contributed by : Mehakmeet Singh
Co-authored-by: Rajesh Balamohan
Co-authored-by: Mehakmeet Singh
2020-06-17 16:15:26 +01:00
5b1a56f9f1
YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak
2020-06-17 14:34:40 +02:00
c8ed33cd2a
HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.
2020-06-17 13:55:40 +05:30
fc4ebb0499
YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars ( #2075 )
2020-06-17 09:26:41 +05:30
730a39d138
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-16 15:59:13 -07:00
81d8a887b0
SPNEGO TLS verification
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-06-15 10:56:16 +09:00
f41a144077
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
2020-06-15 09:15:53 +09:00
719b53a79d
HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.
2020-06-13 09:35:05 -07:00
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
2020-06-12 14:32:19 -07:00
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:16:33 +05:30
7c4de59fc1
YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph.
2020-06-12 22:10:25 +08:00
fed6fecd3a
YARN-10312. Add support for yarn logs -logFile to retain backward compatibility.
...
Contributed by Jim Brennan.
2020-06-11 21:02:41 +00:00
93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G.
2020-06-10 15:00:02 -07:00
b735a77717
HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.
2020-06-10 12:06:16 +05:30
635e6a16d0
HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi.
2020-06-10 11:50:37 +05:30
56247db302
YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger).
2020-06-09 18:43:16 +00:00
ac5d899d40
HADOOP-17050 S3A to support additional token issuers
...
Contributed by Steve Loughran.
S3A delegation token providers will be asked for any additional
token issuers, an array can be returned,
each one will be asked for tokens when DelegationTokenIssuer collects
all the tokens for a filesystem.
2020-06-09 14:39:06 +01:00
8525874561
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:53:33 +05:30
fbb8775430
Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
...
This reverts commit e2322e1117
2020-06-08 20:35:27 +00:00
0c25131ca4
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit
2020-06-08 11:28:56 -07:00
9f242c215e
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina
2020-06-08 10:11:30 -07:00
a8610c15c4
HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena.
2020-06-07 13:29:10 -07:00
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
2020-06-06 08:11:57 -07:00
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
2020-06-05 22:19:38 -07:00
e7dd02768b
HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das.
2020-06-05 14:56:51 -07:00
76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
2020-06-05 10:58:21 -07:00
2326123705
HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.
2020-06-05 19:15:05 +05:30
8abff5151a
YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke
2020-06-05 15:00:39 +02:00
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
2020-06-05 11:11:49 +01:00
Thomas Marquardt
lfengnan
Szilard Nemeth
He Xiaoqiao
Uma Maheswara Rao G
Akira Ajisaka
Prabhu Joseph
Ayush Saxena
Surendra Singh Lilhore
Masatake Iwasaki
belugabehr
Mehakmeet Singh
Stephen O'Donnell
Wei-Chiu Chuang
S O'Donnell
Yiqun Lin
Eric Yang
Vinayakumar B
Takanobu Asanuma
Inigo Goiri
Tao Yang
Eric Badger
Eric E Payne
Steve Loughran
Mingliang Liu
Abhishek Das
Toshihiro Suzuki