Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,145 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24145 Commits

Author SHA1 Message Date
belugabehr 100ec8e870
HADOOP-17009: Embrace Immutability of Java Collections 2020-06-19 10:23:43 -07:00
Mehakmeet Singh 3472c3efc0
HADOOP-17065. Add Network Counters to ABFS (#2056) 
Contributed by Mehakmeet Singh.
 2020-06-19 14:03:49 +01:00
Szilard Nemeth 469841446f YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-19 14:50:24 +02:00
Akira Ajisaka 9821b94c94
HADOOP-16888. [JDK11] Support JDK11 in the precommit job (#2012) 
* Install JDK 11 in the docker image
* Refactor Dockerfile
* Add compile test with JDK11 via Yetus multijdk feature
 2020-06-19 13:29:44 +09:00
Stephen O'Donnell d50e93ce7b HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-18 06:45:15 -07:00
Wei-Chiu Chuang edf716a5c3 Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell." 
This reverts commit 730a39d138.
 2020-06-18 06:45:15 -07:00
S O'Donnell 123777823e HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina 2020-06-18 12:26:22 +01:00
Yiqun Lin 9cbd76cc77 HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
Thomas Marquardt caf3995ac2
HADOOP-17076: ABFS: Delegation SAS Generator Updates 
Contributed by Thomas Marquardt.

DETAILS:
1) The authentication version in the service has been updated from Dec19 to Feb20, so need to update the client.
2) Add support and test cases for getXattr and setXAttr.
3) Update DelegationSASGenerator and related to use Duration instead of int for time periods.
4) Cleanup DelegationSASGenerator switch/case statement that maps operations to permissions.
5) Cleanup SASGenerator classes to use String.equals instead of ==.

TESTS:
Added tests for getXAttr and setXAttr.

All tests are passing against my account in eastus2euap:

 $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
 Tests run: 76, Failures: 0, Errors: 0, Skipped: 0
 Tests run: 441, Failures: 0, Errors: 0, Skipped: 33
 Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
 2020-06-18 02:07:08 +00:00
Eric Yang 89689c52c3 YARN-10308. Update javadoc and variable names for YARN service. 
Contributed by Bilwa S T via eyang
 2020-06-17 09:04:26 -07:00
Mehakmeet Singh 2bfb22840a
HADOOP-17020. Improve RawFileSystem Performance (#2063) 
Contributed by : Mehakmeet Singh

Co-authored-by: Rajesh Balamohan
Co-authored-by: Mehakmeet Singh
 2020-06-17 16:15:26 +01:00
Szilard Nemeth 5b1a56f9f1 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:34:40 +02:00
Ayush Saxena c8ed33cd2a HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor. 2020-06-17 13:55:40 +05:30
Vinayakumar B fc4ebb0499
YARN-10314. YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars (#2075) 2020-06-17 09:26:41 +05:30
Stephen O'Donnell 730a39d138 HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-06-16 15:59:13 -07:00
Eric Yang 81d8a887b0
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-06-15 10:56:16 +09:00
Takanobu Asanuma f41a144077 HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina. 2020-06-15 09:15:53 +09:00
Inigo Goiri 719b53a79d HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 2020-06-13 09:35:05 -07:00
Uma Maheswara Rao G 785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G. 2020-06-12 14:32:19 -07:00
Vinayakumar B e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Tao Yang 7c4de59fc1 YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph. 2020-06-12 22:10:25 +08:00
Eric Badger fed6fecd3a YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan.
 2020-06-11 21:02:41 +00:00
Uma Maheswara Rao G 93b121a971
HADOOP-17060. Clarify listStatus and getFileStatus behaviors inconsistent in the case of ViewFs implementation for isDirectory. Contributed by Uma Maheswara Rao G. 2020-06-10 15:00:02 -07:00
Ayush Saxena b735a77717 HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang. 2020-06-10 12:06:16 +05:30
Ayush Saxena 635e6a16d0 HDFS-15376. Update the error about command line POST in httpfs documentation. Contributed by bianqi. 2020-06-10 11:50:37 +05:30
Eric E Payne 56247db302 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 2020-06-09 18:43:16 +00:00
Steve Loughran ac5d899d40
HADOOP-17050 S3A to support additional token issuers 
Contributed by Steve Loughran.

S3A delegation token providers will be asked for any additional
token issuers, an array can be returned,
each one will be asked for tokens when DelegationTokenIssuer collects
all the tokens for a filesystem.
 2020-06-09 14:39:06 +01:00
Ayush Saxena 8525874561 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 
*Added missed test file.
 2020-06-09 18:53:33 +05:30
Eric Badger fbb8775430 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit e2322e1117.
 2020-06-08 20:35:27 +00:00
Mingliang Liu 0c25131ca4
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:28:56 -07:00
Mingliang Liu 9f242c215e
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-08 10:11:30 -07:00
Ayush Saxena a8610c15c4
HDFS-15396. Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDir. Contributed by Ayush Saxena. 2020-06-07 13:29:10 -07:00
Uma Maheswara Rao G 3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G. 2020-06-06 08:11:57 -07:00
Ayush Saxena cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena 2020-06-05 22:19:38 -07:00
Abhishek Das e7dd02768b
HADOOP-17029. Return correct permission and owner for listing on internal directories in ViewFs. Contributed by Abhishek Das. 2020-06-05 14:56:51 -07:00
Uma Maheswara Rao G 76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G. 2020-06-05 10:58:21 -07:00
Ayush Saxena 2326123705 HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena. 2020-06-05 19:15:05 +05:30
Szilard Nemeth 8abff5151a YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-05 15:00:39 +02:00
Toshihiro Suzuki 545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.
 2020-06-05 11:11:49 +01:00
Szilard Nemeth 8b146c17b3 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-05 11:38:32 +02:00
Szilard Nemeth a4835db95a YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-05 10:11:40 +02:00
Akira Ajisaka 5157118bd7
HADOOP-17056. Addendum patch: Fix typo 2020-06-04 16:34:46 +09:00
Akira Ajisaka 704409d53b
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 2020-06-04 06:04:26 +09:00
Steve Loughran 40d63e02f0
HADOOP-16568. S3A FullCredentialsTokenBinding fails if local credentials are unset. (#1441) 
Contributed by Steve Loughran.

Move the loading to deployUnbonded (where they are required) and add a safety check when a new DT is requested
 2020-06-03 17:07:00 +01:00
Mike 97c98ce531
HADOOP-14566. Add seek support for SFTP FileSystem. (#1999) 
Contributed by Mikhail Pryakhin
 2020-06-03 11:37:40 +01:00
Akira Ajisaka 9c290c08db
HADOOP-17056. shelldoc fails in hadoop-common. (#2045) 
In the docker build image, skip GPG verification when downloading
Yetus tarball via yetus-wrapper.
 2020-06-03 18:01:37 +09:00
Ayush Saxena e8cb2ae409 HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 2020-06-03 12:47:15 +05:30
Ayush Saxena f6453244ab HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan. 2020-06-03 12:16:36 +05:30
Xiaoyu Yao 6288e15118 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 2020-06-02 11:53:32 -07:00
Uma Maheswara Rao G ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G. 2020-06-02 11:09:26 -07:00