Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,143 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5623 Commits

Author SHA1 Message Date
Petre Bogdan Stolojan a218038960
HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem (#3101) 
This work
* Defines the behavior of FileSystem.copyFromLocal in filesystem.md
* Implements a high performance implementation of copyFromLocalOperation
  for S3 
* Adds a contract test for the operation: AbstractContractCopyFromLocalTest
* Implements the contract tests for Local and S3A FileSystems

Contributed by: Bogdan Stolojan
 2021-07-30 19:42:08 +01:00
Tamas Domok 798a0837c1
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty (#3206) 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-07-30 12:16:46 +02:00
hchaverr 3c8a48e681 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 2021-07-28 15:37:56 -07:00
Viraj Jasani e001f8ee39
HADOOP-17814. Provide fallbacks for identity/cost providers and backoff enable (#3230) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-29 02:10:07 +09:00
jianghuazhu fd13970d94
HDFS-16137.Improve the comments related to FairCallQueue#queues. (#3226) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 03:18:04 -07:00
Mehakmeet Singh f813554769
HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK (#2706) 
This (big!) patch adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console. 
  
* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

Contributed by Mehakmeet Singh
 2021-07-27 11:08:51 +01:00
Akira Ajisaka 4c35466359
HADOOP-17317. [JDK 11] Upgrade dnsjava to remove illegal access warnings (#2442) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-07-24 14:48:25 +09:00
Viraj Jasani 3a52bfc5db
HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 2021-07-23 12:38:55 +08:00
Viraj Jasani e1d00addb5
HADOOP-16290. Enable RpcMetrics units to be configurable (#3198) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-19 23:55:49 -07:00
He Xiaoqiao 6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00
Viraj Jasani df44178eb6
HADOOP-17795. Provide fallbacks for callqueue.impl and scheduler.impl (#3192) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-14 20:58:32 +09:00
jianghuazhu 87e0000137
HADOOP-17672.Remove an invalid comment content in the FileContext class. (#2961) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-14 00:15:02 -07:00
Abhishek Das 1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
Akira Ajisaka c81f82e21d
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
 2021-07-12 10:16:13 +09:00
Artem Smotrakov ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
Akira Ajisaka 6ac5d8a1ed
HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 2021-07-09 22:48:47 +09:00
Viraj Jasani 618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
liangxs a5db6831bc
HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 2021-07-06 09:11:03 +08:00
Rafal Wojdyla f639fbc29f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-05 21:07:12 +01:00
Akira Ajisaka 20a4b1ae36
HADOOP-17331. [JDK 16] TestDNS fails (#2884) 2021-06-30 03:06:29 -07:00
Viraj Jasani c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
Takanobu Asanuma 9e7c7ad129
HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-17 09:58:47 +09:00
Wei-Chiu Chuang a77bf7cf07 Make upstream aware of 3.3.1 release 2021-06-15 15:43:36 +00:00
Mingliang Liu 6e5692e7e2
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 4a26a61ecd.
 2021-06-11 00:34:24 -07:00
Viraj Jasani 4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Steve Loughran 762a83e044
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.
 2021-06-08 21:56:40 +01:00
Viraj Jasani f4b24c68e7
HADOOP-17743. Replace Guava Lists usage by Hadoop's own Lists in hadoop-common, hadoop-tools and cloud-storage projects (#3072) 2021-06-07 13:24:09 +09:00
July 4a26a61ecd
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042) 
Contributed by kaifeiYi (yikf).

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2021-06-04 13:36:09 -07:00
Viraj Jasani 59fc4061cb
HADOOP-17152. Provide Hadoop's own Lists utility to reduce dependency on Guava (#3061) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-06-03 18:56:00 +09:00
Konstantin V Shvachko 8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Hui Fei 98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani 986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Steve Loughran 832a3c6a89
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.
 2021-05-25 10:25:41 +01:00
Mehakmeet Singh c665ab02ed
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics 
to be logged at info. (also: `warn` or `error` for even higher
log levels).


Contributed by: Mehakmeet Singh
 2021-05-24 13:02:11 +01:00
Vinayakumar B 2bbeae3240
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 2021-05-24 02:45:39 -07:00
Viraj Jasani c70ee2d548
HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-05-22 19:06:58 +09:00
Viraj Jasani e4062ad027
HADOOP-17115. Replace Guava Sets usage by Hadoop's own Sets in hadoop-common and hadoop-tools (#2985) 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-05-20 10:47:04 -05:00
Hongbing Wang f7247922b7
HDFS-16018. Optimize the display of hdfs "count -e" or "count -t" com… (#2994) 2021-05-20 11:23:54 +08:00
Xiaoyu Yao 86729e130f
HADOOP-17699. Remove hardcoded SunX509 usage from SSLFactory. (#3016) 2021-05-18 10:11:36 -07:00
Akira Ajisaka 35ca1dcb9d
HADOOP-17685. Fix junit deprecation warnings in hadoop-common module. (#2983) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-13 14:22:25 +09:00
Viraj Jasani fdd20a3cf4
HADOOP-17689. Avoid Potential NPE in org.apache.hadoop.fs (#3008) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-13 00:05:58 +09:00
dependabot[bot] 29105ffb63
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-05-12 10:57:45 +09:00
Viraj Jasani c80f07422f
HADOOP-17686. Avoid potential NPE by using Path#getParentPath API in hadoop-huaweicloud (#2990) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-12 10:34:01 +09:00
Borislav Iordanov 91430889a5
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 13:31:48 -07:00
hexiaoqiao 6d6766bc22
HADOOP-17690. Improve the log for The DecayRpcScheduler. Contributed by Bhavik Patel. 2021-05-10 14:11:36 +08:00
Ayush Saxena 7f93349ee7 HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li. 2021-05-09 03:40:08 +05:30
hchaverr f40e3eb059 HADOOP-17680. Allow ProtobufRpcEngine to be extensible (#2905) Contributed by Hector Chaverri. 2021-05-06 16:42:27 -07:00
Viraj Jasani b93e448f9a
HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ (#2973) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-06 04:52:02 +09:00
Istvan Fajth 0d78d73973
HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException (#2965) 2021-05-04 11:33:59 +01:00