ec3311975c
HADOOP-16290. Enable RpcMetrics units to be configurable ( #3198 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e1d00addb5
2021-07-20 14:56:28 +08:00
450dae7383
HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das ( #2260 )
...
(cherry picked from commit 1dd03cc4b5
2021-07-13 18:23:27 -07:00
a1f90883f0
HADOOP-12665. Document hadoop.security.token.service.use_ip. ( #3187 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d
2021-07-12 10:18:19 +09:00
c18ad9a007
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada
2021-07-10 13:43:31 +09:00
6cc9549499
HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS ( #3190 )
...
(cherry picked from commit 6ac5d8a1ed
2021-07-09 22:50:18 +09:00
e3fb63f33f
HADOOP-17402. Add GCS config to the core-site ( #2638 )
...
Contributed by Rafal Wojdyla
2021-07-07 22:43:31 +01:00
24b780820c
HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout ( #3080 )
...
(cherry picked from commit a5db6831bc
2021-07-07 09:41:11 +08:00
b8a98e4f82
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79
2021-06-21 10:28:05 +09:00
25138c98bf
HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml ( #3099 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129
2021-06-17 10:00:36 +09:00
91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp ( #3042 )"
...
This reverts commit 8c0f948054
2021-06-11 00:35:41 -07:00
4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true ( #2977 )
...
Contributed by Steve Loughran.
Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
2021-06-08 22:05:00 +01:00
8c0f948054
HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp ( #3042 )
...
Contributed by kaifeiYi (yikf).
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-06-04 13:37:24 -07:00
c5535caf6e
HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.
...
(cherry picked from commit 8ce30f51f9
2021-05-27 18:53:46 -07:00
464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector ( #2807 )
...
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.
The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.
If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.
Contributed by Steve Loughran.
Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
2021-05-25 12:55:38 +01:00
a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() ( #2963 )
...
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:
org.apache.hadoop.fs.statistics.IOStatisticsLogging
Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).
Contributed by: Mehakmeet Singh
Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
2021-05-24 13:04:20 +01:00
dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist ( #2767 )
...
(cherry picked from commit 2bbeae3240
2021-05-24 18:00:38 +08:00
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )"
...
This reverts commit 4ffe5eb1dd
2021-05-24 17:37:18 +08:00
87e32fbb8d
HADOOP-17700. ExitUtil#halt info log should log HaltException ( #3015 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548
2021-05-22 19:10:28 +09:00
fa4915fdbb
Preparing for 3.3.2 development
2021-05-19 21:52:37 +08:00
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )
...
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb
* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661 ). Contributed by Vinayakumar B.
(cherry picked from commit c4c0683dff
2021-05-17 18:57:46 -07:00
3f9c9ccf46
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider.… ( #3010 )
...
* HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334 )
(cherry picked from commit 4c5ad57818
2021-05-13 16:57:58 -07:00
b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 ( #2974 )
...
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63
2021-05-12 10:58:39 +09:00
f5ef78d46f
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism
2021-05-10 15:41:56 -05:00
2a95ba29bb
HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li.
2021-05-09 04:00:30 +05:30
0f12f3e125
HADOOP-17036. TestFTPFileSystem failing as ftp server dir already exists.
...
Contributed by Mikhail Pryakhin.
(cherry picked from commit 017d24e970
2021-05-07 14:20:29 +09:00
cedebf1c27
HADOOP-17680. Allow ProtobufRpcEngine to be extensible ( #2905 ) Contributed by Hector Chaverri.
...
(cherry picked from commit f40e3eb059
2021-05-06 17:50:32 -07:00
be34c1222a
HADOOP-11616. Remove workaround for Curator's ChildReaper requiring Guava 15+ ( #2973 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b93e448f9a
2021-05-06 04:53:06 +09:00
65bf544118
HADOOP-16954. Add -S option in "Count" command to show only Snapshot Counts. Contributed by hemanthboyina.
...
(cherry picked from commit b89d875f7b
2021-05-04 17:44:34 +01:00
00ba886a22
HADOOP-17675. LdapGroupsMapping$LdapSslSocketFactory ClassNotFoundException ( #2965 )
...
(cherry picked from commit 0d78d73973
2021-05-04 11:36:15 +01:00
98aa4fc32c
HADOOP-17657: implement StreamCapabilities in SequenceFile.Writer and fall back to flush, if hflush is not supported ( #2949 )
...
Co-authored-by: Kishen Das <kishen@cloudera.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit e571025f5b
2021-05-04 16:35:00 +08:00
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds ( #2910 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c
2021-05-02 19:19:55 +09:00
f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 ( #2940 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-27 22:39:25 +09:00
fb71e6c91e
HADOOP-17597. Optionally downgrade on S3A Syncable calls ( #2801 )
...
Followup to HADOOP-13327, which changed S3A output stream hsync/hflush calls
to raise an exception.
Adds a new option fs.s3a.downgrade.syncable.exceptions
When true, calls to Syncable hsync/hflush on S3A output streams will
log once at warn (for entire process life, not just the stream), then
increment IOStats with the relevant operation counter
With the downgrade option false (default)
* IOStats are incremented
* The UnsupportedOperationException current raised includes a link to the
JIRA.
Contributed by Steve Loughran.
Change-Id: I967e077eda1d1a1a3795b4d22e003fe7997b6679
2021-04-24 18:32:39 +01:00
389d3034c6
HADOOP-17471. ABFS to collect IOStatistics ( #2731 ) ( #2950 )
...
The ABFS Filesystem and its input and output streams now implement
the IOStatisticSource interface and provide IOStatistics on
their interactions with Azure Storage.
This includes the min/max/mean durations of all REST API calls.
Contributed by Mehakmeet Singh <mehakmeet.singh@cloudera.com>
2021-04-24 17:59:26 +01:00
930f384e30
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
...
(cherry picked from commit da93cd962c
2021-04-23 18:42:50 +00:00
f7e382c62e
YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko
...
(cherry picked from commit 31477a224f
2021-04-16 20:21:01 +00:00
a3405cb56a
HADOOP-17569. Building native code fails on Fedora 33. ( #2886 )
...
(cherry picked from commit 2717203f85
2021-04-15 12:39:07 +00:00
a3cff8776a
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C.
2021-04-14 12:28:34 +08:00
72355c7b6e
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. ( #2890 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 156ecc89be
2021-04-13 17:10:00 +09:00
77315abe47
HADOOP-17608. Fix TestKMS failure ( #2880 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2bd810a507
2021-04-12 12:54:59 +09:00
dca2bf9dd5
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
...
(cherry picked from commit 95e6892675
2021-04-08 17:20:08 +08:00
236a9a7713
HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree
...
(cherry picked from commit ae88174c29
2021-04-07 23:50:02 +05:30
1110b03752
HADOOP-17621. hadoop-auth to remove jetty-server dependency. ( #2865 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282
2021-04-07 13:17:40 +08:00
8b4b3d6fe6
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. ( #2862 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b
2021-04-06 14:18:31 +09:00
a549b4a82e
HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq.
...
(cherry picked from commit 936bf09c37
2021-04-02 17:11:16 +08:00
90bbaca88b
HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree.
...
(cherry picked from commit bc7689abf5
2021-04-02 10:04:19 +05:30
c60e81b5a8
HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree.
...
(cherry picked from commit 478402cc74
2021-04-02 09:56:47 +05:30
62937d15bb
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync ( #2770 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6
2021-04-02 10:57:05 +09:00
c365149e16
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures.
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:50:28 -07:00
ae2d856515
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C.
2021-03-31 12:24:01 +08:00
Viraj Jasani
Abhishek Das
Akira Ajisaka
Artem Smotrakov
Rafal Wojdyla
liangxs
Takanobu Asanuma
Mingliang Liu
Steve Loughran
July
Konstantin V Shvachko
Mehakmeet Singh
Vinayakumar B
Wei-Chiu Chuang
Xiaoyu Yao
dependabot[bot]
Borislav Iordanov
Ayush Saxena
Mike
hchaverr
Istvan Fajth
kishendas
lfengnan
litao
Eric Badger
Masatake Iwasaki
He Xiaoqiao
touchida
Brahma Reddy Battula