2717203f85
HADOOP-17569. Building native code fails on Fedora 33. ( #2886 )
2021-04-15 21:33:18 +09:00
9179638017
HADOOP-17524. Remove EventCounter and Log counters from JVM Metrics ( #2909 )
...
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-15 18:04:46 +09:00
156ecc89be
HADOOP-17630. [JDK 15] TestPrintableString fails due to Unicode 13.0 support. ( #2890 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-04-13 17:08:49 +09:00
2bd810a507
HADOOP-17608. Fix TestKMS failure ( #2880 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-04-12 12:53:58 +09:00
ae88174c29
HADOOP-17617. Incorrect representation of RESPONSE for Get Key Version in KMS index.md.vm file. Contributed by Ravuri Sushma sree
2021-04-07 23:49:17 +05:30
dac60b8282
HADOOP-17621. hadoop-auth to remove jetty-server dependency. ( #2865 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-06 21:55:01 -07:00
3a89471c36
HADOOP-17613. Log not flushed fully when daemon shutdown. Contributed by Renukaprasad C.
2021-04-07 12:18:23 +08:00
e86050fae5
HADOOP-17588. CryptoInputStream#close() should be syncronized. Contributed by RenukaPrasad C
2021-04-06 18:06:10 +05:30
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. ( #2862 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 13:39:10 +09:00
5eaa1b7230
HADOOP-17619: Fix DelegationTokenRenewer#updateRenewalTime java doc error. ( #2846 ). Contributed by Qi Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-04-04 14:52:03 +05:30
bc7689abf5
HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree.
2021-04-02 10:03:50 +05:30
478402cc74
HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree.
2021-04-02 09:56:00 +05:30
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync ( #2770 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-02 10:33:40 +09:00
a94a23ab9e
Revert "HADOOP-17608. Fix NPE in TestKMS ( #2828 )"
...
This reverts commit d69088a097
2021-04-01 04:24:26 +09:00
2c482fbacf
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:46:35 -07:00
22961a615d
Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2"
...
This reverts commit a2975d2153
2021-03-31 10:43:09 -07:00
a2975d2153
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
2021-03-31 10:40:20 -07:00
5183aaeda2
Revert "Hadoop 16524 - resubmission following some unit test fixes ( #2693 )"
...
Revert to fix the summary message.
This reverts commit 9509bebf7f
2021-03-31 10:39:55 -07:00
9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes ( #2693 )
...
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:07:42 -07:00
d69088a097
HADOOP-17608. Fix NPE in TestKMS ( #2828 )
2021-03-31 09:57:31 -07:00
054e1c5e83
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C.
2021-03-31 12:22:53 +08:00
f5c1557288
HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. ( #2820 ). Contributed by Ayush Saxena.
...
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-27 03:01:41 +05:30
af1f9f43ea
HADOOP-17133. Implement HttpServer2 metrics ( #2145 )
2021-03-25 12:09:43 -07:00
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
2021-03-24 16:56:09 +08:00
b4d97a8dc7
HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel.
2021-03-24 17:51:35 +09:00
03cfc85279
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. ( #2732 ). Contributed by Ayush Saxena.
...
Signed-off-by: Steve Loughran <stevel@apache.org>
2021-03-24 02:36:26 +05:30
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof ( #2804 )
2021-03-23 08:46:11 -07:00
63eb289462
HADOOP-17598. Fix java doc issue introduced by HADOOP-17578. ( #2802 ). Contributed by Xiaoyu Yao.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 10:23:45 +05:30
c8d327a4f1
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. ( #2780 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-22 11:43:32 +09:00
b503de2328
HADOOP-17578. Improve UGI debug log to help troubleshooting TokenCach… ( #2762 )
2021-03-17 10:57:11 -07:00
299b8062f1
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
2021-03-15 20:13:17 +00:00
b1dc6c40a0
HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell.
2021-03-14 18:09:50 +08:00
0f6ba5e031
HADOOP-17514. Remove trace subcommand from hadoop CLI. ( #2680 )
2021-03-12 10:26:46 +09:00
bcd9c67082
HADOOP-16721. Improve S3A rename resilience ( #2742 )
...
The S3A connector's rename() operation now raises FileNotFoundException if
the source doesn't exist; a FileAlreadyExistsException if the destination
exists and is unsuitable for the source file/directory.
When renaming to a path which does not exist, the connector no longer checks
for the destination parent directory existing -instead it simply verifies
that there is no file immediately above the destination path.
This is needed to avoid race conditions with delete() and rename()
calls working on adjacent subdirectories.
Contributed by Steve Loughran.
2021-03-11 12:47:39 +00:00
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
176bd88890
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. ( #2522 )
...
Contributed by Chao Sun.
(Cherry-picked via PR #2575 )
2021-03-09 20:01:29 +00:00
ef7ab535c5
HADOOP-17552. Change ipc.client.rpc-timeout.ms from 0 to 120000 by default to avoid potential hang. ( #2727 )
2021-03-06 22:26:16 +09:00
9fd2198daa
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree.
2021-03-04 14:55:37 +09:00
c3b3b36dee
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
2021-03-01 11:36:41 +00:00
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 13:42:33 -06:00
924fa87010
HADOOP-17510. Hadoop prints sensitive Cookie information. ( #2673 )
2021-02-24 01:29:25 -08:00
7b7c0019f4
HADOOP-17528. SFTP File System: close the connection pool when closing a FileSystem ( #2701 )
...
Contributed by Mike Pryakhin.
2021-02-23 17:03:27 +00:00
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )"
...
This reverts commit 4cf3531583
2021-02-19 11:09:10 +09:00
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )
...
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
78905d7e3f
HADOOP-16906. Abortable ( #2684 )
...
Adds an Abortable.abort() interface for streams to enable output streams to be terminated; this
is implemented by the S3A connector's output stream. It allows for commit protocols
to be implemented which commit/abort work by writing to the final destination and
using the abort() call to cancel any write which is not intended to be committed.
Consult the specification document for information about the interface and its use.
Contributed by Jungtaek Lim and Steve Loughran.
2021-02-11 17:37:20 +00:00
798df6d699
HADOOP-13327 Output Stream Specification. ( #2587 )
...
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.
The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.
Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
52217fce3d
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 ( #2546 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-02-05 16:14:10 +09:00
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type ( #2377 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer ( #2645 )
2021-02-01 13:42:44 +09:00
06e836cd57
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li.
2021-01-31 19:45:40 +08:00
0bb52a42e5
HADOOP-17483. Magic committer is enabled by default. ( #2656 )
...
* core-default.xml updated so that fs.s3a.committer.magic.enabled = true
* CommitConstants updated to match
* All tests which previously enabled the magic committer now rely on
default settings. This helps make sure it is enabled.
* Docs cover the switch, mention its enabled and explain why you may
want to disable it.
Note: this doesn't switch to using the committer -it just enables the path
rewriting magic which it depends on.
Contributed by Steve Loughran.
2021-01-27 19:04:22 +00:00
28cc912a5c
HADOOP-17493. Revert name of DELEGATION_TOKENS_ISSUED constant/statistic ( #2649 )
...
Follow-on to HADOOP-16830/HADOOP-17271.
Contributed by Steve Loughran.
2021-01-27 16:39:29 +00:00
80c7404b51
HADOOP-17414. Magic committer files don't have the count of bytes written collected by spark ( #2530 )
...
This needs SPARK-33739 in the matching spark branch in order to work
Contributed by Steve Loughran.
2021-01-26 19:30:51 +00:00
d82009599a
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file ( #2628 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2021-01-24 22:56:47 +09:00
91d4ba57c5
HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov.
2021-01-22 17:55:37 -08:00
9b4f09a6bb
HDFS-15758. Fix typos in MutableMetric. Contributed by Haibin Huang.
2021-01-18 20:35:37 +05:30
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-16 14:06:56 +08:00
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount ( #2583 ). Contributed by Leon Gao.
2021-01-15 16:28:11 -08:00
724edb0354
HADOOP-17451. IOStatistics test failures in S3A code. ( #2594 )
...
Caused by HADOOP-16830 and HADOOP-17271.
Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.
Contributed by Steve Loughran.
Change-Id: I65ba3f44444e59d298df25ac5c8dc5a8781dfb7d
2021-01-12 17:30:32 +00:00
05c9c2ed02
Revert "HADOOP-17451. IOStatistics test failures in S3A code. ( #2594 )"
...
This reverts commit d3014e01f3
2021-01-12 17:29:59 +00:00
d3014e01f3
HADOOP-17451. IOStatistics test failures in S3A code. ( #2594 )
...
Caused by HADOOP-16380 and HADOOP-17271.
Fixes tests which fail intermittently based on configs and
in the case of the HugeFile tests, bulk runs with existing
FS instances meant statistic probes sometimes ended up probing those
of a previous FS.
Contributed by Steve Loughran.
2021-01-12 17:25:14 +00:00
0a6ddfa145
HADOOP-17272. ABFS Streams to support IOStatistics API ( #2604 )
...
Contributed by Mehakmeet Singh.
2021-01-12 15:48:09 +00:00
d4fd675a95
Revert "HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode ( #2470 )"
...
This reverts commit e306f59421
2021-01-11 08:54:55 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
2021-01-08 13:10:09 -06:00
e306f59421
HADOOP-16524. Reloading SSL keystore for both DataNode and NameNode ( #2470 )
...
Co-authored-by: Borislav Iordanov <biordanov@apple.com>
Signed-off-by: stack <stack@apache.org>
2021-01-08 09:10:21 -08:00
87bd4d2aca
HDFS-15754. Add DataNode packet metrics ( #2578 )
...
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
b1abb10ea2
HADOOP-17430. Restore ability to set Text to empty byte array ( #2545 )
...
Contributed by gaozhan.ding
2021-01-05 21:09:41 +00:00
2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout ( #2533 )
2021-01-04 20:54:44 -08:00
77299ae992
HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun.
2021-01-05 00:05:03 +05:30
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
2021-01-04 09:43:58 -08:00
c98d274db1
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
2021-01-03 16:25:17 +08:00
99d08a19ba
HADOOP-17450. Add Public IOStatistics API. ( #2577 )
...
This is the API and implementation classes of HADOOP-16830,
which allows callers to query IO object instances
(filesystems, streams, remote iterators, ...) and other classes
for statistics on their I/O Usage: operation count and min/max/mean
durations.
New Packages
org.apache.hadoop.fs.statistics.
Public API, including:
IOStatisticsSource
IOStatistics
IOStatisticsSnapshot (seralizable to java objects and json)
+helper classes for logging and integration
BufferedIOStatisticsInputStream
implements IOStatisticsSource and StreamCapabilities
BufferedIOStatisticsOutputStream
implements IOStatisticsSource, Syncable and StreamCapabilities
org.apache.hadoop.fs.statistics.impl
Implementation classes for internal use.
org.apache.hadoop.util.functional
functional programming support for RemoteIterators and
other operations which raise IOEs; all wrapper classes
implement and propagate IOStatisticsSource
Contributed by Steve Loughran.
2020-12-31 11:52:42 +00:00
6de1a8eb67
HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger
2020-12-11 20:16:56 +00:00
ad40715690
HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.
2020-12-07 15:00:19 +05:30
e2c1268ebd
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-04 09:20:09 +08:00
8c234fc0d4
HADOOP-17389. KMS should log full UGI principal. ( #2476 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
2020-12-04 09:33:30 +09:00
db73e994ed
HADOOP-16881. KerberosAuthentication does not disconnect HttpURLConnection leading to CLOSE_WAIT cnxns. Contributed by Attila Magyar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-03 12:05:20 -08:00
f94e927bfb
HADOOP-17392. Remote exception messages should not include the exception class ( #2486 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-03 10:55:51 -06:00
9170eb566b
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs ( #2512 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-12-04 00:14:04 +09:00
44910b53f3
HDFS-15699 Remove lz4 references in vcxproj ( #2498 )
2020-11-29 22:53:54 -08:00
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:32:10 +09:00
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs ( #2488 )" because JIRA ID is missing.
...
This reverts commit ce5b3d7975
2020-11-26 11:30:53 +09:00
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:27:30 +09:00
ac7045b75f
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. ( #2396 )
...
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".
This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.
The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.
If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.
Contributed by Steve Loughran.
2020-11-25 14:31:02 +00:00
8fa699b53f
HDFS-15635. ViewFileSystemOverloadScheme support specifying mount table loader imp through conf ( #2389 ). Contributed by Junfan Zhang.
2020-11-19 20:21:16 -08:00
07050339e0
HADOOP-17367. Add InetAddress api to ProxyUsers.authorize ( #2449 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-11-19 14:37:14 -06:00
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-11-18 12:03:25 -08:00
ce7827c82a
HADOOP-17318. Support concurrent S3A commit jobs with same app attempt ID. ( #2399 )
...
See also [SPARK-33402]: Jobs launched in same second have duplicate MapReduce JobIDs
Contributed by Steve Loughran.
Change-Id: Iae65333cddc84692997aae5d902ad8765b45772a
2020-11-18 13:34:51 +00:00
e3c08f285a
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. ( #2310 )
...
This fixes the S3Guard/Directory Marker Retention integration so that when
fs.s3a.directory.marker.retention=keep, failures during multipart delete
are handled correctly, as are incremental deletes during
directory tree operations.
In both cases, when a directory marker with children is deleted from
S3, the directory entry in S3Guard is not deleted, because it is still
critical to representing the structure of the store.
Contributed by Steve Loughran.
Change-Id: I4ca133a23ea582cd42ec35dbf2dc85b286297d2f
2020-11-18 12:18:11 +00:00
ebe1d1fbf7
HADOOP-17362. reduce RPC calls doing ls on HAR file ( #2444 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-11-13 14:22:35 -06:00
5ce18101cb
HADOOP-17346. Fair call queue is defeated by abusive service principals ( #2431 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-12 13:13:12 -06:00
6f10a0506f
HADOOP-17365. Contract test for renaming over existing file is too lenient ( #2447 )
...
Contributed by Attila Doroszlai.
2020-11-11 21:20:09 +00:00
71071e5c0f
HADOOP-17358. Improve excessive reloading of Configurations ( #2436 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-11 12:39:03 -08:00
45434c93e8
HADOOP-17096. Fix ZStandardCompressor input buffer offset ( #2104 ). Contributed by Stephen Jung (Stripe).
2020-11-10 11:38:22 -08:00
f1d5652359
HDFS-15668. RBF: Fix RouterRPCMetrics annocation and document misplaced error. Contributed by Hongbing Wang.
2020-11-10 13:09:03 +08:00
1ea3f74246
HADOOP-17360. Log the remote address for authentication success ( #2441 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-11-09 14:05:08 -06:00
ae7b00a998
HADOOP-17340. TestLdapGroupsMapping failing -string mismatch in exception validation. ( #2427 ). Contributed by Steve Loughran.
2020-11-07 09:43:24 +05:30
6eacaffeea
HDFS-15623. Respect configured values of rpc.engine ( #2403 ) Contributed by Hector Chaverri.
2020-11-06 14:25:35 -08:00
af389d9897
HADOOP-17342. Creating a token identifier should not do kerberos name
...
resolution. Contributed by Jim Brennan.
2020-11-05 21:46:14 +00:00
638f1fc2b6
Revert "HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 ( #2387 )"
...
This reverts commit d259928035
2020-11-05 17:11:39 +00:00
7e52c0975f
Revert "HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. ( #2291 )"
...
This reverts commit 7f5caca04c
2020-11-04 16:18:03 -08:00
7f5caca04c
HADOOP-17255. JavaKeyStoreProvider fails to create a new key if the keystore is HDFS. ( #2291 )
...
Reviewed-by: Steve Loughran <stevel@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-11-03 11:20:23 -08:00
f17e067d52
HADOOP-17329. mvn site commands fails due to MetricsSystemImpl changes. Contributed by Xiaoqiao He.
2020-10-29 07:19:49 +05:30
da93cd962c
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
2020-10-28 10:35:30 +01:00
872440610f
HADOOP-17328. LazyPersist Overwrite fails in direct write mode. ( #2413 )
2020-10-27 01:37:37 +09:00
d259928035
HADOOP-17306. RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09 ( #2387 )
2020-10-23 11:30:02 +05:30
6a9ceedfb3
HADOOP-17175. [JDK 11] Fix javadoc errors in hadoop-common module. ( #2397 )
2020-10-23 03:15:45 +09:00
7b4359657f
HADOOP-17315. Use shaded guava in ClientCache.java ( #2398 )
...
Added checkstyle rules to warn guava imports
2020-10-20 23:25:24 +09:00
30f06e0c74
HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. ( #2394 )
2020-10-20 19:08:59 +09:00
31477a224f
YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko
2020-10-19 13:17:51 +02:00
4c651103f2
HADOOP-17310. Touch command with -c option is broken. ( #2393 ). Contributed by Ayush Saxena.
2020-10-19 10:54:18 +05:30
42e7e0380f
HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina.
2020-10-18 18:37:46 +05:30
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. ( #2342 ). Contributed by Ayush Saxena.
2020-10-17 12:01:18 +05:30
ddc0ee27fa
HADOOP-17304. KMS ACL: Allow DeleteKey Operation to Invalidate Cache. Contributed by Xiaoyu.
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-10-14 16:00:37 +08:00
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
...
Contributed by Gabor Bota.
2020-10-13 16:17:44 +01:00
b3786d6c3c
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
2020-10-12 17:26:49 -07:00
518a212cff
HDFS-13293. RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer ( #2363 )
2020-10-09 16:12:22 +08:00
52db86b0bb
HADOOP-17021. Add concat fs command ( #1993 )
...
Contributed by Jinglun
2020-10-08 10:36:07 +01:00
82522d60fb
HADOOP-17281 Implement FileSystem.listStatusIterator() in S3AFileSystem ( #2354 )
...
Contains HADOOP-17300: FileSystem.DirListingIterator.next() call should
return NoSuchElementException
Contributed by Mukund Thakur
2020-10-07 13:59:06 +01:00
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec ( #2297 )
...
This switches the SnappyCodec to use the java-snappy codec, rather than the native one.
To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.
This comesin as an avro dependency, so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7
Contributed by DB Tsai and Liang-Chi Hsieh
2020-10-06 17:07:54 +01:00
d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items ( #2327 )
2020-10-04 02:02:28 +09:00
d68d2a5c1e
HADOOP-17287. Support new Instance by non default constructor by ReflectionUtils ( #2341 )
2020-09-30 14:22:55 -07:00
a490d87eb7
HADOOP-17280. Service-user cost shouldn't be accumulated to totalDecayedCallCost and totalRawCallCost. Contributed by Jinglun.
2020-09-30 12:12:52 +08:00
4c5ad57818
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. ( #2334 )
2020-09-29 20:15:27 -07:00
75d10f8499
HADOOP-17267. Add debug-level logs in Filesystem.close() ( #2321 )
...
When a filesystem is closed, the FileSystem log will, at debug level,
log the method calling close/closeAll.
At trace level: the full calling stack.
Contributed by Karen Coppage.
2020-09-29 16:06:50 +01:00
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng
2020-09-27 19:27:13 +05:30
899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. ( #2339 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-25 21:21:01 -07:00
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS ( #2189 )
...
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
2020-09-24 14:27:04 +05:30
474fa80bfb
HADOOP-17277. Correct spelling errors for separator ( #2322 )
...
Contributed by Hui Fei.
2020-09-22 17:10:33 +01:00
6b5d9e2334
HADOOP-17259. Allow SSLFactory fallback to input config if ssl-*.xml … ( #2301 )
2020-09-21 12:41:06 -07:00
c8c1cc43d3
HADOOP-17088.Failed to load XInclude files with relative path. ( #2097 )
...
Contributed by Yushi Hayasaka.
2020-09-21 19:10:51 +01:00
364b958085
HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… ( #2311 )
2020-09-20 00:47:02 +09:00
fc2435cb5c
HADOOP-15136. Correct typos in filesystem.md ( #2314 )
...
Contributed by David Tucker
2020-09-18 18:28:14 +01:00
6adf8462ba
HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances. ( #2259 )
2020-09-17 10:39:19 -07:00
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled ( #2305 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-16 22:43:00 -07:00
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. ( #2298 ). Contributed by Uma Maheswara Rao G.
2020-09-12 17:06:39 -07:00
958cab804e
Revert "HADOOP-17244. S3A directory delete tombstones dir markers prematurely. ( #2280 )"
...
This reverts commit 9960c01a25
2020-09-11 18:07:49 +01:00
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache ( #2241 )
...
Contributed by fanrui.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-09-10 22:30:52 -07:00
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree ( #2185 ). Contributed by Zhenzhao Wang.
...
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
2020-09-10 21:20:32 -07:00
9960c01a25
HADOOP-17244. S3A directory delete tombstones dir markers prematurely. ( #2280 )
...
This changes directory tree deletion so that only files are incrementally deleted
from S3Guard after the objects are deleted; the directories are left alone
until metadataStore.deleteSubtree(path) is invoke.
This avoids directory tombstones being added above files/child directories,
which stop the treewalk and delete phase from working.
Also:
* Callback to delete objects splits files and dirs so that
any problems deleting the dirs doesn't trigger s3guard updates
* New statistic to measure #of objects deleted, alongside request count.
* Callback listFilesAndEmptyDirectories renamed listFilesAndDirectoryMarkers
to clarify behavior.
* Test enhancements to replicate the failure and verify the fix
Contributed by Steve Loughran
2020-09-10 17:03:52 +01:00
e5fe326270
HADOOP-17165. Implement service-user feature in DecayRPCScheduler. ( #2240 )
2020-09-09 09:56:58 -07:00
2029556dbb
HADOOP-17181. Handle transient stream read failures in FileSystem contract tests ( #2286 )
...
Contributed by Steve Loughran.
* Fixes AbstractContractSeekTest test to use readFully
* Doesn't do this to AbstractContractUnbufferTest test as it changes the test too much.
Instead just notes in the error that this may be transient
The issue is that read(buffer) doesn't guarantee that the buffer is filled, only that it will
read up to a point, and that may be just be the amount of data left in the TCP packet.
readFully corrects for this, but using it in the unbuffer test runs the risk that what
is tested for in terms of unbuffering doesn't actually get validated.
2020-09-09 11:58:26 +01:00
8a3952436c
HADOOP-17245. Add OzoneFileSystem classes to core-default.xml. ( #2276 )
2020-09-04 18:38:48 -07:00
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well ( #2234 ). Contributed by Uma Maheswara Rao G.
2020-09-03 11:06:20 -07:00
7e671f3b65
HADOOP-17235. Erasure Coding: Remove dead code from common side. Contributed by Fei Hui.
2020-09-01 11:35:55 +08:00
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing ( #2252 )
...
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
2020-08-28 15:39:09 +01:00
bfd99f31fd
HADOOP-17232. Erasure Coding: Typo in document. Contributed by Fei Hui.
2020-08-28 17:38:23 +05:30
d8aaa8c338
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time ( #2249 )
...
Contributed by Sandeep Guggilam.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
2020-08-26 23:45:21 -07:00
Masatake Iwasaki
Viraj Jasani
Akira Ajisaka
Brahma Reddy Battula
Wei-Chiu Chuang
He Xiaoqiao
Boyina, Hemanth Kumar
zhuqi
litao
Borislav Iordanov
stack
Ayush Saxena
touchida
Takanobu Asanuma
Jack Jiang
Xiaoyu Yao
kwangsun
Jim Brennan
Steve Loughran
Chao Sun
Haoze Wu
S O'Donnell
Ahmed Hussein
Renukaprasad C
Mike
YaYun-Wang
belugabehr
Siyao Meng
Anton Kutuzov
LeonGao
Mehakmeet Singh
lfengnan
dgzdot
Hui Fei
Attila Magyar
Andrea Scarpino
Gautham B A
zhang_jf
Liang-Chi Hsieh
Doroszlai, Attila
Stephen Jung
hchaverr
Eric Badger
Sunil G
Peter Bacsko
Vinayakumar B
Szilard Nemeth
hemanthboyina
Gabor Bota
Konstantin V Shvachko
Hui Fei
Jinglun
Mukund Thakur
maobaolong
Karen Coppage
Uma Maheswara Rao G
huangtianhua
crossfire
David Tucker
1996fanrui
zz
Bharat Viswanadham
sguggilam