1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.
2020-01-21 10:58:32 +09:00
edbbc03ce7
HADOOP-16621. [pb-upgrade] Remove Protobuf classes from signatures of Public APIs. Contributed by Vinayakumar B. ( #1803 )
2020-01-16 23:27:50 +05:30
d7c4f8ab21
HDFS-13616. Batch listing of multiple directories ( #1725 )
2020-01-15 17:22:47 -08:00
5d180463dc
HADOOP-16790. Add Write Convenience Methods.
...
Contributed by David Mollitor.
This adds operations in FileUtil to write text to a file via
either a FileSystem or FileContext instance.
Change-Id: I5fe8fcf1bdbdbc734e137f922a75a822f2b88410
2020-01-15 18:38:34 +00:00
6b86a5110e
HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu.
2020-01-13 10:48:53 -08:00
26a969ec73
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.
2020-01-10 17:06:59 -08:00
e589f6199c
HADOOP-16590. Update OS login modules for IBM JDK.
...
Contributed by Nicholas Marion
Close #1484
2020-01-10 14:33:12 -05:00
c4fb43c942
HADOOP-16749. Configuration parsing of CDATA values are blank
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-10 10:31:39 -06:00
49df838995
HADOOP-16697. Tune/audit S3A authoritative mode.
...
Contains:
HADOOP-16474. S3Guard ProgressiveRenameTracker to mark destination
dirirectory as authoritative on success.
HADOOP-16684. S3guard bucket info to list a bit more about
authoritative paths.
HADOOP-16722. S3GuardTool to support FilterFileSystem.
This patch improves the marking of newly created/import directory
trees in S3Guard DynamoDB tables as authoritative.
Specific changes:
* Renamed directories are marked as authoritative if the entire
operation succeeded (HADOOP-16474).
* When updating parent table entries as part of any table write,
there's no overwriting of their authoritative flag.
s3guard import changes:
* new -verbose flag to print out what is going on.
* The "s3guard import" command lets you declare that a directory tree
is to be marked as authoritative
hadoop s3guard import -authoritative -verbose s3a://bucket/path
When importing a listing and a file is found, the import tool queries
the metastore and only updates the entry if the file is different from
before, where different == new timestamp, etag, or length. S3Guard can get
timestamp differences due to clock skew in PUT operations.
As the recursive list performed by the import command doesn't retrieve the
versionID, the existing entry may in fact be more complete.
When updating an existing due to clock skew the existing version ID
is propagated to the new entry (note: the etags must match; this is needed
to deal with inconsistent listings).
There is a new s3guard command to audit a s3guard bucket/path's
authoritative state:
hadoop s3guard authoritative -check-config s3a://bucket/path
This is primarily for testing/auditing.
The s3guard bucket-info command also provides some more details on the
authoritative state of a store (HADOOP-16684).
Change-Id: I58001341c04f6f3597fcb4fcb1581ccefeb77d91
2020-01-10 11:11:56 +00:00
17aa8f6764
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.
...
This hardens the wasb and abfs output streams' resilience to being invoked
in/after close().
wasb:
Explicity raise IOEs on operations invoked after close,
rather than implicitly raise NPEs.
This ensures that invocations which catch and swallow IOEs will perform as
expected.
abfs:
When rethrowing an IOException in the close() call, explicitly wrap it
with a new instance of the same subclass.
This is needed to handle failures in try-with-resources clauses, where
any exception in closed() is added as a suppressed exception to the one
thrown in the try {} clause
*and you cannot attach the same exception to itself*
Contributed by Steve Loughran.
Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b
2020-01-08 11:46:54 +00:00
bb1aed475b
HADOOP-16751. Followup: move java import. ( #1799 )
...
This moves the import of a java module to the preferred place in the import ordering.
Change-Id: I1a594e3d954554a72c2b71c954eda0ae940a8f70
2020-01-08 11:32:31 +00:00
a7fccc1122
HADOOP-16727. KMS Jetty server does not startup if trust store password is null.
2020-01-07 15:46:14 -08:00
bc366d4ea7
HADOOP-16773. Fix duplicate assertj-core dependency in hadoop-common module. Contributed by Xieming Li.
2020-01-07 20:49:24 +09:00
b19d87c2b7
HADOOP-16751. DurationInfo text parsing/formatting should be moved out of hotpath.
...
Contributed by Rajesh Balamohan
Change-Id: Icc3dcfa81aa69164f2c088f9b533d231138cbb8b
2020-01-02 17:03:07 +00:00
958764479d
HADOOP-16777. Add Tez to LimitedPrivate of ClusterStorageCapacityExceededException
...
Contributed by Wang Yan.
Change-Id: I92dfe7079ba8ebe89d70255bb845309be0603a8e
2020-01-02 15:49:42 +00:00
d79cce20ab
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.
2020-01-02 11:44:00 +05:30
926d0b48f0
HDFS-15081. Typo in RetryCache#waitForCompletion annotation. Contributed by Fei Hui.
2019-12-27 18:32:15 +05:30
d80d86264d
HADOOP-16042. Update the link to HadoopJavaVersion ( #1784 )
2019-12-27 17:18:10 +09:00
e2a5448d2b
HADOOP-16774. TestDiskChecker and TestReadWriteDiskValidator fails when run with -Pparallel-tests ( #1776 ). Contributed by Vinayakumar B.
2019-12-21 00:08:15 +05:30
ab32762f43
HADOOP-16755. Fix broken link in single node cluster setup documentation. Contributed by Denes Gerencser.
2019-12-16 08:44:28 +05:30
fd7de2b82a
HADOOP-16764. Rewrite Python example codes using Python3 ( #1762 )
2019-12-16 11:04:20 +09:00
631dbbc6f2
YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal
2019-12-15 17:28:04 +01:00
d12ad9e8ad
HADOOP-16757. Increase timeout unit test rule for MetadataStoreTestBase ( #1757 )
...
Contributed by Mingliang Liu.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-12-13 08:19:27 -08:00
c174d50b30
HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen.
2019-12-11 09:39:37 -08:00
19512b21e3
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu
...
This closes #1733
2019-12-05 17:37:17 -08:00
83a14559e5
HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui.
2019-12-05 18:52:01 +05:30
f1ab7f18c4
HADOOP-16718. Allow disabling Server Name Indication (SNI) for Jetty. Contributed by Aravindan Vijayan.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Siyao Meng <smeng@cloudera.com>
2019-12-03 15:47:16 -08:00
2b452b4e60
HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
2019-11-28 10:43:35 +08:00
3161813482
HADOOP-16685: FileSystem#listStatusIterator does not check if given path exists ( #1695 )
2019-11-26 11:11:26 -08:00
da9435182a
HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li.
2019-11-25 14:36:27 +09:00
1a0c0e42e7
HADOOP-16276. Fix jsvc startup command in hadoop-functions.sh due to jsvc >= 1.0.11 changed default current working directory ( #1272 )
2019-11-20 16:44:44 -08:00
6f899e9030
HADOOP-15852. Refactor QuotaUsage. Contributed by David Mollitor.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-11-20 14:40:10 -08:00
9664b9c7a6
HADOOP-15686. Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr. Contributed by Wei-Chiu Chuang.
...
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
2019-11-20 13:07:43 -08:00
be77231452
HADOOP-16700. Track response time separately in Server.Call to avoid negative RpcQueueTime values. Contributed by xuzq.
2019-11-20 07:56:33 -08:00
1415ad3800
HADOOP-16707. NPE in UGI.getCurrentUser in ITestAbfsIdentityTransformer setup.
...
Contributed by Steve Loughran.
Change-Id: I38fdba2fa70e534d78b15e61de19368912588b0c
2019-11-20 14:32:12 +00:00
4050471b45
HADOOP-16657. Move remaining log4j APIs over to slf4j in hadoop-common. Contributed by Minni Mittal.
2019-11-19 08:01:17 +05:30
27642367ef
YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko
2019-11-18 16:29:39 +01:00
7f811722b7
HADOOP-16712. Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml. Contributed by Xieming Li.
2019-11-18 19:00:25 +05:30
67f2c491fe
HDFS-14802. The feature of protect directories should be used in RenameOp ( #1669 )
2019-11-15 12:35:37 -08:00
c892a879dd
HDFS-14882. Consider DataNode load when #getBlockLocation. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Istvan Fajth <pifta@cloudera.com>
2019-11-15 12:16:31 -08:00
92c28c100e
HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li.
2019-11-15 13:49:57 +09:00
d0302d3849
HADOOP-16691. Unify Logging in UserGroupInformation.
...
Unify Logging in UserGroupInformation.
2019-11-14 10:28:26 -08:00
c73334a924
HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log.
...
MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.
2019-11-14 09:54:02 -08:00
a139a87610
HADOOP-16708. HadoopExecutors cleanup to only log at debug.
...
Contributed by David Mollitor.
Change-Id: Ib501a3ed6ed299ccd3c773c4dc086441a5728e73
2019-11-14 17:51:39 +00:00
73a386ab76
HADOOP-16585. [Tool:NNloadGeneratorMR] Multiple threads are using same id for creating file LoadGenerator#write. Contributed by Ranith Sardar.
2019-11-14 01:45:46 +05:30
df6b3162c1
HADOOP-16677. Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. ( #1687 )
2019-11-12 14:19:39 -08:00
3ab98eefa5
HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-11-12 11:34:24 -08:00
990063d2af
HADOOP-16665. Filesystems to be closed if they failed during initialize().
...
Contributed by Steve Loughran.
This FileSystem instantiation so if an IOException or RuntimeException is
raised in the invocation of FileSystem.initialize() then a best-effort
attempt is made to close the FS instance; exceptions raised that there
are swallowed.
The S3AFileSystem is also modified to do its own cleanup if an
IOException is raised during its initialize() process, it being the
FS we know has the "potential" to leak threads, especially in
extension points (e.g AWS Authenticators) which spawn threads.
Change-Id: Ib84073a606c9d53bf53cbfca4629876a03894f04
2019-11-12 18:17:21 +00:00
3d249301f4
HADOOP-16683. Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException. Contributed by Adam Antal
2019-11-09 15:43:53 +01:00
dd900259c4
HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko.
2019-11-06 09:56:19 -08:00
ee8addbec4
Revert "HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng."
...
This reverts commit f9b99d2f24
2019-11-05 15:41:52 -08:00
6f0190d8e4
HADOOP-16678: Review of ArrayWritable ( #1692 )
2019-11-04 14:24:44 -08:00
dca19fc3aa
HADOOP-16484. S3A to warn or fail if S3Guard is disabled ( #1661 ). Contributed by Gabor Bota.
2019-11-04 12:55:36 +01:00
ef9d12df24
Add 2.10.0 release notes for HDFS-12943
2019-10-31 20:33:59 -07:00
f9b99d2f24
HADOOP-16656. Document FairCallQueue configs in core-default.xml. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-31 12:59:46 -07:00
d9fbedc4ae
HDFS-14936. Add getNumOfChildren() for interface InnerNode. Contributed by Lisheng Sun.
2019-10-31 23:39:27 +05:30
8c9173c87c
HADOOP-16510. [hadoop-common] Fix order of actual and expected expression in assert statements. Contributed by Adam Antal
2019-10-31 14:35:20 +01:00
e6137d0e50
Make upstream aware of 2.10.0 release
2019-10-30 20:58:32 -07:00
bd498bac86
HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library.
...
Contributed by Steve Loughran.
Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451
2019-10-29 17:09:26 +00:00
7be5508d9b
HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-25 15:06:16 -07:00
2eba262472
HADOOP-16615. Add password check for credential provider,
...
Contributed by hongdongdong.
Change-Id: Iaac01bc8594860064a80c822a0e47981243ab7e1
2019-10-24 18:48:27 +01:00
2ae4b33d48
HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun.
2019-10-19 11:05:54 +05:30
6d92aa7c30
HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 ( #1656 ). Contributed by Norbert Kalmár, Mate Szalay-Beko
...
* HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5
- Add a static initializer for the unit tests using ZooKeeper to enable
the four-letter-words diagnostic telnet commands. (this is an interface
that become disabled by default, so to keep the ZooKeeper 3.4.x behavior
we enabled it for the tests)
- Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt
records. The new ZooKeeper seems to respond quicker during the fail-over
tests than the ZooKeeper, so we made sure to catch all the relevant records
by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().
Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>
2019-10-18 13:26:20 -07:00
3d41f33018
HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.
...
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-18 08:25:02 -07:00
c79a5f2d99
HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal
2019-10-16 13:29:06 +02:00
c39e9fc9a3
HADOOP-15169. "hadoop.ssl.enabled.protocols" should be considered in httpserver2. Contributed by Brahma Reddy Battula, Wei-Chiu Chuang.
...
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-By: Brahma Reddy Battula <brahma@apache.org>
2019-10-15 13:55:23 -07:00
dee9e97075
Revert "HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos."
...
This reverts commit 7a4b3d42c4
2019-10-14 16:56:50 +01:00
44bc095998
HADOOP-13907. Fix TestWebDelegationToken#testKerberosDelegationTokenAuthenticator on Windows. Contributed by Kitti Nanasi.
2019-10-12 13:16:51 +05:30
7a4b3d42c4
HADOOP-15870. S3AInputStream.remainingInFile should use nextReadPos.
...
Contributed by lqjacklee.
Change-Id: I32bb00a683102e7ff8ff8ce0b8d9c3195ca7381c
2019-10-10 21:58:42 +01:00
5462d2176f
YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena
2019-10-08 11:20:13 -07:00
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3
...
Contributed by Steve Loughran.
Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.
This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.
Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
2019-10-01 18:11:05 +01:00
13b427fc05
HDFS-14564: Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable ( #963 ) Contributed by Sahil Takiar.
...
Reviewed-by: Siyao Meng <smeng@cloudera.com>
2019-09-27 12:25:03 -07:00
2870668cfa
Make upstream aware of 3.1.3 release.
2019-09-27 16:44:07 +08:00
587a8eeec8
HDFS-14874. Fix TestHDFSCLI and TestDFSShell test break because of logging change in mkdir ( #1522 ). Contributed by Gabor Bota.
2019-09-26 10:15:13 +05:30
e346e3638c
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
2019-09-25 12:16:41 +01:00
c72457787d
YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko
2019-09-25 10:36:22 +02:00
3f89084ac7
HDFS-14845. Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-09-25 10:19:30 +09:00
aa664d7259
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE ( #1302 ). Contributed by Gabor Bota.
...
Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023
2019-09-23 13:29:01 +02:00
07c81e9bfc
HADOOP-16558. [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes ( #1494 ). Contributed by Vinayakumar B.
2019-09-23 12:37:47 +05:30
4c0a7a9e13
Make upstream aware of 3.2.1 release.
2019-09-23 06:20:54 +00:00
659c88801d
HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He.
2019-09-22 22:02:54 +05:30
dd0834696a
HADOOP-16581. Revise ValueQueue to correctly replenish queues that go below the watermark ( #1463 )
...
In the existing implementation, the ValueQueue::getAtMost() method will only trigger a refill on a key queue if it has gone empty, instead of triggering a refill when it has gone below the watermark. Revise the test suite to correctly verify this behavior.
2019-09-20 09:55:48 -07:00
1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 ( #1432 )
...
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
298f07abce
HADOOP-16069. Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST. Contributed by luhuachao.
2019-09-20 13:55:34 +09:00
56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM.
...
Contributed by Malcolm Taylor.
Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
2019-09-19 16:00:05 +01:00
d4205dce17
HADOOP-16582. LocalFileSystem's mkdirs() does not work as expected under viewfs. Contributed by Kihwal Lee
2019-09-19 08:23:35 -05:00
f8c14326ee
YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu.
2019-09-18 09:12:28 +08:00
3cf6e4272f
HDFS-14846: libhdfs tests are failing on trunk due to jni usage bugs
...
Signed-off-by: Anu Engineer <aengineer@apache.org>
2019-09-17 15:51:39 -07:00
55ce454ce4
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.
...
Contributed by Sahil Takiar.
This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common
as the DelegatingSSLSocketFactory and binds the S3A connector to it so that
it can avoid using those HTTPS algorithms which are underperformant on Java 8.
Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd
2019-09-17 11:32:03 +01:00
9221704f85
HADOOP-16490. Avoid/handle cached 404s during S3A file creation.
...
Contributed by Steve Loughran.
This patch avoids issuing any HEAD path request when creating a file with overwrite=true,
so 404s will not end up in the S3 load balancers unless someone calls getFileStatus/exists/isFile
in their own code.
The Hadoop FsShell CommandWithDestination class is modified to not register uncreated files
for deleteOnExit(), because that calls exists() and so can place the 404 in the cache, even
after S3A is patched to not do it itself.
Because S3Guard knows when a file should be present, it adds a special FileNotFound retry policy
independently configurable from other retry policies; it is also exponential, but with
different parameters. This is because every HEAD request will refresh any 404 cached in
the S3 Load Balancers. It's not enough to retry: we have to have a suitable gap between
attempts to (hopefully) ensure any cached entry wil be gone.
The options and values are:
fs.s3a.s3guard.consistency.retry.interval: 2s
fs.s3a.s3guard.consistency.retry.limit: 7
The S3A copy() method used during rename() raises a RemoteFileChangedException which is not caught
so not downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated.
Copy operations without S3Guard lack the confidence that the file exists, so don't retry the same way:
it will fail fast with a different error message. However, because create(path, overwrite=false) no
longer does HEAD path, we can at least be confident that S3A itself is not creating those cached
404 markers.
Change-Id: Ia7807faad8b9a8546836cb19f816cccf17cca26d
2019-09-11 16:46:25 +01:00
bc2d3a71d6
HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2019-09-10 10:51:47 +08:00
147f98629c
HADOOP-16438. ADLS Gen1 OpenSSL config control.
...
Contributed by Sneha Vijayarajan.
Change-Id: Ib79ea6b4a90ad068033e175f3f59c5185868872d
2019-09-09 17:09:32 +01:00
bb0b922a71
HADOOP-16255. Add ChecksumFs.rename(path, path, boolean)
...
Contributed by Jungtaek Lim
Change-Id: If00a4d7d30456c08eb2b0f7e2b242197bc4ee05d
2019-09-06 21:53:00 +01:00
a23417533e
HADOOP-16531. Log more timing information for slow RPCs. Contributed by Chen Zhang.
2019-09-06 10:28:21 -07:00
c92a3e94d8
HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun.
2019-09-06 10:22:28 -07:00
511df1e837
HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions
...
Contributed by Steve Loughran.
This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.
For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.
Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
337e9b794d
HADOOP-16268. Allow StandbyException to be thrown as CallQueueOverflowException when RPC call queue is filled. Contributed by CR Hota.
2019-09-04 08:22:02 -07:00
5ff76cb8bc
HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina.
2019-09-03 12:37:09 +05:30
915cbc91c0
HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-09-02 09:47:04 -07:00
dc0acceabb
YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko.
2019-08-30 10:13:10 +09:00
29bd6f3fc3
HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun.
2019-08-28 23:58:23 +05:30
55cc115878
HADOOP-16527. Add a whitelist of endpoints to skip Kerberos authentication ( #1336 ) Contributed by Akira Ajisaka.
2019-08-28 14:28:41 +09:00
61b2df2331
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper.
...
Contributed by Steve Loughran.
Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.
Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8
2019-08-22 17:27:56 +01:00
30ce8546f1
HADOOP-16496. Apply HDDS-1870 (ConcurrentModification at PrometheusMetricsSink) to Hadoop common.
...
This closes #1317
Reviewed-by: Bharat Viswanadham <bharat@apache.org>
2019-08-21 10:10:11 +09:00
51b65370b9
HADOOP-14784. [KMS] Improve KeyAuthorizationKeyProvider#toString(). Contributed by Yeliang Cang.
...
Reviewed-by: Dinesh Chitlangia <dchitlangia@cloudera.com>
2019-08-19 11:12:09 -07:00
a707bb7c1b
HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-18 18:44:35 -07:00
971a4c8e83
HDFS-14523. Remove excess read lock for NetworkToplogy. Contributed by Wu Weiwei.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Chen Liang <cliang@apache.org>
2019-08-16 17:19:58 -07:00
e356e4f4b7
HADOOP-16391 Add a prefix to the metric names for MutableRatesWithAggregation used for deferred RPC metrics to avoid collision with non-deferred metrics. Contributed by Bilwa S T.
2019-08-16 09:01:44 -07:00
5882cf94ea
HADOOP-16504. Increase ipc.server.listen.queue.size default from 128 to 256. Contributed by Lisheng Sun.
2019-08-15 15:20:54 -07:00
0f8add8a60
HADOOP-16495. Fix invalid metric types in PrometheusMetricsSink ( #1244 )
2019-08-14 12:24:03 +09:00
da0006fe04
HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-14 08:39:40 +09:00
6b4564f1d5
HADOOP-16453. Update how exceptions are handled in NetUtils. Contributed by Lisheng Sun.
2019-08-11 20:34:36 -07:00
e25a5c2eab
HADOOP-16499. S3A retry policy to be exponential ( #1246 ). Contributed by Steve Loughran.
2019-08-09 15:52:37 +02:00
df30d8ea09
YARN-9727: Allowed Origin pattern is discouraged if regex contains *. Contributed by Zoltan Siegl
2019-08-09 09:34:23 +02:00
b0131bc265
HADOOP-15014. Addendum: KMS should log the IP address of the clients. Contributed by Zsombor Gegesy.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-07 20:57:42 -07:00
a5bb1e8ee8
HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun.
2019-08-07 10:18:11 +08:00
8cef9f89f4
HDFS-14652. Addendum: HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang.
2019-08-06 15:24:11 -07:00
22430c10e2
HADOOP-16457. Fixed Kerberos activation in ServiceAuthorizationManager.
...
Contributed by Prabhu Joseph
2019-08-06 17:04:17 -04:00
71aad60e51
HDFS-14691. Wrong usage hint for hadoop fs command "test".
...
Contributed by Jianfei Jiang.
Change-Id: I9f5e89721ff210641375fbf42a70043f0d74458e
2019-08-05 13:08:47 +01:00
61180f4656
HADOOP-15942. Change the logging level form DEBUG to ERROR for RuntimeErrorException in JMXJsonServlet. Contributed by Anuhan Torgonshar.
2019-08-02 14:58:24 -07:00
797d14e816
Revert "HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm."
...
This reverts commit 076618677d
2019-08-02 08:25:41 -07:00
e872ceb810
HADOOP-15865. ConcurrentModificationException in Configuration.overlay() method. Contributed by Oleksandr Shevchenko.
2019-08-01 19:56:51 -07:00
e20b19543b
HADOOP-15681. AuthenticationFilter should generate valid date format for Set-Cookie header regardless of default Locale. Contributed by Cao Manh Dat.
2019-08-01 17:35:31 -07:00
d086d058d8
HDFS-14652. HealthMonitor connection retry times should be configurable. Contributed by Chen Zhang.
2019-08-01 16:13:10 -07:00
b94eba9f11
HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun.
2019-08-01 15:50:43 -07:00
f86de6f76a
HDFS-13529. Fix default trash policy emptier trigger time correctly. Contributed by He Xiaoqiao.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-01 14:54:39 -07:00
8bda91d20a
HADOOP-16398. Exports Hadoop metrics to Prometheus ( #1170 )
2019-07-31 10:11:36 -07:00
0f2dad6679
HDFS-14569. Result of crypto -listZones is not formatted properly. Contributed by hemanthboyina.
2019-07-30 16:52:42 -07:00
c75f16db79
HADOOP-16452. Increase ipc.maximum.data.length default from 64MB to 128MB. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-07-30 11:22:45 -07:00
204a977f55
HADOOP-15910. Fix Javadoc for LdapAuthenticationHandler#ENABLE_START_TLS
...
Contributed by Don Jeba.
Change-Id: I2755bfb1263fc659078a1af8f0bdfd739fd1ae40
2019-07-30 12:39:48 +01:00
cbfa3f3e98
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich.
2019-07-29 17:37:26 -07:00
62efb63006
HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen.
2019-07-26 11:16:58 -07:00
aebac6d2d2
HADOOP-16461. Regression: FileSystem cache lock parses XML within the lock.
...
Contributed by Gopal V.
Change-Id: If6654f850e9c24ee0d9519a46fd6269b18e1a7a4
2019-07-26 11:32:13 +01:00
07530314c2
HADOOP-9844. NPE when trying to create an error message response of SASL RPC
...
This closes #55
Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-26 17:53:18 +09:00
a1251addff
HADOOP-16431. Remove useless log in IOUtils.java and ExceptionDiags.java.
...
This closes #1091
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-24 10:04:39 +09:00
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories.
...
Contributed by Steve Loughran and Gabor Bota.
This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.
It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".
Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
2019-07-23 14:52:03 +01:00
eb36b09cb7
HADOOP-16443. Improve help text for setfacl --set option.
...
Contributed by S O'Donnell.
Change-Id: I1da46c4c414a5d2b07ee15867508f0799440a413
2019-07-23 10:24:07 +01:00
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration ( #1125 )
2019-07-19 09:49:59 -06:00
d545f9c290
HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise
...
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise
Contributed by: Josh Rosen
2019-07-18 23:19:38 +01:00
b4466a3b0a
HADOOP-16341. ShutDownHookManager: Regressed performance on Hook removals after HADOOP-15679
...
Contributed by Gopal V and Atilla Magyar.
Change-Id: I066d5eece332a1673594de0f9b484443f95530ec
2019-07-17 13:50:02 +01:00
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
...
Contributed by Steve Loughran
Contains
- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently
This patch doesn't fix the underlying problem but it
* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones
The outcome is that tests should pass consistently and if problems occur we have better diagnostics.
Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
2019-07-12 13:02:25 +01:00
8fb5ca3f40
HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang.
2019-07-11 13:23:25 -07:00
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
...
Contributed by Steve Loughran.
This patch
* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.
Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
2019-07-11 18:15:34 +01:00
aa9f0850e8
HADOOP-16385. Namenode crashes with 'RedundancyMonitor thread received Runtime exception'. Contributed by Ayush Saxena.
2019-07-02 13:22:30 -07:00
129576f628
HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian.
2019-06-24 12:09:17 -07:00
95c94dcca7
HADOOP-16350. Ability to tell HDFS client not to request KMS Information from NameNode. Ccontributed by Greg Senia, Ajay Kumar.
2019-06-24 11:59:07 -07:00
b52fd05d42
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
ba221dc50e
HDFS-14526. RBF: Update the document of RBF related metrics. Contributed by Takanobu Asanuma.
2019-06-24 22:03:04 +05:30
559cb11551
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails.
2019-06-24 22:03:03 +05:30
f40e10b349
HDFS-14129. RBF: Create new policy provider for router. Contributed by Ranith Sardar.
2019-06-24 22:03:03 +05:30
371452e260
HADOOP-15989. Synchronized at CompositeService#removeService is not required. Contributed by Prabhu Joseph.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-21 17:05:13 -07:00
840d02ca5b
HADOOP-14385. HttpExceptionUtils#validateResponse swallows exceptions. Contributed by Wei-Chiu Chuang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-20 19:57:48 -07:00
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.
...
Contributed by Steve Loughran.
Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
2019-06-20 09:56:40 +01:00
3ab77d9bc9
HDFS-14201. Ability to disallow safemode NN to become active. Contributed by Xiao Liang and He Xiaoqiao.
2019-06-18 09:58:29 -07:00
dd4a7633ec
HADOOP-16156. [Clean-up] Remove NULL check before instanceof and fix checkstyle in InnerNodeImpl. Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 23:47:57 -07:00
56c621c68f
HADOOP-15914. hadoop jar command has no help argument. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-17 22:56:00 -07:00
f1c239c6a4
HADOOP-9157. Better option for curl in hadoop-auth-examples. Contributed by Andras Bokor.
2019-06-17 21:51:33 -07:00
10311c30b0
HADOOP-14807. should prevent the possibility of NPE about ReconfigurableBase.java. Contributed by hu xiaodong.
2019-06-17 19:55:56 -07:00
1e92db5a1e
HDFS-11949. Add testcase for ensuring that FsShell cann't move file to the target directory that file exists. Contributed by legend.
2019-06-17 18:29:34 -07:00
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).
...
Contributed by Gabor Bota.
Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
2019-06-16 17:05:01 +01:00
076618677d
HADOOP-16336. finish variable is unused in ZStandardCompressor. Contributed by cxorm.
2019-06-14 20:05:20 -07:00
c7554ffd5c
HADOOP-16373. Fix typo in FileSystemShell#test documentation ( #968 )
2019-06-14 14:36:04 -07:00
54f9f75a44
HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles.
2019-06-14 10:17:25 +09:00
4ea6c2f457
HADOOP-16354. Enable AuthFilter as default for WebHDFS.
...
Contributed by Prabhu Joseph
2019-06-11 18:41:08 -04:00
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-11 18:40:35 +09:00
d6d95d2686
HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-10 10:23:18 -07:00
76b94c274f
HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen.
2019-06-07 14:20:44 -07:00
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables.
...
Contributed by Steve Loughran
Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
2019-06-07 18:26:10 +01:00
14552d19e3
HADOOP-16344. Make DurationInfo public unstable.
...
Second attempt: imports fixed up.
Contributed by Huan-Ping Su and Kai Xie.
Change-Id: Ib2a85dd4b2d12b840692cc854cc53ddb3bdde7d5
2019-06-07 12:55:17 +01:00
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java
2019-06-07 10:27:41 +09:00
1f244b4227
Revert "HADOOP-16344. Make DurationInfo public unstable."
...
This reverts commit 829848ba2e
2019-06-06 17:49:31 +01:00
829848ba2e
HADOOP-16344. Make DurationInfo public unstable.
...
Contributed by Huan-Ping Su
Change-Id: I64a94cf382f9db78b4ef49a3912f25f0d906af7c
2019-06-06 12:52:49 +01:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
d1aad44490
HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He.
2019-06-05 21:33:00 +08:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
7724d8031b
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
...
This reverts commit 5906268f0d
2019-06-05 13:54:42 +01:00
827a84778a
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer -follow-on patch.
...
This follow-on patch to HADOOP-16266 fixes up the problem where logs were being full of
stack traces because the timeout passed down to select was in nanos, whereas the API
expected millis.
Contributed by Erik Krogen.
Change-Id: I5c6e9ddf68127b1d7e0ca0e179d036eb9941e445
2019-06-04 15:56:47 +01:00
6f5a36c13c
HADOOP-13656. fs -expunge to take a filesystem. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-30 13:21:58 -07:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
93d7630062
HDFS-14451. Incorrect header or version mismatch log message. Contributed by Shweta.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-24 18:42:46 +02:00
f96a2df38d
HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
2019-05-23 10:28:37 -07:00
ea0b1d8fba
HADOOP-16287. Implement ProxyUserAuthenticationFilter for web protocol impersonation.
...
Contributed by Prabhu Joseph
2019-05-23 11:36:32 -04:00
a771e2a638
HADOOP-12948. Remove the defunct startKdc profile from hadoop-common. Contributed by Wei-Chiu Chuang.
2019-05-23 13:59:42 +09:00
5906268f0d
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
2019-05-21 11:30:45 -06:00
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
4cb3da6ac7
HADOOP-16248. MutableQuantiles leak memory under heavy load.
...
Contributed by Alexis Daboville,
2019-05-17 15:15:22 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
2713dcf6e9
HADOOP-16307. Intern User Name and Group Name in FileStatus.
...
Author: David Mollitor
2019-05-16 16:02:07 +02:00
d4c8858586
HADOOP-16247. NPE in FsUrlConnection. Contributed by Karthik Palanisamy.
2019-05-15 17:41:36 -07:00
389e640f0c
HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao.
2019-05-13 11:46:16 -07:00
f257497b0f
HADOOP-16299. [JDK 11] Build fails without specifying -Djavac.version=11
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-09 14:49:46 +09:00
96dc5cedfe
HADOOP-16293. AuthenticationFilterInitializer doc has speudo instead of pseudo.
...
Author: Prabhu Joseph
2019-05-08 10:18:20 +01:00
713e8a27ae
HADOOP-16238. Add the possbility to set SO_REUSEADDR in IPC Server Listener. Contributed by Peter Bacsko.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-07 17:48:27 -07:00
93f2283a69
HADOOP-16289. Allow extra jsvc startup option in hadoop_start_secure_daemon in hadoop-functions.sh. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-05-06 15:47:33 -07:00
f1875b205e
HADOOP-16059. Use SASL Factories Cache to Improve Performance. Contributed by Ayush Saxena.
2019-05-03 11:22:14 +05:30
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
4877f0aa51
HDFS-3246: pRead equivalent for direct read path ( #597 )
...
HDFS-3246: pRead equivalent for direct read path
Contributed by Sahil Takiar
2019-04-30 14:52:16 -07:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
f4ab9370f5
HADOOP-16026:Replace incorrect use of system property user.name.
...
Contributed by Dinesh Chitlangia.
2019-04-22 14:02:13 -07:00
fb1c549139
HDFS-14374. Expose total number of delegation tokens in AbstractDelegationTokenSecretManager. Contributed by CR Hota.
2019-04-22 13:32:08 -07:00
1ddb48872f
HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin.
2019-04-22 08:16:57 -07:00
008766c119
HADOOP-15014. KMS should log the IP address of the clients. Contributed by Zsombor Gegesy.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-16 05:28:18 -07:00
b1120d27ab
HADOOP-16249. Make CallerContext LimitedPrivate scope to Public.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-16 11:18:21 +09:00
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre.
...
Author: Gabor Bota <gabor.bota@cloudera.com>
2019-04-12 18:28:38 -07:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
626fec652b
HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang.
2019-04-12 17:37:51 -07:00
87407553ef
HADOOP-16243. Change Log Level to trace in NetUtils.java. Contributed by chencan.
2019-04-10 13:21:04 -07:00
813cee1a18
HDFS-14420. Fix typo in KeyShell console. Contributed by Hu Xiaodong.
2019-04-10 11:18:40 -07:00
65deb1ac42
HADOOP-16179. hadoop-common pom should not depend on kerb-simplekdc
...
The hadoop-common pom currently has a dependency on kerb-simplekdc. In
fact, the only classes used from Kerby are in kerb-core and kerb-util
(which is a transitive dependency frmo kerb-core). Depending on
kerb-simplekdc pulls a bunch of other unnecessary classes into the
hadoop-common classpath.
This changes the hadoop-common pom to depend only on kerb-core.
hadoop-minikdc already had the appropriate dependency on kerb-simplekdc
so it continues to pull in what it needs.
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-10 08:49:46 -07:00
bb8dda2bf9
HADOOP-12890. Fix typo in AbstractService. Contributed by Gabor Liptak.
2019-04-08 15:26:12 +09:00
ab645b3caa
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang.
2019-04-08 15:18:45 +09:00
72f4b9cd68
HADOOP-15242. Fix typos in hadoop-functions.sh. Contributed by Ray Chiang.
2019-04-08 13:20:21 +09:00
0d47d283a6
HADOOP-10848. Cleanup calling of sun.security.krb5.Config.
2019-04-08 10:02:34 +09:00
c90736350b
HADOOP-16208. Do Not Log InterruptedException in Client.
...
Contributed by David Mollitor.
2019-04-04 16:15:57 +01:00
7b5b783f66
HDFS-14327. Using FQDN instead of IP to access servers with DNS resolving. Contributed by Fengnan Li.
2019-04-03 16:11:13 -07:00
e62cbcbc83
HADOOP-16011. OsSecureRandom very slow compared to other SecureRandom implementations. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-03 14:29:52 -07:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
3b0c5016b2
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags
...
Signed-off-by: Todd Lipcon <todd@apache.org>
2019-04-03 10:56:33 -07:00
aaaf856f4b
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str
2019-04-03 13:16:59 +09:00
cf268114c9
HDFS-13960. hdfs dfs -checksum command should optionally show block size in output. Contributed by Lokesh Jain.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-02 12:24:55 -07:00
61d19110d4
HADOOP-16218. Findbugs warning of null param to non-nullable method in Configuration with Guava update. ( #655 )
...
Change-Id: I461e518ce9a4730b91a8138ad55b39e9a4b0a4b8
2019-04-02 09:15:11 +01:00
ebd0d21538
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji.
2019-04-02 10:51:47 +09:00
f41f938b2e
HADOOP-16199. KMSLoadBlanceClientProvider does not select token correctly. Contributed by Xiaoyu Yao.
...
This closes #642 .
2019-03-28 21:55:31 -07:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
d18d0859eb
HADOOP-16181. HadoopExecutors shutdown Cleanup.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:29:27 +00:00
246ab77f28
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:26:24 +00:00
5f6e225166
YARN-9363. Replaced debug logging with SLF4J parameterized log message.
...
Contributed by Prabhu Joseph
2019-03-18 13:57:18 -04:00
5446e3cb8a
HADOOP-16167. Fixed Hadoop shell script for Ubuntu 18.
...
Contributed by Daniel Templeton
2019-03-18 13:04:49 -04:00
8c95cb9d6b
HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed.
2019-03-18 08:13:43 -07:00
2db38abffc
HDFS-14328. [Clean-up] Remove NULL check before instanceof in TestGSet
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I5b9f0e66664714d7c5bbfa30492a09f770626711
2019-03-18 15:10:26 +01:00
2064ca015d
YARN-9349. Changed logging to use slf4j api.
...
Contributed by Prabhu Joseph
2019-03-15 19:20:59 -04:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
66357574ae
HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun.
2019-03-13 13:15:56 -07:00
f74159c8fc
HADOOP-16166. TestRawLocalFileSystemContract fails with build Docker container running on Mac.
...
Also provided similar fix for Windows.
2019-03-13 09:33:24 -07:00
2a54feabb2
HDFS-14347. [SBN Read] Restore a comment line mistakenly removed in ProtobufRpcEngine. Contributed by Fengnan Li.
2019-03-11 18:59:15 -07:00
4ad295a4f1
HDFS-14270.[SBN Read] Add trace level logging for stateId in RPC Server. Contributed by Shweta Yakkali.
2019-03-11 13:48:06 -07:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
2b94e51a8f
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host.
...
Author: Praveen Krishna <praveenkrishna@tutanota.com>
2019-03-07 11:06:34 +00:00
a55fc36299
HDDS-1093. Configuration tab in OM/SCM ui is not displaying the correct values.
2019-03-06 17:43:57 -08:00
618e009ac0
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 15:04:06 -08:00
6192c1fe3b
Revert "HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar."
...
This reverts commit f5a4b43a49
2019-03-06 15:02:18 -08:00
f5a4b43a49
HDFS-14111. hdfsOpenFile on HDFS causes unnecessary IO from file offset 0. Contributed by Sahil Takiar.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-03-06 14:58:45 -08:00
3c5b7136e2
HADOOP-16150. Added concat method to ChecksumFS as unsupported operation.
...
Contributed by Steve Loughran
(cherry picked from commit 8b517e7ad6
2019-03-05 13:32:00 -05:00
686c0141ef
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately.
...
Contributed by Stephen O'Donnell.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:09:00 +00:00
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-03-05 14:10:08 +09:00
fe7551f21b
HADOOP-16162. Remove unused Job Summary Appender configurations from log4j.properties
...
This closes #551
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-03-05 13:41:33 +09:00
9fcd89ab93
HADOOP-16148. Cleanup LineReader Unit Test.
...
Contributed by David Mollitor.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-04 23:08:12 +00:00
Akira Ajisaka
Vinayakumar B
Chao Sun
David Mollitor
Wei-Chiu Chuang
Chen Liang
Eric Yang
Daryn Sharp
Steve Loughran
Hanisha Koneru
Rajesh Balamohan
Rakesh Radhakrishnan
Ayush Saxena
Chandra Sanivarapu
Kengo Seki
Szilard Nemeth
Mingliang Liu
Erik Krogen
Aravindan Vijayan
Yiqun Lin
Sahil Takiar
Siyao Meng
Abhishek Modi
Hui Fei
He Xiaoqiao
belugabehr
Lukas Majercak
Surendra Singh Lilhore
Xudong Cao
Siyao Meng
Gabor Bota
Jonathan Hung
Jonathan Hung
hongdongdong
Mate Szalay-Beko
Yuming Wang
Steve Loughran
lqjacklee
Zhankun Tang
Prabhu Joseph
Rohith Sharma K S
Yuval Degani
Takanobu Asanuma
Malcolm Taylor
Kihwal Lee
Weiwei Yang
Sahil Takiar
Sahil Takiar
Daisuke Kobayashi
Sneha Vijayarajan
Jungtaek Lim (HeartSaVioR)
Stephen O'Donnell
Masatake Iwasaki
Inigo Goiri
Zsombor Gegesy
Jianfei Jiang
Don Jeba
Gopal V
sunlisheng
Sean Mackrory
Josh Rosen
Christopher Gregorian
Ajay Kumar
Shweta Yakkali
Adam Antal
cxorm
dineshchitlangia
Wanqiang Ji
Erik Krogen
Huan-Ping Su
Sammi Chen
Ben Roling
Alexis Daboville
Bharat Viswanadham
Peter Bacsko
Giovanni Matteo Fumarola
Ben Roling
Sean Mackrory
Anu Engineer
Kenneth Yang
Arpit Agarwal
Todd Lipcon
Lokesh Jain
Xiaoyu Yao
Matt Foley
Konstantin V Shvachko
Praveen Krishna
Vivek Ratnavel Subramanian
Sahil Takiar
coder-chenzhi