<ahref="http://maven.apache.org/"title="Built by Maven"class="poweredBy">
<imgalt="Built by Maven"src="../../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<divid="bodyColumn">
<divid="contentBox">
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<h1>Apache Hadoop 3.3.0 Release Notes</h1>
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-8587">YARN-8587</a> | <i>Major</i> | <b>Delays are noticed to launch docker container</b></li>
</ul>
<p>add “docker inspect” retries to discover container exit code.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14053">HDFS-14053</a> | <i>Major</i> | <b>Provide ability for NN to re-replicate based on topology changes.</b></li>
</ul>
<p>A new option (-replicate) is added to fsck command to re-trigger the replication for mis-replicated blocks. This option should be used instead of previous workaround of increasing and then decreasing replication factor (using hadoop fs -setrep command).</p><hr/>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-8986">YARN-8986</a> | <i>Minor</i> | <b>publish all exposed ports to random ports when using bridge network</b></li>
</ul>
<p>support -p and -P for bridge type network;</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/MAPREDUCE-6190">MAPREDUCE-6190</a> | <i>Major</i> | <b>If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck</b></li>
</ul>
<p>Added “mapreduce.task.stuck.timeout-ms” parameter to timeout a task before sending the task’s first heartbeat. The default value is 600000 milliseconds (10 minutes).</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-9071">YARN-9071</a> | <i>Critical</i> | <b>NM and service AM don’t have updated status for reinitialized containers</b></li>
</ul>
<p>In progress upgrade status may show READY state sooner than actual upgrade operations. External caller to upgrade API is recommended to wait minimum 30 seconds before querying yarn app -status.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15428">HADOOP-15428</a> | <i>Major</i> | <b>s3guard bucket-info will create s3guard table if FS is set to do this automatically</b></li>
</ul>
<p>If -unguarded flag is passed to `hadoop s3guard bucket-info`, it will now proceed with S3Guard disabled instead of failing if S3Guard is not already disabled.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16000">HADOOP-16000</a> | <i>Major</i> | <b>Remove TLSv1 and SSLv2Hello from the default value of hadoop.ssl.enabled.protocols</b></li>
</ul>
<p>TLSv1 and SSLv2Hello were removed from the default value of “hadoop.ssl.enabled.protocols”.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-9084">YARN-9084</a> | <i>Major</i> | <b>Service Upgrade: With default readiness check, the status of upgrade is reported to be successful prematurely</b></li>
</ul>
<p>Improve transient container status accuracy for upgrade.</p><hr/>
<ul>
<li>
<p><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-8762">YARN-8762</a> | <i>Major</i> | <b>[Umbrella] Support Interactive Docker Shell to running Containers</b></p>
</li>
<li>
<p>Add shell access to YARN containers</p>
</li>
</ul><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15965">HADOOP-15965</a> | <i>Major</i> | <b>Upgrade to ADLS SDK which has major performance improvement for ingress/egress</b></li>
</ul><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15996">HADOOP-15996</a> | <i>Major</i> | <b>Plugin interface to support more complex usernames in Hadoop</b></li>
</ul>
<p>This patch enables “Hadoop” and “MIT” as options for “hadoop.security.auth_to_local.mechanism” and defaults to ‘hadoop’. This should be backward compatible with pre-HADOOP-12751.</p>
<p>This is basically HADOOP-12751 plus configurable + extended tests.</p><hr/>
<ul>
<li>
<p><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-8489">YARN-8489</a> | <i>Major</i> | <b>Need to support “dominant” component concept inside YARN service</b></p>
</li>
<li>
<p>Improved YARN service status report based on dominant component status.</p>
</li>
</ul><hr/>
<ul>
<li>
<p><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15922">HADOOP-15922</a> | <i>Major</i> | <b>DelegationTokenAuthenticationFilter get wrong doAsUser since it does not decode URL</b></p>
</li>
<li>
<p>Fix DelegationTokenAuthentication filter for incorrectly double encode doAs user parameter.</p>
<p>After this change, capacity scheduler queue is able to inherit and override max-allocation from its parent queue. The subqueue’s max-allocation can be larger or smaller than the parent queue’s but can’t exceed the global “yarn.scheduler.capacity.maximum-allocation”. User can set queue-level max-allocation on any level of queues, using configuration property “yarn.scheduler.capacity.%QUEUE_PATH%.maximum-allocation”. And this property allows user to set max-resource-allocation for customized resource types, e.g “memory=20G,vcores=20,gpu=3”.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14158">HDFS-14158</a> | <i>Minor</i> | <b>Checkpointer ignores configured time period > 5 minutes</b></li>
</ul>
<p>Fix the Checkpointer not to ignore the configured “dfs.namenode.checkpoint.period”> 5 minutes</p><hr/>
<ul>
<li>
<p><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-8761">YARN-8761</a> | <i>Major</i> | <b>Service AM support for decommissioning component instances</b></p>
</li>
<li>
<p>Component instance number is not linear increment when decommission feature is used. Application with assumption of linear increment component instance number maybe impacted by introduction of this feature.</p>
<p>Thanks for the patch, [~shwetayakkali], and review, [~knanasi]. Committed to trunk.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14118">HDFS-14118</a> | <i>Major</i> | <b>Support using DNS to resolve nameservices to IP addresses</b></li>
</ul>
<p>HDFS clients can use a single domain name to discover servers (namenodes/routers/observers) instead of explicitly listing out all hosts in the config</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7133">HDFS-7133</a> | <i>Major</i> | <b>Support clearing namespace quota on “/”</b></li>
</ul>
<p>Namespace Quota on root can be cleared now.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16011">HADOOP-16011</a> | <i>Major</i> | <b>OsSecureRandom very slow compared to other SecureRandom implementations</b></li>
</ul>
<p>The default RNG is now OpensslSecureRandom instead of OsSecureRandom.The high-performance hardware random number generator (RDRAND instruction) will be used if available. If not, it will fall back to OpenSSL secure random generator. If you insist on using OsSecureRandom, set hadoop.security.secure.random.impl in core-site.xml to org.apache.hadoop.crypto.random.OsSecureRandom.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16210">HADOOP-16210</a> | <i>Critical</i> | <b>Update guava to 27.0-jre in hadoop-project trunk</b></li>
</ul>
<p>Guava has been updated to 27.0. Code built against this new version may not link against older releases because of new overloads of methods like Preconditions.checkArgument().</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16085">HADOOP-16085</a> | <i>Major</i> | <b>S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite</b></li>
</ul>
<p>S3Guard will now track the etag of uploaded files and, if an S3 bucket is versioned, the object version. You can then control how to react to a mismatch between the data in the DynamoDB table and that in the store: warn, fail, or, when using versions, return the original value.</p>
<p>This adds two new columns to the table: etag and version. This is transparent to older S3A clients -but when such clients add/update data to the S3Guard table, they will not add these values. As a result, the etag/version checks will not work with files uploaded by older clients.</p>
<p>For a consistent experience, upgrade all clients to use the latest hadoop version.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15563">HADOOP-15563</a> | <i>Major</i> | <b>S3Guard to support creating on-demand DDB tables</b></li>
</ul>
<p>S3Guard now defaults to creating DynamoDB tables as “On-Demand”, rather than with a prepaid IO capacity. This reduces costs when idle to only the storage of the metadata entries, while delivering significantly faster performance during query planning and other bursts of IO. Consult the S3Guard documentation for further details.</p><hr/>
<p>This patch changes the format of app activities query URL to <aclass="externalLink"href="http://{rm-host}:{port}/ws/v1/cluster/scheduler/app-activities/{app-id}/">http://{rm-host}:{port}/ws/v1/cluster/scheduler/app-activities/{app-id}/</a>, app-id now is a path parameter instead of a query parameter.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14339">HDFS-14339</a> | <i>Major</i> | <b>Inconsistent log level practices in RpcProgramNfs3.java</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<p>This adds an extension to the IPC FairCallQueue which allows for the consideration of the *cost* of a user’s operations when deciding how they should be prioritized, as opposed to the number of operations. This can be helpful for protecting the NameNode from clients which submit very expensive operations (e.g. large listStatus operations or recursive getContentSummary operations).</p>
<p>This can be enabled by setting the `ipc.<port>.costprovder.impl` configuration to `org.apache.hadoop.ipc.WeightedTimeCostProvider`.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16460">HADOOP-16460</a> | <i>Major</i> | <b>ABFS: fix for Sever Name Indication (SNI)</b></li>
</ul>
<p>ABFS: Bug fix to support Server Name Indication (SNI).</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16452">HADOOP-16452</a> | <i>Major</i> | <b>Increase ipc.maximum.data.length default from 64MB to 128MB</b></li>
</ul>
<p>Default ipc.maximum.data.length is now 128 MB in order to accommodate huge block reports.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-13783">HDFS-13783</a> | <i>Major</i> | <b>Balancer: make balancer to be a long service process for easy to monitor it.</b></li>
</ul>
<p>Adds a new parameter to the Balancer CLI, “-asService”, to enable the process to be long-running.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16398">HADOOP-16398</a> | <i>Major</i> | <b>Exports Hadoop metrics to Prometheus</b></li>
</ul>
<p>If “hadoop.prometheus.endpoint.enabled” is set to true, Prometheus-friendly formatted metrics can be obtained from ‘/prom’ endpoint of Hadoop daemons. The default value of the property is false.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16479">HADOOP-16479</a> | <i>Major</i> | <b>ABFS FileStatus.getModificationTime returns localized time instead of UTC</b></li>
</ul>
<p>private long parseLastModifiedTime(final String lastModifiedTime)</p>
<p>Timezone is part of lastModifiedTime String as it’s last field. But when parsed it ignores timezone field and assumes JVM timezone. Fix: Made timezone field considered in lastModifiedTime when the same is parsed</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16499">HADOOP-16499</a> | <i>Critical</i> | <b>S3A retry policy to be exponential</b></li>
</ul>
<p>The S3A filesystem now backs off exponentially on failures. if you have customized the fs.s3a.retry.limit and fs.s3a.retry.interval options you may wish to review these settings</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-13505">HDFS-13505</a> | <i>Major</i> | <b>Turn on HDFS ACLs by default.</b></li>
</ul>
<p>By default, dfs.namenode.acls.enabled is now set to true. If you haven’t set dfs.namenode.acls.enabled=false before and want to keep ACLs feature disabled, you must explicitly set it to false in hdfs-site.xml. i.e. <property><name>dfs.namenode.acls.enabled</name><value>false</value></property></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-2599">YARN-2599</a> | <i>Major</i> | <b>Standby RM should expose jmx endpoint</b></li>
</ul>
<p>YARN /jmx URL end points will be accessible per resource manager process. Hence there will not be any redirection to active resource manager while accessing /jmx endpoints.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-13101">HDFS-13101</a> | <i>Critical</i> | <b>Yet another fsimage corruption related to snapshot</b></li>
</ul>
<p>Fix a corner case in deleting HDFS snapshots. A regression was later found and fixed by HDFS-15012.</p><hr/>
<p>Increase default bandwidth limit for rebalancing per DataNode (dfs.datanode.balance.bandwidthPerSec) from 10MB/s to 100MB/s.</p>
<p>Increase default maximum threads of DataNode balancer (dfs.datanode.balance.max.concurrent.moves) from 50 to 100.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14617">HDFS-14617</a> | <i>Major</i> | <b>Improve fsimage load time by writing sub-sections to the fsimage index</b></li>
</ul>
<p>This change allows the inode and inode directory sections of the fsimage to be loaded in parallel. Tests on large images have shown this change to reduce the image load time to about 50% of the pre-change run time.</p>
<p>It works by writing sub-section entries to the image index, effectively splitting each image section into many sub-sections which can be processed in parallel. By default 12 sub-sections per image section are created when the image is saved, and 4 threads are used to load the image at startup.</p>
<p>This is disabled by default for any image with more than 1M inodes (dfs.image.parallel.inode.threshold) and can be enabled by setting dfs.image.parallel.load to true. When the feature is enabled, the next HDFS checkpoint will write the image sub-sections and subsequent namenode restarts can load the image in parallel.</p>
<p>A image with the parallel sections can be read even if the feature is disabled, but HDFS versions without this Jira cannot load an image with parallel sections. OIV can process a parallel enabled image without issues.</p>
<p>Key configuration parameters are:</p>
<p>dfs.image.parallel.load=false - enable or disable the feature</p>
<p>dfs.image.parallel.target.sections = 12 - The target number of subsections. Aim for 2 to 3 times the number of dfs.image.parallel.threads.</p>
<p>dfs.image.parallel.inode.threshold = 1000000 - Only save and load in parallel if the image has more than this number of inodes.</p>
<p>dfs.image.parallel.threads = 4 - The number of threads used to load the image. Testing has shown 4 to be optimal, but this may depends on the environment</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14396">HDFS-14396</a> | <i>Blocker</i> | <b>Failed to load image from FSImageFile when downgrade from 3.x to 2.x</b></li>
</ul>
<p>During a rolling upgrade from Hadoop 2.x to 3.x, NameNode cannot persist erasure coding information, and therefore a user cannot start using erasure coding feature until finalize is done.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16554">HADOOP-16554</a> | <i>Major</i> | <b>mvn javadoc:javadoc fails in hadoop-aws</b></li>
</ul>
<p>Fixed the link for javadoc of hadoop-aws CommitOperations</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16557">HADOOP-16557</a> | <i>Major</i> | <b>[pb-upgrade] Upgrade protobuf.version to 3.7.1</b></li>
</ul>
<p>Upgraded the protobuf to 3.7.1.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14845">HDFS-14845</a> | <i>Critical</i> | <b>Ignore AuthenticationFilterInitializer for HttpFSServerWebServer and honor hadoop.http.authentication configs</b></li>
</ul>
<p>httpfs.authentication.* configs become deprecated and hadoop.http.authentication.* configs are honored in HttpFS. If the both configs are set, httpfs.authentication.* configs are effective for compatibility.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-15616">HADOOP-15616</a> | <i>Major</i> | <b>Incorporate Tencent Cloud COS File System Implementation</b></li>
</ul>
<p>Tencent cloud is top 2 cloud vendors in China market and the object store COS is widely used among China’s cloud users. This task implements a COSN filesytem to support Tencent cloud COS natively in Hadoop. With simple configuration, Hadoop applications, like Spark and Hive, can read/write data from COS without any code change.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-13762">HDFS-13762</a> | <i>Major</i> | <b>Support non-volatile storage class memory(SCM) in HDFS cache directives</b></li>
</ul>
<p>Non-volatile storage class memory (SCM, also known as persistent memory) is supported in HDFS cache. To enable SCM cache, user just needs to configure SCM volume for property “dfs.datanode.cache.pmem.dirs” in hdfs-site.xml. And all HDFS cache directives keep unchanged. There are two implementations for HDFS SCM Cache, one is pure java code implementation and the other is native PMDK based implementation. The latter implementation can bring user better performance gain in cache write and cache read. If PMDK native libs could be loaded, it will use PMDK based implementation otherwise it will fallback to java code implementation. To enable PMDK based implementation, user should install PMDK library by referring to the official site <aclass="externalLink"href="http://pmem.io/">http://pmem.io/</a>. Then, build Hadoop with PMDK support by referring to “PMDK library build options” section in `BUILDING.txt` in the source code. If multiple SCM volumes are configured, a round-robin policy is used to select an available volume for caching a block. Consistent with DRAM cache, SCM cache also has no cache eviction mechanism. When DataNode receives a data read request from a client, if the corresponding block is cached into SCM, DataNode will instantiate an InputStream with the block location path on SCM (pure java implementation) or cache address on SCM (PMDK based implementation). Once the InputStream is created, DataNode will send the cached data to the client. Please refer “Centralized Cache Management” guide for more details.</p><hr/>
<ul>
<li>
<p><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14890">HDFS-14890</a> | <i>Blocker</i> | <b>Setting permissions on name directory fails on non posix compliant filesystems</b></p>
</li>
<li>
<p>Fixed namenode/journal startup on Windows.</p>
</li>
</ul><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16152">HADOOP-16152</a> | <i>Major</i> | <b>Upgrade Eclipse Jetty version to 9.4.x</b></li>
</ul>
<p>Upgraded Jetty to 9.4.20.v20190813. Downstream applications that still depend on Jetty 9.3.x may break.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-12943">HDFS-12943</a> | <i>Major</i> | <b>Consistent Reads from Standby Node</b></li>
</ul>
<p>Observer is a new type of a NameNode in addition to Active and Standby Nodes in HA settings. An Observer Node maintains a replica of the namespace same as a Standby Node. It additionally allows execution of clients read requests.</p>
<p>To ensure read-after-write consistency within a single client, a state ID is introduced in RPC headers. The Observer responds to the client request only after its own state has caught up with the client’s state ID, which it previously received from the Active NameNode.</p>
<p>Clients can explicitly invoke a new client protocol call msync(), which ensures that subsequent reads by this client from an Observer are consistent.</p>
<p>A new client-side ObserverReadProxyProvider is introduced to provide automatic switching between Active and Observer NameNodes for submitting respectively write and read requests.</p><hr/>
<p>When dead node blocks DFSInputStream, Deadnode detection can find it and share this information to other DFSInputStreams in the same DFSClient. Thus, these DFSInputStreams will not read from the dead node and be blocked by this dead node.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16771">HADOOP-16771</a> | <i>Major</i> | <b>Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0</b></li>
</ul>
<p>Updated checkstyle to 8.26 and updated maven-checkstyle-plugin to 3.1.0.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-10036">YARN-10036</a> | <i>Major</i> | <b>Install yarnpkg and upgrade nodejs in Dockerfile</b></li>
</ul>
<p>In the Dockerfile, nodejs is upgraded to 8.17.0 and yarn 1.12.1 is installed.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16621">HADOOP-16621</a> | <i>Critical</i> | <b>[pb-upgrade] Remove Protobuf classes from signatures of Public APIs</b></li>
</ul>
<p>Following APIs have been removed from Token.java to avoid protobuf classes in signature. 1. o.a.h.security.token.Token(TokenProto tokenPB) 2. o.a.h.security.token.Token.toTokenProto()</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16670">HADOOP-16670</a> | <i>Blocker</i> | <b>Stripping Submarine code from Hadoop codebase.</b></li>
</ul>
<p>The Submarine subproject is moved to its own Apache Top Level Project. Therefore, the Submarine code is removed from Hadoop 3.3.0 and forward. Please visit <aclass="externalLink"href="https://submarine.apache.org/">https://submarine.apache.org/</a> for more information.</p><hr/>
<p>Improved debugging Docker container on YARN</p>
</li>
<li>Improved security for running Docker containers</li>
<li>Improved cgroup management for docker container.</li>
</ul><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16732">HADOOP-16732</a> | <i>Major</i> | <b>S3Guard to support encrypted DynamoDB table</b></li>
</ul>
<p>Support server-side encrypted DynamoDB table for S3Guard. Users don’t need to do anything (provide any configuration or change application code) if they don’t want to enable server side encryption. Existing tables and the default configuration values will keep existing behavior, which is encrypted using Amazon owned customer master key (CMK).</p>
<p>To enable server side encryption, users can set “fs.s3a.s3guard.ddb.table.sse.enabled” as true. This uses Amazon managed CMK “alias/aws/dynamodb”. When it’s enabled, a user can also specify her own custom KMS CMK with config “fs.s3a.s3guard.ddb.table.sse.cmk”.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16596">HADOOP-16596</a> | <i>Major</i> | <b>[pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency</b></li>
</ul>
<p>All protobuf classes will be used from hadooop-shaded-protobuf_3_7 artifact with package prefix as ‘org.apache.hadoop.thirdparty.protobuf’ instead of ‘com.google.protobuf’</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16823">HADOOP-16823</a> | <i>Minor</i> | <b>Large DeleteObject requests are their own Thundering Herd</b></li>
</ul>
<p>The page size for bulk delete operations has been reduced from 1000 to 250 to reduce the likelihood of overloading an S3 partition, especially because the retry policy on throttling is simply to try again.</p>
<p>The page size can be set in “fs.s3a.bulk.delete.page.size”</p>
<p>There is also an option to control whether or not the AWS client retries requests, or whether it is handled exclusively in the S3A code. This option “fs.s3a.experimental.aws.s3.throttling” is true by default. If set to false: everything is handled in the S3A client. While this means that metrics may be more accurate, it may mean that throttling failures in helper threads of the AWS SDK (especially those used in copy/rename) may not be handled properly. This is experimental, and should be left at “true” except when seeking more detail about throttling rates.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16850">HADOOP-16850</a> | <i>Major</i> | <b>Support getting thread info from thread group for JvmMetrics to improve the performance</b></li>
</ul>
<p>Add a new configuration “hadoop.metrics.jvm.use-thread-mxbean”. If set to true, ThreadMXBean is used for getting thread info in JvmMetrics, otherwise, ThreadGroup is used. The default value is false (use ThreadGroup for better performance).</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16711">HADOOP-16711</a> | <i>Minor</i> | <b>S3A bucket existence checks to support v2 API and “no checks at all”</b></li>
</ul>
<p>The probe for an S3 bucket existing now uses the V2 API, which fails fast if the caller lacks access to the bucket. The property fs.s3a.bucket.probe can be used to change this. If using a third party library which doesn’t support this API call, change it to “1”.</p>
<p>to skip the probe entirely, use “0”. This will make filesystem instantiation slightly faster, at a cost of postponing all issues related to bucket existence and client authentication until the filesystem API calls are first used.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-15186">HDFS-15186</a> | <i>Critical</i> | <b>Erasure Coding: Decommission may generate the parity block’s content with all 0 in some case</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14743">HDFS-14743</a> | <i>Critical</i> | <b>Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc…</b></li>
</ul>
<p>A new INodeAttributeProvider API checkPermissionWithContext(AuthorizationContext) is added. Authorization provider implementations may implement this API to get additional context (operation name and caller context) of an authorization request.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-14820">HDFS-14820</a> | <i>Major</i> | ** The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big**</li>
</ul>
<p>Reduce the output stream buffer size of a DFSClient remote read from 8KB to 512 bytes.</p><hr/>
<p>Starting with Hadoop 3.3.0, TLS 1.3 is supported on Java 11 Runtime (11.0.3 and above). This support does not include cloud connectors.</p>
<p>To enable TLSv1.3, add to the core-site.xml: <property><name>hadoop.ssl.enabled.protocols</name><value>TLSv1.3</value></property></p><hr/>
<p>Lots of enhancements to the S3A code, including * Delegation Token support * better handling of 404 caching * S3guard performance, resilience improvements</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-16986">HADOOP-16986</a> | <i>Major</i> | <b>s3a to not need wildfly on the classpath</b></li>
</ul>
<p>hadoop-aws can use native openssl libraries for better HTTPS performance -consult the S3A performance document for details.</p>
<p>To enable this, wildfly.jar is declared as a compile-time dependency of the hadoop-aws module, so ensuring it ends up on the classpath of the hadoop command line, distribution packages and downstream modules.</p>
<p>It is however, still optional, unless fs.s3a.ssl.channel.mode is set to openssl</p>
