10973 lines
385 KiB
Plaintext
10973 lines
385 KiB
Plaintext
Hadoop HDFS Change Log
|
||
|
||
Release 2.7.0 - UNRELEASED
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-7278. Add a command that allows sysadmins to manually trigger full
|
||
block reports from a DN (cmccabe)
|
||
|
||
HDFS-6663. Admin command to track file and locations from block id.
|
||
(Chen He via kihwal)
|
||
|
||
HDFS-6982. nntop: top-like tool for name node users.
|
||
(Maysam Yabandeh via wang)
|
||
|
||
HDFS-7424. Add web UI for NFS gateway (brandonli)
|
||
|
||
HDFS-7449. Add metrics to NFS gateway (brandonli)
|
||
|
||
HDFS-3107. Introduce truncate. (Plamen Jeliazkov via shv)
|
||
|
||
HDFS-7056. Snapshot support for truncate. (Plamen Jeliazkov and shv)
|
||
|
||
HDFS-6673. Add delimited format support to PB OIV tool. (Eddy Xu via wang)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-7055. Add tracing to DFSInputStream (cmccabe)
|
||
|
||
HDFS-7186. Document the "hadoop trace" command. (Masatake Iwasaki via Colin
|
||
P. McCabe)
|
||
|
||
HDFS-7202. Should be able to omit package name of SpanReceiver on "hadoop
|
||
trace -add" (iwasakims via cmccabe)
|
||
|
||
HDFS-7026. Introduce a string constant for "Failed to obtain user group
|
||
info...". (Yongjun Zhang via atm)
|
||
|
||
HDFS-7209. Populate EDEK cache when creating encryption zone. (Yi Liu via wang)
|
||
|
||
HDFS-6252. Phase out the old web UI in HDFS. (wheat9)
|
||
|
||
HDFS-7266. HDFS Peercache enabled check should not lock on object (awang
|
||
via cmccabe)
|
||
|
||
HDFS-7254. Add documentation for hot swaping DataNode drives (Lei Xu via
|
||
Colin P. McCabe)
|
||
|
||
HDFS-6877. Avoid calling checkDisk when an HDFS volume is removed during a
|
||
write. (Lei Xu via Colin P. McCabe)
|
||
|
||
HDFS-2486. Remove unnecessary priority level checks in
|
||
UnderReplicatedBlocks. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-6824. Additional user documentation for HDFS encryption. (wang)
|
||
|
||
HDFS-7165. Separate block metrics for files with replication count 1.
|
||
(Zhe Zhang via wang)
|
||
|
||
HDFS-7222. Expose DataNode network errors as a metric. (Charles Lamb via wang)
|
||
|
||
HDFS-7257. Add the time of last HA state transition to NN's /jmx page.
|
||
(Charles Lamb via wheat9)
|
||
|
||
HDFS-7223. Tracing span description of IPC client is too long (iwasakims
|
||
via cmccabe)
|
||
|
||
HDFS-7283. Bump DataNode OOM log from WARN to ERROR.
|
||
(Stephen Chu via wheat9)
|
||
|
||
HDFS-5928. Show namespace and namenode ID on NN dfshealth page.
|
||
(Siqi Li via wheat9)
|
||
|
||
HDFS-7213. processIncrementalBlockReport performance degradation.
|
||
(Eric Payne via kihwal)
|
||
|
||
HDFS-7280. Use netty 4 in WebImageViewer. (wheat9)
|
||
|
||
HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could
|
||
have a better error message. (Yongjun Zhang via wang)
|
||
|
||
HDFS-7035. Make adding a new data directory to the DataNode an atomic
|
||
operation and improve error handling (Lei Xu via Colin P. McCabe)
|
||
|
||
HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease,
|
||
etc. (cmccabe)
|
||
|
||
HDFS-7356. Use DirectoryListing.hasMore() directly in nfs. (Li Lu via jing9)
|
||
|
||
HDFS-7357. FSNamesystem.checkFileProgress should log file path.
|
||
(Tsz Wo Nicholas Sze via wheat9)
|
||
|
||
HDFS-7335. Redundant checkOperation() in FSN.analyzeFileState().
|
||
(Milan Desai via shv)
|
||
|
||
HDFS-7333. Improve logging in Storage.tryLock(). (shv)
|
||
|
||
HDFS-7361. TestCheckpoint fails after change of log message related to
|
||
locking violation. (shv)
|
||
|
||
HDFS-7329. Improve logging when MiniDFSCluster fails to start.
|
||
(Byron Wong via shv)
|
||
|
||
HDFS-7336. Unused member DFSInputStream.buffersize. (Milan Desai via shv)
|
||
|
||
HDFS-7365. Remove hdfs.server.blockmanagement.MutableBlockCollection.
|
||
(Li Lu via wheat9)
|
||
|
||
HDFS-7381. Decouple the management of block id and gen stamps from
|
||
FSNamesystem. (wheat9)
|
||
|
||
HDFS-7375. Move FSClusterStats to o.a.h.h.hdfs.server.blockmanagement.
|
||
(wheat9)
|
||
|
||
HDFS-7386. Replace check "port number < 1024" with shared isPrivilegedPort
|
||
method. (Yongjun Zhang via cnauroth)
|
||
|
||
HDFS-7394. Log at INFO level, not WARN level, when InvalidToken is seen in
|
||
ShortCircuitCache (Keith Pak via Colin P. McCabe)
|
||
|
||
HDFS-7279. Use netty to implement DatanodeWebHdfsMethods. (wheat9)
|
||
|
||
HDFS-7404. Remove o.a.h.hdfs.server.datanode.web.resources.
|
||
(Li Lu via wheat9)
|
||
|
||
HDFS-7398. Reset cached thread-local FSEditLogOp's on every
|
||
FSEditLog#logEdit. (Gera Shegalov via cnauroth)
|
||
|
||
HDFS-7409. Allow dead nodes to finish decommissioning if all files are
|
||
fully replicated. (wang)
|
||
|
||
HDFS-7413. Some unit tests should use NameNodeProtocols instead of
|
||
FSNameSystem. (wheat9)
|
||
|
||
HDFS-7415. Move FSNameSystem.resolvePath() to FSDirectory. (wheat9)
|
||
|
||
HDFS-7420. Delegate permission checks to FSDirectory. (wheat9)
|
||
|
||
HDFS-7331. Add Datanode network counts to datanode jmx page. (Charles Lamb
|
||
via atm)
|
||
|
||
HDFS-7412. Move RetryCache to NameNodeRpcServer. (wheat9)
|
||
|
||
HDFS-7419. Improve error messages for DataNode hot swap drive feature (Lei
|
||
Xu via Colin P. Mccabe)
|
||
|
||
HDFS-7436. Consolidate implementation of concat(). (wheat9)
|
||
|
||
HDFS-7440. Consolidate snapshot related operations in a single class.
|
||
(wheat9)
|
||
|
||
HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading
|
||
in concurrent context. (stack via stevel)
|
||
|
||
HDFS-7310. Mover can give first priority to local DN if it has target storage type
|
||
available in local DN. (Vinayakumar B via umamahesh)
|
||
|
||
HDFS-7210. Avoid two separate RPC's namenode.append() and namenode.getFileInfo()
|
||
for an append call from DFSClient. (Vinayakumar B via umamahesh)
|
||
|
||
HDFS-7450. Consolidate the implementation of GetFileInfo(), GetListings() and
|
||
GetContentSummary() into a single class. (wheat9)
|
||
|
||
HDFS-7438. Consolidate the implementation of rename() into a single class.
|
||
(wheat9)
|
||
|
||
HDFS-7462. Consolidate implementation of mkdirs() into a single class.
|
||
(wheat9)
|
||
|
||
HDFS-7446. HDFS inotify should have the ability to determine what txid it
|
||
has read up to (cmccabe)
|
||
|
||
HDFS-6735. A minor optimization to avoid pread() be blocked by read()
|
||
inside the same DFSInputStream (Lars Hofhansl via stack)
|
||
|
||
HDFS-7458. Add description to the nfs ports in core-site.xml used by nfs
|
||
test to avoid confusion (Yongjun Zhang via brandonli)
|
||
|
||
HDFS-7468. Moving verify* functions to corresponding classes.
|
||
(Li Lu via wheat9)
|
||
|
||
HDFS-7478. Move org.apache.hadoop.hdfs.server.namenode.NNConf to
|
||
FSNamesystem. (Li Lu via wheat9)
|
||
|
||
HDFS-7474. Avoid resolving path in FSPermissionChecker. (jing9)
|
||
|
||
HDFS-7459. Consolidate cache-related implementation in FSNamesystem into
|
||
a single class. (wheat9)
|
||
|
||
HDFS-7476. Consolidate ACL-related operations to a single class.
|
||
(wheat9 via cnauroth)
|
||
|
||
HDFS-7384. 'getfacl' command and 'getAclStatus' output should be in sync.
|
||
(Vinayakumar B via cnauroth)
|
||
|
||
HDFS-7486. Consolidate XAttr-related implementation into a single class.
|
||
(wheat9)
|
||
|
||
HDFS-7498. Simplify the logic in INodesInPath. (jing9)
|
||
|
||
HDFS-7463. Simplify FSNamesystem#getBlockLocationsUpdateTimes. (wheat9)
|
||
|
||
HDFS-7509. Avoid resolving path multiple times. (jing9)
|
||
|
||
HDFS-7426. Change nntop JMX format to be a JSON blob. (wang)
|
||
|
||
HDFS-7513. HDFS inotify: add defaultBlockSize to CreateEvent (cmccabe)
|
||
|
||
HDFS-7536. Remove unused CryptoCodec in org.apache.hadoop.fs.Hdfs.
|
||
(Yi Liu via wheat9)
|
||
|
||
HDFS-7528. Consolidate symlink-related implementation into a single class.
|
||
(wheat9)
|
||
|
||
HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin
|
||
P. McCabe)
|
||
|
||
HDFS-7373. Clean up temporary files after fsimage transfer failures.
|
||
(kihwal)
|
||
|
||
HDFS-7543. Avoid path resolution when getting FileStatus for audit logs.
|
||
(wheat9)
|
||
|
||
HDFS-7530. Allow renaming of encryption zone roots. (Charles Lamb via wang)
|
||
|
||
HDFS-7484. Make FSDirectory#addINode take existing INodes as its parameter.
|
||
(jing9)
|
||
|
||
HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch
|
||
(ozawa)
|
||
|
||
HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop
|
||
codebase. (Sangjin Lee via Colin P. McCabe)
|
||
|
||
HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9)
|
||
|
||
HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin.
|
||
(jing9 via yliu)
|
||
|
||
HDFS-7326: Add documentation for hdfs debug commands (Vijay Bhat via Colin
|
||
P. McCabe)
|
||
|
||
HDFS-7598. Remove dependency on old version of guava in
|
||
TestDFSClientCache#testEviction. (Sangjin Lee via Colin P. McCabe)
|
||
|
||
HDFS-7600. Refine hdfs admin classes to reuse common code. (jing9)
|
||
|
||
HDFS-2219. Change fsck to support fully qualified paths so that a
|
||
particular namenode in a federated cluster with multiple namenodes
|
||
can be specified in the path parameter. (szetszwo)
|
||
|
||
HDFS-7457. DatanodeID generates excessive garbage. (daryn via kihwal)
|
||
|
||
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P.
|
||
McCabe via yliu)
|
||
|
||
HDFS-7591. hdfs classpath command should support same options as hadoop
|
||
classpath. (Varun Saxena via Arpit Agarwal)
|
||
|
||
HDFS-7573. Consolidate the implementation of delete() into a single class.
|
||
(wheat9)
|
||
|
||
HDFS-7640. Print NFS Client in the NFS log. (Brandon Li via wheat9)
|
||
|
||
HDFS-7623. Add htrace configuration properties to core-default.xml and
|
||
update user doc about how to enable htrace. (yliu)
|
||
|
||
HDFS-7224. Allow reuse of NN connections via webhdfs (Eric Payne via
|
||
kihwal)
|
||
|
||
HDFS-7683. Combine usages and percent stats in NameNode UI.
|
||
(Vinayakumar B via wheat9)
|
||
|
||
HDFS-7675. Remove unused member DFSClient#spanReceiverHost (cmccabe)
|
||
|
||
HDFS-7603. The background replication queue initialization may not let
|
||
others run (kihwal)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-7454. Reduce memory footprint for AclEntries in NameNode.
|
||
(Vinayakumar B via wheat9)
|
||
|
||
HDFS-7615. Remove longReadLock (kihwal)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-6741. Improve permission denied message when
|
||
FSPermissionChecker#checkOwner fails (Stephen Chu and harsh).
|
||
|
||
HDFS-6538. Comment format error in ShortCircuitRegistry javadoc.
|
||
(David Luo via harsh).
|
||
|
||
HDFS-7194. Fix findbugs "inefficient new String constructor" warning in
|
||
DFSClient#PATH (yzhang via cmccabe)
|
||
|
||
HDFS-7198. Fix or suppress findbugs "unchecked conversion" warning in
|
||
DFSClient#getPathTraceScope (cmccabe)
|
||
|
||
HDFS-6657. Remove link to 'Legacy UI' in the Namenode UI.
|
||
(Vinayakumar B via wheat9)
|
||
|
||
HDFS-7201. Fix typos in hdfs-default.xml. (Dawson Choong via wheat9)
|
||
|
||
HDFS-7190. Bad use of Preconditions in startFileInternal().
|
||
(Dawson Choong via wheat9)
|
||
|
||
HDFS-7242. Code improvement for FSN#checkUnreadableBySuperuser.
|
||
(Yi Liu via vinayakumarb)
|
||
|
||
HDFS-7252. small refinement to the use of isInAnEZ in FSNamesystem.
|
||
(Yi Liu via vinayakumarb)
|
||
|
||
HDFS-7277. Remove explicit dependency on netty 3.2 in BKJournal. (wheat9)
|
||
|
||
HDFS-7232. Populate hostname in httpfs audit log (Zoran Dimitrijevic
|
||
via aw)
|
||
|
||
HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG
|
||
level instead of INFO level. (Xiaoyu Yao via wheat9)
|
||
|
||
HDFS-7282. Fix intermittent TestShortCircuitCache and
|
||
TestBlockReaderFactory failures resulting from TemporarySocketDirectory GC.
|
||
(Jinghui Wang via Colin Patrick McCabe)
|
||
|
||
HDFS-7301. TestMissingBlocksAlert should use MXBeans instead of old web UI.
|
||
(Zhe Zhang via wheat9)
|
||
|
||
HDFS-7235. DataNode#transferBlock should report blocks that don't exist
|
||
using reportBadBlock (yzhang via cmccabe)
|
||
|
||
HDFS-7263. Snapshot read can reveal future bytes for appended files.
|
||
(Tao Luo via shv)
|
||
|
||
HDFS-7315. DFSTestUtil.readFileBuffer opens extra FSDataInputStream.
|
||
(Plamen Jeliazkov via wheat9)
|
||
|
||
HDFS-7324. haadmin command usage prints incorrect command name.
|
||
(Brahma Reddy Battula via suresh)
|
||
|
||
HDFS-7366. BlockInfo should take replication as an short in the constructor.
|
||
(Li Lu via wheat9)
|
||
|
||
HDFS-7389. Named user ACL cannot stop the user from accessing the FS entity.
|
||
(Vinayakumar B via cnauroth)
|
||
|
||
HDFS-6938. Cleanup javac warnings in FSNamesystem (Charles Lamb via wheat9)
|
||
|
||
HDFS-7358. Clients may get stuck waiting when using ByteArrayManager.
|
||
(szetszwo)
|
||
|
||
HDFS-7395. BlockIdManager#clear() bails out when resetting the
|
||
GenerationStampV1Limit. (wheat9)
|
||
|
||
HDFS-7399. Lack of synchronization in
|
||
DFSOutputStream#Packet#getLastByteOffsetBlock() (vinayakumarb)
|
||
|
||
HDFS-7146. NFS ID/Group lookup requires SSSD enumeration on the server
|
||
(Yongjun Zhang via brandonli)
|
||
|
||
HDFS-7406. SimpleHttpProxyHandler puts incorrect "Connection: Close"
|
||
header. (wheat9)
|
||
|
||
HDFS-7225. Remove stale block invalidation work when DN re-registers with
|
||
different UUID. (Zhe Zhang and Andrew Wang)
|
||
|
||
HDFS-7374. Allow decommissioning of dead DataNodes. (Zhe Zhang)
|
||
|
||
HDFS-7403. Inaccurate javadoc of BlockUCState#COMPLETE state. (
|
||
Yongjun Zhang via yliu)
|
||
|
||
HDFS-7303. NN UI fails to distinguish datanodes on the same host.
|
||
(Benoy Antony via wheat9)
|
||
|
||
HDFS-7097. Allow block reports to be processed during checkpointing on
|
||
standby name node. (kihwal via wang)
|
||
|
||
HDFS-7444. convertToBlockUnderConstruction should preserve BlockCollection.
|
||
(wheat9)
|
||
|
||
HDFS-7448 TestBookKeeperHACheckpoints fails in trunk build
|
||
(Akira Ajisaka via stevel)
|
||
|
||
HDFS-7472. Fix typo in message of ReplicaNotFoundException.
|
||
(Masatake Iwasaki via wheat9)
|
||
|
||
HDFS-7473. Document setting dfs.namenode.fs-limits.max-directory-items to 0
|
||
is invalid. (Akira AJISAKA via cnauroth)
|
||
|
||
HDFS-7481. Add ACL indicator to the "Permission Denied" exception.
|
||
(vinayakumarb)
|
||
|
||
HDFS-7502. Fix findbugs warning in hdfs-nfs project.
|
||
(Brandon Li via wheat9)
|
||
|
||
HDFS-5578. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags
|
||
in doc comments. (Andrew Purtell via wheat9)
|
||
|
||
HDFS-7475. Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved
|
||
deterministic. (Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-7515. Fix new findbugs warnings in hadoop-hdfs. (wheat9)
|
||
|
||
HDFS-7497. Inconsistent report of decommissioning DataNodes between
|
||
dfsadmin and NameNode webui. (Yongjun Zhang via wang)
|
||
|
||
HDFS-7517. Remove redundant non-null checks in FSNamesystem#
|
||
getBlockLocations. (wheat9)
|
||
|
||
HDFS-7514. TestTextCommand fails on Windows. (Arpit Agarwal)
|
||
|
||
HDFS-7506. Consolidate implementation of setting inode attributes into a
|
||
single class. (wheat9)
|
||
|
||
HDFS-7516. Fix findbugs warnings in hdfs-nfs project. (brandonli)
|
||
|
||
HDFS-6425. Large postponedMisreplicatedBlocks has impact on blockReport
|
||
latency. (Ming Ma via kihwal)
|
||
|
||
HDFS-7494. Checking of closed in DFSInputStream#pread() should be protected
|
||
by synchronization (Ted Yu via Colin P. McCabe)
|
||
|
||
HDFS-7431. log message for InvalidMagicNumberException may be incorrect.
|
||
(Yi Liu via cnauroth)
|
||
|
||
HDFS-7552. Change FsVolumeList toString() to fix
|
||
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe)
|
||
|
||
HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java
|
||
(Colin P.McCabe)
|
||
|
||
HDFS-7560. ACLs removed by removeDefaultAcl() will be back after NameNode
|
||
restart/failover. (Vinayakumar B via cnauroth)
|
||
|
||
HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce
|
||
memory footprint of NameNode (vinayakumarb)
|
||
|
||
HDFS-7563. NFS gateway parseStaticMap NumberFormatException
|
||
(Yongjun Zhang via brandonli)
|
||
|
||
HDFS-7572. TestLazyPersistFiles#testDnRestartWithSavedReplicas is flaky on
|
||
Windows. (Arpit Agarwal via cnauroth)
|
||
|
||
HDFS-7583. Fix findbug in TransferFsImage.java (vinayakumarb)
|
||
|
||
HDFS-7564. NFS gateway dynamically reload UID/GID mapping file /etc/nfs.map
|
||
(Yongjun Zhang via brandonli)
|
||
|
||
HDFS-7561. TestFetchImage should write fetched-image-dir under target.
|
||
(Liang Xie via shv)
|
||
|
||
HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs.
|
||
(Zhanwei Wang via cnauroth)
|
||
|
||
HDFS-7579. Improve log reporting during block report rpc failure.
|
||
(Charles Lamb via cnauroth)
|
||
|
||
HDFS-7596. NameNode should prune dead storages from storageMap.
|
||
(Arpit Agarwal via cnauroth)
|
||
|
||
HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade
|
||
shutdown command (Eric Payne via kihwal)
|
||
|
||
HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader
|
||
incorrectly (Jonathan Mace via Colin P. McCabe)
|
||
|
||
HDFS-7470. SecondaryNameNode need twice memory when calling
|
||
reloadFromImageFile. (zhaoyunjiong via cnauroth)
|
||
|
||
HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures
|
||
with page sizes other than 4096 (Sam Liu via Colin P. McCabe)
|
||
|
||
HDFS-7635. Remove TestCorruptFilesJsp from branch-2. (cnauroth)
|
||
|
||
HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if
|
||
using more than 1 DataNode and more than 2 storage locations per DataNode.
|
||
(cnauroth)
|
||
|
||
HDFS-7637. Fix the check condition for reserved path. (Yi Liu via jing9)
|
||
|
||
HDFS-7641. Update archival storage user doc for list/set/get block storage
|
||
policies. (yliu)
|
||
|
||
HDFS-7496. Fix FsVolume removal race conditions on the DataNode by
|
||
reference-counting the volume instances (lei via cmccabe)
|
||
|
||
HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via
|
||
Colin P. McCabe)
|
||
|
||
HDFS-7548. Corrupt block reporting delayed until datablock scanner thread
|
||
detects it (Rushabh Shah via kihwal)
|
||
|
||
HDFS-7575. Upgrade should generate a unique storage ID for each
|
||
volume. (Arpit Agarwal)
|
||
|
||
HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace.
|
||
(Ming Ma via cnauroth)
|
||
|
||
HDFS-7644. minor typo in HttpFS doc (Charles Lamb via aw)
|
||
|
||
HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). (Byron Wong via shv)
|
||
|
||
HDFS-7638: Small fix and few refinements for FSN#truncate. (yliu)
|
||
|
||
HDFS-7634. Disallow truncation of Lazy persist files. (Yi Liu via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-7643. Test case to ensure lazy persist files cannot be truncated.
|
||
(Yi Liu via Arpit Agarwal)
|
||
|
||
HDFS-7659. truncate should check negative value of the new length.
|
||
(Yi Liu via shv)
|
||
|
||
HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611. (shv)
|
||
|
||
HDFS-49. MiniDFSCluster.stopDataNode will always shut down a node in
|
||
the cluster if a matching name is not found. (stevel)
|
||
|
||
HDFS-7566. Remove obsolete entries from hdfs-default.xml (Ray Chiang
|
||
via aw)
|
||
|
||
HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu)
|
||
|
||
HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks
|
||
in the blocksMap on NameNode restart. (jing9 and Byron Wong)
|
||
|
||
HDFS-7423. various typos and message formatting fixes in nfs daemon and
|
||
doc. (Charles Lamb via yliu)
|
||
|
||
Release 2.6.1 - UNRELEASED
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-7425. NameNode block deletion logging uses incorrect appender.
|
||
(cnauroth)
|
||
|
||
HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in
|
||
checkLeases (Ravi Prakash via Colin P. McCabe)
|
||
|
||
HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes
|
||
(Noah Lorang via Colin P. McCabe)
|
||
|
||
HDFS-7503. Namenode restart after large deletions can cause slow
|
||
processReport. (Arpit Agarwal)
|
||
|
||
HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate
|
||
block files are present in the same volume (cmccabe)
|
||
|
||
HDFS-3443. Fix NPE when namenode transition to active during startup by
|
||
adding checkNNStartup() in NameNodeRpcServer. (Vinayakumar B via szetszwo)
|
||
|
||
Release 2.6.0 - 2014-11-18
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-6584. Support archival storage. (See breakdown of tasks below for
|
||
features and contributors)
|
||
|
||
HDFS-6581. Support writing to a single replica in RAM. (See breakdown of
|
||
tasks below for features and contributors).
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-6613. Improve logging in caching classes. (wang)
|
||
|
||
HDFS-6511. BlockManager#computeInvalidateWork() could do nothing. (Juan Yu via wang)
|
||
|
||
HDFS-6638. Shorten test run time with a smaller retry timeout setting.
|
||
(Liang Xie via cnauroth)
|
||
|
||
HDFS-6627. Rename DataNode#checkWriteAccess to checkReadAccess.
|
||
(Liang Xie via cnauroth)
|
||
|
||
HDFS-6645. Add test for successive Snapshots between XAttr modifications.
|
||
(Stephen Chu via jing9)
|
||
|
||
HDFS-6643. Refactor INodeWithAdditionalFields.PermissionStatusFormat and
|
||
INodeFile.HeaderFormat. (szetszwo)
|
||
|
||
HDFS-6640. Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly
|
||
in WebHdfs document (missed webhdfs/v1). (Stephen Chu via jing9)
|
||
|
||
HDFS-5202. Support Centralized Cache Management on Windows. (cnauroth)
|
||
|
||
HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient.
|
||
(Uma Maheswara Rao G)
|
||
|
||
HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh)
|
||
|
||
HDFS-2856. Fix block protocol so that Datanodes don't require root or jsvc.
|
||
(cnauroth)
|
||
|
||
HDFS-5624. Add HDFS tests for ACLs in combination with viewfs.
|
||
(Stephen Chu via cnauroth)
|
||
|
||
HDFS-6655. Add 'header banner' to 'explorer.html' also in Namenode UI
|
||
(vinayakumarb)
|
||
|
||
HDFS-4120. Add a new "-skipSharedEditsCheck" option for BootstrapStandby
|
||
(Liang Xie and Rakesh R via vinayakumarb)
|
||
|
||
HDFS-6597. Add a new option to NN upgrade to terminate the process after
|
||
upgrade on NN is completed. (Danilo Vunjak via cnauroth)
|
||
|
||
HDFS-6700. BlockPlacementPolicy shoud choose storage but not datanode for
|
||
deletion. (szetszwo)
|
||
|
||
HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it
|
||
will not redirect retries to the same datanode. (zhaoyunjiong via szetszwo)
|
||
|
||
HDFS-6702. Change DFSClient to pass the StorageType from the namenode to
|
||
datanodes and change datanode to write block replicas using the specified
|
||
storage type. (szetszwo)
|
||
|
||
HDFS-6701. Make seed optional in NetworkTopology#sortByDistance.
|
||
(Ashwin Shankar via wang)
|
||
|
||
HDFS-6755. There is an unnecessary sleep in the code path where
|
||
DFSOutputStream#close gives up its attempt to contact the namenode
|
||
(mitdesai21 via cmccabe)
|
||
|
||
HDFS-6750. The DataNode should use its shared memory segment to mark
|
||
short-circuit replicas that have been unlinked as stale (cmccabe)
|
||
|
||
HDFS-6739. Add getDatanodeStorageReport to ClientProtocol. (szetszwo)
|
||
|
||
HDFS-6665. Add tests for XAttrs in combination with viewfs.
|
||
(Stephen Chu via wang)
|
||
|
||
HDFS-6778. The extended attributes javadoc should simply refer to the
|
||
user docs. (clamb via wang)
|
||
|
||
HDFS-6570. add api that enables checking if a user has certain permissions on
|
||
a file. (Jitendra Pandey via cnauroth)
|
||
|
||
HDFS-6441. Add ability to exclude/include specific datanodes while
|
||
balancing. (Benoy Antony and Yu Li via Arpit Agarwal)
|
||
|
||
HDFS-6685. Balancer should preserve storage type of replicas. (szetszwo)
|
||
|
||
HDFS-6798. Add test case for incorrect data node condition during
|
||
balancing. (Benoy Antony via Arpit Agarwal)
|
||
|
||
HDFS-6796. Improve the argument check during balancer command line parsing.
|
||
(Benoy Antony via szetszwo)
|
||
|
||
HDFS-6794. Update BlockManager methods to use DatanodeStorageInfo
|
||
where possible (Arpit Agarwal)
|
||
|
||
HDFS-6802. Some tests in TestDFSClientFailover are missing @Test
|
||
annotation. (Akira Ajisaka via wang)
|
||
|
||
HDFS-6788. Improve synchronization in BPOfferService with read write lock.
|
||
(Yongjun Zhang via wang)
|
||
|
||
HDFS-6787. Remove duplicate code in FSDirectory#unprotectedConcat. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6809. Move Balancer's inner classes MovedBlocks and Matcher as to
|
||
standalone classes and separates KeyManager from NameNodeConnector.
|
||
(szetszwo)
|
||
|
||
HDFS-6812. Remove addBlock and replaceBlock from DatanodeDescriptor.
|
||
(szetszwo)
|
||
|
||
HDFS-6781. Separate HDFS commands from CommandsManual.apt.vm. (Akira
|
||
Ajisaka via Arpit Agarwal)
|
||
|
||
HDFS-6728. Dynamically add new volumes to DataStorage, formatted if
|
||
necessary. (Lei Xu vi atm)
|
||
|
||
HDFS-6740. Make FSDataset support adding data volumes dynamically. (Lei
|
||
Xu via atm)
|
||
|
||
HDFS-6722. Display readable last contact time for dead nodes on NN webUI.
|
||
(Ming Ma via wheat9)
|
||
|
||
HDFS-6772. Get DN storages out of blockContentsStale state faster after
|
||
NN restarts. (Ming Ma via Arpit Agarwal)
|
||
|
||
HDFS-573. Porting libhdfs to Windows. (cnauroth)
|
||
|
||
HDFS-6828. Separate block replica dispatching from Balancer. (szetszwo via
|
||
jing9)
|
||
|
||
HDFS-6837. Code cleanup for Balancer and Dispatcher. (szetszwo via
|
||
jing9)
|
||
|
||
HDFS-6838. Code cleanup for unnecessary INode replacement.
|
||
(Jing Zhao via wheat9)
|
||
|
||
HDFS-6836. HDFS INFO logging is verbose & uses file appenders. (Xiaoyu
|
||
Yao via Arpit Agarwal)
|
||
|
||
HDFS-6567. Normalize the order of public final in HdfsFileStatus.
|
||
(Tassapol Athiapinya via wheat9)
|
||
|
||
HDFS-6849. Replace HttpFS custom proxyuser handling with common
|
||
implementation. (tucu)
|
||
|
||
HDFS-6850. Move NFS out of order write unit tests into TestWrites class.
|
||
(Zhe Zhang via atm)
|
||
|
||
HDFS-6188. An ip whitelist based implementation of TrustedChannelResolver.
|
||
(Benoy Antony via Arpit Agarwal)
|
||
|
||
HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to
|
||
hadoop.security.saslproperties.resolver.class. (Benoy Antony via cnauroth)
|
||
|
||
HDFS-6878. Change MiniDFSCluster to support StorageType configuration
|
||
for individual directories. (Arpit Agarwal)
|
||
|
||
HDFS-6758. block writer should pass the expected block size to
|
||
DataXceiverServer. (Arpit Agarwal)
|
||
|
||
HDFS-6899. Allow changing MiniDFSCluster volumes per DN and capacity
|
||
per volume. (Arpit Agarwal)
|
||
|
||
HDFS-4486. Add log category for long-running DFSClient notices (Zhe Zhang
|
||
via Colin Patrick McCabe)
|
||
|
||
HDFS-6879. Adding tracing to Hadoop RPC (Masatake Iwasaki via Colin Patrick
|
||
McCabe)
|
||
|
||
HDFS-6774. Make FsDataset and DataStore support removing volumes. (Lei Xu
|
||
via atm)
|
||
|
||
HDFS-6634. inotify in HDFS. (James Thomas via wang)
|
||
|
||
HDFS-4257. The ReplaceDatanodeOnFailure policies could have a forgiving
|
||
option (szetszwo via cmccabe)
|
||
|
||
HDFS-6959. Make the HDFS home directory location customizable. (yzhang via
|
||
cmccabe)
|
||
|
||
HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable
|
||
directory. (Jing Zhao via wheat9)
|
||
|
||
HDFS-6886. Use single editlog record for creating file + overwrite. (Yi Liu
|
||
via jing9)
|
||
|
||
HDFS-6376. Distcp data between two HA clusters requires another configuration.
|
||
(Dave Marion and Haohui Mai via jing9)
|
||
|
||
HDFS-6943. Improve NN allocateBlock log to include replicas' datanode IPs.
|
||
(Ming Ma via wheat9)
|
||
|
||
HDFS-6036. Forcibly timeout misbehaving DFSClients that try to do
|
||
no-checksum reads that extend too long (cmccabe)
|
||
|
||
HDFS-6482. Use block ID-based block layout on datanodes (James Thomas via
|
||
Colin Patrick McCabe)
|
||
|
||
HDFS-7061. Add test to verify encryption zone creation after NameNode
|
||
restart without saving namespace. (Stephen Chu via wang)
|
||
|
||
HDFS-7059. HAadmin transtionToActive with forceActive option can show
|
||
confusing message.
|
||
|
||
HDFS-6880. Adding tracing to DataNode data transfer protocol. (iwasakims
|
||
via cmccabe)
|
||
|
||
HDFS-7006. Test encryption zones with KMS. (Anthony Young-Garner and tucu)
|
||
|
||
HDFS-6851. Refactor EncryptionZoneWithId and EncryptionZone. (clamb via wang)
|
||
|
||
HDFS-6705. Create an XAttr that disallows the HDFS admin from accessing a
|
||
file. (clamb via wang)
|
||
|
||
HDFS-6843. Create FileStatus isEncrypted() method (clamb via cmccabe)
|
||
|
||
HDFS-7004. Update KeyProvider instantiation to create by URI. (wang)
|
||
|
||
HDFS-7047. Expose FileStatus#isEncrypted in libhdfs (cmccabe)
|
||
|
||
HDFS-7003. Add NFS Gateway support for reading and writing to
|
||
encryption zones. (clamb via wang)
|
||
|
||
HDFS-6727. Refresh data volumes on DataNode based on configuration changes
|
||
(Lei Xu via cmccabe)
|
||
|
||
HDFS-6970. Move startFile EDEK retries to the DFSClient. (wang)
|
||
|
||
HDFS-6948. DN rejects blocks if it has older UC block
|
||
(Eric Payne via kihwal)
|
||
|
||
HDFS-6987. Move CipherSuite xattr information up to the encryption zone
|
||
root. (Zhe Zhang via wang)
|
||
|
||
HDFS-7139. Unit test for creating encryption zone on root path. (Zhe Zhang via wang)
|
||
|
||
HDFS-7138. Fix hftp to work with encryption. (clamb via wang)
|
||
|
||
HDFS-7118. Improve diagnostics on storage directory rename operations by
|
||
using NativeIO#renameTo in Storage#rename. (cnauroth)
|
||
|
||
HDFS-6808. Add command line option to ask DataNode reload configuration.
|
||
(Lei Xu via Colin Patrick McCabe)
|
||
|
||
HDFS-7119. Split error checks in AtomicFileOutputStream#close into separate
|
||
conditions to improve diagnostics. (cnauroth)
|
||
|
||
HDFS-7077. Separate CipherSuite from crypto protocol version. (wang)
|
||
|
||
HDFS-6956. Allow dynamically changing the tracing level in Hadoop servers
|
||
(cmccabe)
|
||
|
||
HDFS-7156. Update fsck documentation. (Masahiro Yamaguch via shv)
|
||
|
||
HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal)
|
||
|
||
HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw)
|
||
|
||
HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota.
|
||
(Binglin Chang via suresh)
|
||
|
||
HDFS-7104. Fix and clarify INodeInPath getter functions. (Zhe Zhang via wang)
|
||
|
||
HDFS-7124. Remove EncryptionZoneManager.NULL_EZ. (clamb via wang)
|
||
|
||
HDFS-6779. Add missing version subcommand for hdfs.
|
||
(Sasaki Toru via wheat9)
|
||
|
||
HDFS-7153. Add storagePolicy to NN edit log during file creation.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7158. Reduce the memory usage of WebImageViewer. (wheat9)
|
||
|
||
HDFS-6894. Add XDR parser method for each NFS response.
|
||
(Brandon Li via wheat9)
|
||
|
||
HDFS-7169. Add SE_BAD_FIELD to findbugsExcludeFile.xml. (szetszwo)
|
||
|
||
HDFS-7217. Better batching of IBRs. (kihwal)
|
||
|
||
HDFS-7195. Update user doc of secure mode about Datanodes don't require root
|
||
or jsvc. (cnauroth)
|
||
|
||
HDFS-7228. Add an SSD policy into the default BlockStoragePolicySuite.
|
||
(jing9)
|
||
|
||
HDFS-6904. YARN unable to renew delegation token fetched via webhdfs
|
||
due to incorrect service port. (jitendra)
|
||
|
||
HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin
|
||
P. McCabe)
|
||
|
||
HDFS-7230. Add rolling downgrade documentation. (szetszwo via jing9)
|
||
|
||
HDFS-6385. Show when block deletion will start after NameNode startup in
|
||
WebUI. (cnauroth)
|
||
|
||
HDFS-7313. Support optional configuration of AES cipher suite on
|
||
DataTransferProtocol. (cnauroth)
|
||
|
||
HDFS-7276. Limit the number of byte arrays used by DFSOutputStream and
|
||
provide a mechanism for recycling arrays. (szetszwo)
|
||
|
||
HDFS-7233. NN logs unnecessary org.apache.hadoop.hdfs.protocol.UnresolvedPathException.
|
||
(Rushabh S Shah via jing9)
|
||
|
||
HDFS-7221. TestDNFencingWithReplication fails consistently. (Charles Lamb via wang)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-6690. Deduplicate xattr names in memory. (wang)
|
||
|
||
HDFS-6773. MiniDFSCluster should skip edit log fsync by default (Stephen
|
||
Chu via Colin Patrick McCabe)
|
||
|
||
HDFS-6865. Byte array native checksumming on client side
|
||
(James Thomas via todd)
|
||
|
||
HDFS-7122. Use of ThreadLocal<Random> results in poor block placement.
|
||
(wang)
|
||
|
||
HDFS-6606. Optimize HDFS Encrypted Transport performance. (yliu)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-7309. XMLUtils.mangleXmlString doesn't seem to handle less than sign
|
||
(Colin Patrick McCabe via raviprak)
|
||
|
||
HDFS-6823. dfs.web.authentication.kerberos.principal shows up in logs for
|
||
insecure HDFS (Allen Wittenauer via raviprak)
|
||
|
||
HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira
|
||
AJISAKA via aw)
|
||
|
||
HDFS-6617. Flake TestDFSZKFailoverController.testManualFailoverWithDFSHAAdmin
|
||
due to a long edit log sync op. (Liang Xie via cnauroth)
|
||
|
||
HDFS-6646. [ HDFS Rolling Upgrade - Shell ] shutdownDatanode and getDatanodeInfo
|
||
usage is missed ( Brahma Reddy Battula via vinayakumarb)
|
||
|
||
HDFS-6630. Unable to fetch the block information by Browsing the file system on
|
||
Namenode UI through IE9 ( Haohui Mai via vinayakumarb)
|
||
|
||
HADOOP-8158. Interrupting hadoop fs -put from the command line
|
||
causes a LeaseExpiredException. (daryn via harsh)
|
||
|
||
HDFS-6678. MiniDFSCluster may still be partially running after initialization
|
||
fails. (cnauroth)
|
||
|
||
HDFS-5809. BlockPoolSliceScanner and high speed hdfs appending make
|
||
datanode to drop into infinite loop (cmccabe)
|
||
|
||
HDFS-6456. NFS should throw error for invalid entry in
|
||
dfs.nfs.exports.allowed.hosts (Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6689. NFS doesn't return correct lookup access for direcories (brandonli)
|
||
|
||
HDFS-6478. RemoteException can't be retried properly for non-HA scenario.
|
||
(Ming Ma via jing9)
|
||
|
||
HDFS-6693. TestDFSAdminWithHA fails on windows ( vinayakumarb )
|
||
|
||
HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails
|
||
with Client cannot authenticate via:[TOKEN, KERBEROS] error. (jing9)
|
||
|
||
HDFS-6704. Fix the command to launch JournalNode in HDFS-HA document.
|
||
(Akira AJISAKA via jing9)
|
||
|
||
HDFS-6731. Run "hdfs zkfc-formatZK" on a server in a non-namenode will cause
|
||
a null pointer exception. (Masatake Iwasaki via brandonli)
|
||
|
||
HDFS-6114. Block Scan log rolling will never happen if blocks written
|
||
continuously leading to huge size of dncp_block_verification.log.curr
|
||
(vinayakumarb via cmccabe)
|
||
|
||
HDFS-6455. NFS: Exception should be added in NFS log for invalid separator in
|
||
nfs.exports.allowed.hosts. (Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6715. Webhdfs wont fail over when it gets java.io.IOException: Namenode
|
||
is in startup mode. (jing9)
|
||
|
||
HDFS-5919. FileJournalManager doesn't purge empty and corrupt inprogress edits
|
||
files (vinayakumarb)
|
||
|
||
HDFS-6752. Avoid Address bind errors in TestDatanodeConfig#testMemlockLimit
|
||
(vinayakumarb)
|
||
|
||
HDFS-6749. FSNamesystem methods should call resolvePath.
|
||
(Charles Lamb via cnauroth)
|
||
|
||
HDFS-4629. Using com.sun.org.apache.xml.internal.serialize.* in
|
||
XmlEditsVisitor.java is JVM vendor specific. Breaks IBM JAVA.
|
||
(Amir Sanjar via stevel)
|
||
|
||
HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException
|
||
if option is specified without values. ( Madhukara Phatak via umamahesh)
|
||
|
||
HDFS-6797. DataNode logs wrong layoutversion during upgrade. (Benoy Antony
|
||
via Arpit Agarwal)
|
||
|
||
HDFS-6810. StorageReport array is initialized with wrong size in
|
||
DatanodeDescriptor#getStorageReports. (szetszwo via Arpit Agarwal)
|
||
|
||
HDFS-5723. Append failed FINALIZED replica should not be accepted as valid
|
||
when that block is underconstruction (vinayakumarb)
|
||
|
||
HDFS-5185. DN fails to startup if one of the data dir is full. (vinayakumarb)
|
||
|
||
HDFS-6451. NFS should not return NFS3ERR_IO for AccessControlException
|
||
(Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6717. JIRA HDFS-5804 breaks default nfs-gateway behavior for unsecured config
|
||
(brandonli)
|
||
|
||
HDFS-6790. DFSUtil Should Use configuration.getPassword for SSL passwords
|
||
(Larry McCay via brandonli)
|
||
|
||
HDFS-6791. A block could remain under replicated if all of its replicas are on
|
||
decommissioned nodes. (Ming Ma via jing9)
|
||
|
||
HDFS-6582. Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
|
||
(Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6830. BlockInfo.addStorage fails when DN changes the storage for a
|
||
block replica (Arpit Agarwal)
|
||
|
||
HDFS-6247. Avoid timeouts for replaceBlock() call by sending intermediate
|
||
responses to Balancer (vinayakumarb)
|
||
|
||
HDFS-6783. Fix HDFS CacheReplicationMonitor rescan logic. (Yi Liu and Colin Patrick McCabe via umamahesh)
|
||
|
||
HDFS-6825. Edit log corruption due to delayed block removal.
|
||
(Yongjun Zhang via wang)
|
||
|
||
HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade
|
||
(brandonli)
|
||
|
||
HDFS-6868. portmap and nfs3 are documented as hadoop commands instead of hdfs
|
||
(brandonli)
|
||
|
||
HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag. (Yi
|
||
Liu via jing9)
|
||
|
||
HDFS-6890. NFS readdirplus doesn't return dotdot attributes (brandonli)
|
||
|
||
HDFS-6829. DFSAdmin refreshSuperUserGroupsConfiguration failed in
|
||
security cluster (zhaoyunjiong via Arpit Agarwal)
|
||
|
||
HDFS-4852. libhdfs documentation is out of date. (cnauroth)
|
||
|
||
HDFS-6908. Incorrect snapshot directory diff generated by snapshot deletion.
|
||
(Juan Yu and jing9 via jing9)
|
||
|
||
HDFS-6892. Add XDR packaging method for each NFS request (brandonli)
|
||
|
||
HDFS-6902. FileWriter should be closed in finally block in
|
||
BlockReceiver#receiveBlock() (Tsuyoshi OZAWA via Colin Patrick McCabe)
|
||
|
||
HDFS-6972. TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration
|
||
doesn't decode url correctly. (Yongjun Zhang via wang)
|
||
|
||
HDFS-6942. Fix typos in log messages. (Ray Chiang via wheat9)
|
||
|
||
HDFS-6848. Lack of synchronization on access to datanodeUuid in
|
||
DataStorage#format(). (Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-6996. SnapshotDiff report can hit IndexOutOfBoundsException when there
|
||
are nested renamed directory/file. (jing9)
|
||
|
||
HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'.
|
||
(Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-6979. hdfs.dll does not produce .pdb files. (cnauroth)
|
||
|
||
HDFS-6862. Add missing timeout annotations to tests. (Xiaoyu Yao via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6898. DN must reserve space for a full block when an RBW block is
|
||
created. (Arpit Agarwal)
|
||
|
||
HDFS-7025. HDFS Credential Provider related Unit Test Failure.
|
||
(Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-7005. DFS input streams do not timeout.
|
||
|
||
HDFS-6951. Correctly persist raw namespace xattrs to edit log and fsimage.
|
||
(clamb via wang)
|
||
|
||
HDFS-6800. Support Datanode layout changes with rolling upgrade.
|
||
(James Thomas via Arpit Agarwal)
|
||
|
||
HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal)
|
||
|
||
HDFS-6506. Newly moved block replica been invalidated and deleted in
|
||
TestBalancer. (Binglin Chang via cnauroth)
|
||
|
||
HDFS-6966. Add additional unit tests for encryption zones.
|
||
(Stephen Chu via wang)
|
||
|
||
HDFS-6621. Hadoop Balancer prematurely exits iterations.
|
||
(Rafal Wojdyla and Benjamin Bowman via wang)
|
||
|
||
HDFS-7045. Fix NameNode deadlock when opening file under /.reserved path.
|
||
(Yi Liu via wang)
|
||
|
||
HDFS-7032. Add WebHDFS support for reading and writing to encryption zones.
|
||
(clamb via wang)
|
||
|
||
HDFS-6965. NN continues to issue block locations for DNs with full disks.
|
||
(Rushabh Shah via kihwal)
|
||
|
||
HDFS-6789. TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI
|
||
and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7.
|
||
(Akira Ajisaka via wang)
|
||
|
||
HDFS-6912. SharedFileDescriptorFactory should not allocate sparse files
|
||
(cmccabe)
|
||
|
||
HDFS-7075. hadoop-fuse-dfs fails because it cannot find
|
||
JavaKeyStoreProvider$Factory (cmccabe)
|
||
|
||
HDFS-7078. Fix listEZs to work correctly with snapshots. (wang)
|
||
|
||
HDFS-6840. Clients are always sent to the same datanode when read
|
||
is off rack. (wang)
|
||
|
||
HDFS-7065. Pipeline close recovery race can cause block corruption (kihwal)
|
||
|
||
HDFS-7096. Fix TestRpcProgramNfs3 to use DFS_ENCRYPTION_KEY_PROVIDER_URI
|
||
(clamb via cmccabe)
|
||
|
||
HDFS-7046. HA NN can NPE upon transition to active. (kihwal)
|
||
|
||
HDFS-7106. Reconfiguring DataNode volumes does not release the lock files
|
||
in removed volumes. (cnauroth via cmccabe)
|
||
|
||
HDFS-7001. Tests in TestTracing depends on the order of execution
|
||
(iwasakims via cmccabe)
|
||
|
||
HDFS-7132. hdfs namenode -metadataVersion command does not honor
|
||
configured name dirs. (Charles Lamb via wang)
|
||
|
||
HDFS-7049. TestByteRangeInputStream.testPropagatedClose fails and throw
|
||
NPE on branch-2. (Juan Yu via wheat9)
|
||
|
||
HDFS-7148. TestEncryptionZones#testIsEncryptedMethod fails on branch-2
|
||
after archival storage merge. (wang)
|
||
|
||
HDFS-7157. Using Time.now() for recording start/end time of reconfiguration
|
||
tasks (Lei Xu via cmccabe)
|
||
|
||
HDFS-6664. HDFS permissions guide documentation states incorrect default
|
||
group mapping class. (Ray Chiang via aw)
|
||
|
||
HDFS-4227. Document dfs.namenode.resource.* (Daisuke Kobayashi via aw)
|
||
|
||
HDFS-6754. TestNamenodeCapacityReport may sometimes fail due to lack of
|
||
retry. (Mit Desai via kihwal)
|
||
|
||
HDFS-7172. Test data files may be checked out of git with incorrect line
|
||
endings, causing test failures in TestHDFSCLI. (Chris Nauroth via wheat9)
|
||
|
||
HDFS-7176. The namenode usage message doesn't include "-rollingupgrade
|
||
started" (cmccabe)
|
||
|
||
HDFS-7178. Additional unit test for replica write with full disk.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7162. Wrong path when deleting through fuse-dfs a file which already
|
||
exists in trash (Chengbing Liu via cmccabe)
|
||
|
||
HDFS-7179. DFSClient should instantiate a KeyProvider, not a
|
||
KeyProviderCryptoExtension. (wang)
|
||
|
||
HDFS-7181. Remove incorrect precondition check on key length in
|
||
FileEncryptionInfo. (wang)
|
||
|
||
HDFS-7203. Concurrent appending to the same file can cause data corruption
|
||
(kihwal)
|
||
|
||
HDFS-7236. Fix TestOpenFilesWithSnapshot#testOpenFilesWithMultipleSnapshots.
|
||
(Yongjun Zhang via jing9)
|
||
|
||
HDFS-6544. Broken Link for GFS in package.html. (Suraj Nayak M via wheat9)
|
||
|
||
HDFS-7237. The command "hdfs namenode -rollingUpgrade" throws
|
||
ArrayIndexOutOfBoundsException. (szetszwo)
|
||
|
||
HDFS-7185. The active NameNode will not accept an fsimage sent from the
|
||
standby during rolling upgrade. (jing9)
|
||
|
||
HDFS-7208. NN doesn't schedule replication when a DN storage fails.
|
||
(Ming Ma via szetszwo)
|
||
|
||
HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support
|
||
FSIMAGE_NAME_OPTIMIZATION. (szetszwo)
|
||
|
||
HDFS-7260. Change DFSOutputStream.MAX_PACKETS to be configurable. (szetszwo)
|
||
|
||
HDFS-7259. Unresponseive NFS mount point due to deferred COMMIT response.
|
||
(brandonli)
|
||
|
||
HDFS-7215.Add JvmPauseMonitor to NFS gateway (brandonli)
|
||
|
||
HDFS-7180. NFSv3 gateway frequently gets stuck due to GC (brandonli)
|
||
|
||
HDFS-7243. HDFS concat operation should not be allowed in Encryption Zone.
|
||
(clamb via yliu)
|
||
|
||
HADOOP-11233. hadoop.security.kms.client.encrypted.key.cache.expiry
|
||
property spelled wrong in core-default. (Stephen Chu via yliu)
|
||
|
||
HDFS-7218. FSNamesystem ACL operations should write to audit log on
|
||
failure. (clamb via yliu)
|
||
|
||
HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer
|
||
crashes with an unchecked exception (rushabhs via cmccabe)
|
||
|
||
HDFS-7383. DataNode.requestShortCircuitFdsForRead may throw
|
||
NullPointerException. (szetszwo via suresh)
|
||
|
||
HDFS-7147. Update archival storage user documentation.
|
||
(Tsz Wo Nicholas Sze via wheat9)
|
||
|
||
HDFS-7340. Make rollingUpgrade start/finalize idempotent. (jing9)
|
||
|
||
HDFS-7334. Fix periodic failures of TestCheckpoint
|
||
#testTooManyEditReplayFailures. (Charles Lamb via wheat9)
|
||
|
||
HDFS-7355. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure
|
||
fails on Windows, because we cannot deny access to the file owner.
|
||
(Chris Nauroth via wheat9)
|
||
|
||
HDFS-7367. HDFS short-circuit read cannot negotiate shared memory slot and
|
||
file descriptors when SASL is enabled on DataTransferProtocol.
|
||
(Chris Nauroth via wheat9)
|
||
|
||
HDFS-7364. Balancer always shows zero Bytes Already Moved.
|
||
(Tsz Wo Nicholas Sze via jing9)
|
||
|
||
HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files
|
||
incorrectly. (Xiaoyu Yao via wheat9)
|
||
|
||
BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-6387. HDFS CLI admin tool for creating & deleting an
|
||
encryption zone. (clamb)
|
||
|
||
HDFS-6386. HDFS Encryption Zones (clamb)
|
||
|
||
HDFS-6388. HDFS integration with KeyProvider. (clamb)
|
||
|
||
HDFS-6473. Protocol and API for Encryption Zones (clamb)
|
||
|
||
HDFS-6392. Wire crypto streams for encrypted files in
|
||
DFSClient. (clamb and yliu)
|
||
|
||
HDFS-6476. Print out the KeyProvider after finding KP successfully on
|
||
startup. (Juan Yu via wang)
|
||
|
||
HDFS-6391. Get the Key/IV from the NameNode for encrypted files in
|
||
DFSClient. (Charles Lamb and wang)
|
||
|
||
HDFS-6389. Rename restrictions for encryption zones. (clamb)
|
||
|
||
HDFS-6605. Client server negotiation of cipher suite. (wang)
|
||
|
||
HDFS-6625. Remove the Delete Encryption Zone function (clamb)
|
||
|
||
HDFS-6516. List of Encryption Zones should be based on inodes (clamb)
|
||
|
||
HDFS-6629. Not able to create symlinks after HDFS-6516 (umamaheswararao)
|
||
|
||
HDFS-6635. Refactor encryption zone functionality into new
|
||
EncryptionZoneManager class. (wang)
|
||
|
||
HDFS-6474. Namenode needs to get the actual keys and iv from the
|
||
KeyProvider. (wang)
|
||
|
||
HDFS-6619. Clean up encryption-related tests. (wang)
|
||
|
||
HDFS-6405. Test Crypto streams in HDFS. (yliu via wang)
|
||
|
||
HDFS-6490. Fix the keyid format for generated keys in
|
||
FSNamesystem.createEncryptionZone (clamb)
|
||
|
||
HDFS-6716. Update usage of KeyProviderCryptoExtension APIs on NameNode.
|
||
(wang)
|
||
|
||
HDFS-6718. Remove EncryptionZoneManager lock. (wang)
|
||
|
||
HDFS-6720. Remove KeyProvider in EncryptionZoneManager. (wang)
|
||
|
||
HDFS-6738. Remove unnecessary getEncryptionZoneForPath call in
|
||
EZManager#createEncryptionZone. (clamb)
|
||
|
||
HDFS-6724. Decrypt EDEK before creating
|
||
CryptoInputStream/CryptoOutputStream. (wang)
|
||
|
||
HDFS-6509. Create a special /.reserved/raw directory for raw access to
|
||
encrypted data. (clamb via wang)
|
||
|
||
HDFS-6771. Require specification of an encryption key when creating
|
||
an encryption zone. (wang)
|
||
|
||
HDFS-6730. Create a .RAW extended attribute namespace. (clamb)
|
||
|
||
HDFS-6692. Add more HDFS encryption tests. (wang)
|
||
|
||
HDFS-6780. Batch the encryption zones listing API. (wang)
|
||
|
||
HDFS-6394. HDFS encryption documentation. (wang)
|
||
|
||
HDFS-6834. Improve the configuration guidance in DFSClient when there
|
||
are no Codec classes found in configs. (umamahesh)
|
||
|
||
HDFS-6546. Add non-superuser capability to get the encryption zone
|
||
for a specific path. (clamb)
|
||
|
||
HDFS-6733. Creating encryption zone results in NPE when
|
||
KeyProvider is null. (clamb)
|
||
|
||
HDFS-6785. Should not be able to create encryption zone using path
|
||
to a non-directory file. (clamb)
|
||
|
||
HDFS-6807. Fix TestReservedRawPaths. (clamb)
|
||
|
||
HDFS-6814. Mistakenly dfs.namenode.list.encryption.zones.num.responses configured
|
||
as boolean. (umamahesh)
|
||
|
||
HDFS-6817. Fix findbugs and other warnings. (yliu)
|
||
|
||
HDFS-6839. Fix TestCLI to expect new output. (clamb)
|
||
|
||
HDFS-6954. With crypto, no native lib systems are too verbose. (clamb via wang)
|
||
|
||
HDFS-2975. Rename with overwrite flag true can make NameNode to stuck in safemode
|
||
on NN (crash + restart). (Yi Liu via umamahesh)
|
||
|
||
HDFS-6905. fs-encryption merge triggered release audit failures. (clamb via tucu)
|
||
|
||
HDFS-6714. TestBlocksScheduledCounter#testBlocksScheduledCounter should
|
||
shutdown cluster (vinayakumarb)
|
||
|
||
HDFS-6986. DistributedFileSystem must get delegation tokens from configured
|
||
KeyProvider. (zhz via tucu)
|
||
|
||
HDFS-6776. Using distcp to copy data between insecure and secure cluster via webdhfs
|
||
doesn't work. (yzhangal via tucu)
|
||
|
||
HDFS-7042. Upgrade fails for Windows HA cluster due to file locks held during
|
||
rename in JournalNode. (cnauroth)
|
||
|
||
HDFS-7051. TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file
|
||
path separator. (cnauroth)
|
||
|
||
HDFS-7105. Fix TestJournalNode#testFailToStartWithBadConfig to match log
|
||
output change. (Ray Chiang via cnauroth)
|
||
|
||
HDFS-7105. Allow falling back to a non-SASL connection on
|
||
DataTransferProtocol in several edge cases. (cnauroth)
|
||
|
||
HDFS-7107. Avoid Findbugs warning for synchronization on
|
||
AbstractNNFailoverProxyProvider#fallbackToSimpleAuth. (cnauroth)
|
||
|
||
HDFS-7109. TestDataStorage does not release file locks between tests.
|
||
(cnauroth)
|
||
|
||
HDFS-7110. Skip tests related to short-circuit read on platforms that do not
|
||
currently implement short-circuit read. (cnauroth)
|
||
|
||
HDFS-7115. TestEncryptionZones assumes Unix path separator for KMS key store
|
||
path. (Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-7115. TestEncryptionZonesWithHA assumes Unix path separator for KMS key
|
||
store path. (Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows.
|
||
(cnauroth)
|
||
|
||
HDFS-6534. Fix build on macosx: HDFS parts (Binglin Chang via aw)
|
||
|
||
HDFS-7111. TestSafeMode assumes Unix line endings in safe mode tip.
|
||
(cnauroth)
|
||
|
||
HDFS-7127. TestLeaseRecovery leaks MiniDFSCluster instances. (cnauroth)
|
||
|
||
HDFS-7131. During HA upgrade, JournalNode should create a new committedTxnId
|
||
file in the current directory. (jing9)
|
||
|
||
HDFS-6995. Block should be placed in the client's 'rack-local' node
|
||
if 'client-local' node is not available (vinayakumarb)
|
||
|
||
HDFS-7128. Decommission slows way down when it gets towards the end.
|
||
(Ming Ma via cnauroth)
|
||
|
||
HDFS-7287. The OfflineImageViewer (OIV) can output invalid XML depending on
|
||
the filename (Ravi Prakash via Colin P. McCabe)
|
||
|
||
HDFS-7300. The getMaxNodesPerRack() method in BlockPlacementPolicyDefault
|
||
is flawed (kihwal)
|
||
|
||
HDFS-7305. NPE seen in wbhdfs FS while running SLive. (jing9)
|
||
|
||
BREAKDOWN OF HDFS-6584 ARCHIVAL STORAGE
|
||
|
||
HDFS-6677. Change INodeFile and FSImage to support storage policy ID.
|
||
(szetszwo)
|
||
|
||
HDFS-6670. Add block storage policy support with default HOT, WARM and COLD
|
||
policies. (szetszwo)
|
||
|
||
HDFS-6671. Change BlockPlacementPolicy to consider block storage policy
|
||
in replicaiton. (szetszwo)
|
||
|
||
HDFS-6710. Change BlockPlacementPolicy to consider block storage policy
|
||
in replica deletion. (szetszwo)
|
||
|
||
HDFS-6679. Bump NameNodeLayoutVersion and update editsStored test files.
|
||
(vinayakumarb via szetszwo)
|
||
|
||
HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage
|
||
types are unavailable. (szetszwo)
|
||
|
||
HDFS-6835. Add a new API to set storage policy. (jing9)
|
||
|
||
HDFS-6847. Support storage policy on directories and include storage policy
|
||
in HdfsFileStatus. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-6801. Add a new data migration tool, Mover, for archiving data.
|
||
(szetszwo via jing9)
|
||
|
||
HDFS-6863. Support migration for snapshot paths. (jing9)
|
||
|
||
HDFS-6906. Add more tests for BlockStoragePolicy. (szetszwo via jing9)
|
||
|
||
HDFS-6911. check if a block is already scheduled in Mover.
|
||
(szetszwo via jing9)
|
||
|
||
HDFS-6920. Check the storage type of delNodeHintStorage when deleting
|
||
a replica. (szetszwo via jing9)
|
||
|
||
HDFS-6944. Add retry and termination logic for Mover. (jing9)
|
||
|
||
HDFS-6969. INode#getStoragePolicyID should always return the latest
|
||
storage policy. (jing9)
|
||
|
||
HDFS-6961. BlockPlacementPolicy#chooseTarget should check each valid
|
||
storage type in each choosing round. (jing9)
|
||
|
||
HDFS-6876. support set/get storage policy in DFSAdmin. (jing9)
|
||
|
||
HDFS-6997. Add more tests for data migration and replicaion. (szetszwo)
|
||
|
||
HDFS-6875. Support migration for a list of specified paths. (jing9)
|
||
|
||
HDFS-7027. Mover does not terminate when some storage type is out of space.
|
||
(szetszwo via jing9)
|
||
|
||
HDFS-7029. Fix TestDFSInotifyEventInputStream and TestDistributedFileSystem.
|
||
(szetszwo via jing9)
|
||
|
||
HDFS-7028. FSDirectory should not get storage policy id from symlinks.
|
||
(szetszwo)
|
||
|
||
HDFS-7034. Fix TestBlockPlacement and TestStorageMover. (jing9)
|
||
|
||
HDFS-7039. Fix Balancer tests. (szetszwo via jing9)
|
||
|
||
HDFS-7062. Skip under construction block for migration. (jing9)
|
||
|
||
HDFS-7052. Add Mover into hdfs script. (jing9)
|
||
|
||
HDFS-7072. Fix TestBlockManager and TestStorageMover. (jing9 via szetszwo)
|
||
|
||
HDFS-6864. Archival Storage: add user documentation. (szetszwo via jing9)
|
||
|
||
HDFS-7088. Archival Storage: fix TestBalancer and
|
||
TestBalancerWithMultipleNameNodes. (szetszwo via jing9)
|
||
|
||
HDFS-7095. TestStorageMover often fails in Jenkins. (jing9)
|
||
|
||
HDFS-7081. Add new DistributedFileSystem API for getting all the existing
|
||
storage policies. (jing9)
|
||
|
||
HDFS-7140. Add a tool to list all the existing block storage policies.
|
||
(jing9)
|
||
|
||
HDFS-7167. NPE while running Mover if the given path is for a file. (jing9)
|
||
|
||
HDFS-7154. Fix returning value of starting reconfiguration task (Lei Xu via
|
||
Colin P. McCabe)
|
||
|
||
BREAKDOWN OF HDFS-6581 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-6921. Add LazyPersist flag to FileStatus. (Arpit Agarwal)
|
||
|
||
HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal)
|
||
|
||
HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and
|
||
edit logs. (Arpit Agarwal)
|
||
|
||
HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-6925. DataNode should attempt to place replicas on transient storage
|
||
first if lazyPersist flag is received. (Arpit Agarwal)
|
||
|
||
HDFS-6926. DN support for saving replicas to persistent storage and
|
||
evicting in-memory replicas. (Arpit Agarwal)
|
||
|
||
HDFS-6927. Initial unit tests for lazy persist files. (Arpit Agarwal)
|
||
|
||
HDFS-6929. NN periodically unlinks lazy persist files with missing
|
||
replicas from namespace. (Arpit Agarwal)
|
||
|
||
HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-6931. Move lazily persisted replicas to finalized directory on DN
|
||
startup. (Arpit Agarwal)
|
||
|
||
HDFS-6950. Add Additional unit tests for HDFS-6581. (Xiaoyu Yao via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal)
|
||
|
||
HDFS-6977. Delete all copies when a block is deleted from the block space.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-6991. Notify NN of evicted block before deleting it from RAM disk.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-6978. Directory scanner should correctly reconcile blocks on RAM
|
||
disk. (Arpit Agarwal)
|
||
|
||
HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState.
|
||
(Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Xiaoyu Yao via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6581. Few more unit test fixes for HDFS-6581. (Arpit Agarwal)
|
||
|
||
HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal)
|
||
|
||
HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved.
|
||
(Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-7091. Add forwarding constructor for INodeFile for existing callers.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal)
|
||
|
||
HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal)
|
||
|
||
HDFS-7071. Updated editsStored and editsStored.xml to bump layout
|
||
version and add LazyPersist flag. (Xiaoyu Yao and Arpit Agarwal via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6990. Add unit test for evict/delete RAM_DISK block with open
|
||
handle. (Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (szetszwo via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK.
|
||
(Xiaoyu Yao via Arpit Agarwal)
|
||
|
||
HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (szetszwo via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7153. Add storagePolicy to NN edit log during file creation.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7159. Use block storage policy to set lazy persist preference.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-7129. Metrics to track usage of memory for writes. (Xiaoyu Yao
|
||
via Arpit Agarwal)
|
||
|
||
HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal)
|
||
|
||
HDFS-7112. LazyWriter should use either async IO or one thread per physical
|
||
disk. (Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-7090. Use unbuffered writes when persisting in-memory replicas.
|
||
(Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-6934. Move checksum computation off the hot path when writing to RAM
|
||
disk. (cnauroth)
|
||
|
||
HDFS-7291. Persist in-memory replicas with appropriate unbuffered copy API
|
||
on POSIX and Windows. (Xiaoyu Yao via cnauroth)
|
||
|
||
HDFS-7328. TestTraceAdmin assumes Unix line endings. (cnauroth)
|
||
|
||
HDFS-7359. NameNode in secured HA cluster fails to start if
|
||
dfs.namenode.secondary.http-address cannot be interpreted as a network
|
||
address. (cnauroth)
|
||
|
||
HDFS-7226. Fix TestDNFencing.testQueueingWithAppend. (Yongjun Zhang via jing9)
|
||
|
||
HDFS-7382. DataNode in secure mode may throw NullPointerException if client
|
||
connects before DataNode registers itself with NameNode. (cnauroth)
|
||
|
||
HDFS-7387. NFS may only do partial commit due to a race between COMMIT and write
|
||
(brandonli)
|
||
|
||
HDFS-7391. Renable SSLv2Hello in HttpFS. (rkanter via acmurthy)
|
||
|
||
HDFS-7385. ThreadLocal used in FSEditLog class causes FSImage permission mess
|
||
up. (jiangyu via cnauroth)
|
||
|
||
Release 2.5.2 - 2014-11-19
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-7274. Disable SSLv3 in HttpFS. (Robert Kanter via kasha)
|
||
|
||
Release 2.5.1 - 2014-09-05
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HADOOP-10957. The globber will sometimes erroneously return a permission
|
||
denied exception when there is a non-terminal wildcard (cmccabe)
|
||
|
||
Release 2.5.0 - 2014-08-11
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure,
|
||
getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from
|
||
DistributedFileSystem. (szetszwo)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-6281. Provide option to use the NFS Gateway without having to use the
|
||
Hadoop portmapper. (atm)
|
||
|
||
HDFS-5168. Add cross node dependency support to BlockPlacementPolicy.
|
||
(Nikola Vujic via szetszwo)
|
||
|
||
HDFS-6334. Client failover proxy provider for IP failover based NN HA.
|
||
(kihwal)
|
||
|
||
HDFS-6406. Add capability for NFS gateway to reject connections from
|
||
unprivileged ports. (atm)
|
||
|
||
HDFS-2006. Ability to support storing extended attributes per file.
|
||
|
||
HDFS-5978. Create a tool to take fsimage and expose read-only WebHDFS API.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6278. Create HTML5-based UI for SNN. (wheat9)
|
||
|
||
HDFS-6279. Create new index page for JN / DN. (wheat9)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-6007. Update documentation about short-circuit local reads (iwasakims
|
||
via cmccabe)
|
||
|
||
HDFS-6125. Cleanup unnecessary cast in HDFS code base. (suresh)
|
||
|
||
HDFS-5196. Provide more snapshot information in WebUI.
|
||
(Shinichi Yamashita via wheat9)
|
||
|
||
HDFS-6155. Fix Boxing/unboxing to parse a primitive findbugs warnings.
|
||
(suresh)
|
||
|
||
HDFS-6119. FSNamesystem code cleanup. (suresh)
|
||
|
||
HDFS-6158. Clean up dead code for OfflineImageViewer. (wheat9)
|
||
|
||
HDFS-6164. Remove lsr in OfflineImageViewer. (wheat9)
|
||
|
||
HDFS-6167. Relocate the non-public API classes in the hdfs.client package.
|
||
(szetszwo)
|
||
|
||
HDFS-6191. Disable quota checks when replaying edit log. (kihwal)
|
||
|
||
HDFS-6170. Support GETFILESTATUS operation in WebImageViewer.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6225. Remove the o.a.h.hdfs.server.common.UpgradeStatusReport.
|
||
(wheat9)
|
||
|
||
HDFS-6224. Add a unit test to TestAuditLogger for file permissions
|
||
passed to logAuditEvent. (Charles Lamb via wang)
|
||
|
||
HDFS-6194. Create new tests for ByteRangeInputStream.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6219. Proxy superuser configuration should use true client IP for
|
||
address checks. (daryn via kihwal)
|
||
|
||
HDFS-6256. Clean up ImageVisitor and SpotCheckImageVisitor.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6265. Prepare HDFS codebase for JUnit 4.11. (cnauroth)
|
||
|
||
HDFS-5693. Few NN metrics data points were collected via JMX when NN
|
||
is under heavy load. (Ming Ma via jing9)
|
||
|
||
HDFS-6273. Config options to allow wildcard endpoints for namenode HTTP
|
||
and HTTPS servers. (Arpit Agarwal)
|
||
|
||
HDFS-6282. Re-add testIncludeByRegistrationName. (cmccabe)
|
||
|
||
HDFS-6266. Identify full path for a given INode. (jing9)
|
||
|
||
HDFS-6210. Support GETACLSTATUS operation in WebImageViewer.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6269. NameNode Audit Log should differentiate between webHDFS open and
|
||
HDFS open. (Eric Payne via jeagles)
|
||
|
||
HDFS-6304. Consolidate the logic of path resolution in FSDirectory.
|
||
(wheat9)
|
||
|
||
HDFS-6295. Add "decommissioning" state and node state filtering to
|
||
dfsadmin. (wang)
|
||
|
||
HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is
|
||
renamed. (cmccabe)
|
||
|
||
HDFS-6328. Clean up dead code in FSDirectory. (wheat9)
|
||
|
||
HDFS-6230. Expose upgrade status through NameNode web UI.
|
||
(Mit Desai via wheat9)
|
||
|
||
HDFS-6186. Pause deletion of blocks when the namenode starts up. (jing9)
|
||
|
||
HDFS-6293. Issues with OIV processing PB-based fsimages. (kihwal)
|
||
|
||
HDFS-2949. Add check to active state transition to prevent operator-induced
|
||
split brain. (Rushabh S Shah via kihwal)
|
||
|
||
HDFS-6287. Add vecsum test of libhdfs read access times (cmccabe)
|
||
|
||
HDFS-5683. Better audit log messages for caching operations.
|
||
(Abhiraj Butala via wang)
|
||
|
||
HDFS-6345. DFS.listCacheDirectives() should allow filtering based on
|
||
cache directive ID. (wang)
|
||
|
||
HDFS-6432. Add snapshot related APIs to webhdfs. (jing9)
|
||
|
||
HDFS-6396. Remove support for ACL feature from INodeSymlink.
|
||
(Charles Lamb via wang)
|
||
|
||
HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS
|
||
gateway. (atm via wang)
|
||
|
||
HDFS-6110 adding more slow action log in critical write path
|
||
(Liang Xie via stack)
|
||
|
||
HDFS-6416. Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to
|
||
avoid system clock bugs (Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6356. Fix typo in DatanodeLayoutVersion. (Tulasi G via wang)
|
||
|
||
HDFS-6447. balancer should timestamp the completion message.
|
||
(Juan Yu via wang)
|
||
|
||
HDFS-6463. Clarify behavior of AclStorage#createFsPermissionForExtendedAcl
|
||
in comments. (cnauroth)
|
||
|
||
HDFS-6472. fix typo in webapps/hdfs/explorer.js. (Juan Yu via wang)
|
||
|
||
HDFS-6056. Clean up NFS config settings (brandonli)
|
||
|
||
HDFS-6109 let sync_file_range() system call run in background
|
||
(Liang Xie via stack)
|
||
|
||
HDFS-6268. Better sorting in NetworkTopology#pseudoSortByDistance when
|
||
no local node is found. (wang)
|
||
|
||
HDFS-6369. Document that BlockReader#available() can return more bytes than
|
||
are remaining in the block (Ted Yu via Colin Patrick McCabe)
|
||
|
||
HDFS-6487. TestStandbyCheckpoint#testSBNCheckpoints is racy.
|
||
(Mit Desai via wang)
|
||
|
||
HDFS-6297. Add CLI testcases to reflect new features of dfs and dfsadmin
|
||
(Dasha Boudnik via cos)
|
||
|
||
HDFS-6399. Add note about setfacl in HDFS permissions guide.
|
||
(cnauroth via wang)
|
||
|
||
HDFS-6315. Decouple recording edit logs from FSDirectory. (wheat9)
|
||
|
||
HDFS-6379. HTTPFS - Implement ACLs support. (yoderme via tucu)
|
||
|
||
HDFS-6471. Make moveFromLocal CLI testcases to be non-disruptive
|
||
(Dasha Boudnik via cos)
|
||
|
||
HDFS-6395. Skip checking xattr limits for non-user-visible namespaces.
|
||
(Yi Liu via wang).
|
||
|
||
HDFS-3493. Invalidate excess corrupted blocks as long as minimum
|
||
replication is satisfied. (Juan Yu and Vinayakumar B via wang)
|
||
|
||
HDFS-6330. Move mkdirs() to FSNamesystem. (wheat9)
|
||
|
||
HDFS-6470. TestBPOfferService.testBPInitErrorHandling is flaky.
|
||
(Ming Ma via wang)
|
||
|
||
HDFS-6529. Trace logging for RemoteBlockReader2 to identify remote datanode
|
||
and file being read. (Anubhav Dhoot via atm)
|
||
|
||
HDFS-6499. Use NativeIO#renameTo instead of File#renameTo in
|
||
FileJournalManager. (Yongjun Zhang via atm)
|
||
|
||
HDFS-6518. TestCacheDirectives#testExceedsCapacity should
|
||
take FSN read lock when accessing pendingCached list.
|
||
(wang)
|
||
|
||
HDFS-6528. Add XAttrs to TestOfflineImageViewer. (Stephen Chu via wang)
|
||
|
||
HDFS-6545. Finalizing rolling upgrade can make NN unavailable for a long
|
||
duration. (kihwal)
|
||
|
||
HDFS-6530. Fix Balancer documentation. (szetszwo)
|
||
|
||
HDFS-6480. Move waitForReady() from FSDirectory to FSNamesystem. (wheat9)
|
||
|
||
HDFS-6403. Add metrics for log warnings reported by JVM pauses. (Yongjun
|
||
Zhang via atm)
|
||
|
||
HDFS-6557. Move the reference of fsimage to FSNamesystem. (wheat9)
|
||
|
||
HDFS-4667. Capture renamed files/directories in snapshot diff report. (jing9
|
||
and Binglin Chang via jing9)
|
||
|
||
HDFS-6507. Improve DFSAdmin to support HA cluster better.
|
||
(Zesheng Wu via vinayakumarb)
|
||
|
||
HDFS-6578. add toString method to DatanodeStorage for easier debugging.
|
||
(Yongjun Zhang via Arpit Agarwal)
|
||
|
||
HDFS-6562. Refactor rename() in FSDirectory. (wheat9)
|
||
|
||
HDFS-6486. Add user doc for XAttrs via WebHDFS. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6430. HTTPFS - Implement XAttr support. (Yi Liu via tucu)
|
||
|
||
HDFS-6593. Move SnapshotDiffInfo out of INodeDirectorySnapshottable.
|
||
(Jing Zhao via wheat9)
|
||
|
||
HDFS-6595. Allow the maximum threads for balancing on datanodes to be
|
||
configurable. (Benoy Antony via szetszwo)
|
||
|
||
HDFS-6572. Add an option to the NameNode that prints the software and
|
||
on-disk image versions. (Charles Lamb via cnauroth)
|
||
|
||
HDFS-6603. Add XAttr with ACL test. (Stephen Chu via cnauroth)
|
||
|
||
HDFS-6612. MiniDFSNNTopology#simpleFederatedTopology(int)
|
||
always hardcode nameservice ID. (Juan Yu via wang)
|
||
|
||
HDFS-6614. shorten TestPread run time with a smaller retry timeout setting.
|
||
(Liang Xie via cnauroth)
|
||
|
||
HDFS-6610. TestShortCircuitLocalRead tests sometimes timeout on slow
|
||
machines. (Charles Lamb via wang)
|
||
|
||
HDFS-6620. Snapshot docs should specify about preserve options with cp command
|
||
(Stephen Chu via umamahesh)
|
||
|
||
HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second
|
||
instead of millisecond. (Juan Yu via wang)
|
||
|
||
HDFS-6680. BlockPlacementPolicyDefault does not choose favored nodes
|
||
correctly. (szetszwo)
|
||
|
||
HDFS-6712. Document HDFS Multihoming Settings. (Arpit Agarwal)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-6214. Webhdfs has poor throughput for files >2GB (daryn)
|
||
|
||
HDFS-6460. Ignore stale and decommissioned nodes in
|
||
NetworkTopology#sortByDistance. (Yongjun Zhang via wang)
|
||
|
||
HDFS-6583. Remove clientNode in FileUnderConstructionFeature. (wheat9)
|
||
|
||
HDFS-6599. 2.4 addBlock is 10 to 20 times slower compared to 0.23 (daryn)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-6112. NFS Gateway docs are incorrect for allowed hosts configuration.
|
||
(atm)
|
||
|
||
HDFS-3087. Decomissioning on NN restart can complete without blocks being
|
||
replicated. (Rushabh S Shah via kihwal)
|
||
|
||
HDFS-6162. Format strings should use platform independent line separator.
|
||
(suresh)
|
||
|
||
HDFS-6156. Simplify the JMX API that provides snapshot information.
|
||
(wheat9)
|
||
|
||
HDFS-6173. Move the default processor from Ls to Web in OfflineImageViewer.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-5591. Checkpointing should use monotonic time when calculating period.
|
||
(Charles Lamb via wang)
|
||
|
||
HDFS-6190. Minor textual fixes in DFSClient. (Charles Lamb via wheat9)
|
||
|
||
HDFS-6159. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if
|
||
there is block missing after balancer success (Chen He via kihwal)
|
||
|
||
HDFS-6181. Fix the wrong property names in NFS user guide (brandonli)
|
||
|
||
HDFS-6180. dead node count / listing is very broken in JMX and old GUI.
|
||
(wheat9)
|
||
|
||
HDFS-6143. WebHdfsFileSystem open should throw FileNotFoundException for
|
||
non-existing paths. (Gera Shegalov via wheat9)
|
||
|
||
HDFS-6169. Move the address in WebImageViewer. (Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6160. TestSafeMode occasionally fails. (Arpit Agarwal)
|
||
|
||
HDFS-5669. Storage#tryLock() should check for null before logging successfull message
|
||
(Vinayakumar B via umamahesh)
|
||
|
||
HDFS-6237. TestDFSShell#testGet fails on Windows due to invalid file system
|
||
path. (cnauroth)
|
||
|
||
HDFS-6238. TestDirectoryScanner leaks file descriptors. (cnauroth)
|
||
|
||
HDFS-6243. HA NameNode transition to active or shutdown may leave lingering
|
||
image transfer thread. (cnauroth)
|
||
|
||
HDFS-5409. TestOfflineEditsViewer#testStored fails on Windows due to CRLF
|
||
line endings in editsStored.xml from git checkout. (cnauroth)
|
||
|
||
HDFS-4909. Avoid protocol buffer RPC namespace clashes. (cmccabe)
|
||
|
||
HDFS-6153. Document "fileId" and "childrenNum" fields in the FileStatus Json schema
|
||
(Akira Ajisaka via vinayakumarb)
|
||
|
||
HDFS-6178. Decommission on standby NN couldn't finish. (Ming Ma via jing9)
|
||
|
||
HDFS-6213. TestDataNodeConfig failing on Jenkins runs due to DN web port
|
||
in use. (wang)
|
||
|
||
HDFS-6274. Cleanup javadoc warnings in HDFS code. (suresh)
|
||
|
||
HDFS-6275. Fix warnings - type arguments can be inferred and redudant
|
||
local variable. (suresh)
|
||
|
||
HDFS-6217. Webhdfs PUT operations may not work via a http proxy.
|
||
(Daryn Sharp via kihwal)
|
||
|
||
HDFS-6276. Remove unnecessary conditions and null check. (suresh)
|
||
|
||
HDFS-5865. Update OfflineImageViewer document. (Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6270. Secondary namenode status page shows transaction count in bytes.
|
||
(Benoy Antony via wheat9)
|
||
|
||
HDFS-6218. Audit log should use true client IP for proxied webhdfs
|
||
operations. (daryn via kihwal)
|
||
|
||
HDFS-6288. DFSInputStream Pread doesn't update ReadStatistics.
|
||
(Juan Yu via wang)
|
||
|
||
HDFS-5892. TestDeleteBlockPool fails in branch-2. (Ted Yu via wheat9)
|
||
|
||
HDFS-6289. HA failover can fail if there are pending DN messages for DNs
|
||
which no longer exist. (atm)
|
||
|
||
HDFS-6337. Setfacl testcase is failing due to dash character in username
|
||
in TestAclCLI (umamahesh)
|
||
|
||
HDFS-5381. ExtendedBlock#hashCode should use both blockId and block pool ID
|
||
(Benoy Antony via Colin Patrick McCabe)
|
||
|
||
HDFS-6240. WebImageViewer returns 404 if LISTSTATUS to an empty directory.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6351. Command hdfs dfs -rm -r can't remove empty directory.
|
||
(Yongjun Zhang via wang)
|
||
|
||
HDFS-5522. Datanode disk error check may be incorrectly skipped.
|
||
(Rushabh S Shah via kihwal)
|
||
|
||
HDFS-6367. EnumSetParam$Domain#parse fails for parameter containing more than one enum.
|
||
(Yi Liu via umamahesh)
|
||
|
||
HDFS-6305. WebHdfs response decoding may throw RuntimeExceptions (Daryn
|
||
Sharp via jeagles)
|
||
|
||
HDFS-6355. Fix divide-by-zero, improper use of wall-clock time in
|
||
BlockPoolSliceScanner (cmccabe)
|
||
|
||
HDFS-6370. Web UI fails to display in intranet under IE.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-6381. Fix a typo in INodeReference.java. (Binglin Chang via jing9)
|
||
|
||
HDFS-6400. Cannot execute hdfs oiv_legacy. (Akira AJISAKA via kihwal)
|
||
|
||
HDFS-6250. Fix test failed in TestBalancerWithNodeGroup.testBalancerWithRackLocality
|
||
(Binglin Chang and Chen He via junping_du)
|
||
|
||
HDFS-4913. Deleting file through fuse-dfs when using trash fails requiring
|
||
root permissions (cmccabe)
|
||
|
||
HDFS-6421. Fix vecsum.c compile on BSD and some other systems. (Mit Desai
|
||
via Colin Patrick McCabe)
|
||
|
||
HDFS-6419. TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk.
|
||
(Akira AJISAKA via kihwal)
|
||
|
||
HDFS-6409. Fix typo in log message about NameNode layout version upgrade.
|
||
(Chen He via cnauroth)
|
||
|
||
HDFS-6433. Replace BytesMoved class with AtomicLong.
|
||
(Benoy Antony via cnauroth)
|
||
|
||
HDFS-6438. DeleteSnapshot should be a DELETE request in WebHdfs. (jing9)
|
||
|
||
HDFS-6423. Diskspace quota usage should be updated when appending data to
|
||
partial block. (jing9)
|
||
|
||
HDFS-6443. Fix MiniQJMHACluster related test failures. (Zesheng Wu via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6227. ShortCircuitCache#unref should purge ShortCircuitReplicas whose
|
||
streams have been closed by java interrupts. (Colin Patrick McCabe via jing9)
|
||
|
||
HDFS-6442. Fix TestEditLogAutoroll and TestStandbyCheckpoints failure
|
||
caused by port conficts. (Zesheng Wu via Arpit Agarwal)
|
||
|
||
HDFS-6448. BlockReaderLocalLegacy should set socket timeout based on
|
||
conf.socketTimeout (liangxie via cmccabe)
|
||
|
||
HDFS-6453. Use Time#monotonicNow to avoid system clock reset.
|
||
(Liang Xie via wang)
|
||
|
||
HDFS-6461. Use Time#monotonicNow to compute duration in DataNode#shutDown.
|
||
(James Thomas via wang)
|
||
|
||
HDFS-6462. NFS: fsstat request fails with the secure hdfs (brandonli)
|
||
|
||
HDFS-6404. HttpFS should use a 000 umask for mkdir and create
|
||
operations. (yoderme via tucu)
|
||
|
||
HDFS-6424. blockReport doesn't need to invalidate blocks on SBN. (Ming Ma
|
||
via jing9)
|
||
|
||
HDFS-6497. Make TestAvailableSpaceVolumeChoosingPolicy deterministic
|
||
(cmccabe)
|
||
|
||
HDFS-6500. Snapshot shouldn't be removed silently after renaming to an
|
||
existing snapshot. (Nicholas SZE via junping_du)
|
||
|
||
HDFS-6257. TestCacheDirectives#testExceedsCapacity fails occasionally
|
||
(cmccabe)
|
||
|
||
HDFS-6364. Incorrect check for unknown datanode in Balancer. (Benoy
|
||
Antony via Arpit Agarwal)
|
||
|
||
HDFS-6503. Fix typo of DFSAdmin restoreFailedStorage.
|
||
(Zesheng Wu via wheat9)
|
||
|
||
HDFS-6464. Support multiple xattr.name parameters for WebHDFS getXAttrs.
|
||
(Yi Liu via umamahesh)
|
||
|
||
HDFS-6539. test_native_mini_dfs is skipped in hadoop-hdfs/pom.xml
|
||
(decstery via cmccabe)
|
||
|
||
HDFS-6527. Edit log corruption due to defered INode removal. (kihwal and
|
||
jing9 via jing9)
|
||
|
||
HDFS-6552. add DN storage to a BlockInfo will not replace the different
|
||
storage from same DN. (Amir Langer via Arpit Agarwal)
|
||
|
||
HDFS-6551. Rename with OVERWRITE option may throw NPE when the target
|
||
file/directory is a reference INode. (jing9)
|
||
|
||
HDFS-6439. NFS should not reject NFS requests to the NULL procedure whether
|
||
port monitoring is enabled or not. (brandonli)
|
||
|
||
HDFS-6559. Fix wrong option "dfsadmin -rollingUpgrade start" in the
|
||
document. (Akira Ajisaka via Arpit Agarwal)
|
||
|
||
HDFS-6553. Add missing DeprecationDeltas for NFS Kerberos configurations
|
||
(Stephen Chu via brandonli)
|
||
|
||
HDFS-6563. NameNode cannot save fsimage in certain circumstances when
|
||
snapshots are in use. (atm)
|
||
|
||
HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class
|
||
(Hooman Peiro Sajjad and Chen He via kihwal)
|
||
|
||
HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS
|
||
client. (atm)
|
||
|
||
HDFS-6535. HDFS quota update is wrong when file is appended. (George Wong
|
||
via jing9)
|
||
|
||
HDFS-6222. Remove background token renewer from webhdfs.
|
||
(Rushabh Shah and Daryn Sharp via cnauroth)
|
||
|
||
HDFS-6580. FSNamesystem.mkdirsInt should call the getAuditFileInfo()
|
||
wrapper. (Zhilei Xu via wheat9)
|
||
|
||
HDFS-6587. Bug in TestBPOfferService can cause test failure. (Zhilei Xu
|
||
via Arpit Agarwal)
|
||
|
||
HDFS-6598. Fix a typo in message issued from explorer.js. (Yongjun Zhang
|
||
via wheat9)
|
||
|
||
HDFS-6475. WebHdfs clients fail without retry because incorrect handling
|
||
of StandbyException. (Yongjun Zhang via atm)
|
||
|
||
HADOOP-10701. NFS should not validate the access premission only based on
|
||
the user's primary group (Harsh J via atm)
|
||
|
||
HDFS-6556. Refine XAttr permissions (umamahesh)
|
||
|
||
HDFS-6601. Issues in finalizing rolling upgrade when there is a layout
|
||
version change (kihwal)
|
||
|
||
HDFS-6418. Regression: DFS_NAMENODE_USER_NAME_KEY missing
|
||
(szetszwo via stevel)
|
||
|
||
HDFS-6558. Missing newline in the description of dfsadmin -rollingUpgrade.
|
||
(Chen He via kihwal)
|
||
|
||
HDFS-6591. while loop is executed tens of thousands of times in Hedged Read
|
||
(Liang Xie via cnauroth)
|
||
|
||
HDFS-6604. The short-circuit cache doesn't correctly time out replicas that
|
||
haven't been used in a while (cmccabe)
|
||
|
||
HDFS-4286. Changes from BOOKKEEPER-203 broken capability of including
|
||
bookkeeper-server jar in hidden package of BKJM (Rakesh R via umamahesh)
|
||
|
||
HDFS-4221. Remove the format limitation point from BKJM documentation as HDFS-3810
|
||
closed. (Rakesh R via umamahesh)
|
||
|
||
HDFS-5411. Update Bookkeeper dependency to 4.2.3. (Rakesh R via umamahesh)
|
||
|
||
HDFS-6631. TestPread#testHedgedReadLoopTooManyTimes fails intermittently.
|
||
(Liang Xie via cnauroth)
|
||
|
||
HDFS-6647. Edit log corruption when pipeline recovery occurs for deleted
|
||
file present in snapshot (kihwal)
|
||
|
||
HDFS-6378. NFS registration should timeout instead of hanging when
|
||
portmap/rpcbind is not available (Abhiraj Butala via brandonli)
|
||
|
||
HDFS-6632. Reintroduce dfs.http.port / dfs.https.port in branch-2.
|
||
(Yongjun Zhang via wheat9)
|
||
|
||
HDFS-6703. NFS: Files can be deleted from a read-only mount
|
||
(Srikanth Upputuri via brandonli)
|
||
|
||
HDFS-6422. getfattr in CLI doesn't throw exception or return non-0 return code
|
||
when xattr doesn't exist. (Charles Lamb via umamahesh)
|
||
|
||
HDFS-6696. Name node cannot start if the path of a file under
|
||
construction contains ".snapshot". (wang)
|
||
|
||
HDFS-6312. WebHdfs HA failover is broken on secure clusters.
|
||
(daryn via tucu)
|
||
|
||
HDFS-6618. FSNamesystem#delete drops the FSN lock between removing INodes
|
||
from the tree and deleting them from the inode map (kihwal via cmccabe)
|
||
|
||
HDFS-6622. Rename and AddBlock may race and produce invalid edits (kihwal
|
||
via cmccabe)
|
||
|
||
HDFS-6723. New NN webUI no longer displays decommissioned state for dead node.
|
||
(Ming Ma via wheat9)
|
||
|
||
HDFS-6768. Fix a few unit tests that use hard-coded port numbers. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-6793. Missing changes in HftpFileSystem when Reintroduce
|
||
dfs.http.port / dfs.https.port in branch-2. (Juan Yu via wang).
|
||
|
||
BREAKDOWN OF HDFS-2006 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-6299. Protobuf for XAttr and client-side implementation. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6302. Implement XAttr as a INode feature. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6309. Javadocs for Xattrs apis in DFSClient and other minor fixups. (Charles Lamb via umamahesh)
|
||
|
||
HDFS-6258. Namenode server-side storage for XAttrs. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6303. HDFS implementation of FileContext API for XAttrs. (Yi Liu and Charles Lamb via umamahesh)
|
||
|
||
HDFS-6324. Shift XAttr helper code out for reuse. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6301. NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log.
|
||
(Yi Liu via umamahesh)
|
||
|
||
HDFS-6298. XML based End-to-End test for getfattr and setfattr commands. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6314. Test cases for XAttrs. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6344. Maximum limit on the size of an xattr. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6377. Unify xattr name and value limits into a single limit. (wang)
|
||
|
||
HDFS-6373. Remove support for extended attributes on symlinks. (Charles Lamb via wang)
|
||
|
||
HDFS-6283. Write end user documentation for xattrs. (wang)
|
||
|
||
HDFS-6412. Interface audience and stability annotations missing from
|
||
several new classes related to xattrs. (wang)
|
||
|
||
HDFS-6259. Support extended attributes via WebHDFS. (yliu)
|
||
|
||
HDFS-6346. Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
|
||
(Yi Liu via umamahesh)
|
||
|
||
HDFS-6331. ClientProtocol#setXattr should not be annotated idempotent.
|
||
(umamahesh via wang)
|
||
|
||
HDFS-6335. TestOfflineEditsViewer for XAttr. (Yi Liu via umamahesh)
|
||
|
||
HDFS-6343. fix TestNamenodeRetryCache and TestRetryCacheWithHA failures. (umamahesh)
|
||
|
||
HDFS-6366. FsImage loading failed with RemoveXattr op (umamahesh)
|
||
|
||
HDFS-6357. SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA
|
||
(umamahesh)
|
||
|
||
HDFS-6372. Handle setXattr rpcIDs for OfflineEditsViewer. (umamahesh)
|
||
|
||
HDFS-6410. DFSClient unwraps AclException in xattr methods, but those
|
||
methods cannot throw AclException. (wang)
|
||
|
||
HDFS-6413. xattr names erroneously handled as case-insensitive.
|
||
(Charles Lamb via cnauroth)
|
||
|
||
HDFS-6414. xattr modification operations are based on state of latest
|
||
snapshot instead of current version of inode. (Andrew Wang via cnauroth)
|
||
|
||
HDFS-6374. setXAttr should require the user to be the owner of the file
|
||
or directory (Charles Lamb via wang)
|
||
|
||
HDFS-6492. Support create-time xattrs and atomically setting multiple
|
||
xattrs. (wang)
|
||
|
||
Release 2.4.1 - 2014-06-23
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-6215. Wrong error message for upgrade. (Kihwal Lee via jeagles)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-6189. Multiple HDFS tests fail on Windows attempting to use a test
|
||
root path containing a colon. (cnauroth via szetszwo)
|
||
|
||
HDFS-6197. Rolling upgrade rollback on Windows can fail attempting to rename
|
||
edit log segment files to a destination that already exists. (cnauroth)
|
||
|
||
HDFS-6198. DataNode rolling upgrade does not correctly identify current
|
||
block pool directory and replace with trash on Windows. (cnauroth)
|
||
|
||
HDFS-6206. Fix NullPointerException in DFSUtil.substituteForWildcardAddress.
|
||
(szetszwo)
|
||
|
||
HDFS-6204. Fix TestRBWBlockInvalidation: change the last sleep to a loop.
|
||
(szetszwo)
|
||
|
||
HDFS-6209. TestValidateConfigurationSettings should use random ports.
|
||
(Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-6208. DataNode caching can leak file descriptors. (cnauroth)
|
||
|
||
HDFS-6231. DFSClient hangs infinitely if using hedged reads and all eligible
|
||
datanodes die. (cnauroth)
|
||
|
||
HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to
|
||
invalid file path. (cnauroth)
|
||
|
||
HDFS-6235. TestFileJournalManager can fail on Windows due to file locking if
|
||
tests run out of order. (cnauroth)
|
||
|
||
HDFS-6229. Race condition in failover can cause RetryCache fail to work.
|
||
(jing9)
|
||
|
||
HDFS-6232. OfflineEditsViewer throws a NPE on edits containing ACL
|
||
modifications (ajisakaa via cmccabe)
|
||
|
||
HDFS-6236. ImageServlet should use Time#monotonicNow to measure latency.
|
||
(cnauroth)
|
||
|
||
HDFS-6245. datanode fails to start with a bad disk even when failed
|
||
volumes is set. (Arpit Agarwal)
|
||
|
||
HDFS-2882. DN continues to start up, even if block pool fails to initialize
|
||
(vinayakumarb)
|
||
|
||
HDFS-6340. DN can't finalize upgrade. (Rahul Singhal via Arpit Agarwal)
|
||
|
||
HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is
|
||
not configured. (kihwal)
|
||
|
||
HDFS-6313. WebHdfs may use the wrong NN when configured for multiple HA NNs
|
||
(kihwal)
|
||
|
||
HDFS-6326. WebHdfs ACL compatibility is broken. (cnauroth)
|
||
|
||
HDFS-6361. TestIdUserGroup.testUserUpdateSetting failed due to out of range
|
||
nfsnobody Id. (Yongjun Zhang via brandonli)
|
||
|
||
HDFS-4052. BlockManager#invalidateWork should print log outside the lock.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-6362. InvalidateBlocks is inconsistent in usage of DatanodeUuid and
|
||
StorageID. (Arpit Agarwal)
|
||
|
||
HDFS-6402. Suppress findbugs warning for failure to override equals and
|
||
hashCode in FsAclPermission. (cnauroth)
|
||
|
||
HDFS-6325. Append should fail if the last block has insufficient number of
|
||
replicas (Keith Pak via cos)
|
||
|
||
HDFS-6397. NN shows inconsistent value in deadnode count.
|
||
(Mohammad Kamrul Islam via kihwal)
|
||
|
||
HDFS-6411. nfs-hdfs-gateway mount raises I/O error and hangs when a
|
||
unauthorized user attempts to access it (brandonli)
|
||
|
||
Release 2.4.0 - 2014-04-07
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-5698. Use protobuf to serialize / deserialize FSImage. (See breakdown
|
||
of tasks below for features and contributors)
|
||
|
||
HDFS-5776 Support 'hedged' reads in DFSClient (Liang Xie via stack)
|
||
|
||
HDFS-4685. Implementation of ACLs in HDFS. (See breakdown of tasks below for
|
||
features and contributors)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-5781. Use an array to record the mapping between FSEditLogOpCode and
|
||
the corresponding byte value. (jing9)
|
||
|
||
HDFS-5153. Datanode should send block reports for each storage in a
|
||
separate message. (Arpit Agarwal)
|
||
|
||
HDFS-5804. HDFS NFS Gateway fails to mount and proxy when using Kerberos.
|
||
(Abin Shahab via jing9)
|
||
|
||
HDFS-5859. DataNode#checkBlockToken should check block tokens even if
|
||
security is not enabled. (cmccabe)
|
||
|
||
HDFS-5746. Add ShortCircuitSharedMemorySegment (cmccabe)
|
||
|
||
HDFS-4911. Reduce PeerCache timeout to be commensurate with
|
||
dfs.datanode.socket.reuse.keepalive (cmccabe)
|
||
|
||
HDFS-4370. Fix typo Blanacer in DataNode. (Chu Tong via shv)
|
||
|
||
HDFS-5929. Add blockpool % usage to HDFS federated nn page.
|
||
(Siqi Li via suresh)
|
||
|
||
HDFS-5810. Unify mmap cache and short-circuit file descriptor cache
|
||
(cmccabe)
|
||
|
||
HDFS-5940. Minor cleanups to ShortCircuitReplica, FsDatasetCache, and
|
||
DomainSocketWatcher (cmccabe)
|
||
|
||
HDFS-5531. Combine the getNsQuota() and getDsQuota() methods in INode.
|
||
(szetszwo)
|
||
|
||
HDFS-5285. Flatten INodeFile hierarchy: Replace INodeFileUnderConstruction
|
||
and INodeFileUnderConstructionWithSnapshot with FileUnderContructionFeature.
|
||
(jing9 via szetszwo)
|
||
|
||
HDFS-5286. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithQuota
|
||
with DirectoryWithQuotaFeature. (szetszwo)
|
||
|
||
HDFS-5537. Remove FileWithSnapshot interface. (jing9 via szetszwo)
|
||
|
||
HDFS-5554. Flatten INodeFile hierarchy: Replace INodeFileWithSnapshot with
|
||
FileWithSnapshotFeature. (jing9 via szetszwo)
|
||
|
||
HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-5632. Flatten INodeDirectory hierarchy: Replace
|
||
INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature.
|
||
(jing9 via szetszwo)
|
||
|
||
HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a
|
||
FileDiff/DirectoryDiff. (jing9)
|
||
|
||
HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. (jing9)
|
||
|
||
HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5775. Consolidate the code for serialization in CacheManager
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5973. add DomainSocket#shutdown method (cmccabe)
|
||
|
||
HDFS-5318. Support read-only and read-write paths to shared replicas.
|
||
(Eric Sirianni via Arpit Agarwal)
|
||
|
||
HDFS-5868. Make hsync implementation pluggable on the DataNode.
|
||
(Buddy Taylor via Arpit Agarwal)
|
||
|
||
HDFS-5935. New Namenode UI FS browser should throw smarter error messages.
|
||
(Travis Thompson via jing9)
|
||
|
||
HDFS-5939. WebHdfs returns misleading error code and logs nothing if trying
|
||
to create a file with no DNs in cluster. (Yongjun Zhang via jing9)
|
||
|
||
HDFS-6006. Remove duplicate code in FSNameSystem#getFileInfo.
|
||
(Akira Ajisaka via cnauroth)
|
||
|
||
HDFS-6018. Exception recorded in LOG when IPCLoggerChannel#close is called.
|
||
(jing9)
|
||
|
||
HDFS-3969. Small bug fixes and improvements for disk locations API.
|
||
(Todd Lipcon and Andrew Wang)
|
||
|
||
HDFS-6025. Update findbugsExcludeFile.xml. (szetszwo)
|
||
|
||
HDFS-6030. Remove an unused constructor in INode.java. (yzhang via
|
||
cmccabe)
|
||
|
||
HDFS-4200. Reduce the size of synchronized sections in PacketResponder.
|
||
(Suresh Srinivas, backported by Andrew Wang, committed by jing9)
|
||
|
||
HDFS-5950. The DFSClient and DataNode should use shared memory segments to
|
||
communicate short-circuit information. (cmccabe)
|
||
|
||
HDFS-6046. add dfs.client.mmap.enabled (cmccabe)
|
||
|
||
HDFS-5321. Clean up the HTTP-related configuration in HDFS (wheat9)
|
||
|
||
HDFS-5167. Add metrics about the NameNode retry cache. (Tsuyoshi OZAWA via
|
||
jing9)
|
||
|
||
HDFS-6043. Give HDFS daemons NFS3 and Portmap their own OPTS (brandonli)
|
||
|
||
HDFS-6044. Add property for setting the NFS look up time for users
|
||
(brandonli)
|
||
|
||
HDFS-6061. Allow dfs.datanode.shared.file.descriptor.path to contain
|
||
multiple entries and fall back when needed (cmccabe)
|
||
|
||
HDFS-5986. Capture the number of blocks pending deletion on namenode webUI.
|
||
(cnauroth)
|
||
|
||
HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream. (wang)
|
||
|
||
HDFS-6055. Change default configuration to limit file name length in HDFS.
|
||
(cnauroth)
|
||
|
||
HDFS-3405. Checkpointing should use HTTP POST or PUT instead of GET-GET
|
||
to send merged fsimages. (Vinayakumar B via wang)
|
||
|
||
HDFS-6085. Improve CacheReplicationMonitor log messages a bit (cmccabe)
|
||
|
||
HDFS-6072. Clean up dead code of FSImage. (wheat9)
|
||
|
||
HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax
|
||
configurable. (Abin Shahab via brandonli)
|
||
|
||
HDFS-6084. Namenode UI - "Hadoop" logo link shouldn't go to hadoop
|
||
homepage. (Travis Thompson via wheat9)
|
||
|
||
HDFS-6106. Reduce default for
|
||
dfs.namenode.path.based.cache.refresh.interval.ms (cmccabe)
|
||
|
||
HDFS-6090. Use MiniDFSCluster.Builder instead of deprecated constructors.
|
||
(Akira AJISAKA via jing9)
|
||
|
||
HDFS-6068. Disallow snapshot names that are also invalid directory names.
|
||
(sathish via szetszwo)
|
||
|
||
HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and
|
||
SocketTimeoutException. (szetszwo)
|
||
|
||
HDFS-6129. When a replica is not found for deletion, do not throw an
|
||
exception. (szetszwo)
|
||
|
||
HDFS-6138. Add a user guide for how to use viewfs with federation.
|
||
(sanjay and szetszwo via szetszwo)
|
||
|
||
HDFS-6120. Fix and improve safe mode log messages. (Arpit Agarwal)
|
||
|
||
HDFS-6050. NFS does not handle exceptions correctly in a few places
|
||
(brandonli)
|
||
|
||
HDFS-5138. Support HDFS upgrade in HA. (atm via todd)
|
||
|
||
HDFS-6124. Add final modifier to class members. (Suresh Srinivas via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5910. Enhance DataTransferProtocol to allow per-connection choice
|
||
of encryption/plain-text. (Benoy Antony via Arpit Agarwal)
|
||
|
||
HDFS-6150. Add inode id information in the logs to make debugging easier.
|
||
(suresh)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-5790. LeaseManager.findPath is very slow when many leases need recovery
|
||
(todd)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-5492. Port HDFS-2069 (Incorrect default trash interval in the
|
||
docs) to trunk. (Akira Ajisaka via Arpit Agarwal)
|
||
|
||
HDFS-5843. DFSClient.getFileChecksum() throws IOException if checksum is
|
||
disabled. (Laurent Goujon via jing9)
|
||
|
||
HDFS-5856. DataNode.checkDiskError might throw NPE.
|
||
(Josh Elser via suresh)
|
||
|
||
HDFS-5828. BlockPlacementPolicyWithNodeGroup can place multiple replicas on
|
||
the same node group when dfs.namenode.avoid.write.stale.datanode is true.
|
||
(Buddy via junping_du)
|
||
|
||
HDFS-5767. NFS implementation assumes userName userId mapping to be unique,
|
||
which is not true sometimes (Yongjun Zhang via brandonli)
|
||
|
||
HDFS-5791. TestHttpsFileSystem should use a random port to avoid binding
|
||
error during testing (Haohui Mai via brandonli)
|
||
|
||
HDFS-5709. Improve NameNode upgrade with existing reserved paths and path
|
||
components. (Andrew Wang via atm)
|
||
|
||
HDFS-5881. Fix skip() of the short-circuit local reader(legacy). (kihwal)
|
||
|
||
HDFS-5895. HDFS cacheadmin -listPools has exit_code of 1 when the command
|
||
returns 0 result. (Tassapol Athiapinya via cnauroth)
|
||
|
||
HDFS-5807. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails
|
||
intermittently. (Chen He via kihwal)
|
||
|
||
HDFS-5882. TestAuditLogs is flaky (jxiang via cmccabe)
|
||
|
||
HDFS-5900. Cannot set cache pool limit of "unlimited" via CacheAdmin.
|
||
(wang)
|
||
|
||
HDFS-5886. Potential null pointer deference in RpcProgramNfs3#readlink()
|
||
(brandonli)
|
||
|
||
HDFS-4858. HDFS DataNode to NameNode RPC should timeout.
|
||
(Henry Wang via shv)
|
||
|
||
HDFS-5879. Some TestHftpFileSystem tests do not close streams.
|
||
(Gera Shegalov via suresh)
|
||
|
||
HDFS-5938. Make BlockReaderFactory#BlockReaderPeer a static class to avoid
|
||
a findbugs warning. (cmccabe)
|
||
|
||
HDFS-5891. webhdfs should not try connecting the DN during redirection
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5904. TestFileStatus fails intermittently. (Mit Desai via kihwal)
|
||
|
||
HDFS-5941. add dfs.namenode.secondary.https-address and
|
||
dfs.namenode.secondary.https-address in hdfs-default.xml.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-5913. Nfs3Utils#getWccAttr() should check attr parameter against null
|
||
(brandonli)
|
||
|
||
HDFS-5934. New Namenode UI back button doesn't work as expected
|
||
(Travis Thompson via brandonli)
|
||
|
||
HDFS-5901. NameNode new UI doesn't support IE8 and IE9 on windows 7
|
||
(Vinayakumar B via brandonli)
|
||
|
||
HDFS-5943. 'dfs.namenode.https-address' property is not loaded from
|
||
configuration in federation setup. (suresh)
|
||
|
||
HDFS-3128. Unit tests should not use a test root in /tmp. (wang)
|
||
|
||
HDFS-5948. TestBackupNode flakes with port in use error. (Haohui Mai
|
||
via Arpit Agarwal)
|
||
|
||
HDFS-5949. New Namenode UI when trying to download a file, the browser
|
||
doesn't know the file name. (Haohui Mai via brandonli)
|
||
|
||
HDFS-5716. Allow WebHDFS to use pluggable authentication filter
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5953. TestBlockReaderFactory fails if libhadoop.so has not been built.
|
||
(Akira Ajisaka via wang)
|
||
|
||
HDFS-5759. Web UI does not show up during the period of loading FSImage.
|
||
(Haohui Mai via Arpit Agarwal)
|
||
|
||
HDFS-5942. Fix javadoc in OfflineImageViewer. (Akira Ajisaka via cnauroth)
|
||
|
||
HDFS-5780. TestRBWBlockInvalidation times out intemittently. (Mit Desai
|
||
via kihwal)
|
||
|
||
HDFS-5803. TestBalancer.testBalancer0 fails. (Chen He via kihwal)
|
||
|
||
HDFS-5893. HftpFileSystem.RangeHeaderUrlOpener uses the default
|
||
URLConnectionFactory which does not import SSL certificates. (Haohui Mai via
|
||
jing9)
|
||
|
||
HDFS-5961. OIV cannot load fsimages containing a symbolic link. (kihwal)
|
||
|
||
HDFS-5483. NN should gracefully handle multiple block replicas on same DN.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-5742. DatanodeCluster (mini cluster of DNs) fails to start.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-5979. Typo and logger fix for fsimage PB code. (wang)
|
||
|
||
HDFS-5962. Mtime and atime are not persisted for symbolic links. (Akira
|
||
Ajisaka via kihwal)
|
||
|
||
HDFS-5944. LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/
|
||
and cause SecondaryNameNode failed do checkpoint (Yunjiong Zhao via brandonli)
|
||
|
||
HDFS-5982. Need to update snapshot manager when applying editlog for deleting
|
||
a snapshottable directory. (jing9)
|
||
|
||
HDFS-5988. Bad fsimage always generated after upgrade. (wang)
|
||
|
||
HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal)
|
||
|
||
HDFS-6008. Namenode dead node link is giving HTTP error 500.
|
||
(Benoy Antony via cnauroth)
|
||
|
||
HDFS-5936. MiniDFSCluster does not clean data left behind by
|
||
SecondaryNameNode. (Binglin Chang via cnauroth)
|
||
|
||
HDFS-5339. WebHDFS URI does not accept logical nameservices when security is
|
||
enabled. (Haohui Mai via jing9)
|
||
|
||
HDFS-6033. PBImageXmlWriter incorrectly handles processing cache
|
||
directives. (atm)
|
||
|
||
HDFS-5821. TestHDFSCLI fails for user names with the dash character.
|
||
(Gera Shegalov via Arpit Agarwal)
|
||
|
||
HDFS-5956. A file size is multiplied by the replication factor in 'hdfs oiv
|
||
-p FileDistribution' option. (Akira Ajisaka via wheat9)
|
||
|
||
HDFS-5866. '-maxSize' and '-step' option fail in OfflineImageViewer.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6040. fix DFSClient issue without libhadoop.so and some other
|
||
ShortCircuitShm cleanups (cmccabe)
|
||
|
||
HDFS-6053. Fix TestDecommissioningStatus and TestDecommission in branch-2.
|
||
(jing9)
|
||
|
||
HDFS-6047 TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange
|
||
(stack)
|
||
|
||
HDFS-6051. HDFS cannot run on Windows since short-circuit shared memory
|
||
segment changes. (cmccabe)
|
||
|
||
HDFS-5857. TestWebHDFS#testNamenodeRestart fails intermittently with NPE.
|
||
(Mit Desai via wheat9)
|
||
|
||
HDFS-5898. Allow NFS gateway to login/relogin from its kerberos keytab.
|
||
(Abin Shahab via atm)
|
||
|
||
HDFS-6057. DomainSocketWatcher.watcherThread should be marked as daemon
|
||
thread (cmccabe)
|
||
|
||
HDFS-6058. Fix TestHDFSCLI failures after HADOOP-8691 change.
|
||
(Akira Ajisaka via wheat9)
|
||
|
||
HDFS-6062. TestRetryCacheWithHA#testConcat is flaky. (Jing Zhao via wheat9)
|
||
|
||
HDFS-6059. TestBlockReaderLocal fails if native library is not available.
|
||
(Akira AJISAKA via Colin Patrick McCabe)
|
||
|
||
HDFS-6084. DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is
|
||
not updated with latest block report interval of 6 hrs.
|
||
(Vinayakumar B via wheat9)
|
||
|
||
HDFS-6067. TestPread.testMaxOutHedgedReadPool is flaky (cmccabe)
|
||
|
||
HDFS-6065. HDFS zero-copy reads should return null on EOF when doing ZCR
|
||
(cmccabe)
|
||
|
||
HDFS-5064. Standby checkpoints should not block concurrent readers.
|
||
(atm via wang)
|
||
|
||
HDFS-6078. TestIncrementalBlockReports is flaky. (Arpit Agarwal)
|
||
|
||
HDFS-6071. BlockReaderLocal doesn't return -1 on EOF when doing a
|
||
zero-length read on a short file (cmccabe)
|
||
|
||
HDFS-6077. Running slive with webhdfs on secure HA cluster fails with unkown
|
||
host exception. (jing9)
|
||
|
||
HDFS-6086. Fix a case where zero-copy or no-checksum reads were not allowed
|
||
even when the block was cached (cmccabe)
|
||
|
||
HDFS-6079. Timeout for getFileBlockStorageLocations does not work. (wang)
|
||
|
||
HDFS-5705. TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due
|
||
to ConcurrentModificationException. (Ted Yu via brandonli)
|
||
|
||
HDFS-6096. TestWebHdfsTokens may timeout. (szetszwo via Arpit Agarwal)
|
||
|
||
HDFS-5244. TestNNStorageRetentionManager#testPurgeMultipleDirs fails.
|
||
(Jinghui Wang via suresh)
|
||
|
||
HDFS-6097. zero-copy reads are incorrectly disabled on file offsets above
|
||
2GB (cmccabe)
|
||
|
||
HDFS-6102. Lower the default maximum items per directory to fix PB fsimage
|
||
loading. (wang)
|
||
|
||
HDFS-6094. The same block can be counted twice towards safe mode
|
||
threshold. (Arpit Agarwal)
|
||
|
||
HDFS-5516. WebHDFS does not require user name when anonymous http requests
|
||
are disallowed. (Miodrag Radulovic via cnauroth)
|
||
|
||
HDFS-6117. Print file path information in FileNotFoundException on INode
|
||
ID mismatch. (suresh)
|
||
|
||
HDFS-6099. HDFS file system limits not enforced on renames. (cnauroth)
|
||
|
||
HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-6105. NN web UI for DN list loads the same jmx page multiple times.
|
||
(wheat9)
|
||
|
||
HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. (wheat9)
|
||
|
||
HDFS-6131. Move HDFSHighAvailabilityWithNFS.apt.vm and
|
||
HDFSHighAvailabilityWithQJM.apt.vm from Yarn to HDFS. (jing9)
|
||
|
||
HDFS-6140. WebHDFS cannot create a file with spaces in the name after HA
|
||
failover changes. (cnauroth)
|
||
|
||
HDFS-6135. In HDFS upgrade with HA setup, JournalNode cannot handle layout
|
||
version bump when rolling back. (jing9)
|
||
|
||
HDFS-5846. Assigning DEFAULT_RACK in resolveNetworkLocation method can break
|
||
data resiliency. (Nikola Vujic via cnauroth)
|
||
|
||
HDFS-5840. Follow-up to HDFS-5138 to improve error handling during partial
|
||
upgrade failures. (atm, jing9 and suresh via jing9)
|
||
|
||
HDFS-6130. NPE when upgrading namenode from fsimages older than -32.
|
||
(wheat9)
|
||
|
||
HDFS-6115. Call flush() for every append on block scan verification log.
|
||
(Vinayakumar B via szetszwo)
|
||
|
||
HDFS-5672. TestHASafeMode#testSafeBlockTracking fails in trunk. (jing9)
|
||
|
||
HDFS-6157. Fix the entry point of OfflineImageViewer for hdfs.cmd. (wheat9)
|
||
|
||
HDFS-6163. Fix a minor bug in the HA upgrade document. (Fengdong Yu via
|
||
jing9)
|
||
|
||
HDFS-6166. Change Balancer socket read timeout to 20 minutes and add
|
||
10 seconds delay after error. (Nathan Roberts via szetszwo)
|
||
|
||
BREAKDOWN OF HDFS-5698 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-5717. Save FSImage header in protobuf. (Haohui Mai via jing9)
|
||
|
||
HDFS-5738. Serialize INode information in protobuf. (Haohui Mai via jing9)
|
||
|
||
HDFS-5772. Serialize under-construction file information in FSImage. (jing9)
|
||
|
||
HDFS-5783. Compute the digest before loading FSImage. (Haohui Mai via jing9)
|
||
|
||
HDFS-5785. Serialize symlink in protobuf. (Haohui Mai via jing9)
|
||
|
||
HDFS-5793. Optimize the serialization of PermissionStatus. (Haohui Mai via
|
||
jing9)
|
||
|
||
HDFS-5743. Use protobuf to serialize snapshot information. (jing9)
|
||
|
||
HDFS-5774. Serialize CachePool directives in protobuf. (Haohui Mai via jing9)
|
||
|
||
HDFS-5744. Serialize information for token managers in protobuf. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-5824. Add a Type field in Snapshot DiffEntry's protobuf definition.
|
||
(jing9)
|
||
|
||
HDFS-5808. Implement cancellation when saving FSImage. (Haohui Mai via jing9)
|
||
|
||
HDFS-5826. Update the stored edit logs to be consistent with the changes in
|
||
HDFS-5698 branch. (Haohui Mai via jing9)
|
||
|
||
HDFS-5797. Implement offline image viewer. (Haohui Mai via jing9)
|
||
|
||
HDFS-5771. Track progress when loading fsimage. (Haohui Mai via cnauroth)
|
||
|
||
HDFS-5871. Use PBHelper to serialize CacheDirectiveInfoExpirationProto.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5884. LoadDelegator should use IOUtils.readFully() to read the magic
|
||
header. (Haohui Mai via jing9)
|
||
|
||
HDFS-5885. Add annotation for repeated fields in the protobuf definition.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5906. Fixing findbugs and javadoc warnings in the HDFS-5698 branch.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5911. The id of a CacheDirective instance does not get serialized in
|
||
the protobuf-fsimage. (Haohui Mai via jing9)
|
||
|
||
HDFS-5915. Refactor FSImageFormatProtobuf to simplify cross section reads.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-5847. Consolidate INodeReference into a separate section. (jing9)
|
||
|
||
HDFS-5959. Fix typo at section name in FSImageFormatProtobuf.java.
|
||
(Akira Ajisaka via suresh)
|
||
|
||
HDFS-5981. PBImageXmlWriter generates malformed XML.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-6107. When a block can't be cached due to limited space on the
|
||
DataNode, that block becomes uncacheable (cmccabe)
|
||
|
||
HDFS-6089. Standby NN while transitioning to active throws a connection
|
||
refused error when the prior active NN process is suspended.
|
||
(Jing Zhao via Andrew Wang)
|
||
|
||
BREAKDOWN OF HDFS-4685 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-5596. Implement RPC stubs. (Haohui Mai via cnauroth)
|
||
|
||
HDFS-5685. Implement ACL as a INode feature. (Haohui Mai via cnauroth)
|
||
|
||
HDFS-5618. NameNode: persist ACLs in fsimage. (Haohui Mai via cnauroth)
|
||
|
||
HDFS-5619. NameNode: record ACL modifications to edit log.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-5673. Implement logic for modification of ACLs. (cnauroth)
|
||
|
||
HDFS-5758. NameNode: complete implementation of inode modifications for
|
||
ACLs. (Chris Nauroth via wheat9)
|
||
|
||
HDFS-5612. NameNode: change all permission checks to enforce ACLs in
|
||
addition to permissions. (Chris Nauroth via wheat9)
|
||
|
||
HDFS-5613. NameNode: implement handling of ACLs in combination with
|
||
symlinks. (Chris Nauroth via wheat9)
|
||
|
||
HDFS-5615. NameNode: implement handling of ACLs in combination with sticky
|
||
bit. (Chris Nauroth via wheat9)
|
||
|
||
HDFS-5702. FsShell Cli: Add XML based End-to-End test for getfacl and
|
||
setfacl commands. (Vinay via cnauroth)
|
||
|
||
HDFS-5608. WebHDFS: implement ACL APIs.
|
||
(Sachin Jose and Renil Joseph via cnauroth)
|
||
|
||
HDFS-5614. NameNode: implement handling of ACLs in combination with
|
||
snapshots. (cnauroth)
|
||
|
||
HDFS-5858. Refactor common ACL test cases to be run through multiple
|
||
FileSystem implementations. (cnauroth)
|
||
|
||
HDFS-5860. Refactor INodeDirectory getDirectoryXFeature methods to use
|
||
common getFeature helper method. (Jing Zhao via cnauroth)
|
||
|
||
HDFS-5861. Add CLI test for Ls output for extended ACL marker.
|
||
(Vinay via cnauroth)
|
||
|
||
HDFS-5616. NameNode: implement default ACL handling. (cnauroth)
|
||
|
||
HDFS-5899. Add configuration flag to disable/enable support for ACLs.
|
||
(cnauroth)
|
||
|
||
HDFS-5914. Incorporate ACLs with the changes from HDFS-5698.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-5625. Write end user documentation for HDFS ACLs. (cnauroth)
|
||
|
||
HDFS-5925. ACL configuration flag must only reject ACL API calls, not ACLs
|
||
present in fsimage or edits. (cnauroth)
|
||
|
||
HDFS-5923. Do not persist the ACL bit in the FsPermission.
|
||
(Haohui Mai via cnauroth)
|
||
|
||
HDFS-5933. Optimize the FSImage layout for ACLs (Haohui Mai via cnauroth)
|
||
|
||
HDFS-5932. Ls should display the ACL bit (Chris Nauroth via wheat9)
|
||
|
||
HDFS-5937. Fix TestOfflineEditsViewer on HDFS-4685 branch. (cnauroth)
|
||
|
||
HDFS-5737. Replacing only the default ACL can fail to copy unspecified base
|
||
entries from the access ACL. (cnauroth)
|
||
|
||
HDFS-5739. ACL RPC must allow null name or unspecified permissions in ACL
|
||
entries. (cnauroth)
|
||
|
||
HDFS-5799. Make audit logging consistent across ACL APIs. (cnauroth)
|
||
|
||
HDFS-5849. Removing ACL from an inode fails if it has only a default ACL.
|
||
(cnauroth)
|
||
|
||
HDFS-5623. NameNode: add tests for skipping ACL enforcement when permission
|
||
checks are disabled, user is superuser or user is member of supergroup.
|
||
(cnauroth)
|
||
|
||
HDFS-5908. Change AclFeature to capture list of ACL entries in an
|
||
ImmutableList. (cnauroth)
|
||
|
||
HDFS-6028. Print clearer error message when user attempts to delete required
|
||
mask entry from ACL. (cnauroth)
|
||
|
||
HDFS-6039. Uploading a File under a Dir with default acls throws "Duplicated
|
||
ACLFeature". (cnauroth)
|
||
|
||
HDFS-6063. TestAclCLI fails intermittently when running test 24:
|
||
copyFromLocal. (cnauroth)
|
||
|
||
HDFS-6069. Quash stack traces when ACLs are disabled. (cnauroth)
|
||
|
||
HDFS-5638. HDFS implementation of FileContext API for ACLs.
|
||
(Vinayakumar B via cnauroth)
|
||
|
||
HDFS-5535 subtasks:
|
||
|
||
HDFS-5496. Make replication queue initialization asynchronous. (Vinay via
|
||
jing9)
|
||
|
||
HDFS-5645. Support upgrade marker in editlog streams. (szetszwo)
|
||
|
||
HDFS-5752. Add a new DFSAdmin command to query, start and finalize rolling
|
||
upgrade. (szetszwo)
|
||
|
||
HDFS-5786. Support QUERY and FINALIZE actions of rolling upgrade. (szetszwo)
|
||
|
||
HDFS-5753. Add new Namenode startup options for downgrade and rollback using
|
||
upgrade marker. (szetszwo)
|
||
|
||
HDFS-5835. Add a new option for starting Namenode when rolling upgrade is
|
||
in progress. (szetszwo)
|
||
|
||
HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and
|
||
DataNodeLayoutVersion. (Brandon Li via szetszwo)
|
||
|
||
HDFS-5848. Add rolling upgrade status to heartbeat response. (szetszwo)
|
||
|
||
HDFS-5890. Avoid NPE in Datanode heartbeat. (Vinay via brandonli)
|
||
|
||
HDFS-5869. When starting rolling upgrade or NN restarts, NN should create
|
||
a checkpoint right before the upgrade marker. (szetszwo)
|
||
|
||
HDFS-5874. Should not compare DataNode current layout version with that of
|
||
NameNode in DataStrorage. (brandonli)
|
||
|
||
HDFS-5889. When starting rolling upgrade, create a fs image for rollback
|
||
so that the standby namenode can create checkpoints during upgrade.
|
||
(szetszwo & jing9)
|
||
|
||
HDFS-5907. Add BlockPoolSliceStorage 'trash' to handle block deletions
|
||
during rolling upgrades. (Arpit Agarwal)
|
||
|
||
HDFS-5494. Merge Protobuf-based-FSImage code from trunk - fix build
|
||
break after the merge. (Jing Zhao via Arpit Agarwal)
|
||
|
||
HDFS-5585. Provide admin commands for data node upgrade (kihwal)
|
||
|
||
HDFS-5920. Support rollback of rolling upgrade in NameNode and JournalNodes.
|
||
(jing9)
|
||
|
||
HDFS-5945. Add rolling upgrade information to fsimage; and disallow upgrade
|
||
and rolling upgrade to be started simultaneously. (szetszwo & jing9)
|
||
|
||
HDFS-5966. Fix rollback of rolling upgrade in NameNode HA setup. (jing9
|
||
via szetszwo)
|
||
|
||
HDFS-5974. Fix compilation error, NameNodeLayoutVersion and
|
||
DataNodeLayoutVersion after merge from trunk. (szetszwo)
|
||
|
||
HDFS-5963. TestRollingUpgrade#testSecondaryNameNode causes subsequent
|
||
tests to fail. (szetszwo via Arpit Agarwal)
|
||
|
||
HDFS-5976. Create unit tests for downgrade and finalize rolling upgrade.
|
||
(Haohui Mai via Arpit Agarwal)
|
||
|
||
HDFS-5980. Rollback does not need to load edits. (jing9 via szetszwo)
|
||
|
||
HDFS-5984. Fix TestEditLog and TestStandbyCheckpoints. (jing9 via szetszwo)
|
||
|
||
HDFS-5985. SimulatedFSDataset#disableAndPurgeTrashStorage should not throw
|
||
UnsupportedOperationException. (jing9 via kihwal)
|
||
|
||
HDFS-5987. Fix findbugs warnings in Rolling Upgrade branch. (seztszwo via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5992. Fix NPE in MD5FileUtils and update editsStored for
|
||
TestOfflineEditsViewer. (szetszwo)
|
||
|
||
HDFS-5994. Fix TestDataNodeRollingUpgrade. (Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-5999. Do not create rollback fsimage when it already exists. (jing9)
|
||
|
||
HDFS-6005. Simplify Datanode rollback and downgrade. (Suresh Srinivas via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6004. Change DFSAdmin for rolling upgrade commands. (szetszwo via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5583. Make DN send an OOB Ack on shutdown before restarting. (kihwal)
|
||
|
||
HDFS-5778. Add rolling upgrade user document. (szetszwo)
|
||
|
||
HDFS-6003. Add the new -rollingUpgrade startup option to the namenode
|
||
usage message. (Vinayakumar B via szetszwo)
|
||
|
||
HDFS-6014. Fix findbug warnings introduced by HDFS-5583. (kihwal)
|
||
|
||
HDFS-6015. Fix TestBlockRecovery
|
||
#testRaceBetweenReplicaRecoveryAndFinalizeBlock. (kihwal)
|
||
|
||
HDFS-5924. Utilize OOB upgrade message processing for writes. (kihwal)
|
||
|
||
HDFS-5498. Improve datanode startup time. (kihwal)
|
||
|
||
HDFS-6000. Avoid saving namespace when starting rolling upgrade. (jing9)
|
||
|
||
HDFS-6017. Query the status of rolling upgrade in the preparation stage in
|
||
TestRollingUpgrade and TestRollingUpgradeRollback. (Haohui Mai via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-6020. Fix the five findbugs warnings. (kihwal)
|
||
|
||
HDFS-6019. Standby NN might not checkpoint when processing the rolling
|
||
upgrade marker. (Haohui Mai via jing9)
|
||
|
||
HDFS-6023. Test whether the standby NN continues to checkpoint after the
|
||
prepare stage. (Haohui Mai via jing9)
|
||
|
||
HDFS-6024. Test whether the NN will reject the downgrade if it has a
|
||
fsimage from a newer release. (Haohui Mai via jing9)
|
||
|
||
HDFS-6026. Fix TestDFSUpgrade and TestDataNodeRollingUpgrade.
|
||
(jing9 via szetszwo)
|
||
|
||
HDFS-6029. Secondary NN fails to checkpoint after -rollingUpgrade prepare.
|
||
(jing9)
|
||
|
||
HDFS-6032. -rollingUpgrade query hits NPE after the NN restarts. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-6031. Add back the "-rollingUpgrade started" namenode startup option;
|
||
otherwise, namenode cannot start when the layout version is changed.
|
||
(szetszwo)
|
||
|
||
HDFS-6034. Use DataNodeLayoutVersion for DN registration check and do not
|
||
verify layout version if there is a rolling upgrade in progress. (szetszwo)
|
||
|
||
HDFS-6013. add rollingUpgrade information to latest UI.
|
||
(Vinayakumar B via wheat9)
|
||
|
||
HDFS-6042. Fix rolling upgrade documentation and error messages. (szetszwo
|
||
via Arpit Agarwal)
|
||
|
||
HDFS-6041. Downgrade/Finalize should rename the rollback image instead of
|
||
purging it. (jing9)
|
||
|
||
HDFS-6060. NameNode should not check DataNode layout version (brandonli)
|
||
|
||
HDFS-6076. DataNode with SimulatedDataSet should not create
|
||
DatanodeRegistration with namenode layout version and namenode node type.
|
||
(szetszwo)
|
||
|
||
HDFS-6038. Allow JournalNode to handle editlog produced by new release with
|
||
future layoutversion. (jing9)
|
||
|
||
HDFS-4564. Ensure webhdfs returns correct HTTP response codes for denied
|
||
operations. (daryn via acmurthy)
|
||
|
||
Release 2.3.1 - UNRELEASED
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
Release 2.3.0 - 2014-02-18
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-4953. Enable HDFS local reads via mmap.
|
||
(Colin Patrick McCabe via wang).
|
||
|
||
HDFS-5342. Provide more information in the FSNamesystem JMX interfaces.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5334. Implement dfshealth.jsp in HTML pages. (Haohui Mai via jing9)
|
||
|
||
HDFS-5379. Update links to datanode information in dfshealth.html. (Haohui
|
||
Mai via jing9)
|
||
|
||
HDFS-5382. Implement the UI of browsing filesystems in HTML 5 page. (Haohui
|
||
Mai via jing9)
|
||
|
||
HDFS-3987. Support webhdfs over HTTPS. (Haohui Mai via jing9)
|
||
|
||
HDFS-5444. Choose default web UI based on browser capabilities. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn)
|
||
|
||
HDFS-2832. Heterogeneous Storages support in HDFS phase 1 - treat DataNode
|
||
as a collection of storages (see breakdown of tasks below for features and
|
||
contributors).
|
||
|
||
HDFS-5703. Add support for HTTPS and swebhdfs to HttpFS. (tucu)
|
||
|
||
HDFS-4949. Centralized cache management in HDFS (wang and cmccabe)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-5360. Improvement of usage message of renameSnapshot and
|
||
deleteSnapshot. (Shinichi Yamashita via wang)
|
||
|
||
HDFS-5331. make SnapshotDiff.java to a o.a.h.util.Tool interface implementation.
|
||
(Vinayakumar B via umamahesh)
|
||
|
||
HDFS-4657. Limit the number of blocks logged by the NN after a block
|
||
report to a configurable value. (Aaron T. Myers via Colin Patrick
|
||
McCabe)
|
||
|
||
HDFS-5344. Make LsSnapshottableDir as Tool interface implementation. (Sathish via umamahesh)
|
||
|
||
HDFS-5544. Adding Test case For Checking dfs.checksum type as NULL value. (Sathish via umamahesh)
|
||
|
||
HDFS-5568. Support includeSnapshots option with Fsck command. (Vinayakumar B via umamahesh)
|
||
|
||
HDFS-4983. Numeric usernames do not work with WebHDFS FS. (Yongjun Zhang via
|
||
jing9)
|
||
|
||
HDFS-5592. statechangeLog of completeFile should be logged only in case of success.
|
||
(Vinayakumar via umamahesh)
|
||
|
||
HDFS-5662. Can't decommission a DataNode due to file's replication factor
|
||
larger than the rest of the cluster size. (brandonli)
|
||
|
||
HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options.
|
||
(shv)
|
||
|
||
HDFS-5675. Add Mkdirs operation to NNThroughputBenchmark.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-5677. Need error checking for HA cluster configuration.
|
||
(Vincent Sheffer via cos)
|
||
|
||
HDFS-5825. Use FileUtils.copyFile() to implement DFSTestUtils.copyFile().
|
||
(Haohui Mai via Arpit Agarwal)
|
||
|
||
HDFS-5267. Remove volatile from LightWeightHashSet. (Junping Du via llu)
|
||
|
||
HDFS-4278. Log an ERROR when DFS_BLOCK_ACCESS_TOKEN_ENABLE config is
|
||
disabled but security is turned on. (Kousuke Saruta via harsh)
|
||
|
||
HDFS-5004. Add additional JMX bean for NameNode status data. Contributed
|
||
by Trevor Lorimer.
|
||
|
||
HDFS-4994. Audit log getContentSummary() calls. (Robert Parker via kihwal)
|
||
|
||
HDFS-5144. Document time unit to NameNodeMetrics. (Akira Ajisaka via
|
||
suresh)
|
||
|
||
HDFS-4491. Parallel testing HDFS. (Andrey Klochkov via cnauroth)
|
||
|
||
HDFS-4879. Add "blocked ArrayList" collection to avoid CMS full GCs
|
||
(Todd Lipcon via Colin Patrick McCabe)
|
||
|
||
HDFS-4096. Add snapshot information to namenode WebUI. (Haohui Mai via
|
||
jing9)
|
||
|
||
HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..)
|
||
methods; replace HashMap with Map in parameter declarations and cleanup
|
||
some related code. (szetszwo)
|
||
|
||
HDFS-5207. In BlockPlacementPolicy.chooseTarget(..), change the writer
|
||
and the excludedNodes parameter types respectively to Node and Set.
|
||
(Junping Du via szetszwo)
|
||
|
||
HDFS-5240. Separate formatting from logging in the audit logger API (daryn)
|
||
|
||
HDFS-5191. Revisit zero-copy API in FSDataInputStream to make it more
|
||
intuitive. (Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5260. Merge zero-copy memory-mapped HDFS client reads to trunk and
|
||
branch-2. (cnauroth)
|
||
|
||
HDFS-4517. Cover class RemoteBlockReader with unit tests. (Vadim Bondarev
|
||
and Dennis Y via kihwal)
|
||
|
||
HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with tests.
|
||
(Vadim Bondarev via kihwal)
|
||
|
||
HDFS-4510. Cover classes ClusterJspHelper/NamenodeJspHelper with unit
|
||
tests. (Andrey Klochkov via kihwal)
|
||
|
||
HDFS-5323. Remove some deadcode in BlockManager. (Colin Patrick McCabe)
|
||
|
||
HDFS-5338. Add a conf to disable hostname check in datanode registration.
|
||
(szetszwo)
|
||
|
||
HDFS-5130. Add test for snapshot related FsShell and DFSAdmin commands.
|
||
(Binglin Chang via jing9)
|
||
|
||
HDFS-5374. Remove deadcode in DFSOutputStream. (suresh)
|
||
|
||
HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test
|
||
(Andrey Klochkov via jeagles)
|
||
|
||
HDFS-4885. Improve the verifyBlockPlacement() API in BlockPlacementPolicy.
|
||
(Junping Du via szetszwo)
|
||
|
||
HDFS-5363. Refactor WebHdfsFileSystem: move SPENGO-authenticated connection
|
||
creation to URLConnectionFactory. (Haohui Mai via jing9)
|
||
|
||
HDFS-5436. Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web
|
||
(Haohui Mai via Arpit Agarwal)
|
||
|
||
HDFS-5371. Let client retry the same NN when
|
||
"dfs.client.test.drop.namenode.response.number" is enabled. (jing9)
|
||
|
||
HDFS-5467. Remove tab characters in hdfs-default.xml.
|
||
(Shinichi Yamashita via Andrew Wang)
|
||
|
||
HDFS-5495. Remove further JUnit3 usages from HDFS.
|
||
(Jarek Jarcec Cecho via Andrew Wang)
|
||
|
||
HDFS-5325. Remove WebHdfsFileSystem#ConnRunner. (Haohui Mai via jing9)
|
||
|
||
HDFS-5488. Clean up TestHftpURLTimeout. (Haohui Mai via jing9)
|
||
|
||
HDFS-5440. Extract the logic of handling delegation tokens in HftpFileSystem
|
||
to the TokenAspect class. (Haohui Mai via jing9)
|
||
|
||
HDFS-5487. Introduce unit test for TokenAspect. (Haohui Mai via jing9)
|
||
|
||
HDFS-4995. Make getContentSummary less expensive. (kihwal)
|
||
|
||
HDFS-5506. Use URLConnectionFactory in DelegationTokenFetcher. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-5489. Use TokenAspect in WebHDFSFileSystem. (Haohui Mai via jing9)
|
||
|
||
HDFS-5393. Serve bootstrap and jQuery locally. (Haohui Mai via jing9)
|
||
|
||
HDFS-5073. TestListCorruptFileBlocks fails intermittently. (Arpit Agarwal)
|
||
|
||
HDFS-1386. TestJMXGet fails in jdk7 (jeagles)
|
||
|
||
HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. (Vinay
|
||
via jing9)
|
||
|
||
HDFS-5525. Inline dust templates for new Web UI. (Haohui Mai via jing9)
|
||
|
||
HDFS-5561. FSNameSystem#getNameJournalStatus() in JMX should return plain
|
||
text instead of HTML. (Haohui Mai via jing9)
|
||
|
||
HDFS-5581. NameNodeFsck should use only one instance of
|
||
BlockPlacementPolicy. (vinay via cmccabe)
|
||
|
||
HDFS-5633. Improve OfflineImageViewer to use less memory. (jing9)
|
||
|
||
HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7
|
||
(Mit Desai via jeagles)
|
||
|
||
HDFS-5637. Try to refeatchToken while local read InvalidToken occurred.
|
||
(Liang Xie via junping_du)
|
||
|
||
HDFS-5652. Refactor invalid block token exception handling in DFSInputStream.
|
||
(Liang Xie via junping_du)
|
||
|
||
HDFS-5350. Name Node should report fsimage transfer time as a metric.
|
||
(Jimmy Xiang via wang)
|
||
|
||
HDFS-5538. URLConnectionFactory should pick up the SSL related configuration
|
||
by default. (Haohui Mai via jing9)
|
||
|
||
HDFS-5545. Allow specifying endpoints for listeners in HttpServer. (Haohui
|
||
Mai via jing9)
|
||
|
||
HDFS-5536. Implement HTTP policy for Namenode and DataNode. (Haohui Mai via
|
||
jing9)
|
||
|
||
HDFS-5312. Generate HTTP / HTTPS URL in DFSUtil#getInfoServer() based on the
|
||
configured http policy. (Haohui Mai via jing9)
|
||
|
||
HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5674. Editlog code cleanup: remove @SuppressWarnings("deprecation") in
|
||
FSEditLogOp; change FSEditLogOpCodes.fromByte(..) to be more efficient; and
|
||
change Some fields in FSEditLog to final. (szetszwo)
|
||
|
||
HDFS-5634. Allow BlockReaderLocal to switch between checksumming and not
|
||
(cmccabe)
|
||
|
||
HDFS-5663 make the retry time and interval value configurable in openInfo()
|
||
(Liang Xie via stack)
|
||
|
||
HDFS-5540. Fix intermittent failure in TestBlocksWithNotEnoughRacks.
|
||
(Binglin Chang via junping_du)
|
||
|
||
HDFS-5695. Clean up TestOfflineEditsViewer and OfflineEditsViewerHelper.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5220. Expose group resolution time as metric (jxiang via cmccabe)
|
||
|
||
HDFS-5762. BlockReaderLocal doesn't return -1 on EOF when doing zero-length
|
||
reads (Colin Patrick McCabe)
|
||
|
||
HDFS-5766. In DFSInputStream, do not add datanode to deadNodes after
|
||
InvalidEncryptionKeyException in fetchBlockByteRange (Liang Xie via Colin
|
||
Patrick McCabe)
|
||
|
||
HDFS-5704. Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK. (jing9)
|
||
|
||
HDFS-5784. Reserve space in edit log header and fsimage header for feature
|
||
flag section. (Colin Patrick McCabe)
|
||
|
||
HDFS-5434. Change block placement policy constructors from package private
|
||
to protected. (Buddy Taylor via Arpit Agarwal)
|
||
|
||
HDFS-5788. listLocatedStatus response can be very large. (Nathan Roberts
|
||
via kihwal)
|
||
|
||
HDFS-5841. Update HDFS caching documentation with new changes. (wang)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-5239. Allow FSNamesystem lock fairness to be configurable (daryn)
|
||
|
||
HDFS-5341. Reduce fsdataset lock duration during directory scanning.
|
||
(Qus-Jiawei via kihwal)
|
||
|
||
HDFS-5681. renewLease should not hold fsn write lock. (daryn via Kihwal)
|
||
|
||
HDFS-5241. Provide alternate queuing audit logger to reduce logging
|
||
contention (daryn)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-5307. Support both HTTP and HTTPS in jsp pages (Haohui Mai via
|
||
brandonli)
|
||
|
||
HDFS-5291. Standby namenode after transition to active goes into safemode.
|
||
(jing9)
|
||
|
||
HDFS-5317. Go back to DFS Home link does not work on datanode webUI
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5316. Namenode ignores the default https port (Haohui Mai via
|
||
brandonli)
|
||
|
||
HDFS-5281. COMMIT request should not block. (brandonli)
|
||
|
||
HDFS-5337. should do hsync for a commit request even there is no pending
|
||
writes (brandonli)
|
||
|
||
HDFS-5335. Hive query failed with possible race in dfs output stream.
|
||
(Haohui Mai via suresh)
|
||
|
||
HDFS-5322. HDFS delegation token not found in cache errors seen on secure HA
|
||
clusters. (jing9)
|
||
|
||
HDFS-5329. Update FSNamesystem#getListing() to handle inode path in startAfter
|
||
token. (brandonli)
|
||
|
||
HDFS-5330. fix readdir and readdirplus for large directories (brandonli)
|
||
|
||
HDFS-5370. Typo in Error Message: different between range in condition
|
||
and range in error message. (Kousuke Saruta via suresh)
|
||
|
||
HDFS-5365. Fix libhdfs compile error on FreeBSD9. (Radim Kolar via cnauroth)
|
||
|
||
HDFS-5347. Add HDFS NFS user guide. (brandonli)
|
||
|
||
HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers
|
||
post HDFS-5306. (atm)
|
||
|
||
HDFS-5171. NFS should create input stream for a file and try to share it
|
||
with multiple read requests. (Haohui Mai via brandonli)
|
||
|
||
HDFS-5413. hdfs.cmd does not support passthrough to any arbitrary class.
|
||
(cnauroth)
|
||
|
||
HDFS-5433. When reloading fsimage during checkpointing, we should clear
|
||
existing snapshottable directories. (Aaron T. Myers via wang)
|
||
|
||
HDFS-5432. TestDatanodeJsp fails on Windows due to assumption that loopback
|
||
address resolves to host name localhost. (cnauroth)
|
||
|
||
HDFS-5065. TestSymlinkHdfsDisable fails on Windows. (ivanmi)
|
||
|
||
HDFS-4633 TestDFSClientExcludedNodes fails sporadically if excluded nodes
|
||
cache expires too quickly (Chris Nauroth via Sanjay)
|
||
|
||
HDFS-5037. Active NN should trigger its own edit log rolls. (wang)
|
||
|
||
HDFS-5035. getFileLinkStatus and rename do not correctly check permissions
|
||
of symlinks. (Andrew Wang via Colin Patrick McCabe)
|
||
|
||
HDFS-5456. NameNode startup progress creates new steps if caller attempts to
|
||
create a counter for a step that doesn't already exist. (cnauroth)
|
||
|
||
HDFS-5458. Datanode failed volume threshold ignored if exception is thrown
|
||
in getDataDirsFromURIs. (Mike Mellenthin via wang)
|
||
|
||
HDFS-5252. Stable write is not handled correctly in someplace. (brandonli)
|
||
|
||
HDFS-5364. Add OpenFileCtx cache. (brandonli)
|
||
|
||
HDFS-5469. Add configuration property for the sub-directroy export path
|
||
(brandonli)
|
||
|
||
HDFS-5519. COMMIT handler should update the commit status after sync
|
||
(brandonli)
|
||
|
||
HDFS-5372. In FSNamesystem, hasReadLock() returns false if the current
|
||
thread holds the write lock (Vinaykumar B via umamahesh)
|
||
|
||
HDFS-4516. Client crash after block allocation and NN switch before lease recovery for
|
||
the same file can cause readers to fail forever (VinaayKumar B via umamahesh)
|
||
|
||
HDFS-5014. Process register commands with out holding BPOfferService lock.
|
||
(Vinaykumar B via umamahesh)
|
||
|
||
HDFS-5288. Close idle connections in portmap (Haohui Mai via brandonli)
|
||
|
||
HDFS-5407. Fix typos in DFSClientCache (Haohui Mai via brandonli)
|
||
|
||
HDFS-5548. Use ConcurrentHashMap in portmap (Haohui Mai via brandonli)
|
||
|
||
HDFS-5577. NFS user guide update (brandonli)
|
||
|
||
HDFS-5563. NFS gateway should commit the buffered data when read request comes
|
||
after write to the same file (brandonli)
|
||
|
||
HDFS-4997. libhdfs doesn't return correct error codes in most cases (cmccabe)
|
||
|
||
HDFS-5587. add debug information when NFS fails to start with duplicate user
|
||
or group names (brandonli)
|
||
|
||
HDFS-5590. Block ID and generation stamp may be reused when persistBlocks is
|
||
set to false. (jing9)
|
||
|
||
HDFS-5353. Short circuit reads fail when dfs.encrypt.data.transfer is
|
||
enabled. (Colin Patrick McCabe via jing9)
|
||
|
||
HDFS-5283. Under construction blocks only inside snapshots should not be
|
||
counted in safemode threshhold. (Vinay via szetszwo)
|
||
|
||
HDFS-5257. addBlock() retry should return LocatedBlock with locations else client
|
||
will get AIOBE. (Vinay via jing9)
|
||
|
||
HDFS-5427. Not able to read deleted files from snapshot directly under
|
||
snapshottable dir after checkpoint and NN restart. (Vinay via jing9)
|
||
|
||
HDFS-5443. Delete 0-sized block when deleting an under-construction file that
|
||
is included in snapshot. (jing9)
|
||
|
||
HDFS-5476. Snapshot: clean the blocks/files/directories under a renamed
|
||
file/directory while deletion. (jing9)
|
||
|
||
HDFS-5425. Renaming underconstruction file with snapshots can make NN failure on
|
||
restart. (jing9 and Vinay)
|
||
|
||
HDFS-5474. Deletesnapshot can make Namenode in safemode on NN restarts.
|
||
(Sathish via jing9)
|
||
|
||
HDFS-5504. In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold,
|
||
leads to NN safemode. (Vinay via jing9)
|
||
|
||
HDFS-5428. Under construction files deletion after snapshot+checkpoint+nn restart
|
||
leads nn safemode. (jing9)
|
||
|
||
HDFS-5074. Allow starting up from an fsimage checkpoint in the middle of a
|
||
segment. (Todd Lipcon via atm)
|
||
|
||
HDFS-4201. NPE in BPServiceActor#sendHeartBeat. (jxiang via cmccabe)
|
||
|
||
HDFS-5666. Fix inconsistent synchronization in BPOfferService (jxiang via cmccabe)
|
||
|
||
HDFS-5657. race condition causes writeback state error in NFS gateway (brandonli)
|
||
|
||
HDFS-5661. Browsing FileSystem via web ui, should use datanode's fqdn instead of ip
|
||
address. (Benoy Antony via jing9)
|
||
|
||
HDFS-5582. hdfs getconf -excludeFile or -includeFile always failed (sathish
|
||
via cmccabe)
|
||
|
||
HDFS-5671. Fix socket leak in DFSInputStream#getBlockReader. (JamesLi via umamahesh)
|
||
|
||
HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down.
|
||
(brandonli)
|
||
|
||
HDFS-5789. Some of snapshot APIs missing checkOperation double check in fsn. (umamahesh)
|
||
|
||
HDFS-5343. When cat command is issued on snapshot files getting unexpected result.
|
||
(Sathish via umamahesh)
|
||
|
||
HDFS-5297. Fix dead links in HDFS site documents. (Akira Ajisaka via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5830. WebHdfsFileSystem.getFileBlockLocations throws
|
||
IllegalArgumentException when accessing another cluster. (Yongjun Zhang via
|
||
Colin Patrick McCabe)
|
||
|
||
HDFS-5833. Fix SecondaryNameNode javadoc. (Bangtao Zhou via Arpit Agarwal)
|
||
|
||
HDFS-5844. Fix broken link in WebHDFS.apt.vm. (Akira Ajisaka via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5034. Remove debug prints from GetFileLinkInfo (Andrew Wang via Colin
|
||
Patrick McCabe)
|
||
|
||
HDFS-4816. transitionToActive blocks if the SBN is doing checkpoint image
|
||
transfer. (Andrew Wang)
|
||
|
||
HDFS-5164. deleteSnapshot should check if OperationCategory.WRITE is
|
||
possible before taking write lock. (Colin Patrick McCabe)
|
||
|
||
HDFS-5170. BlockPlacementPolicyDefault uses the wrong classname when
|
||
alerting to enable debug logging. (Andrew Wang)
|
||
|
||
HDFS-5266. ElasticByteBufferPool#Key does not implement equals. (cnauroth)
|
||
|
||
HDFS-5352. Server#initLog() doesn't close InputStream in httpfs. (Ted Yu via
|
||
jing9)
|
||
|
||
HDFS-4376. Fix race conditions in Balancer. (Junping Du via szetszwo)
|
||
|
||
HDFS-5375. hdfs.cmd does not expose several snapshot commands. (cnauroth)
|
||
|
||
HDFS-5336. DataNode should not output 'StartupProgress' metrics.
|
||
(Akira Ajisaka via cnauroth)
|
||
|
||
HDFS-5400. DFS_CLIENT_MMAP_CACHE_THREAD_RUNS_PER_TIMEOUT constant is set
|
||
to the wrong value. (Colin Patrick McCabe)
|
||
|
||
HDFS-5075. httpfs-config.sh calls out incorrect env script name
|
||
(Timothy St. Clair via stevel)
|
||
|
||
HDFS-5438. Flaws in block report processing can cause data loss. (kihwal)
|
||
|
||
HDFS-5502. Fix HTTPS support in HsftpFileSystem. (Haohui Mai via jing9)
|
||
|
||
HDFS-5552. Fix wrong information of "Cluster summay" in dfshealth.html.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5533. Symlink delete/create should be treated as DELETE/CREATE in snapshot diff
|
||
report. (Binglin Chang via jing9)
|
||
|
||
HDFS-5580. Fix infinite loop in Balancer.waitForMoveCompletion.
|
||
(Binglin Chang via junping_du)
|
||
|
||
HDFS-5676. fix inconsistent synchronization of CachingStrategy (cmccabe)
|
||
|
||
HDFS-5691. Fix typo in ShortCircuitLocalRead document.
|
||
(Akira Ajisaka via suresh)
|
||
|
||
HDFS-5690. DataNode fails to start in secure mode when dfs.http.policy equals to
|
||
HTTP_ONLY. (Haohui Mai via jing9)
|
||
|
||
HDFS-5449. WebHdfs compatibility broken between 2.2 and 1.x / 23.x (kihwal)
|
||
|
||
HDFS-5756. hadoopRzOptionsSetByteBufferPool does not accept NULL argument,
|
||
contrary to docs. (cmccabe via wang)
|
||
|
||
HDFS-5747. Fix NPEs in BlockManager. (Arpit Agarwal)
|
||
|
||
HDFS-5710. FSDirectory#getFullPathName should check inodes against null.
|
||
(Uma Maheswara Rao G via jing9)
|
||
|
||
HDFS-5579. Under construction files make DataNode decommission take very long
|
||
hours. (zhaoyunjiong via jing9)
|
||
|
||
HDFS-5777. Update LayoutVersion for the new editlog op OP_ADD_BLOCK. (jing9)
|
||
|
||
HDFS-5800. Fix a typo in DFSClient.renewLease(). (Kousuke Saruta
|
||
via szetszwo)
|
||
|
||
HDFS-5748. Too much information shown in the dfs health page.
|
||
(Haohui Mai via brandonli)
|
||
|
||
HDFS-5806. Balancer should set SoTimeout to avoid indefinite hangs.
|
||
(Nathan Roberts via Andrew Wang)
|
||
|
||
HDFS-5728. Block recovery will fail if the metafile does not have crc
|
||
for all chunks of the block (Vinay via kihwal)
|
||
|
||
HDFS-5719. FSImage#doRollback() should close prevState before return
|
||
(Ted Yu via todd)
|
||
|
||
HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be
|
||
closed before method returns (Ted Yu via todd)
|
||
|
||
HDFS-5845. SecondaryNameNode dies when checkpointing with cache pools.
|
||
(wang)
|
||
|
||
HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs
|
||
on a secure cluster. (jing9)
|
||
|
||
HDFS-5399. Revisit SafeModeException and corresponding retry policies.
|
||
(Jing Zhao via todd)
|
||
|
||
HDFS-5876. SecureDataNodeStarter does not pick up configuration in
|
||
hdfs-site.xml. (Haohui Mai via jing9)
|
||
|
||
HDFS-5873. dfs.http.policy should have higher precedence over dfs.https.enable.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5837. dfs.namenode.replication.considerLoad should consider
|
||
decommissioned nodes. (Tao Luo via shv)
|
||
|
||
HDFS-5921. Cannot browse file system via NN web UI if any directory has
|
||
the sticky bit set. (atm)
|
||
|
||
BREAKDOWN OF HDFS-2832 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-4985. Add storage type to the protocol and expose it in block report
|
||
and block locations. (Arpit Agarwal)
|
||
|
||
HDFS-5115. Make StorageID a UUID. (Arpit Agarwal)
|
||
|
||
HDFS-5000. DataNode configuration should allow specifying storage type.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-4987. Namenode changes to track multiple storages per datanode.
|
||
(szetszwo)
|
||
|
||
HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987.
|
||
(Junping Du via szetszwo)
|
||
|
||
HDFS-5009. Include storage information in the LocatedBlock. (szetszwo)
|
||
|
||
HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and
|
||
firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and
|
||
fix a synchronization problem in DatanodeStorageInfo. (szetszwo)
|
||
|
||
HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo)
|
||
|
||
HDFS-4990. Change BlockPlacementPolicy to choose storages instead of
|
||
datanodes. (szetszwo)
|
||
|
||
HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal)
|
||
|
||
HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal)
|
||
|
||
HDFS-5222. Move block schedule information from DatanodeDescriptor to
|
||
DatanodeStorageInfo. (szetszwo)
|
||
|
||
HDFS-4988. Datanode must support all the volumes as individual storages.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-5377. Heartbeats from Datandode should include one storage report
|
||
per storage directory. (Arpit Agarwal)
|
||
|
||
HDFS-5398. NameNode changes to process storage reports per storage
|
||
directory. (Arpit Agarwal)
|
||
|
||
HDFS-5390. Send one incremental block report per storage directory.
|
||
(Arpit Agarwal)
|
||
|
||
HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal)
|
||
|
||
HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo)
|
||
|
||
HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal)
|
||
|
||
HDFS-5435. File append fails to initialize storageIDs. (Junping Du via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5447. Fix TestJspHelper. (Arpit Agarwal)
|
||
|
||
HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter.
|
||
|
||
HDFS-5448. Datanode should generate its ID on first registration. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal)
|
||
|
||
HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry.
|
||
(Contributed by szetszwo)
|
||
|
||
HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed
|
||
by szetszwo)
|
||
|
||
HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5470. Add back trunk's reportDiff algorithm to the branch.
|
||
(Contributed by szetszwo)
|
||
|
||
HDFS-5472. Fix TestDatanodeManager, TestSafeMode and
|
||
TestNNThroughputBenchmark (Contributed by szetszwo)
|
||
|
||
HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed
|
||
by Junping Du)
|
||
|
||
HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo)
|
||
|
||
HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal)
|
||
|
||
HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal)
|
||
|
||
HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal)
|
||
|
||
HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo)
|
||
|
||
HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch.
|
||
(Junping Du via Arpit Agarwal)
|
||
|
||
HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal)
|
||
|
||
HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by
|
||
szetszwo)
|
||
|
||
HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo)
|
||
|
||
HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is
|
||
not accurate. (Eric Sirianni via Arpit Agarwal)
|
||
|
||
HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal)
|
||
|
||
HDFS-5454. DataNode UUID should be assigned prior to FsDataset
|
||
initialization. (Arpit Agarwal)
|
||
|
||
HDFS-5406. Send incremental block reports for all storages in a
|
||
single call. (Arpit Agarwal)
|
||
|
||
HDFS-5667. Include DatanodeStorage in StorageReport. (Arpit Agarwal)
|
||
|
||
BREAKDOWN OF HDFS-4949 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-5049. Add JNI mlock support. (Andrew Wang via Colin Patrick McCabe)
|
||
|
||
HDFS-5051. Propagate cache status information from the DataNode to the
|
||
NameNode (Andrew Wang via Colin Patrick McCabe)
|
||
|
||
HDFS-5052. Add cacheRequest/uncacheRequest support to NameNode.
|
||
(Contributed by Colin Patrick McCabe.)
|
||
|
||
HDFS-5050. Add DataNode support for mlock and munlock (contributed by
|
||
Andrew Wang)
|
||
|
||
HDFS-5141. Add cache status information to datanode heartbeat. (Contributed
|
||
by Andrew Wang)
|
||
|
||
HDFS-5121. Add RPCs for creating and manipulating cache pools.
|
||
(Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5163. Miscellaneous cache pool RPC fixes (Contributed by Colin Patrick
|
||
McCabe)
|
||
|
||
HDFS-5169. hdfs.c: translateZCRException: null pointer deref when
|
||
translating some exceptions (Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5120. Add command-line support for manipulating cache pools. (cmccabe)
|
||
|
||
HDFS-5158. Add command-line support for manipulating cache directives.
|
||
(cmccabe)
|
||
|
||
HDFS-5198. NameNodeRpcServer must not send back DNA_FINALIZE in reply to a
|
||
cache report. (cmccabe)
|
||
|
||
HDFS-5195. Prevent passing null pointer to mlock and munlock. Contributed
|
||
by Chris Nauroth.
|
||
|
||
HDFS-5053. NameNode should invoke DataNode APIs to coordinate caching.
|
||
(Andrew Wang)
|
||
|
||
HDFS-5201. NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit.
|
||
(Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5197. Document dfs.cachereport.intervalMsec in hdfs-default.xml.
|
||
Contributed by Chris Nauroth.
|
||
|
||
HDFS-5210. Fix some failing unit tests on HDFS-4949 branch. (Contributed by
|
||
Andrew Wang)
|
||
|
||
HDFS-5213. Separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId.
|
||
Contributed by Colin Patrick McCabe.
|
||
|
||
HDFS-5236. Change PathBasedCacheDirective APIs to be a single value rather
|
||
than batch. (Contributed by Andrew Wang)
|
||
|
||
HDFS-5119. Persist CacheManager state in the edit log. (Contributed by
|
||
Andrew Wang)
|
||
|
||
HDFS-5190. Move cache pool related CLI commands to CacheAdmin. (Contributed
|
||
by Andrew Wang)
|
||
|
||
HDFS-5309. Fix failing caching unit tests. (Andrew Wang)
|
||
|
||
HDFS-5314. Do not expose CachePool type in AddCachePoolOp (Colin Patrick
|
||
McCabe)
|
||
|
||
HDFS-5304. Expose if a block replica is cached in getFileBlockLocations.
|
||
(Contributed by Andrew Wang)
|
||
|
||
HDFS-5224. Refactor PathBasedCache* methods to use a Path rather than a
|
||
String. Contributed by Chris Nauroth.
|
||
|
||
HDFS-5348. Fix error message when dfs.datanode.max.locked.memory is
|
||
improperly configured. (Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5349. DNA_CACHE and DNA_UNCACHE should be by blockId only (cmccabe)
|
||
|
||
HDFS-5358. Add replication field to PathBasedCacheDirective. (Contributed
|
||
by Colin Patrick McCabe)
|
||
|
||
HDFS-5359. Allow LightWeightGSet#Iterator to remove elements. (Contributed
|
||
by Colin Patrick McCabe)
|
||
|
||
HDFS-5373. hdfs cacheadmin -addDirective short usage does not mention
|
||
-replication parameter. Contributed by Chris Nauroth.
|
||
|
||
HDFS-5096. Automatically cache new data added to a cached path (contributed
|
||
by Colin Patrick McCabe)
|
||
|
||
HDFS-5383. fix broken caching unit tests (Andrew Wang)
|
||
|
||
HDFS-5388. Loading fsimage fails to find cache pools during namenode
|
||
startup (Chris Nauroth via Colin Patrick McCabe)
|
||
|
||
HDFS-5203. Concurrent clients that add a cache directive on the same path
|
||
may prematurely uncache each other. (Chris Nauroth via Colin Patrick McCabe)
|
||
|
||
HDFS-5378. In CacheReport, don't send genstamp and length on the wire
|
||
(Contributed by Colin Patrick McCabe)
|
||
|
||
HDFS-5385. Caching RPCs are AtMostOnce, but do not persist client ID and
|
||
call ID to edit log. (Chris Nauroth via Colin Patrick McCabe)
|
||
|
||
HDFS-5404 Resolve regressions in Windows compatibility on HDFS-4949 branch.
|
||
Contributed by Chris Nauroth.
|
||
|
||
HDFS-5405. Fix possible RetryCache hang for caching RPC handlers in
|
||
FSNamesystem. (Contributed by Andrew Wang)
|
||
|
||
HDFS-5419. Fixup test-patch.sh warnings on HDFS-4949 branch. (wang)
|
||
|
||
HDFS-5386. Add feature documentation for datanode caching. Contributed by
|
||
Colin Patrick McCabe.
|
||
|
||
HDFS-5468. CacheAdmin help command does not recognize commands (Stephen
|
||
Chu via Colin Patrick McCabe)
|
||
|
||
HDFS-5326. add modifyDirective to cacheAdmin (cmccabe)
|
||
|
||
HDFS-5394: Fix race conditions in DN caching and uncaching (cmccabe)
|
||
|
||
HDFS-5320. Add datanode caching metrics. Contributed by Andrew Wang.
|
||
|
||
HDFS-5482. DistributedFileSystem#listPathBasedCacheDirectives must support
|
||
relative paths. Contributed by Colin Patrick McCabe.
|
||
|
||
HDFS-5471. CacheAdmin -listPools fails when user lacks permissions to view
|
||
all pools (Andrew Wang via Colin Patrick McCabe)
|
||
|
||
HDFS-5450. better API for getting the cached blocks locations. Contributed
|
||
by Andrew Wang.
|
||
|
||
HDFS-5485. add command-line support for modifyDirective (cmccabe)
|
||
|
||
HDFS-5366. recaching improvements (cmccabe)
|
||
|
||
HDFS-5520. loading cache path directives from edit log doesnt update
|
||
nextEntryId (cmccabe)
|
||
|
||
HDFS-5512. CacheAdmin -listPools fails with NPE when user lacks permissions
|
||
to view all pools (awang via cmccabe)
|
||
|
||
HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed
|
||
by Andrew Wang.
|
||
|
||
HDFS-5511. improve CacheManipulator interface to allow better unit testing
|
||
(cmccabe)
|
||
|
||
HDFS-5451. Add byte and file statistics to PathBasedCacheEntry. Contributed
|
||
by Colin Patrick McCabe.
|
||
|
||
HDFS-5473. Consistent naming of user-visible caching classes and methods
|
||
(cmccabe)
|
||
|
||
HDFS-5543. Fix narrow race condition in TestPathBasedCacheRequests
|
||
(cmccabe)
|
||
|
||
HDFS-5565. CacheAdmin help should match against non-dashed commands (wang
|
||
via cmccabe)
|
||
|
||
HDFS-5556. Add some more NameNode cache statistics, cache pool stats
|
||
(cmccabe)
|
||
|
||
HDFS-5562. TestCacheDirectives and TestFsDatasetCache should stub out
|
||
native mlock. Contributed by Colin Patrick McCabe and Akira Ajisaka.
|
||
|
||
HDFS-5430. Support TTL on CacheDirectives. Contributed by Andrew Wang.
|
||
|
||
HDFS-5555. CacheAdmin commands fail when first listed NameNode is in
|
||
Standby (jxiang via cmccabe)
|
||
|
||
HDFS-5626. dfsadmin report shows incorrect values (cmccabe)
|
||
|
||
HDFS-5630. Hook up cache directive and pool usage statistics. (wang)
|
||
|
||
HDFS-5665. Remove the unnecessary writeLock while initializing CacheManager
|
||
in FsNameSystem Ctor. (Uma Maheswara Rao G via Andrew Wang)
|
||
|
||
HDFS-5431. Support cachepool-based limit management in path-based caching.
|
||
(awang via cmccabe)
|
||
|
||
HDFS-5679. TestCacheDirectives should handle the case where native code is
|
||
not available. (wang)
|
||
|
||
HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe)
|
||
|
||
HDFS-5701. Fix the CacheAdmin -addPool -maxTtl option name. Contributed by
|
||
Stephen Chu.
|
||
|
||
HDFS-5708. The CacheManager throws a NPE in the DataNode logs when
|
||
processing cache reports that refer to a block not known to the BlockManager.
|
||
Contributed by Colin Patrick McCabe.
|
||
|
||
HDFS-5659. dfsadmin -report doesn't output cache information properly.
|
||
Contributed by Andrew Wang.
|
||
|
||
HDFS-5651. Remove dfs.namenode.caching.enabled and improve CRM locking.
|
||
Contributed by Colin Patrick McCabe.
|
||
|
||
HDFS-5589. Namenode loops caching and uncaching when data should be
|
||
uncached. (awang via cmccabe)
|
||
|
||
HDFS-5724. modifyCacheDirective logging audit log command wrongly as
|
||
addCacheDirective (Uma Maheswara Rao G via Colin Patrick McCabe)
|
||
|
||
Release 2.2.0 - 2013-10-13
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-4817. Make HDFS advisory caching configurable on a per-file basis.
|
||
(Colin Patrick McCabe)
|
||
|
||
HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API.
|
||
(Haohui Mai via brandonli)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-5246. Make Hadoop nfs server port and mount daemon port
|
||
configurable. (Jinghui Wang via brandonli)
|
||
|
||
HDFS-5256. Use guava LoadingCache to implement DFSClientCache. (Haohui Mai
|
||
via brandonli)
|
||
|
||
HDFS-5308. Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS
|
||
JSP. (Haohui Mai via jing9)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-5139. Remove redundant -R option from setrep.
|
||
|
||
HDFS-5251. Race between the initialization of NameNode and the http
|
||
server. (Haohui Mai via suresh)
|
||
|
||
HDFS-5258. Skip tests in TestHDFSCLI that are not applicable on Windows.
|
||
(Chuan Liu via cnauroth)
|
||
|
||
HDFS-5186. TestFileJournalManager fails on Windows due to file handle leaks.
|
||
(Chuan Liu via cnauroth)
|
||
|
||
HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit
|
||
Agarwal)
|
||
|
||
HDFS-5268. NFS write commit verifier is not set in a few places (brandonli)
|
||
|
||
HDFS-5265. Namenode fails to start when dfs.https.port is unspecified.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure
|
||
cluster. (Arpit Agarwal)
|
||
|
||
HDFS-5279. Guard against NullPointerException in NameNode JSP pages before
|
||
initialization of FSNamesystem. (cnauroth)
|
||
|
||
HDFS-5289. Race condition in TestRetryCacheWithHA#testCreateSymlink causes
|
||
spurious test failure. (atm)
|
||
|
||
HDFS-5300. FSNameSystem#deleteSnapshot() should not check owner in case of
|
||
permissions disabled. (Vinay via jing9)
|
||
|
||
HDFS-5306. Datanode https port is not available at the namenode. (Suresh
|
||
Srinivas via brandonli)
|
||
|
||
HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened.
|
||
(Vinay via jing9)
|
||
|
||
HDFS-5259. Support client which combines appended data with old data
|
||
before sends it to NFS server. (brandonli)
|
||
|
||
Release 2.1.1-beta - 2013-09-23
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-4962 Use enum for nfs constants (Nicholas SZE via jing9)
|
||
|
||
HDFS-5071 Change hdfs-nfs parent project to hadoop-project (brandonli)
|
||
|
||
HDFS-4763 Add script changes/utility for starting NFS gateway (brandonli)
|
||
|
||
HDFS-5076 Add MXBean methods to query NN's transaction information and
|
||
JournalNode's journal status. (jing9)
|
||
|
||
HDFS-5104 Support dotdot name in NFS LOOKUP operation (brandonli)
|
||
|
||
HDFS-5107 Fix array copy error in Readdir and Readdirplus responses
|
||
(brandonli)
|
||
|
||
HDFS-5110 Change FSDataOutputStream to HdfsDataOutputStream for opened
|
||
streams to fix type cast error. (brandonli)
|
||
|
||
HDFS-5069 Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist for
|
||
NFS deployment (brandonli)
|
||
|
||
HDFS-4947 Add NFS server export table to control export by hostname or
|
||
IP range (Jing Zhao via brandonli)
|
||
|
||
HDFS-5078 Support file append in NFSv3 gateway to enable data streaming
|
||
to HDFS (brandonli)
|
||
|
||
HDFS-5136 MNT EXPORT should give the full group list which can mount the
|
||
exports (brandonli)
|
||
|
||
HDFS-5118. Provide testing support for DFSClient to drop RPC responses.
|
||
(jing9)
|
||
|
||
HDFS-5085. Refactor o.a.h.nfs to support different types of
|
||
authentications. (jing9)
|
||
|
||
HDFS-5067 Support symlink operations in NFS gateway. (brandonli)
|
||
|
||
HDFS-5199 Add more debug trace for NFS READ and WRITE. (brandonli)
|
||
|
||
HDFS-5234 Move RpcFrameDecoder out of the public API.
|
||
(Haohui Mai via brandonli)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-4513. Clarify in the WebHDFS REST API that all JSON respsonses may
|
||
contain additional properties. (szetszwo)
|
||
|
||
HDFS-5061. Make FSNameSystem#auditLoggers an unmodifiable list.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
HDFS-4905. Add appendToFile command to "hdfs dfs". (Arpit Agarwal via
|
||
cnauroth)
|
||
|
||
HDFS-4926. Namenode webserver's page has a tooltip that is inconsistent
|
||
with the datanode HTML link. (Vivek Ganesan via jing9)
|
||
|
||
HDFS-5047. Supress logging of full stack trace of quota and lease
|
||
exceptions. (Robert Parker via kihwal)
|
||
|
||
HDFS-2933. Improve DataNode Web UI Index Page. (Vivek Ganesan via
|
||
Arpit Agarwal)
|
||
|
||
HDFS-5111. Remove duplicated error message for snapshot commands when
|
||
processing invalid arguments. (jing9)
|
||
|
||
HDFS-5045. Add more unit tests for retry cache to cover all AtMostOnce
|
||
methods. (jing9)
|
||
|
||
HDFS-3245. Add metrics and web UI for cluster version summary. (Ravi
|
||
Prakash via kihwal)
|
||
|
||
HDFS-5128. Allow multiple net interfaces to be used with HA namenode RPC
|
||
server. (kihwal)
|
||
|
||
HDFS-5150. Allow per NN SPN for internal SPNEGO. (kihwal)
|
||
|
||
HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang)
|
||
|
||
HDFS-5212. Refactor RpcMessage and NFS3Response to support different
|
||
types of authentication information. (jing9)
|
||
|
||
HDFS-4971. Move IO operations out of locking in OpenFileCtx. (brandonli and
|
||
jing9)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-5028. LeaseRenewer throws ConcurrentModificationException when timeout.
|
||
(zhaoyunjiong via szetszwo)
|
||
|
||
HDFS-5043. For HdfsFileStatus, set default value of childrenNum to -1
|
||
instead of 0 to avoid confusing applications. (brandonli)
|
||
|
||
HDFS-4993. Fsck can fail if a file is renamed or deleted. (Robert Parker
|
||
via kihwal)
|
||
|
||
HDFS-5091. Support for spnego keytab separate from the JournalNode keytab
|
||
for secure HA. (jing9)
|
||
|
||
HDFS-5055. nn fails to download checkpointed image from snn in some
|
||
setups. (Vinay and suresh via suresh)
|
||
|
||
HDFS-4898. BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to
|
||
properly fallback to local rack. (szetszwo)
|
||
|
||
HDFS-4632. globStatus using backslash for escaping does not work on Windows.
|
||
(Chuan Liu via cnauroth)
|
||
|
||
HDFS-5093. TestGlobPaths should re-use the MiniDFSCluster to avoid failure
|
||
on Windows. (Chuan Liu via cnauroth)
|
||
|
||
HDFS-5080. BootstrapStandby not working with QJM when the existing NN is
|
||
active. (jing9)
|
||
|
||
HDFS-5099. Namenode#copyEditLogSegmentsToSharedDir should close
|
||
EditLogInputStreams upon finishing. (Chuan Liu via cnauroth)
|
||
|
||
HDFS-2994. If lease soft limit is recovered successfully
|
||
the append can fail. (Tao Luo via shv)
|
||
|
||
HDFS-5100. TestNamenodeRetryCache fails on Windows due to incorrect cleanup.
|
||
(Chuan Liu via cnauroth)
|
||
|
||
HDFS-5103. TestDirectoryScanner fails on Windows. (Chuan Liu via cnauroth)
|
||
|
||
HDFS-5102. Snapshot names should not be allowed to contain slash characters.
|
||
(jing9)
|
||
|
||
HDFS-5105. TestFsck fails on Windows. (Chuan Liu via arp)
|
||
|
||
HDFS-5106. TestDatanodeBlockScanner fails on Windows due to incorrect path
|
||
format. (Chuan Liu via cnauroth)
|
||
|
||
HDFS-4594. WebHDFS open sets Content-Length header to what is specified by
|
||
length parameter rather than how much data is actually returned. (cnauroth)
|
||
|
||
HDFS-5124. DelegationTokenSecretManager#retrievePassword can cause deadlock
|
||
in NameNode. (Daryn Sharp via jing9)
|
||
|
||
HDFS-5132. Deadlock in NameNode between SafeModeMonitor#run and
|
||
DatanodeManager#handleHeartbeat. (kihwal)
|
||
|
||
HDFS-5077. NPE in FSNamesystem.commitBlockSynchronization().
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-5140. Too many safemode monitor threads being created in the standby
|
||
namenode causing it to fail with out of memory error. (jing9)
|
||
|
||
HDFS-5159. Secondary NameNode fails to checkpoint if error occurs
|
||
downloading edits on first checkpoint. (atm)
|
||
|
||
HDFS-5192. NameNode may fail to start when
|
||
dfs.client.test.drop.namenode.response.number is set. (jing9)
|
||
|
||
HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem.
|
||
(Haohui Mai via jing9)
|
||
|
||
HDFS-5231. Fix broken links in the document of HDFS Federation. (Haohui Mai
|
||
via jing9)
|
||
|
||
HDFS-5249. Fix dumper thread which may die silently. (brandonli)
|
||
|
||
Release 2.1.0-beta - 2013-08-22
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-4053. Increase the default block size. (eli)
|
||
|
||
HDFS-4305. Add a configurable limit on number of blocks per file, and min
|
||
block size. (Andrew Wang via atm)
|
||
|
||
HDFS-4434. Provide a mapping from INodeId to INode. (suresh)
|
||
|
||
HDFS-2802. Add HDFS Snapshot feature. (See breakdown of tasks below for
|
||
subtasks and contributors)
|
||
|
||
HDFS-4866. Protocol buffer support cannot compile under C. (Arpit Agarwal via
|
||
cnauroth)
|
||
|
||
HDFS-5083. Update the HDFS compatibility version range. (kihwal)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-1804. Add a new block-volume device choosing policy that looks at
|
||
free space. (atm)
|
||
|
||
HDFS-4296. Reserve layout version for release 1.2.0. (suresh)
|
||
|
||
HDFS-4334. Add a unique id to INode. (Brandon Li via szetszwo)
|
||
|
||
HDFS-4339. Persist inode id in fsimage and editlog. (Brandon Li via
|
||
suresh)
|
||
|
||
HDFS-4340. Update addBlock() to inculde inode id as additional argument.
|
||
(Brandon Li via suresh)
|
||
|
||
HDFS-4502. JsonUtil.toFileStatus(..) should check if the fileId property
|
||
exists. (Brandon Li via suresh)
|
||
|
||
HDFS-2576. Enhances the DistributedFileSystem's create API so that clients
|
||
can specify favored datanodes for a file's blocks. (ddas)
|
||
|
||
HDFS-347. DFS read performance suboptimal when client co-located on nodes
|
||
with data. (Colin Patrick McCabe via todd and atm)
|
||
|
||
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
|
||
Azure environments. (See breakdown of tasks below for subtasks and
|
||
contributors)
|
||
|
||
HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement
|
||
with 4-layer network topology. (Junping Du via szetszwo)
|
||
|
||
HDFS-3495. Update Balancer to support new NetworkTopology with NodeGroup.
|
||
(Junping Du via szetszwo)
|
||
|
||
HDFS-4659 Support setting execution bit for regular files (Brandon Li via sanjay)
|
||
|
||
HDFS-4762 Provide HDFS based NFSv3 and Mountd implementation (brandonli)
|
||
|
||
HDFS-4372. Track NameNode startup progress. (cnauroth)
|
||
|
||
HDFS-4373. Add HTTP API for querying NameNode startup progress. (cnauroth)
|
||
|
||
HDFS-4374. Display NameNode startup progress in UI. (cnauroth)
|
||
|
||
HDFS-4974. Add Idempotent and AtMostOnce annotations to namenode
|
||
protocol methods. (suresh)
|
||
|
||
HDFS-4979. Implement retry cache on Namenode. (suresh)
|
||
|
||
HDFS-5025. Record ClientId and CallId in EditLog to enable rebuilding
|
||
retry cache in case of HA failover. (Jing Zhao via suresh)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-4461. DirectoryScanner: volume path prefix takes up memory for every
|
||
block that is scanned (Colin Patrick McCabe)
|
||
|
||
HDFS-4222. NN is unresponsive and loses heartbeats from DNs when
|
||
configured to use LDAP and LDAP has issues. (Xiaobo Peng, suresh)
|
||
|
||
HDFS-4304. Make FSEditLogOp.MAX_OP_SIZE configurable. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-4518. Finer grained metrics for HDFS capacity.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
HDFS-4519. Support overriding jsvc binary and log file locations
|
||
when launching secure datanode. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4569. Small image transfer related cleanups.
|
||
(Andrew Wang via suresh)
|
||
|
||
HDFS-4521. Invalid network toploogies should not be cached. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-4246. The exclude node list should be more forgiving, for each output
|
||
stream. (harsh via atm)
|
||
|
||
HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. (suresh)
|
||
|
||
HDFS-4621. Additional logging to help diagnose slow QJM syncs. (todd)
|
||
|
||
HDFS-4618. Default transaction interval for checkpoints is too low. (todd)
|
||
|
||
HDFS-4525. Provide an API for knowing that whether file is closed or not.
|
||
(SreeHari via umamahesh)
|
||
|
||
HDFS-3940. Add Gset#clear method and clear the block map when namenode is
|
||
shutdown. (suresh)
|
||
|
||
HDFS-4679. Namenode operation checks should be done in a consistent
|
||
manner. (suresh)
|
||
|
||
HDFS-4693. Some test cases in TestCheckpoint do not clean up after
|
||
themselves. (Arpit Agarwal, suresh via suresh)
|
||
|
||
HDFS-3817. Avoid printing SafeModeException stack trace.
|
||
(Brandon Li via suresh)
|
||
|
||
HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable
|
||
returning more than INode array. (Jing Zhao via suresh)
|
||
|
||
HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead
|
||
of INode[] as a parameter. (szetszwo)
|
||
|
||
HDFS-4129. Add utility methods to dump NameNode in memory tree for
|
||
testing. (szetszwo via suresh)
|
||
|
||
HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in
|
||
INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4206. Change the fields in INode and its subclasses to private.
|
||
(szetszwo)
|
||
|
||
HDFS-4215. Remove locking from addToParent(..) since it is used in image
|
||
loading, and add INode.isFile(). (szetszwo)
|
||
|
||
HDFS-4243. When replacing an INodeDirectory, the parent pointers of the
|
||
children of the child have to be updated to the new child. (Jing Zhao
|
||
via szetszwo)
|
||
|
||
HDFS-4209. Clean up the addNode/addChild/addChildNoQuotaCheck methods in
|
||
FSDirectory and INodeDirectory. (szetszwo)
|
||
|
||
HDFS-4346. Add SequentialNumber as a base class for INodeId and
|
||
GenerationStamp. (szetszwo)
|
||
|
||
HDFS-4721. Speed up lease recovery by avoiding stale datanodes and choosing
|
||
the datanode with the most recent heartbeat as the primary. (Varun Sharma
|
||
via szetszwo)
|
||
|
||
HDFS-4804. WARN when users set the block balanced preference percent below
|
||
0.5 or above 1.0. (Stephen Chu via atm)
|
||
|
||
HDFS-4698. Provide client-side metrics for remote reads, local reads, and
|
||
short-circuit reads. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3498. Support replica removal in BlockPlacementPolicy and make
|
||
BlockPlacementPolicyDefault extensible for reusing code in subclasses.
|
||
(Junping Du via szetszwo)
|
||
|
||
HDFS-4234. Use generic code for choosing datanode in Balancer. (szetszwo)
|
||
|
||
HDFS-4880. Print the image and edits file loaded by the namenode in the
|
||
logs. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh)
|
||
|
||
HDFS-2857. Cleanup BlockInfo class. (suresh)
|
||
|
||
HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using
|
||
NetUtils. (Hari Mankude via suresh)
|
||
|
||
HDFS-4914. Use DFSClient.Conf instead of Configuration. (szetszwo)
|
||
|
||
HDFS-4883. complete() should verify fileId. (Tao Luo via shv)
|
||
|
||
HDFS-4772. Add number of children in HdfsFileStatus. (brandonli)
|
||
|
||
HDFS-4932. Avoid a wide line on the name node webUI if we have more Journal
|
||
nodes. (Fengdong Yu via cnauroth)
|
||
|
||
HDFS-4908. Reduce snapshot inode memory usage. (szetszwo)
|
||
|
||
HDFS-4645. Move from randomly generated block ID to sequentially generated
|
||
block ID. (Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4912. Cleanup FSNamesystem#startFileInternal. (suresh)
|
||
|
||
HDFS-4903. Print trash configuration and trash emptier state in
|
||
namenode log. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4992. Make balancer's mover thread count and dispatcher thread count
|
||
configurable. (Max Lapan via szetszwo)
|
||
|
||
HDFS-4996. ClientProtocol#metaSave can be made idempotent by overwriting the
|
||
output file instead of appending to it. (cnauroth)
|
||
|
||
HADOOP-9418. Add symlink support to DistributedFileSystem (Andrew Wang via
|
||
Colin Patrick McCabe)
|
||
|
||
HDFS-5007. Replace hard-coded property keys with DFSConfigKeys fields.
|
||
(Kousuke Saruta via jing9)
|
||
|
||
HDFS-5008. Make ClientProtocol#abandonBlock() idempotent. (jing9)
|
||
|
||
HADOOP-9760. Move GSet and related classes to common from HDFS.
|
||
(suresh)
|
||
|
||
HDFS-5020. Make DatanodeProtocol#blockReceivedAndDeleted idempotent.
|
||
(jing9)
|
||
|
||
HDFS-5024. Make DatanodeProtocol#commitBlockSynchronization idempotent.
|
||
(Arpit Agarwal via jing9)
|
||
|
||
HDFS-3880. Use Builder to build RPC server in HDFS.
|
||
(Brandon Li and Junping Du via szetszwo)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-4465. Optimize datanode ReplicasMap and ReplicaInfo. (atm)
|
||
|
||
HDFS-5027. On startup, DN should scan volumes in parallel. (atm)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4626. ClientProtocol#getLinkTarget should throw an exception for
|
||
non-symlink and non-existent paths. (Andrew Wang via cmccabe)
|
||
|
||
HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick
|
||
McCabe)
|
||
|
||
HDFS-4470. Several HDFS tests attempt file operations on invalid HDFS
|
||
paths when running on Windows. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4471. Namenode WebUI file browsing does not work with wildcard
|
||
addresses configured. (Andrew Wang via atm)
|
||
|
||
HDFS-4342. Directories configured in dfs.namenode.edits.dir.required
|
||
but not in dfs.namenode.edits.dir are silently ignored. (Arpit Agarwal
|
||
via szetszwo)
|
||
|
||
HDFS-4482. ReplicationMonitor thread can exit with NPE due to the race
|
||
between delete and replication of same file. (umamahesh)
|
||
|
||
HDFS-4269. Datanode rejects all datanode registrations from localhost
|
||
in single-node developer setup on Windows. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4235. When outputting XML, OfflineEditsViewer can't handle some edits
|
||
containing non-ASCII strings. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4541. Set hadoop.log.dir and hadoop.id.str when starting secure
|
||
datanode to write the logs to right dir by default. (Arpit Gupta via
|
||
suresh)
|
||
|
||
HDFS-4540. Namenode http server should use the web authentication
|
||
keytab for spnego principal. (Arpit Gupta via suresh)
|
||
|
||
HDFS-4544. Error in deleting blocks should not do check disk, for
|
||
all types of errors. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4565. Use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key
|
||
in secondary namenode and namenode http server. (Arpit Gupta via suresh)
|
||
|
||
HDFS-4571. WebHDFS should not set the service hostname on the server side.
|
||
(tucu)
|
||
|
||
HDFS-4013. TestHftpURLTimeouts throws NPE. (Chao Shi via suresh)
|
||
|
||
HDFS-4592. Default values for access time precision are out of sync between
|
||
hdfs-default.xml and the code. (atm)
|
||
|
||
HDFS-4522. LightWeightGSet expects incrementing a volatile to be atomic.
|
||
(Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4484. libwebhdfs compilation broken with gcc 4.6.2. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-4595. When short circuit read is fails, DFSClient does not fallback
|
||
to regular reads. (suresh)
|
||
|
||
HDFS-4583. TestNodeCount fails. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4591. HA clients can fail to fail over while Standby NN is performing
|
||
long checkpoint. (atm)
|
||
|
||
HDFS-3277. fail over to loading a different FSImage if the first one we
|
||
try to load is corrupt. (Colin Patrick McCabe and Andrew Wang via atm)
|
||
|
||
HDFS-4596. Shutting down namenode during checkpointing can lead to md5sum
|
||
error. (Andrew Wang via atm)
|
||
|
||
HDFS-4614. FSNamesystem#getContentSummary should use getPermissionChecker
|
||
helper method. (atm)
|
||
|
||
HDFS-4620. Documentation for dfs.namenode.rpc-address specifies wrong
|
||
format. (Sandy Ryza via atm)
|
||
|
||
HDFS-4609. TestAuditLogs should release log handles between tests.
|
||
(Ivan Mitic via szetszwo)
|
||
|
||
HDFS-4598. Fix the default value of ConcatSourcesParam and the WebHDFS doc.
|
||
(szetszwo)
|
||
|
||
HDFS-4655. DNA_FINALIZE is logged as being an unknown command by the DN
|
||
when received from the standby NN. (atm)
|
||
|
||
HDFS-4656. DN heartbeat loop can be briefly tight. (atm)
|
||
|
||
HDFS-4658. Standby NN will log that it has received a block report "after
|
||
becoming active" (atm)
|
||
|
||
HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout
|
||
value (Jagane Sundar via cos)
|
||
|
||
HDFS-3981. Fix handling of FSN lock in getBlockLocations. (Xiaobo Peng
|
||
and todd via todd)
|
||
|
||
HDFS-4676. TestHDFSFileSystemContract should set MiniDFSCluster variable
|
||
to null to free up memory. (suresh)
|
||
|
||
HDFS-4669. TestBlockPoolManager fails using IBM java. (Tian Hong Wang via
|
||
suresh)
|
||
|
||
HDFS-4643. Fix flakiness in TestQuorumJournalManager. (todd)
|
||
|
||
HDFS-4639. startFileInternal() should not increment generation stamp.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-4695. TestEditLog leaks open file handles between tests.
|
||
(Ivan Mitic via suresh)
|
||
|
||
HDFS-4737. JVM path embedded in fuse binaries. (Sean Mackrory via atm)
|
||
|
||
HDFS-4739. NN can miscalculate the number of extra edit log segments to
|
||
retain. (atm)
|
||
|
||
HDFS-4745. TestDataTransferKeepalive#testSlowReader has race condition that
|
||
causes sporadic failure. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4768. File handle leak in datanode when a block pool is removed.
|
||
(Chris Nauroth via suresh)
|
||
|
||
HDFS-4748. MiniJournalCluster#restartJournalNode leaks resources, which
|
||
causes sporadic test failures. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4733. Make HttpFS username pattern configurable. (tucu via atm)
|
||
|
||
HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed.
|
||
(ddas)
|
||
|
||
HDFS-4785. Concat operation does not remove concatenated files from
|
||
InodeMap. (suresh)
|
||
|
||
HDFS-4784. NPE in FSDirectory.resolvePath(). (Brandon Li via suresh)
|
||
|
||
HDFS-4810. several HDFS HA tests have timeouts that are too short. (Chris
|
||
Nauroth via atm)
|
||
|
||
HDFS-4799. Corrupt replica can be prematurely removed from
|
||
corruptReplicas map. (todd via kihwal)
|
||
|
||
HDFS-4751. TestLeaseRenewer#testThreadName flakes. (Andrew Wang via atm)
|
||
|
||
HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade.
|
||
(Fengdong Yu via suresh)
|
||
|
||
HDFS-4765. Permission check of symlink deletion incorrectly throws
|
||
UnresolvedLinkException. (Andrew Wang via atm)
|
||
|
||
HDFS-4300. TransferFsImage.downloadEditsToStorage should use a tmp file for
|
||
destination. (Andrew Wang via atm)
|
||
|
||
HDFS-4813. Add volatile to BlocksMap.blocks so that the replication thread
|
||
can see the updated value. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-3180. Add socket timeouts to WebHdfsFileSystem. (Chris Nauroth via
|
||
szetszwo)
|
||
|
||
HDFS-4787. Create a new HdfsConfiguration before each TestDFSClientRetries
|
||
testcases. (Tian Hong Wang via atm)
|
||
|
||
HDFS-4830. Typo in config settings for AvailableSpaceVolumeChoosingPolicy
|
||
in hdfs-default.xml. (atm)
|
||
|
||
HDFS-4824. FileInputStreamCache.close leaves dangling reference to
|
||
FileInputStreamCache.cacheCleaner. (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4298. StorageRetentionManager spews warnings when used with QJM. (atm)
|
||
|
||
HDFS-4725. Fix HDFS file handle leaks in FSEditLog, NameNode,
|
||
OfflineEditsBinaryLoader and some tests. (Chris Nauroth via szetszwo)
|
||
|
||
HDFS-4825. webhdfs / httpfs tests broken because of min block size change.
|
||
(Andrew Wang via suresh)
|
||
|
||
HDFS-4780. Use the correct relogin method for services. (Robert Parker via
|
||
kihwal)
|
||
|
||
HDFS-4827. Slight update to the implementation of API for handling favored
|
||
nodes in DFSClient (ddas)
|
||
|
||
HDFS-4865. Remove sub resource warning from httpfs log at startup time.
|
||
(ywskycn via tucu)
|
||
|
||
HDFS-4240. For nodegroup-aware block placement, when a node is excluded,
|
||
the nodes in the same nodegroup should also be excluded. (Junping Du
|
||
via szetszwo)
|
||
|
||
HDFS-4261. Fix bugs in Balaner causing infinite loop and
|
||
TestBalancerWithNodeGroup timeing out. (Junping Du via szetszwo)
|
||
|
||
HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. (Ted Yu via suresh)
|
||
|
||
HDFS-4840. ReplicationMonitor gets NPE during shutdown. (kihwal)
|
||
|
||
HDFS-4815. TestRBWBlockInvalidation: Double call countReplicas() to fetch
|
||
corruptReplicas and liveReplicas is not needed. (Tian Hong Wang via atm)
|
||
|
||
HADOOP-8957 HDFS tests for AbstractFileSystem#IsValidName should be overridden for
|
||
embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia)
|
||
|
||
HDFS-4586. TestDataDirs.testGetDataDirsFromURIs fails with all directories
|
||
in dfs.datanode.data.dir are invalid. (Ivan Mitic via atm)
|
||
|
||
HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd)
|
||
|
||
HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar
|
||
via cmccabe)
|
||
|
||
HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase.
|
||
(Brandon Li via atm)
|
||
|
||
HDFS-4845. FSNamesystem.deleteInternal should acquire write-lock before
|
||
changing the inode map. (Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4910. TestPermission failed in branch-2. (Chuan Liu via cnauroth)
|
||
|
||
HDFS-4906. HDFS Output streams should not accept writes after being
|
||
closed. (atm)
|
||
|
||
HDFS-4917. Start-dfs.sh cannot pass the parameters correctly.
|
||
(Fengdong Yu via suresh)
|
||
|
||
HDFS-4205. fsck fails with symlinks. (jlowe)
|
||
|
||
HDFS-4927. CreateEditsLog creates inodes with an invalid inode ID, which then
|
||
cannot be loaded by a namenode. (cnauroth)
|
||
|
||
HDFS-4944. WebHDFS cannot create a file path containing characters that must
|
||
be URI-encoded, such as space. (cnauroth)
|
||
|
||
HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. (Ravi Prakash via
|
||
kihwal)
|
||
|
||
HDFS-4943. WebHdfsFileSystem does not work when original file path has
|
||
encoded chars. (Jerry He via szetszwo)
|
||
|
||
HDFS-4954. In nfs, OpenFileCtx.getFlushedOffset() should handle IOException.
|
||
(Brandon Li via szetszwo)
|
||
|
||
HDFS-4948. mvn site for hadoop-hdfs-nfs fails. (brandonli)
|
||
|
||
HDFS-4887. TestNNThroughputBenchmark exits abruptly. (kihwal)
|
||
|
||
HDFS-4980. Incorrect logging.properties file for hadoop-httpfs.
|
||
(Mark Grover via suresh)
|
||
|
||
HDFS-4999. Fix TestShortCircuitLocalRead on branch-2. (cmccabe via kihwal)
|
||
|
||
HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with
|
||
JDK7. (Andrew Wang via atm)
|
||
|
||
HDFS-5003. TestNNThroughputBenchmark failed caused by existing directories.
|
||
(Xi Fang via cnauroth)
|
||
|
||
HDFS-5018. Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT
|
||
in javadoc of DatanodeInfo#isStale(). (Ted Yu via jing9)
|
||
|
||
HDFS-4602. TestBookKeeperHACheckpoints fails. (umamahesh)
|
||
|
||
HDFS-5016. Deadlock in pipeline recovery causes Datanode to be marked dead.
|
||
(suresh)
|
||
|
||
HDFS-5228. The RemoteIterator returned by DistributedFileSystem.listFiles
|
||
may throw NullPointerException. (szetszwo and cnauroth via szetszwo)
|
||
|
||
BREAKDOWN OF HDFS-347 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes.
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4354. Create DomainSocket and DomainPeer and associated unit tests.
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4356. BlockReaderLocal should use passed file descriptors rather than paths.
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4388. DomainSocket should throw AsynchronousCloseException when appropriate.
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4390. Bypass UNIX domain socket unit tests when they cannot be run.
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4400. DFSInputStream#getBlockReader: last retries should ignore the cache
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4401. Fix bug in DomainSocket path validation
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4402. Some small DomainSocket fixes: avoid findbugs warning, change
|
||
log level, etc. (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4418. increase default FileInputStreamCache size (todd)
|
||
|
||
HDFS-4416. Rename dfs.datanode.domain.socket.path to dfs.domain.socket.path
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4417. Fix case where local reads get disabled incorrectly
|
||
(Colin Patrick McCabe and todd via todd)
|
||
|
||
HDFS-4433. Make TestPeerCache not flaky (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4438. TestDomainSocket fails when system umask is set to 0002. (Colin
|
||
Patrick McCabe via atm)
|
||
|
||
HDFS-4440. Avoid annoying log message when dfs.domain.socket.path is not
|
||
set. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4453. Make a simple doc to describe the usage and design of the
|
||
shortcircuit read feature. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4496. DFSClient: don't create a domain socket unless we need it (Colin
|
||
Patrick McCabe via todd)
|
||
|
||
HDFS-347: style cleanups (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4538. Allow use of legacy blockreader (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4661. A few little code cleanups of some HDFS-347-related code. (Colin
|
||
Patrick McCabe via atm)
|
||
|
||
BREAKDOWN OF HADOOP-8562 and HDFS-3602 SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-4145. Merge hdfs cmd line scripts from branch-1-win. (David Lao,
|
||
Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh)
|
||
|
||
HDFS-4163. HDFS distribution build fails on Windows. (Chris Nauroth via
|
||
suresh)
|
||
|
||
HDFS-4316. branch-trunk-win contains test code accidentally added during
|
||
work on fixing tests on Windows. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4297. Fix issues related to datanode concurrent reading and writing on
|
||
Windows. (Arpit Agarwal, Chuan Liu via suresh)
|
||
|
||
HDFS-4573. Fix TestINodeFile on Windows. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4572. Fix TestJournal failures on Windows. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4287. HTTPFS tests fail on Windows. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4593. TestSaveNamespace fails on Windows. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4582. TestHostsFiles fails on Windows. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4603. TestMiniDFSCluster fails on Windows. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4604. TestJournalNode fails on Windows. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4607. In TestGetConf.testGetSpecificKey(), use a platform-specific
|
||
line separator; otherwise, it fails on Windows. (Ivan Mitic via szetszwo)
|
||
|
||
HDFS-4625. Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on
|
||
Windows. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4674. TestBPOfferService fails on Windows due to failure parsing
|
||
datanode data directory as URI. (Chris Nauroth via suresh)
|
||
|
||
HDFS-4615. Fix TestDFSShell failures on Windows. (Arpit Agarwal
|
||
via szetszwo)
|
||
|
||
HDFS-4584. Skip TestNNWithQJM.testNewNamenodeTakesOverWriter() on Windows.
|
||
(Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4732. Fix TestDFSUpgradeFromImage which fails on Windows due to
|
||
failure to unpack old image tarball that contains hard links.
|
||
(Chris Nauroth via szetszwo)
|
||
|
||
HDFS-4741. TestStorageRestore#testStorageRestoreFailure fails on Windows.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
HDFS-4743. TestNNStorageRetentionManager fails on Windows.
|
||
(Chris Nauroth via suresh)
|
||
|
||
HDFS-4740. Fixes for a few test failures on Windows.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
HDFS-4722. TestGetConf#testFederation times out on Windows.
|
||
(Ivan Mitic via suresh)
|
||
|
||
HDFS-4705. Address HDFS test failures on Windows because of invalid
|
||
dfs.namenode.name.dir. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and
|
||
FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh)
|
||
|
||
HDFS-4677. Editlog should support synchronous writes. (ivanmi)
|
||
|
||
HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking.
|
||
(Chris Nauroth via suresh)
|
||
|
||
HDFS-4783. TestDelegationTokensWithHA#testHAUtilClonesDelegationTokens fails
|
||
on Windows. (cnauroth)
|
||
|
||
HDFS-4818. Several HDFS tests that attempt to make directories unusable do
|
||
not work correctly on Windows. (cnauroth)
|
||
|
||
BREAKDOWN OF HDFS-2802 HDFS SNAPSHOT SUBTASKS AND RELATED JIRAS
|
||
|
||
HDFS-4076. Support snapshot of single files. (szetszwo)
|
||
|
||
HDFS-4082. Add editlog opcodes for snapshot create and delete operations.
|
||
(suresh via szetszwo)
|
||
|
||
HDFS-4086. Add editlog opcodes to allow and disallow snapshots on a
|
||
directory. (Brandon Li via suresh)
|
||
|
||
HDFS-4083. Protocol changes for snapshots. (suresh)
|
||
|
||
HDFS-4077. Add support for Snapshottable Directory. (szetszwo via suresh)
|
||
|
||
HDFS-4087. Protocol changes for listSnapshots functionality.
|
||
(Brandon Li via suresh)
|
||
|
||
HDFS-4079. Add SnapshotManager which maintains a list for all the
|
||
snapshottable directories and supports snapshot methods such as setting a
|
||
directory to snapshottable and creating a snapshot. (szetszwo)
|
||
|
||
HDFS-4078. Handle replication in snapshots. (szetszwo)
|
||
|
||
HDFS-4084. Provide CLI support to allow and disallow snapshot
|
||
on a directory. (Brondon Li via suresh)
|
||
|
||
HDFS-4091. Add snapshot quota to limit the number of snapshots allowed.
|
||
(szetszwo)
|
||
|
||
HDFS-4097. Provide CLI support for createSnapshot. (Brandon Li via suresh)
|
||
|
||
HDFS-4092. Update file deletion logic for snapshot so that the current inode
|
||
is removed from the circular linked list; and if some blocks at the end of
|
||
the block list no longer belong to any other inode, collect them and update
|
||
the block list. (szetszwo)
|
||
|
||
HDFS-4111. Support snapshot of subtrees. (szetszwo via suresh)
|
||
|
||
HDFS-4119. Complete the allowSnapshot code and add a test for it. (szetszwo)
|
||
|
||
HDFS-4133. Add testcases for testing basic snapshot functionalities.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4116. Add auditlog for some snapshot operations. (Jing Zhao via suresh)
|
||
|
||
HDFS-4095. Add some snapshot related metrics. (Jing Zhao via suresh)
|
||
|
||
HDFS-4141. Support directory diff - the difference between the current state
|
||
and a previous snapshot of an INodeDirectory. (szetszwo)
|
||
|
||
HDFS-4146. Use getter and setter in INodeFileWithLink to access blocks and
|
||
initialize root directory as snapshottable. (szetszwo)
|
||
|
||
HDFS-4149. Implement the disallowSnapshot(..) in FSNamesystem and add
|
||
resetSnapshottable(..) to SnapshotManager. (szetszwo)
|
||
|
||
HDFS-4147. When there is a snapshot in a subtree, deletion of the subtree
|
||
should fail. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4150. Update the inode in the block map when a snapshotted file or a
|
||
snapshot file is deleted. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4159. Rename should fail when the destination directory is
|
||
snapshottable and has snapshots. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4170. Add snapshot information to INodesInPath. (szetszwo)
|
||
|
||
HDFS-4177. Add a snapshot parameter to INodeDirectory.getChildrenList() for
|
||
selecting particular snapshot children list views. (szetszwo)
|
||
|
||
HDFS-4148. Disallow write/modify operations on files and directories in a
|
||
snapshot. (Brandon Li via suresh)
|
||
|
||
HDFS-4188. Add Snapshot.ID_COMPARATOR for comparing IDs and fix a bug in
|
||
ReadOnlyList.Util.binarySearch(..). (szetszwo)
|
||
|
||
HDFS-4187. Add tests for replication handling in snapshots. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4196. Support renaming of snapshots. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4175. Additional snapshot tests for more complicated directory
|
||
structure and modifications. (Jing Zhao via suresh)
|
||
|
||
HDFS-4293. Fix TestSnapshot failure. (Jing Zhao via suresh)
|
||
|
||
HDFS-4317. Change INode and its subclasses to support HDFS-4103. (szetszwo)
|
||
|
||
HDFS-4103. Support O(1) snapshot creation. (szetszwo)
|
||
|
||
HDFS-4330. Support snapshots up to the snapshot limit. (szetszwo)
|
||
|
||
HDFS-4357. Fix a bug that if an inode is replaced, further INode operations
|
||
should apply to the new inode. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4230. Support listing of all the snapshottable directories. (Jing Zhao
|
||
via szetszwo)
|
||
|
||
HDFS-4244. Support snapshot deletion. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4245. Include snapshot related operations in TestOfflineEditsViewer.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4395. In INodeDirectorySnapshottable's constructor, the passed-in dir
|
||
could be an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4397. Fix a bug in INodeDirectoryWithSnapshot.Diff.combinePostDiff(..)
|
||
that it may put the wrong node into the deleted list. (szetszwo)
|
||
|
||
HDFS-4407. Change INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) to
|
||
merge-sort like and keep the postDiff parameter unmodified. (szetszwo)
|
||
|
||
HDFS-4098. Add FileWithSnapshot, INodeFileUnderConstructionWithSnapshot and
|
||
INodeFileUnderConstructionSnapshot for supporting append to snapshotted
|
||
files. (szetszwo)
|
||
|
||
HDFS-4126. Add reading/writing snapshot information to FSImage.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4436. Change INode.recordModification(..) to return only the current
|
||
inode and remove the updateCircularList parameter from some methods in
|
||
INodeDirectoryWithSnapshot.Diff. (szetszwo)
|
||
|
||
HDFS-4429. When the latest snapshot exists, INodeFileUnderConstruction
|
||
should be replaced with INodeFileWithSnapshot but not INodeFile.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4441. Move INodeDirectoryWithSnapshot.Diff and the related classes to a
|
||
package. (szetszwo)
|
||
|
||
HDFS-4432. Support INodeFileUnderConstructionWithSnapshot in FSImage
|
||
saving/loading. (Jing Zhao via suresh)
|
||
|
||
HDFS-4131. Add capability to namenode to get snapshot diff. (Jing Zhao via
|
||
suresh)
|
||
|
||
HDFS-4447. Refactor INodeDirectoryWithSnapshot for supporting general INode
|
||
diff lists. (szetszwo)
|
||
|
||
HDFS-4189. Renames the getMutableXxx methods to getXxx4Write and fix a bug
|
||
that some getExistingPathINodes calls should be getINodesInPath4Write.
|
||
(szetszwo)
|
||
|
||
HDFS-4361. When listing snapshottable directories, only return those
|
||
where the user has permission to take snapshots. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4464. Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot
|
||
and rename it to destroySubtreeAndCollectBlocks. (szetszwo)
|
||
|
||
HDFS-4414. Add support for getting snapshot diff from DistributedFileSystem.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4446. Support file snapshots with diff lists. (szetszwo)
|
||
|
||
HDFS-4480. Eliminate the file snapshot circular linked list. (szetszwo)
|
||
|
||
HDFS-4481. Change fsimage to support snapshot file diffs. (szetszwo)
|
||
|
||
HDFS-4500. Refactor snapshot INode methods. (szetszwo)
|
||
|
||
HDFS-4487. Fix snapshot diff report for HDFS-4446. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4431. Support snapshot in OfflineImageViewer. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4503. Update computeContentSummary(..), spaceConsumedInTree(..) and
|
||
diskspaceConsumed(..) in INode for snapshot. (szetszwo)
|
||
|
||
HDFS-4499. Fix file/directory/snapshot deletion for file diff. (Jing Zhao
|
||
via szetszwo)
|
||
|
||
HDFS-4524. Update SnapshotManager#snapshottables when loading fsimage.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4520. Support listing snapshots under a snapshottable directory using
|
||
ls. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4514. Add CLI for supporting snapshot rename, diff report, and
|
||
snapshottable directory listing. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4523. Fix INodeFile replacement, TestQuota and javac errors from trunk
|
||
merge. (szetszwo)
|
||
|
||
HDFS-4507. Update quota verification for snapshots. (szetszwo)
|
||
|
||
HDFS-4545. With snapshots, FSDirectory.unprotectedSetReplication(..) always
|
||
changes file replication but it may or may not changes block replication.
|
||
(szetszwo)
|
||
|
||
HDFS-4557. Fix FSDirectory#delete when INode#cleanSubtree returns 0.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4579. Annotate snapshot tests. (Arpit Agarwal via suresh)
|
||
|
||
HDFS-4574. Move Diff to the util package. (szetszwo)
|
||
|
||
HDFS-4563. Update namespace/diskspace usage after deleting snapshots.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4144. Create test for all snapshot-related metrics.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4556. Add snapshotdiff and LsSnapshottableDir tools to hdfs script.
|
||
(Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4534. Add INodeReference in order to support rename with snapshots.
|
||
(szetszwo)
|
||
|
||
HDFS-4616. Update the FilesDeleted metric while deleting file/dir in the
|
||
current tree. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4627. Fix FSImageFormat#Loader NPE and synchronization issues.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4612. Not to use INode.getParent() when generating snapshot diff
|
||
report. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4636. Update quota usage when deleting files/dirs that were created
|
||
after taking the latest snapshot. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4648. For snapshot deletion, when merging the diff from to-delete
|
||
snapshot to the prior snapshot, make sure files/directories created after
|
||
the prior snapshot get deleted. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4637. INodeDirectory#replaceSelf4Quota may incorrectly convert a newly
|
||
created directory to an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4611. Update FSImage for INodeReference. (szetszwo)
|
||
|
||
HDFS-4647. Rename should call setLocalName after an inode is removed from
|
||
snapshots. (Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4684. Use INode id for image serialization when writing INodeReference.
|
||
(szetszwo)
|
||
|
||
HDFS-4675. Fix rename across snapshottable directories. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4692. Use timestamp as default snapshot names. (szetszwo)
|
||
|
||
HDFS-4666. Define ".snapshot" as a reserved inode name so that users cannot
|
||
create a file/directory with ".snapshot" as the name. If ".snapshot" is
|
||
used in a previous version of HDFS, it must be renamed before upgrade;
|
||
otherwise, upgrade will fail. (szetszwo)
|
||
|
||
HDFS-4700. Fix the undo section of rename with snapshots. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4529. Disallow concat when one of the src files is in some snapshot.
|
||
(szetszwo)
|
||
|
||
HDFS-4550. Refactor INodeDirectory.INodesInPath to a standalone class.
|
||
(szetszwo)
|
||
|
||
HDFS-4707. Add snapshot methods to FilterFileSystem and fix findbugs
|
||
warnings. (szetszwo)
|
||
|
||
HDFS-4706. Do not replace root inode for disallowSnapshot. (szetszwo)
|
||
|
||
HDFS-4717. Change the path parameter type of the snapshot methods in
|
||
HdfsAdmin from String to Path. (szetszwo)
|
||
|
||
HDFS-4708. Add snapshot user documentation. (szetszwo)
|
||
|
||
HDFS-4726. Fix test failures after merging the INodeId-INode mapping
|
||
from trunk. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4727. Update inodeMap after deleting files/directories/snapshots.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4719. Remove AbstractINodeDiff.Factory and move its methods to
|
||
AbstractINodeDiffList. (Arpit Agarwal via szetszwo)
|
||
|
||
HDFS-4735. DisallowSnapshot throws IllegalStateException for nested
|
||
snapshottable directories. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4738. Changes AbstractINodeDiff to implement Comparable<Integer>, and
|
||
fix javadoc and other warnings. (szetszwo)
|
||
|
||
HDFS-4686. Update quota computation for rename and INodeReference.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4729. Fix OfflineImageViewer and permission checking for snapshot
|
||
operations. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4749. Use INodeId to identify the corresponding directory node in
|
||
FSImage saving/loading. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4742. Fix appending to a renamed file with snapshot. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4755. Fix AccessControlException message and moves "implements
|
||
LinkedElement" from INode to INodeWithAdditionalFields. (szetszwo)
|
||
|
||
HDFS-4650. Fix a bug in FSDirectory and add more unit tests for rename with
|
||
existence of snapshottable directories and snapshots. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4650. When passing two non-existing snapshot names to snapshotDiff, it
|
||
returns success if the names are the same. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4767. If a directory is snapshottable, do not replace it when clearing
|
||
quota. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4578. Restrict snapshot IDs to 24-bit wide. (Arpit Agarwal via
|
||
szetszwo)
|
||
|
||
HDFS-4773. Fix bugs in quota usage computation and OfflineImageViewer.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4760. Update inodeMap after node replacement. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4758. Disallow nested snapshottable directories and unwrap
|
||
RemoteException. (szetszwo)
|
||
|
||
HDFS-4781. Fix a NullPointerException when listing .snapshot under
|
||
a non-existing directory. (szetszwo)
|
||
|
||
HDFS-4791. Update and fix deletion of reference inode. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4798. Update computeContentSummary() for the reference nodes in
|
||
snapshots. (szetszwo)
|
||
|
||
HDFS-4800. Fix INodeDirectoryWithSnapshot#cleanDeletedINode. (Jing Zhao via
|
||
szetszwo)
|
||
|
||
HDFS-4801. lsSnapshottableDir throws IllegalArgumentException when root is
|
||
snapshottable. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4802. Disallowing snapshot on / twice should throw SnapshotException
|
||
but not IllegalStateException. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4806. In INodeDirectoryWithSnapshot, use isInLatestSnapshot() to
|
||
determine if an added/removed child should be recorded in the snapshot diff.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-4809. When a QuotaExceededException is thrown during rename, the quota
|
||
usage should be subtracted back. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4842. Identify the correct prior snapshot when deleting a
|
||
snapshot under a renamed subtree. (jing9)
|
||
|
||
HDFS-4846. Clean up snapshot CLI commands output stacktrace for invalid
|
||
arguments. (Jing Zhao via brandonli)
|
||
|
||
HDFS-4857. Snapshot.Root and AbstractINodeDiff#snapshotINode should not be
|
||
put into INodeMap when loading FSImage. (jing9)
|
||
|
||
HDFS-4863. The root directory should be added to the snapshottable
|
||
directory list while loading fsimage. (jing9)
|
||
|
||
HDFS-4848. copyFromLocal and renaming a file to ".snapshot" should output
|
||
that ".snapshot" is a reserved name. (Jing Zhao via brandonli)
|
||
|
||
HDFS-4826. TestNestedSnapshots times out due to repeated slow edit log
|
||
flushes when running on virtualized disk. (Chris Nauroth via szetszwo)
|
||
|
||
HDFS-4876. Fix the javadoc of FileWithSnapshot and move FileDiffList to
|
||
FileWithSnapshot. (szetszwo)
|
||
|
||
HDFS-4850. Fix OfflineImageViewer to work on fsimages with empty files or
|
||
snapshots. (jing9)
|
||
|
||
HDFS-4877. Snapshot: fix the scenario where a directory is renamed under
|
||
its prior descendant. (jing9)
|
||
|
||
HDFS-4873. callGetBlockLocations returns incorrect number of blocks for
|
||
snapshotted files. (jing9)
|
||
|
||
HDFS-4819. Update Snapshot doc to clarify that nested snapshots are not
|
||
allowed. (szetszwo)
|
||
|
||
HDFS-4902. DFSClient.getSnapshotDiffReport should use string path rather
|
||
than o.a.h.fs.Path. (Binglin Chang via jing9)
|
||
|
||
HDFS-4875. Add a test for testing snapshot file length.
|
||
(Arpit Agarwal via jing9)
|
||
|
||
HDFS-4841. FsShell commands using secure webhfds fail ClientFinalizer
|
||
shutdown hook. (rkanter via tucu)
|
||
|
||
HDFS-4951. FsShell commands using secure httpfs throw exceptions due
|
||
to missing TokenRenewer. (rknater via tucu)
|
||
|
||
HDFS-4969. WebhdfsFileSystem expects non-standard WEBHDFS Json element.
|
||
(rkanter via tucu)
|
||
|
||
HDFS-4797. BlockScanInfo does not override equals(..) and hashCode()
|
||
consistently. (szetszwo)
|
||
|
||
HDFS-4978. Make disallowSnapshot idempotent. (jing9)
|
||
|
||
HDFS-5005. Move SnapshotException and SnapshotAccessControlException
|
||
to o.a.h.hdfs.protocol. (jing9)
|
||
|
||
HDFS-4982. JournalNode should relogin from keytab before fetching logs
|
||
from other JNs (todd)
|
||
|
||
Release 2.0.6-alpha - 08/22/2013
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
Release 2.0.5-alpha - 06/06/2013
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
Release 2.0.4-alpha - 2013-04-25
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
Release 2.0.3-alpha - 2013-02-06
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages.
|
||
(suresh)
|
||
|
||
HDFS-4362. GetDelegationTokenResponseProto does not handle null token.
|
||
(suresh)
|
||
|
||
HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null
|
||
response. (suresh)
|
||
|
||
HDFS-4364. GetLinkTargetResponseProto does not handle null path. (suresh)
|
||
|
||
HDFS-4369. GetBlockKeysResponseProto does not handle null response.
|
||
(suresh)
|
||
|
||
HDFS-4451. hdfs balancer command returns exit code 1 on success instead
|
||
of 0. (Joshua Blatt via suresh)
|
||
|
||
HDFS-4350. Make enabling of stale marking on read and write paths
|
||
independent. (Andrew Wang via suresh)
|
||
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS.
|
||
(Jaimin D Jetly and Jing Zhao via szetszwo)
|
||
|
||
HDFS-3912. Detect and avoid stale datanodes for writes.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh)
|
||
|
||
HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd)
|
||
|
||
HDFS-4213. Add an API to hsync for updating the last block length at the
|
||
namenode. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for
|
||
reliably storing HDFS edit logs. See dedicated section below for breakdown
|
||
of subtasks.
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-3925. Prettify PipelineAck#toString() for printing to a log
|
||
(Andrew Wang via todd)
|
||
|
||
HDFS-3939. NN RPC address cleanup. (eli)
|
||
|
||
HDFS-3373. Change DFSClient input stream socket cache to global static and
|
||
add a thread to cleanup expired cache entries. (John George via szetszwo)
|
||
|
||
HDFS-3896. Add descriptions for dfs.namenode.rpc-address and
|
||
dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm)
|
||
|
||
HDFS-3996. Add debug log removed in HDFS-3873 back. (eli)
|
||
|
||
HDFS-3916. libwebhdfs (C client) code cleanups.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3813. Log error message if security and WebHDFS are enabled but
|
||
principal/keytab are not configured. (Stephen Chu via atm)
|
||
|
||
HDFS-3483. Better error message when hdfs fsck is run against a ViewFS
|
||
config. (Stephen Fritz via atm)
|
||
|
||
HDFS-3682. MiniDFSCluster#init should provide more info when it fails.
|
||
(todd via eli)
|
||
|
||
HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli)
|
||
|
||
HDFS-4007. Rehabilitate bit-rotted unit tests under
|
||
hadoop-hdfs-project/hadoop-hdfs/src/test/unit/
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external
|
||
sh script. (Chris Nauroth via suresh)
|
||
|
||
HADOOP-8911. CRLF characters in source and text files.
|
||
(Raja Aluri via suresh)
|
||
|
||
HDFS-4037. Rename the getReplication() method in BlockCollection to
|
||
getBlockReplication(). (szetszwo)
|
||
|
||
HDFS-4036. Remove "throws UnresolvedLinkException" from
|
||
FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo)
|
||
|
||
HDFS-2946. HA: Put a cap on the number of completed edits files retained
|
||
by the NN. (atm)
|
||
|
||
HDFS-4029. GenerationStamp should use an AtomicLong. (eli)
|
||
|
||
HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli)
|
||
|
||
HDFS-4073. Two minor improvements to FSDirectory. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4074. Remove the unused default constructor from INode. (Brandon Li
|
||
via szetszwo)
|
||
|
||
HDFS-4088. Remove "throws QuotaExceededException" from an
|
||
INodeDirectoryWithQuota constructor. (szetszwo)
|
||
|
||
HDFS-4099. Clean up replication code and add more javadoc. (szetszwo)
|
||
|
||
HDFS-4107. Add utility methods for casting INode to INodeFile and
|
||
INodeFileUnderConstruction. (szetszwo)
|
||
|
||
HDFS-4112. A few improvements on INodeDirectory include adding a utility
|
||
method for casting; avoiding creation of new empty lists; cleaning up
|
||
some code and rewriting some javadoc. (szetszwo)
|
||
|
||
HDFS-4121. Add namespace declarations in hdfs .proto files for languages
|
||
other than java. (Binglin Chang via suresh)
|
||
|
||
HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers.
|
||
(todd and ivank via umamahesh)
|
||
|
||
HDFS-3695. Genericize format() to non-file JournalManagers.
|
||
(todd via umamahesh)
|
||
|
||
HDFS-3789. JournalManager#format() should be able to throw IOException.
|
||
(Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3809. Make BKJM use protobufs for all serialization with ZK.
|
||
(Ivan Kelly via umamhesh)
|
||
|
||
HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh)
|
||
|
||
HDFS-4143. Change blocks to private in INodeFile and renames isLink() to
|
||
isSymlink() in INode. (szetszwo)
|
||
|
||
HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in
|
||
C/C++. (Binglin Chang via suresh)
|
||
|
||
HDFS-4048. Use ERROR instead of INFO for volume failure logs.
|
||
(Stephen Chu via eli)
|
||
|
||
HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-4038. Override toString() for BookKeeperEditLogInputStream.
|
||
(Vinay via umamahesh)
|
||
|
||
HDFS-4214. OfflineEditsViewer should print out the offset at which it
|
||
encountered an error. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4199. Provide test for HdfsVolumeId. (Ivan A. Veselovsky via atm)
|
||
|
||
HDFS-3049. During the normal NN startup process, fall back on a different
|
||
edit log if we see one that is corrupt (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd)
|
||
|
||
HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh)
|
||
|
||
HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm)
|
||
|
||
HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd)
|
||
|
||
HDFS-4268. Remove redundant enum NNHAStatusHeartbeat.State. (shv)
|
||
|
||
HDFS-3680. Allow customized audit logging in HDFS FSNamesystem. (Marcelo
|
||
Vanzin via atm)
|
||
|
||
HDFS-4130. BKJM: The reading for editlog at NN starting using bkjm is not efficient.
|
||
(Han Xiao via umamahesh)
|
||
|
||
HDFS-4326. bump up Tomcat version for HttpFS to 6.0.36. (tucu via acmurthy)
|
||
|
||
HDFS-4270. Introduce soft and hard limits for max replication so that
|
||
replications of the highest priority are allowed to choose a source datanode
|
||
that has reached its soft limit but not the hard limit. (Derek Dagit via
|
||
szetszwo)
|
||
|
||
HADOOP-9173. Add security token protobuf definition to common and
|
||
use it in hdfs. (suresh)
|
||
|
||
HDFS-4030. BlockManager excessBlocksCount and
|
||
postponedMisreplicatedBlocksCount should be AtomicLongs. (eli)
|
||
|
||
HDFS-4031. Update findbugsExcludeFile.xml to include findbugs 2
|
||
exclusions. (eli)
|
||
|
||
HDFS-4033. Miscellaneous findbugs 2 fixes. (eli)
|
||
|
||
HDFS-4034. Remove redundant null checks. (eli)
|
||
|
||
HDFS-4035. LightWeightGSet and LightWeightHashSet increment a
|
||
volatile without synchronization. (eli)
|
||
|
||
HDFS-4032. Specify the charset explicitly rather than rely on the
|
||
default. (eli)
|
||
|
||
HDFS-4363. Combine PBHelper and HdfsProtoUtil and remove redundant
|
||
methods. (suresh)
|
||
|
||
HDFS-4377. Some trivial DN comment cleanup. (eli)
|
||
|
||
HDFS-4381. Document fsimage format details in FSImageFormat class javadoc.
|
||
(Jing Zhao via suresh)
|
||
|
||
HDFS-4375. Use token request messages defined in hadoop common.
|
||
(suresh)
|
||
|
||
HDFS-4392. Use NetUtils#getFreeSocketPort in MiniDFSCluster.
|
||
(Andrew Purtell via suresh)
|
||
|
||
HDFS-4393. Make empty request and responses in protocol translators can be
|
||
static final members. (Brandon Li via suresh)
|
||
|
||
HDFS-4403. DFSClient can infer checksum type when not provided by reading
|
||
first byte (todd)
|
||
|
||
HDFS-4259. Improve pipeline DN replacement failure message (harsh)
|
||
|
||
HDFS-3598. WebHDFS support for file concat. (Plamen Jeliazkov via shv)
|
||
|
||
HDFS-4456. Add concat to HttpFS and WebHDFS REST API docs. (plamenj2003 via tucu)
|
||
|
||
HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-3429. DataNode reads checksums even if client does not need them (todd)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever.
|
||
(Andy Isaacson via eli)
|
||
|
||
HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm)
|
||
|
||
HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli)
|
||
|
||
HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu)
|
||
|
||
HDFS-3949. NameNodeRpcServer#join should join on both client and
|
||
server RPC servers. (eli)
|
||
|
||
HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken.
|
||
(Andy Isaacson via eli)
|
||
|
||
HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli)
|
||
|
||
HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown()
|
||
sometimes throws NPEs. (Ivan A. Veselovsky via atm)
|
||
|
||
HDFS-3753. Tests don't run with native libraries.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers
|
||
must be direct buffers". (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu)
|
||
|
||
HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage
|
||
occasionally fails due to unexpected exit. (todd via eli)
|
||
|
||
HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some
|
||
cluster cleanup. (eli)
|
||
|
||
HDFS-4020. TestRBWBlockInvalidation may time out. (eli)
|
||
|
||
HDFS-4021. Misleading error message when resources are low on the NameNode.
|
||
(Christopher Conner via atm)
|
||
|
||
HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files.
|
||
(Binglin Chang via suresh)
|
||
|
||
HDFS-4049. Fix hflush performance regression due to nagling delays
|
||
(todd)
|
||
|
||
HDFS-3678. Edit log files are never being purged from 2NN. (atm)
|
||
|
||
HDFS-4058. DirectoryScanner may fail with IOOB if the directory
|
||
scanning threads return out of volume order. (eli)
|
||
|
||
HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli)
|
||
|
||
HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli)
|
||
|
||
HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when
|
||
visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm)
|
||
|
||
HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli)
|
||
|
||
HDFS-4072. On file deletion remove corresponding blocks pending
|
||
replications. (Jing Zhao via suresh)
|
||
|
||
HDFS-4022. Replication not happening for appended block.
|
||
(Vinay via umamahesh)
|
||
|
||
HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the
|
||
out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao
|
||
via szetszwo)
|
||
|
||
HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads
|
||
may not be shutdown properly in DataNode. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-4127. Log message is not correct in case of short of replica.
|
||
(Junping Du via suresh)
|
||
|
||
HADOOP-8994. TestDFSShell creates file named "noFileHere", making further
|
||
tests hard to understand (Andy Isaacson via daryn)
|
||
|
||
HDFS-3804. TestHftpFileSystem fails intermittently with JDK7
|
||
(Trevor Robinson via daryn)
|
||
|
||
HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees
|
||
uninitialized memory (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn)
|
||
|
||
HDFS-3979. For hsync, datanode should wait for the local sync to complete
|
||
before sending ack. (Lars Hofhansl via szetszwo)
|
||
|
||
HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3625. Fix TestBackupNode by properly initializing edit log during
|
||
startup. (Junping Du via todd)
|
||
|
||
HDFS-4138. BackupNode startup fails due to uninitialized edit log.
|
||
(Kihwal Lee via shv)
|
||
|
||
HDFS-4162. Some malformed and unquoted HTML strings are returned from
|
||
datanode web ui. (Darek Dagit via suresh)
|
||
|
||
HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3921. NN will prematurely consider blocks missing when entering active
|
||
state while still in safe mode. (atm)
|
||
|
||
HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and
|
||
lastDeletedReport should be volatile. (Jing Zhao via suresh)
|
||
|
||
HDFS-4139. fuse-dfs RO mode still allows file truncation.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory
|
||
(Andy Isaacson via daryn)
|
||
|
||
HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead.
|
||
(umamahesh)
|
||
|
||
HDFS-4100. Fix all findbug security warings. (Liang Xie via eli)
|
||
|
||
HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode.
|
||
(Vinay via atm)
|
||
|
||
HDFS-4105. The SPNEGO user for secondary namenode should use the web
|
||
keytab. (Arpit Gupta via jitendra)
|
||
|
||
HDFS-4156. Seeking to a negative position should throw an IOE.
|
||
(Eli Reisman via eli)
|
||
|
||
HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user
|
||
names. (tucu)
|
||
|
||
HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn)
|
||
|
||
HDFS-4179. BackupNode: allow reads, fix checkpointing, safeMode. (shv)
|
||
|
||
HDFS-4216. Do not ignore QuotaExceededException when adding symlinks.
|
||
(szetszwo)
|
||
|
||
HDFS-4242. Map.Entry is incorrectly used in LeaseManager since the behavior
|
||
of it is undefined after the iteration or modifications of the map.
|
||
(szetszwo)
|
||
|
||
HDFS-4231. BackupNode: Introduce BackupState. (shv)
|
||
|
||
HDFS-4238. Standby namenode should not do purging of shared
|
||
storage edits. (todd)
|
||
|
||
HDFS-4282. TestEditLog.testFuzzSequences FAILED in all pre-commit test
|
||
(todd)
|
||
|
||
HDFS-4236. Remove artificial limit on username length introduced in
|
||
HDFS-4171. (tucu via suresh)
|
||
|
||
HDFS-4279. NameNode does not initialize generic conf keys when started
|
||
with -recover. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4291. edit log unit tests leave stray test_edit_log_file around
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-4292. Sanity check not correct in RemoteBlockReader2.newBlockReader
|
||
(Binglin Chang via todd)
|
||
|
||
HDFS-4295. Using port 1023 should be valid when starting Secure DataNode
|
||
(Stephen Chu via todd)
|
||
|
||
HDFS-4294. Backwards compatibility is not maintained for TestVolumeId.
|
||
(Ivan A. Veselovsky and Robert Parker via atm)
|
||
|
||
HDFS-2264. NamenodeProtocol has the wrong value for clientPrincipal in
|
||
KerberosInfo annotation. (atm)
|
||
|
||
HDFS-4307. SocketCache should use monotonic time. (Colin Patrick McCabe
|
||
via atm)
|
||
|
||
HDFS-4315. DNs with multiple BPs can have BPOfferServices fail to start
|
||
due to unsynchronized map access. (atm)
|
||
|
||
HDFS-4140. fuse-dfs handles open(O_TRUNC) poorly. (Colin Patrick McCabe
|
||
via atm)
|
||
|
||
HDFS-4308. addBlock() should persist file blocks once.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-4347. Avoid infinite waiting checkpoint to complete in TestBackupNode.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-4349. Add test for reading files from BackupNode. (shv)
|
||
|
||
HDFS-4302. Fix fatal exception when starting NameNode with DEBUG logs
|
||
(Eugene Koontz via todd)
|
||
|
||
HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad
|
||
VERSION file. (Vinay and Andrew Wang via atm)
|
||
|
||
HDFS-4306. PBHelper.convertLocatedBlock miss convert BlockToken. (Binglin
|
||
Chang via atm)
|
||
|
||
HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be
|
||
initialized. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. (Chris Nauroth
|
||
via atm)
|
||
|
||
HDFS-4274. BlockPoolSliceScanner does not close verification log during
|
||
shutdown. (Chris Nauroth via suresh)
|
||
|
||
HDFS-1245. Pluggable block id generation. (shv)
|
||
|
||
HDFS-4415. HostnameFilter should handle hostname resolution failures and
|
||
continue processing. (Robert Kanter via atm)
|
||
|
||
HDFS-4359. Slow RPC responses from NN can prevent metrics collection on
|
||
DNs. (liang xie via atm)
|
||
|
||
HDFS-4444. Add space between total transaction time and number of
|
||
transactions in FSEditLog#printStatistics. (Stephen Chu via suresh)
|
||
|
||
HDFS-4428. FsDatasetImpl should disclose what the error is when a rename
|
||
fails. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-4452. getAdditionalBlock() can create multiple blocks if the client
|
||
times out and retries. (shv)
|
||
|
||
HDFS-4445. All BKJM ledgers are not checked while tailing, So failover will fail.
|
||
(Vinay via umamahesh)
|
||
|
||
HDFS-4462. 2NN will fail to checkpoint after an HDFS upgrade from a
|
||
pre-federation version of HDFS. (atm)
|
||
|
||
HDFS-4404. Create file failure when the machine of first attempted NameNode
|
||
is down. (Todd Lipcon via atm)
|
||
|
||
HDFS-4344. dfshealth.jsp throws NumberFormatException when
|
||
dfs.hosts/dfs.hosts.exclude includes port number. (Andy Isaacson via atm)
|
||
|
||
HDFS-4468. Use the new StringUtils methods added by HADOOP-9252 and fix
|
||
TestHDFSCLI and TestQuota. (szetszwo)
|
||
|
||
HDFS-4458. In DFSUtil.getNameServiceUris(..), convert default fs URI using
|
||
NetUtils.createSocketAddr(..) for being consistent with other addresses.
|
||
(Binglin Chang via szetszwo)
|
||
|
||
BREAKDOWN OF HDFS-3077 SUBTASKS
|
||
|
||
HDFS-3077. Quorum-based protocol for reading and writing edit logs.
|
||
(todd, Brandon Li, and Hari Mankude via todd)
|
||
|
||
HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd)
|
||
|
||
HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs
|
||
(todd)
|
||
|
||
HDFS-3693. JNStorage should read its storage info even before a writer
|
||
becomes active (todd)
|
||
|
||
HDFS-3725. Fix QJM startup when individual JNs have gaps (todd)
|
||
|
||
HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd)
|
||
|
||
HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm)
|
||
|
||
HDFS-3793. Implement genericized format() in QJM (todd)
|
||
|
||
HDFS-3795. QJM: validate journal dir at startup (todd)
|
||
|
||
HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid
|
||
segment (todd)
|
||
|
||
HDFS-3799. QJM: handle empty log segments during recovery (todd)
|
||
|
||
HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd)
|
||
|
||
HDFS-3800. improvements to QJM fault testing (todd)
|
||
|
||
HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed
|
||
setting of HTTP port. (todd and atm)
|
||
|
||
HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd
|
||
and atm)
|
||
|
||
HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode"
|
||
(eli)
|
||
|
||
HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd)
|
||
|
||
HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli)
|
||
|
||
HDFS-3863. Track last "committed" txid in QJM (todd)
|
||
|
||
HDFS-3869. Expose non-file journal manager details in web UI (todd)
|
||
|
||
HDFS-3884. Journal format() should reset cached values (todd)
|
||
|
||
HDFS-3870. Add metrics to JournalNode (todd)
|
||
|
||
HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd)
|
||
|
||
HDFS-3726. If a logger misses an RPC, don't retry that logger until next
|
||
segment (todd)
|
||
|
||
HDFS-3893. QJM: Make QJM work with security enabled. (atm)
|
||
|
||
HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm)
|
||
|
||
HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd)
|
||
|
||
HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd)
|
||
|
||
HDFS-3900. QJM: avoid validating log segments on log rolls (todd)
|
||
|
||
HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are
|
||
out-of-sync (todd)
|
||
|
||
HDFS-3899. QJM: Add client-side metrics (todd)
|
||
|
||
HDFS-3914. QJM: acceptRecovery should abort current segment (todd)
|
||
|
||
HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd)
|
||
|
||
HDFS-3906. QJM: quorum timeout on failover with large log segment (todd)
|
||
|
||
HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error
|
||
before being formatted (todd)
|
||
|
||
HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC
|
||
client caching (todd)
|
||
|
||
HDFS-3926. QJM: Add user documentation for QJM. (atm)
|
||
|
||
HDFS-3943. QJM: remove currently-unused md5sum field (todd)
|
||
|
||
HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd)
|
||
|
||
HDFS-3955. QJM: Make acceptRecovery() atomic. (todd)
|
||
|
||
HDFS-3956. QJM: purge temporary files when no longer within retention
|
||
period (todd)
|
||
|
||
HDFS-4004. TestJournalNode#testJournal fails because of test case execution
|
||
order (Chao Shi via todd)
|
||
|
||
HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet
|
||
(Chao Shi via todd)
|
||
|
||
HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas
|
||
needs to be updated when avoiding stale nodes. (Andrew Wang via szetszwo)
|
||
|
||
HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li
|
||
via tgraves)
|
||
|
||
HDFS-4399. Fix RAT warnings by excluding images sub-dir in docs. (Thomas
|
||
Graves via acmurthy)
|
||
|
||
Release 2.0.2-alpha - 2012-09-07
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-3446. HostsFileReader silently ignores bad includes/excludes
|
||
(Matthew Jacobs via todd)
|
||
|
||
HDFS-3755. Creating an already-open-for-write file with overwrite=true fails
|
||
(todd)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo)
|
||
|
||
HDFS-3042. Automatic failover support for NameNode HA (todd)
|
||
(see dedicated section below for breakdown of subtasks)
|
||
|
||
HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if
|
||
the given HDFS is healthy. (szetszwo)
|
||
|
||
HDFS-3113. httpfs does not support delegation tokens. (tucu)
|
||
|
||
HDFS-3513. HttpFS should cache filesystems. (tucu)
|
||
|
||
HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm)
|
||
|
||
HDFS-3150. Add option for clients to contact DNs via hostname. (eli)
|
||
|
||
HDFS-2793. Add an admin command to trigger an edit log roll. (todd)
|
||
|
||
HDFS-3703. Datanodes are marked stale if heartbeat is not received in
|
||
configured timeout and are selected as the last location to read from.
|
||
(Jing Zhao via suresh)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak
|
||
via atm)
|
||
|
||
HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG
|
||
logging is enabled. (atm)
|
||
|
||
HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of
|
||
final releases. (todd)
|
||
|
||
HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and
|
||
BlocksMap to {get|set|add}BlockCollection(..). (John George via szetszwo)
|
||
|
||
HDFS-3134. harden edit log loader against malformed or malicious input.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3230. Cleanup DatanodeID creation in the tests. (eli)
|
||
|
||
HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli)
|
||
|
||
HDFS-3400. DNs should be able start with jsvc even if security is disabled.
|
||
(atm via eli)
|
||
|
||
HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch
|
||
from request. (atm)
|
||
|
||
HDFS-3335. check for edit log corruption at the end of the log
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be
|
||
consistent with Datanode. (eli)
|
||
|
||
HDFS-3416. Cleanup DatanodeID and DatanodeRegistration
|
||
constructors used by testing. (eli)
|
||
|
||
HDFS-3419. Cleanup LocatedBlock. (eli)
|
||
|
||
HDFS-3440. More effectively limit stream memory consumption when reading
|
||
corrupt edit logs (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-3438. BootstrapStandby should not require a rollEdits on active node
|
||
(todd)
|
||
|
||
HDFS-2885. Remove "federation" from the nameservice config options.
|
||
(Tsz Wo (Nicholas) Sze via eli)
|
||
|
||
HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time
|
||
ClassCastException check is useless since generic type information is only
|
||
available in compile-time. (szetszwo)
|
||
|
||
HDFS-3454. Balancer unconditionally logs InterruptedException at
|
||
INFO level on shutdown if security is enabled. (eli)
|
||
|
||
HDFS-1013. Miscellaneous improvements to HTML markup for web UIs
|
||
(Eugene Koontz via todd)
|
||
|
||
HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package
|
||
private. (szetszwo)
|
||
|
||
HDFS-3520. Add transfer rate logging to TransferFsImage. (eli)
|
||
|
||
HDFS-3504. Support configurable retry policy in DFSClient for RPC
|
||
connections and RPC calls, and add MultipleLinearRandomRetry, a new retry
|
||
policy. (szetszwo)
|
||
|
||
HDFS-3372. offlineEditsViewer should be able to read a binary
|
||
edits file with recovery mode. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3516. Check content-type in WebHdfsFileSystem. (szetszwo)
|
||
|
||
HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli)
|
||
|
||
HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu)
|
||
|
||
HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd)
|
||
|
||
HDFS-3475. Make the replication monitor multipliers configurable.
|
||
(harsh via eli)
|
||
|
||
HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd)
|
||
|
||
HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via
|
||
todd)
|
||
|
||
HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli)
|
||
|
||
HDFS-2988. Improve error message when storage directory lock fails
|
||
(Miomir Boljanovic via harsh)
|
||
|
||
HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere.
|
||
(harsh)
|
||
|
||
HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on
|
||
corrupted block. (Henry Robinson via atm)
|
||
|
||
HDFS-3555. idle client socket triggers DN ERROR log
|
||
(should be INFO or DEBUG). (Andy Isaacson via harsh)
|
||
|
||
HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm)
|
||
|
||
HDFS-3629. Fix the typo in the error message about inconsistent
|
||
storage layout version. (Brandon Li via harsh)
|
||
|
||
HDFS-3613. GSet prints some INFO level values, which aren't
|
||
really very useful to all (Andrew Wang via harsh)
|
||
|
||
HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping
|
||
a few bytes. (Colin Patrick McCabe via harsh)
|
||
|
||
HDFS-3582. Hook daemon process exit for testing. (eli)
|
||
|
||
HDFS-3641. Move server Util time methods to common and use now
|
||
instead of System#currentTimeMillis. (eli)
|
||
|
||
HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3306. fuse_dfs: don't lock release operations.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3612. Single namenode image directory config warning can
|
||
be improved. (Andy Isaacson via harsh)
|
||
|
||
HDFS-3606. libhdfs: create self-contained unit test.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3663. MiniDFSCluster should capture the code path that led to
|
||
the first ExitException. (eli)
|
||
|
||
HDFS-3659. Add missing @Override to methods across the hadoop-hdfs
|
||
project. (Brandon Li via harsh)
|
||
|
||
HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3665. Add a test for renaming across file systems via a symlink. (eli)
|
||
|
||
HDFS-3666. Plumb more exception messages to terminate. (eli)
|
||
|
||
HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-1249. With fuse-dfs, chown which only has owner (or only group)
|
||
argument fails with Input/output error. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm)
|
||
|
||
HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm)
|
||
|
||
HDFS-3650. Use MutableQuantiles to provide latency histograms for various
|
||
operations. (Andrew Wang via atm)
|
||
|
||
HDFS-3667. Add retry support to WebHdfsFileSystem. (szetszwo)
|
||
|
||
HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop
|
||
filesystem (tucu)
|
||
|
||
HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-3190. Simple refactors in existing NN code to assist
|
||
QuorumJournalManager extension. (todd)
|
||
|
||
HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd)
|
||
|
||
HDFS-3765. namenode -initializeSharedEdits should be able to initialize
|
||
all shared storages. (Vinay and todd via todd)
|
||
|
||
HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via
|
||
suresh)
|
||
|
||
HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG.
|
||
(Andrew Purtell via suresh)
|
||
|
||
HDFS-3802. StartupOption.name in HdfsServerConstants should be final.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-3796. Speed up edit log tests by avoiding fsync() (todd)
|
||
|
||
HDFS-2963. Console Output is confusing while executing metasave
|
||
(dfsadmin command). (Andrew Wang via eli)
|
||
|
||
HDFS-3672. Expose disk-location information for blocks to enable better
|
||
scheduling. (Andrew Wang via atm)
|
||
|
||
HDFS-2727. libhdfs should get the default block size from the server.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh)
|
||
|
||
HDFS-3819. Should check whether invalidate work percentage default value is
|
||
not greater than 1.0f. (Jing Zhao via jitendra)
|
||
|
||
HDFS-3177. Update DFSClient and DataXceiver to handle different checkum
|
||
types in file checksum computation. (Kihwal Lee via szetszwo)
|
||
|
||
HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary
|
||
imports. (Jing Zhao via suresh)
|
||
|
||
HDFS-3853. Port MiniDFSCluster enableManagedDfsDirsRedundancy
|
||
option to branch-2. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3871. Change NameNodeProxies to use RetryUtils. (Arun C Murthy
|
||
via szetszwo)
|
||
|
||
HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy.
|
||
(Jing Zhao via szetszwo)
|
||
|
||
HDFS-3888. Clean up BlockPlacementPolicyDefault. (Jing Zhao via szetszwo)
|
||
|
||
HDFS-3907. Allow multiple users for local block readers. (eli)
|
||
|
||
HDFS-3510. Editlog pre-allocation is performed prior to writing edits
|
||
to avoid partial edits case disk out of space. (Colin McCabe via todd)
|
||
|
||
HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli)
|
||
|
||
HDFS-3920. libwebdhfs string processing and using strerror consistently
|
||
to handle all errors. (Jing Zhao via suresh)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-2982. Startup performance suffers when there are many edit log
|
||
segments. (Colin Patrick McCabe via todd)
|
||
|
||
HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream.
|
||
(Henry Robinson via todd)
|
||
|
||
HDFS-3110. Use directRead API to reduce the number of buffer copies in
|
||
libhdfs (Henry Robinson via todd)
|
||
|
||
HDFS-3697. Enable fadvise readahead by default. (todd)
|
||
|
||
HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode.
|
||
(Jing Zhao and Weiyan Wang via szetszwo)
|
||
|
||
HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-3385. The last block of INodeFileUnderConstruction is not
|
||
necessarily a BlockInfoUnderConstruction, so do not cast it in
|
||
FSNamesystem.recoverLeaseInternal(..). (szetszwo)
|
||
|
||
HDFS-3414. Balancer does not find NameNode if rpc-address or
|
||
servicerpc-address are not set in client configs. (atm)
|
||
|
||
HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent
|
||
(todd)
|
||
|
||
HDFS-2759. Pre-allocate HDFS edit log files after writing version number.
|
||
(atm)
|
||
|
||
HDFS-3413. TestFailureToReadEdits timing out. (atm)
|
||
|
||
HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd)
|
||
|
||
HDFS-3433. GetImageServlet should allow administrative requestors when
|
||
security is enabled. (atm)
|
||
|
||
HDFS-1153. dfsnodelist.jsp should handle invalid input parameters.
|
||
(Ravi Phulari via eli)
|
||
|
||
HDFS-3434. InvalidProtocolBufferException when visiting DN
|
||
browseDirectory.jsp (eli)
|
||
|
||
HDFS-2800. Fix cancellation of checkpoints in the standby node to be more
|
||
reliable. (todd)
|
||
|
||
HDFS-3391. Fix InvalidateBlocks to compare blocks including their
|
||
generation stamps. (todd)
|
||
|
||
HDFS-3444. hdfs groups command doesn't work with security enabled. (atm)
|
||
|
||
HDFS-2717. BookKeeper Journal output stream doesn't check addComplete rc.
|
||
(Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3415. Make sure all layout versions are the same for all storage
|
||
directories in the Namenode. (Brandon Li via szetszwo)
|
||
|
||
HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should
|
||
use the stored generation stamp to check if the block is valid. (Vinay
|
||
via szetszwo)
|
||
|
||
HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full
|
||
principal name. (tucu)
|
||
|
||
HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes.
|
||
(Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv)
|
||
|
||
HDFS-3452. BKJM:Switch from standby to active fails and NN gets shut down
|
||
due to delay in clearing of lock. (umamahesh)
|
||
|
||
HDFS-3398. Client will not retry when primaryDN is down once it's just got pipeline.
|
||
(Amith D K via umamahesh)
|
||
|
||
HDFS-3474. Cleanup Exception handling in BookKeeper journal manager.
|
||
(Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3468. Make BKJM-ZK session timeout configurable. (umamahesh)
|
||
|
||
HDFS-3423. BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments()
|
||
a bad inProgress_ ZNodes. (Ivan Kelly and Uma via umamahesh)
|
||
|
||
HDFS-3441. Race condition between rolling logs at active NN and purging at standby.
|
||
(Rakesh R via umamahesh)
|
||
|
||
HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to
|
||
0.0.0.0 even if NN RPC address is configured. (atm via eli)
|
||
|
||
HDFS-3486. offlineimageviewer can't read fsimage files that contain
|
||
persistent delegation tokens. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3487. offlineimageviewer should give byte offset information
|
||
when it encounters an exception. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew
|
||
Wang via atm)
|
||
|
||
HDFS-2025. Go Back to File View link is not working in tail.jsp.
|
||
(Ashish and Sravan via umamahesh)
|
||
|
||
HDFS-3501. Checkpointing with security enabled will stop working
|
||
after ticket lifetime expires. (atm via eli)
|
||
|
||
HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks.
|
||
(Madhukara Phatak via atm)
|
||
|
||
HDFS-3505. DirectoryScanner does not join all threads in shutdown.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis
|
||
jumps (Andy Isaacson via todd)
|
||
|
||
HDFS-2914. HA: Standby should not enter safemode when resources are low.
|
||
(Vinay via atm)
|
||
|
||
HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option.
|
||
(Henry Robinson via atm)
|
||
|
||
HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm)
|
||
|
||
HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd)
|
||
|
||
HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if
|
||
NamenodeRpcAddressParam is not set. (szetszwo)
|
||
|
||
HDFS-2797. Fix misuses of InputStream#skip in the edit log code.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3517. TestStartup should bind ephemeral ports. (eli)
|
||
|
||
HDFS-3522. If a namenode is in safemode, it should throw SafeModeException
|
||
when getBlockLocations has zero locations. (Brandon Li via szetszwo)
|
||
|
||
HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh)
|
||
|
||
HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh)
|
||
|
||
HDFS-3531. EditLogFileOutputStream#preallocate should check for
|
||
incomplete writes. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-766. Error message not clear for set space quota out of boundary
|
||
values. (Jon Zuanich via atm)
|
||
|
||
HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli)
|
||
|
||
HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522. (Brandon
|
||
Li via szetszwo)
|
||
|
||
HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil
|
||
instances. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection.
|
||
(szetszwo)
|
||
|
||
HDFS-3157. Fix a bug in the case that the generation stamps of the stored
|
||
block in a namenode and the reported block from a datanode do not match.
|
||
(Ashish Singhi via szetszwo)
|
||
|
||
HDFS-3575. HttpFS does not log Exception Stacktraces (brocknoland via tucu)
|
||
|
||
HDFS-3574. Fix small race and do some cleanup in GetImageServlet (todd)
|
||
|
||
HDFS-3581. FSPermissionChecker#checkPermission sticky bit check
|
||
missing range check. (eli)
|
||
|
||
HDFS-3541. Deadlock between recovery, xceiver and packet responder.
|
||
(Vinay via umamahesh)
|
||
|
||
HDFS-3428. Move DelegationTokenRenewer to common (tucu)
|
||
|
||
HDFS-3491. HttpFs does not set permissions correctly (tucu)
|
||
|
||
HDFS-3580. incompatible types; no instance(s) of type variable(s) V exist
|
||
so that V conforms to boolean compiling HttpFSServer.java with OpenJDK
|
||
(adi2 via tucu)
|
||
|
||
HDFS-3603. Decouple TestHDFSTrash from TestTrash. (Jason Lowe via eli)
|
||
|
||
HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3548. NamenodeFsck.copyBlock fails to create a Block Reader.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3615. Two BlockTokenSecretManager findbugs warnings. (atm)
|
||
|
||
HDFS-470. libhdfs should handle 0-length reads from FSInputStream
|
||
correctly. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3492. fix some misuses of InputStream#skip.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-2966 TestNameNodeMetrics tests can fail under load. (stevel)
|
||
|
||
HDFS-3605. Block mistakenly marked corrupt during edit log catchup
|
||
phase of failover. (todd and Brahma Reddy Battula via todd)
|
||
|
||
HDFS-3690. BlockPlacementPolicyDefault incorrectly casts LOG. (eli)
|
||
|
||
HDFS-3597. SNN fails to start after DFS upgrade. (Andy Isaacson via todd)
|
||
|
||
HDFS-3608. fuse_dfs: detect changes in UID ticket cache. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-3709. TestStartup tests still binding to the ephemeral port. (eli)
|
||
|
||
HDFS-3720. hdfs.h must get packaged. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3626. Creating file with invalid path can corrupt edit log (todd)
|
||
|
||
HDFS-3679. fuse_dfs notrash option sets usetrash. (Conrad Meyer via suresh)
|
||
|
||
HDFS-3732. fuse_dfs: incorrect configuration value checked for connection
|
||
expiry timer period. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect
|
||
timeout config. (atm)
|
||
|
||
HDFS-3756. DelegationTokenFetcher creates 2 HTTP connections, the second
|
||
one not properly configured. (tucu)
|
||
|
||
HDFS-3579. libhdfs: fix exception handling. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. (eli)
|
||
|
||
HDFS-3760. primitiveCreate is a write, not a read. (Andy Isaacson via atm)
|
||
|
||
HDFS-3710. libhdfs misuses O_RDONLY/WRONLY/RDWR. (Andy Isaacson via atm)
|
||
|
||
HDFS-3721. hsync support broke wire compatibility. (todd and atm)
|
||
|
||
HDFS-3758. TestFuseDFS test failing. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-2330. In NNStorage.java, IOExceptions of stream closures can mask
|
||
root exceptions. (umamahesh via todd)
|
||
|
||
HDFS-3790. test_fuse_dfs.c doesn't compile on centos 5. (Colin Patrick
|
||
McCabe via atm)
|
||
|
||
HDFS-3658. Fix bugs in TestDFSClientRetries and add more tests. (szetszwo)
|
||
|
||
HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP
|
||
header when offset is specified and length is omitted.
|
||
(Ravi Prakash via szetszwo)
|
||
|
||
HDFS-3048. Small race in BlockManager#close. (Andy Isaacson via eli)
|
||
|
||
HDFS-3194. DataNode block scanner is running too frequently.
|
||
(Andy Isaacson via eli)
|
||
|
||
HDFS-3808. fuse_dfs: postpone libhdfs intialization until after fork.
|
||
(Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3788. ByteRangeInputStream should not expect HTTP Content-Length header
|
||
when chunked transfer-encoding is used. (szetszwo)
|
||
|
||
HDFS-3816. Invalidate work percentage default value should be 0.32f
|
||
instead of 32. (Jing Zhao via suresh)
|
||
|
||
HDFS-3707. TestFSInputChecker: improper use of skip.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3830. test_libhdfs_threaded: use forceNewInstance.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is
|
||
enabled and the NN restarts with outstanding delegation tokens. (atm)
|
||
|
||
HDFS-3715. Fix TestFileCreation#testFileCreationNamenodeRestart.
|
||
(Andrew Whang via eli)
|
||
|
||
HDFS-3683. Edit log replay progress indicator shows >100% complete. (Plamen
|
||
Jeliazkov via atm)
|
||
|
||
HDFS-3731. Release upgrade must handle blocks being written from 1.0.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure.
|
||
(eli)
|
||
|
||
HDFS-3860. HeartbeatManager#Monitor may wrongly hold the writelock of
|
||
namesystem. (Jing Zhao via atm)
|
||
|
||
HDFS-3849. When re-loading the FSImage, we should clear the existing
|
||
genStamp and leases. (Colin Patrick McCabe via atm)
|
||
|
||
HDFS-3864. NN does not update internal file mtime for OP_CLOSE when reading
|
||
from the edit log. (atm)
|
||
|
||
HDFS-3837. Fix DataNode.recoverBlock findbugs warning. (eli)
|
||
|
||
HDFS-3733. Audit logs should include WebHDFS access. (Andy Isaacson via
|
||
eli)
|
||
|
||
HDFS-2686. Remove DistributedUpgrade related code. (suresh)
|
||
|
||
HDFS-3833. TestDFSShell fails on windows due to concurrent file
|
||
read/write. (Brandon Li via suresh)
|
||
|
||
HDFS-3466. Get HTTP kerberos principal from the web authentication keytab.
|
||
(omalley)
|
||
|
||
HDFS-3879. Fix findbugs warning in TransferFsImage on branch-2. (eli)
|
||
|
||
HDFS-3469. start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly.
|
||
(Vinay via umamahesh)
|
||
|
||
HDFS-1490. TransferFSImage should timeout (Dmytro Molkov and Vinay via todd)
|
||
|
||
HDFS-3828. Block Scanner rescans blocks too frequently.
|
||
(Andy Isaacson via eli)
|
||
|
||
HDFS-3895. hadoop-client must include commons-cli (tucu)
|
||
|
||
HDFS-2757. Cannot read a local block that's being written to when
|
||
using the local read short circuit. (Jean-Daniel Cryans via eli)
|
||
|
||
HDFS-3664. BlockManager race when stopping active services.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3928. MiniDFSCluster should reset the first ExitException on shutdown. (eli)
|
||
|
||
HDFS-3938. remove current limitations from HttpFS docs. (tucu)
|
||
|
||
HDFS-3944. Httpfs resolveAuthority() is not resolving host correctly. (tucu)
|
||
|
||
HDFS-3972. Trash emptier fails in secure HA cluster. (todd via eli)
|
||
|
||
HDFS-4443. Remove a trailing '`' character from the HTML code generated by
|
||
NamenodeJspHelper.generateNodeData(..). (Christian Rohling via szetszwo)
|
||
|
||
BREAKDOWN OF HDFS-3042 SUBTASKS
|
||
|
||
HDFS-2185. HDFS portion of ZK-based FailoverController (todd)
|
||
|
||
HDFS-3200. Scope all ZKFC configurations by nameservice (todd)
|
||
|
||
HDFS-3223. add zkfc to hadoop-daemon.sh script (todd)
|
||
|
||
HDFS-3261. TestHASafeMode fails on HDFS-3042 branch (todd)
|
||
|
||
HDFS-3159. Document NN auto-failover setup and configuration (todd)
|
||
|
||
HDFS-3412. Fix findbugs warnings in auto-HA branch (todd)
|
||
|
||
HDFS-3432. TestDFSZKFailoverController tries to fail over too early (todd)
|
||
|
||
HDFS-3902. TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken.
|
||
(Andy Isaacson via eli)
|
||
|
||
Release 2.0.0-alpha - 05-23-2012
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-2676. Remove Avro RPC. (suresh)
|
||
|
||
HDFS-2303. Unbundle jsvc. (Roman Shaposhnik and Mingjie Lai via eli)
|
||
|
||
HDFS-3137. Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16. (eli)
|
||
|
||
HDFS-3138. Move DatanodeInfo#ipcPort to DatanodeID. (eli)
|
||
|
||
HDFS-3164. Move DatanodeInfo#hostName to DatanodeID. (eli)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm)
|
||
|
||
HDFS-395. DFS Scalability: Incremental block reports. (Tomasz Nykiel
|
||
via hairong)
|
||
|
||
HDFS-2517. Add protobuf service for JounralProtocol. (suresh)
|
||
|
||
HDFS-2518. Add protobuf service for NamenodeProtocol. (suresh)
|
||
|
||
HDFS-2520. Add protobuf service for InterDatanodeProtocol. (suresh)
|
||
|
||
HDFS-2519. Add protobuf service for DatanodeProtocol. (suresh)
|
||
|
||
HDFS-2581. Implement protobuf service for JournalProtocol. (suresh)
|
||
|
||
HDFS-2618. Implement protobuf service for NamenodeProtocol. (suresh)
|
||
|
||
HDFS-2629. Implement protobuf service for InterDatanodeProtocol. (suresh)
|
||
|
||
HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. (suresh)
|
||
|
||
HDFS-2642. Protobuf translators for DatanodeProtocol. (jitendra)
|
||
|
||
HDFS-2647. Used protobuf based RPC for InterDatanodeProtocol,
|
||
ClientDatanodeProtocol, JournalProtocol, NamenodeProtocol. (suresh)
|
||
|
||
HDFS-2661. Enable protobuf RPC for DatanodeProtocol. (jitendra)
|
||
|
||
HDFS-2697. Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings
|
||
protocol to protocol buffers. (jitendra)
|
||
|
||
HDFS-2880. Protobuf changes in DatanodeProtocol to add multiple storages.
|
||
(suresh)
|
||
|
||
HDFS-2899. Service protocol changes in DatanodeProtocol to add multiple
|
||
storages. (suresh)
|
||
|
||
HDFS-2430. The number of failed or low-resource volumes the NN can tolerate
|
||
should be configurable. (atm)
|
||
|
||
HDFS-1623. High Availability Framework for HDFS NN. Contributed by Todd
|
||
Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, Bikas Saha,
|
||
Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay
|
||
Radia, Mingjie Lai, and Gregory Chanan
|
||
|
||
HDFS-2941. Add an administrative command to download a copy of the fsimage
|
||
from the NN. (atm)
|
||
|
||
HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change
|
||
DistributedFileSystem to @InterfaceAudience.LimitedPrivate.
|
||
(harsh via szetszwo)
|
||
|
||
HDFS-3167. CLI-based driver for MiniDFSCluster. (Henry Robinson via atm)
|
||
|
||
HDFS-3148. The client should be able to use multiple local interfaces
|
||
for data transfer. (eli)
|
||
|
||
HDFS-3000. Add a public API for setting quotas. (atm)
|
||
|
||
HDFS-3102. Add CLI tool to initialize the shared-edits dir. (atm)
|
||
|
||
HDFS-3004. Implement Recovery Mode. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3282. Add HdfsDataInputStream as a public API. (umamahesh)
|
||
|
||
HDFS-3298. Add HdfsDataOutputStream as a public API. (szetszwo)
|
||
|
||
HDFS-234. Integration with BookKeeper logging system. (Ivan Kelly
|
||
via umamahesh)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-2018. Move all journal stream management code into one place.
|
||
(Ivan Kelly via jitendra)
|
||
|
||
HDFS-2223. Untangle depencencies between NN components (todd)
|
||
|
||
HDFS-2351. Change Namenode and Datanode to register each of their protocols
|
||
seperately (sanjay)
|
||
|
||
HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm)
|
||
|
||
HDFS-2181. Separate HDFS Client wire protocol data types (sanjay)
|
||
|
||
HDFS-2459. Separate datatypes for Journal Protocol. (suresh)
|
||
|
||
HDFS-2480. Separate datatypes for NamenodeProtocol. (suresh)
|
||
|
||
HDFS-2489. Move Finalize and Register to separate file out of
|
||
DatanodeCommand.java. (suresh)
|
||
|
||
HDFS-2488. Separate datatypes for InterDatanodeProtocol. (suresh)
|
||
|
||
HDFS-2496. Separate datatypes for DatanodeProtocol. (suresh)
|
||
|
||
HDFS-2479. HDFS Client Data Types in Protocol Buffers (sanjay)
|
||
|
||
HADOOP-7862. Hdfs changes to work with HADOOP-7862: Move the support for
|
||
multiple protocols to lower layer so that Writable, PB and Avro can all
|
||
use it. (sanjay)
|
||
|
||
HDFS-2597. ClientNameNodeProtocol in Protocol Buffers. (sanjay)
|
||
|
||
HDFS-2651. ClientNameNodeProtocol Translators for Protocol Buffers. (sanjay)
|
||
|
||
HDFS-2650. Replace @inheritDoc with @Override. (Hari Mankude via suresh).
|
||
|
||
HDFS-2669. Enable protobuf rpc for ClientNamenodeProtocol. (sanjay)
|
||
|
||
HDFS-2801. Provide a method in client side translators to check for a
|
||
methods supported in underlying protocol. (jitendra)
|
||
|
||
HDFS-2895. Remove Writable wire protocol types and translators to
|
||
complete transition to protocol buffers. (suresh)
|
||
|
||
HDFS-2992. Edit log failure trace should include transaction ID of
|
||
error. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3030. Remove getProtocolVersion and getProtocolSignature from
|
||
translators. (jitendra)
|
||
|
||
HDFS-2158. Add JournalSet to manage the set of journals. (jitendra)
|
||
|
||
HDFS-2334. Add Closeable to JournalManager. (Ivan Kelly via jitendra)
|
||
|
||
HDFS-1580. Add interface for generic Write Ahead Logging mechanisms.
|
||
(Ivan Kelly via jitendra)
|
||
|
||
HDFS-3060. Bump TestDistributedUpgrade#testDistributedUpgrade timeout (eli)
|
||
|
||
HDFS-2410. Further cleanup of hardcoded configuration keys and values.
|
||
(suresh)
|
||
|
||
HDFS-2878. Fix TestBlockRecovery and move it back into main test directory.
|
||
(todd)
|
||
|
||
HDFS-3003. Remove getHostPortString() from NameNode, replace it with
|
||
NetUtils.getHostPortString(). (Brandon Li via atm)
|
||
|
||
HDFS-3014. FSEditLogOp and its subclasses should have toString() method.
|
||
(Sho Shimauchi via atm)
|
||
|
||
HDFS-3021. Use generic type to declare FSDatasetInterface. (szetszwo)
|
||
|
||
HDFS-3056. Add a new interface RollingLogs for DataBlockScanner logging.
|
||
(szetszwo)
|
||
|
||
HDFS-2731. Add command to bootstrap the Standby Node's name directories
|
||
from the Active NameNode. (todd)
|
||
|
||
HDFS-3082. Clean up FSDatasetInterface and change DataNode.data to package
|
||
private. (szetszwo)
|
||
|
||
HDFS-3057. httpfs and hdfs launcher scripts should honor CATALINA_HOME
|
||
and HADOOP_LIBEXEC_DIR (rvs via tucu)
|
||
|
||
HDFS-3088. Move FSDatasetInterface inner classes to a package. (szetszwo)
|
||
|
||
HDFS-3111. Missing license headers in trunk. (umamahesh)
|
||
|
||
HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in
|
||
the config description for the smaller clusters. (szetszwo via umamahesh)
|
||
|
||
HDFS-3105. Add DatanodeStorage information to block recovery. (szetszwo)
|
||
|
||
HDFS-3086. Change Datanode not to send storage list in registration.
|
||
(szetszwo)
|
||
|
||
HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi
|
||
via todd)
|
||
|
||
HDFS-3044. fsck move should be non-destructive by default.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3071. haadmin failover command does not provide enough detail when
|
||
target NN is not ready to be active. (todd)
|
||
|
||
HDFS-3089. Move FSDatasetInterface and the related classes to a package.
|
||
(szetszwo)
|
||
|
||
HDFS-3129. NetworkTopology: add test that getLeaf should check for
|
||
invalid topologies. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3155. Clean up FSDataset implemenation related code. (szetszwo)
|
||
|
||
HDFS-3158. LiveNodes member of NameNodeMXBean should list non-DFS used
|
||
space and capacity per DN. (atm)
|
||
|
||
HDFS-3172. dfs.upgrade.permission is dead code. (eli)
|
||
|
||
HDFS-3171. The DatanodeID "name" field is overloaded. (eli)
|
||
|
||
HDFS-3144. Refactor DatanodeID#getName by use. (eli)
|
||
|
||
HDFS-3130. Move fsdataset implementation to a package. (szetszwo)
|
||
|
||
HDFS-3120. Enable hsync and hflush by default. (eli)
|
||
|
||
HDFS-3187. Upgrade guava to 11.0.2 (todd)
|
||
|
||
HDFS-3168. Remove unnecessary "throw IOException" and change fields to
|
||
final in FSNamesystem and BlockManager. (szetszwo)
|
||
|
||
HDFS-2564. Cleanup unnecessary exceptions thrown and unnecessary casts.
|
||
(Hari Mankude via eli)
|
||
|
||
HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass
|
||
namenodeId as well as host:port (todd)
|
||
|
||
HDFS-3050. rework OEV to share more code with the NameNode.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3204. Minor modification to JournalProtocol.proto to make
|
||
it generic. (suresh)
|
||
|
||
HDFS-3226. Allow GetConf tool to print arbitrary keys (todd)
|
||
|
||
HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
|
||
(todd)
|
||
|
||
HDFS-3238. ServerCommand and friends don't need to be writables. (eli)
|
||
|
||
HDFS-3094. add -nonInteractive and -force option to namenode -format
|
||
command (Arpit Gupta via todd)
|
||
|
||
HDFS-3244. Remove dead writable code from hdfs/protocol. (eli)
|
||
|
||
HDFS-3247. Improve bootstrapStandby behavior when original NN is not active
|
||
(todd)
|
||
|
||
HDFS-3249. Use ToolRunner.confirmPrompt in NameNode (todd)
|
||
|
||
HDFS-3179. Improve the exception message thrown by DataStreamer when
|
||
it failed to add a datanode. (szetszwo)
|
||
|
||
HDFS-2983. Relax the build version check to permit rolling upgrades within a release. (atm)
|
||
|
||
HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir
|
||
with edit log segments. (atm)
|
||
|
||
HDFS-2708. Stats for the # of blocks per DN. (atm)
|
||
|
||
HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to
|
||
TestEditLog. (Arpit Gupta via szetszwo)
|
||
|
||
HDFS-3294. Fix code indentation in NamenodeWebHdfsMethods and
|
||
DatanodeWebHdfsMethods. (szetszwo)
|
||
|
||
HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu)
|
||
|
||
HDFS-3206. Miscellaneous xml cleanups for OEV.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3169. TestFsck should test multiple -move operations in a row.
|
||
(Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in
|
||
NetworkTopology for first rack local node). (Junping Du via eli)
|
||
|
||
HDFS-3322. Use HdfsDataInputStream and HdfsDataOutputStream in Hdfs.
|
||
(szetszwo)
|
||
|
||
HDFS-3339. Change INode to package private. (John George via szetszwo)
|
||
|
||
HDFS-3303. Remove Writable implementation from RemoteEditLogManifest.
|
||
(Brandon Li via szetszwo)
|
||
|
||
HDFS-2617. Replaced Kerberized SSL for image transfer and fsck
|
||
with SPNEGO-based solution. (jghoman, omalley, tucu, and atm via eli)
|
||
|
||
HDFS-3365. Enable users to disable socket caching in DFS client
|
||
configuration (todd)
|
||
|
||
HDFS-3375. Put client name in DataXceiver thread name for readBlock
|
||
and keepalive (todd)
|
||
|
||
HDFS-3363. Define BlockCollection and MutableBlockCollection interfaces
|
||
so that INodeFile and INodeFileUnderConstruction do not have to be used in
|
||
block management. (John George via szetszwo)
|
||
|
||
HDFS-3211. Add fence(..) and replace NamenodeRegistration with JournalInfo
|
||
and epoch in JournalProtocol. (suresh via szetszwo)
|
||
|
||
HADOOP-8285 HDFS changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
|
||
|
||
HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs.
|
||
(eli)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-2477. Optimize computing the diff between a block report and the
|
||
namenode state. (Tomasz Nykiel via hairong)
|
||
|
||
HDFS-2495. Increase granularity of write operations in ReplicationMonitor
|
||
thus reducing contention for write lock. (Tomasz Nykiel via hairong)
|
||
|
||
HDFS-2476. More CPU efficient data structure for under-replicated,
|
||
over-replicated, and invalidated blocks. (Tomasz Nykiel via todd)
|
||
|
||
HDFS-3036. Remove unused method DFSUtil#isDefaultNamenodeAddress. (atm)
|
||
|
||
HDFS-3378. Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT. (eli)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-2481. Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol.
|
||
(sanjay)
|
||
|
||
HDFS-2497. Fix TestBackupNode failure. (suresh)
|
||
|
||
HDFS-2499. RPC client is created incorrectly introduced in HDFS-2459.
|
||
(suresh)
|
||
|
||
HDFS-2526. (Client)NamenodeProtocolTranslatorR23 do not need to keep a
|
||
reference to rpcProxyWithoutRetry (atm)
|
||
|
||
HDFS-2532. TestDfsOverAvroRpc timing out in trunk (Uma Maheswara Rao G
|
||
via todd)
|
||
|
||
HDFS-2666. Fix TestBackupNode failure. (suresh)
|
||
|
||
HDFS-2663. Optional protobuf parameters are not handled correctly. (suresh)
|
||
|
||
HDFS-2694. Removal of Avro broke non-PB NN services. (atm)
|
||
|
||
HDFS-2687. Tests failing with ClassCastException post protobuf RPC
|
||
changes. (suresh)
|
||
|
||
HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk
|
||
(Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-2739. SecondaryNameNode doesn't start up. (jitendra)
|
||
|
||
HDFS-2768. BackupNode stop can not close proxy connections because
|
||
it is not a proxy instance. (Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-2968. Protocol translator for BlockRecoveryCommand broken when
|
||
multiple blocks need recovery. (todd)
|
||
|
||
HDFS-3020. Fix editlog to automatically sync when buffer is full. (todd)
|
||
|
||
HDFS-3038. Add FSEditLog.metrics to findbugs exclude list. (todd via atm)
|
||
|
||
HDFS-2188. Make FSEditLog create its journals from a list of URIs rather
|
||
than NNStorage. (Ivan Kelly via jitendra)
|
||
|
||
HDFS-1765. Block Replication should respect under-replication
|
||
block priority. (Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-2285. BackupNode should reject requests to modify namespace.
|
||
(shv and Uma Maheswara Rao)
|
||
|
||
HDFS-2764. TestBackupNode is racy. (atm)
|
||
|
||
HDFS-3093. Fix bug where namenode -format interpreted the -force flag in
|
||
reverse. (todd)
|
||
|
||
HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (szetszwo)
|
||
|
||
HDFS-3099. SecondaryNameNode does not properly initialize metrics system.
|
||
(atm)
|
||
|
||
HDFS-3062. Fix bug which prevented MR job submission from creating
|
||
delegation tokens on an HA cluster. (Mingjie Lai via todd)
|
||
|
||
HDFS-3083. Cannot run an MR job with HA and security enabled when
|
||
second-listed NN active. (atm)
|
||
|
||
HDFS-3100. In BlockSender, throw an exception when it needs to verify
|
||
checksum but the meta data does not exist. (Brandon Li via szetszwo)
|
||
|
||
HDFS-3132. Fix findbugs warning on HDFS trunk. (todd)
|
||
|
||
HDFS-3156. TestDFSHAAdmin is failing post HADOOP-8202. (atm)
|
||
|
||
HDFS-3143. TestGetBlocks.testGetBlocks is failing. (Arpit Gupta via atm)
|
||
|
||
HDFS-3142. TestHDFSCLI.testAll is failing. (Brandon Li via atm)
|
||
|
||
HDFS-3070. HDFS balancer doesn't ensure that hdfs-site.xml is loaded. (atm)
|
||
|
||
HDFS-2995. start-dfs.sh should only start the 2NN for namenodes
|
||
with dfs.namenode.secondary.http-address configured. (eli)
|
||
|
||
HDFS-3174. Fix assert in TestPendingDataNodeMessages. (eli)
|
||
|
||
HDFS-3199. TestValidateConfigurationSettings is failing. (todd via eli)
|
||
|
||
HDFS-3202. NamespaceInfo PB translation drops build version. (atm)
|
||
|
||
HDFS-3109. Remove hsqldf exclusions from pom.xml. (Ravi Prakash
|
||
via suresh)
|
||
|
||
HDFS-3210. JsonUtil#toJsonMap for for a DatanodeInfo should use
|
||
"ipAddr" instead of "name". (eli)
|
||
|
||
HDFS-3208. Bogus entries in hosts files are incorrectly displayed
|
||
in the report. (eli)
|
||
|
||
HDFS-3136. Remove SLF4J dependency as HDFS does not need it to fix
|
||
unnecessary warnings. (Jason Lowe via suresh)
|
||
|
||
HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle
|
||
null response from initReplicaRecovery (todd)
|
||
|
||
HDFS-3119. Overreplicated block is not deleted even after the replication
|
||
factor is reduced after sync follwed by closing that file. (Ashish Singhi
|
||
via umamahesh)
|
||
|
||
HDFS-3234. Accidentally left log message in GetConf after HDFS-3226 (todd)
|
||
|
||
HDFS-3236. NameNode does not initialize generic conf keys when started
|
||
with -initializeSharedEditsDir (atm)
|
||
|
||
HDFS-3248. bootstrapStandby repeated twice in hdfs namenode usage message
|
||
(Colin Patrick McCabe via todd)
|
||
|
||
HDFS-2696. Fix the fuse-fds build. (Bruno Mahé via eli)
|
||
|
||
HDFS-3254. Branch-2 build broken due to wrong version number in
|
||
fuse-dfs' pom.xml. (Anupam Seth via eli)
|
||
|
||
HDFS-3260. TestDatanodeRegistration should set minimum DN version in
|
||
addition to minimum NN version. (atm)
|
||
|
||
HDFS-3255. HA DFS returns wrong token service (Daryn Sharp via todd)
|
||
|
||
HDFS-3256. HDFS considers blocks under-replicated if topology script is
|
||
configured with only 1 rack. (atm)
|
||
|
||
HDFS-2799. Trim fs.checkpoint.dir values. (Amith D K via eli)
|
||
|
||
HDFS-2765. TestNameEditsConfigs is incorrectly swallowing IOE. (atm)
|
||
|
||
HDFS-3280. DFSOutputStream.sync should not be synchronized (todd)
|
||
|
||
HDFS-3268. FileContext API mishandles token service and incompatible with
|
||
HA (Daryn Sharp via todd)
|
||
|
||
HDFS-3284. bootstrapStandby fails in secure cluster (todd)
|
||
|
||
HDFS-3165. HDFS Balancer scripts are refering to wrong path of
|
||
hadoop-daemon.sh (Amith D K via eli)
|
||
|
||
HDFS-891. DataNode no longer needs to check for dfs.network.script.
|
||
(harsh via eli)
|
||
|
||
HDFS-3305. GetImageServlet should consider SBN a valid requestor in a
|
||
secure HA setup. (atm)
|
||
|
||
HDFS-3314. HttpFS operation for getHomeDirectory is incorrect. (tucu)
|
||
|
||
HDFS-3319. Change DFSOutputStream to not to start a thread in constructors.
|
||
(szetszwo)
|
||
|
||
HDFS-3222. DFSInputStream#openInfo should not silently get the length as 0
|
||
when locations length is zero for last partial block. (umamahesh)
|
||
|
||
HDFS-3181. Fix a test case in TestLeaseRecovery2. (szetszwo)
|
||
|
||
HDFS-3309. HttpFS (Hoop) chmod not supporting octal and sticky bit
|
||
permissions. (tucu)
|
||
|
||
HDFS-3326. Append enabled log message uses the wrong variable.
|
||
(Matthew Jacobs via eli)
|
||
|
||
HDFS-3275. Skip format for non-file based directories.
|
||
(Amith D K via umamahesh)
|
||
|
||
HDFS-3286. When the threshold value for balancer is zero, unexpected output is displayed.
|
||
(Ashish Singhi via umamahesh)
|
||
|
||
HDFS-3336. hdfs launcher script will be better off not special casing
|
||
namenode command with regards to hadoop.security.logger (rvs via tucu)
|
||
|
||
HDFS-3330. If GetImageServlet throws an Error or RTE, response should not
|
||
have HTTP "OK" status. (todd)
|
||
|
||
HDFS-3351. NameNode#initializeGenericKeys should always set fs.defaultFS
|
||
regardless of whether HA or Federation is enabled. (atm)
|
||
|
||
HDFS-3332. NullPointerException in DN when directoryscanner is trying to
|
||
report bad blocks. (Amith D K via umamahesh)
|
||
|
||
HDFS-3359. DFSClient.close should close cached sockets. (todd)
|
||
|
||
HDFS-3350. In INode, add final to compareTo(..), equals(..) and hashCode(),
|
||
and remove synchronized from updatePermissionStatus(..). (szetszwo)
|
||
|
||
HDFS-3357. DataXceiver reads from client socket with incorrect/no timeout
|
||
(todd)
|
||
|
||
HDFS-3376. DFSClient fails to make connection to DN if there are many
|
||
unusable cached sockets (todd)
|
||
|
||
HDFS-3328. NPE in DataNode.getIpcPort. (eli)
|
||
|
||
HDFS-3396. FUSE build fails on Ubuntu 12.04. (Colin Patrick McCabe via eli)
|
||
|
||
HDFS-3395. NN doesn't start with HA+security enabled and HTTP address set to 0.0.0.0. (atm)
|
||
|
||
HDFS-3026. HA: Handle failure during HA state transition. (atm)
|
||
|
||
HDFS-860. fuse-dfs truncate behavior causes issues with scp.
|
||
(Brian Bockelman via eli)
|
||
|
||
BREAKDOWN OF HDFS-1623 SUBTASKS
|
||
|
||
HDFS-2179. Add fencing framework and mechanisms for NameNode HA. (todd)
|
||
|
||
HDFS-1974. Introduce active and standy states to the namenode. (suresh)
|
||
|
||
HDFS-2407. getServerDefaults and getStats don't check operation category (atm)
|
||
|
||
HDFS-1973. HA: HDFS clients must handle namenode failover and switch over to
|
||
the new active namenode. (atm)
|
||
|
||
HDFS-2301. Start/stop appropriate namenode services when transition to active
|
||
and standby states. (suresh)
|
||
|
||
HDFS-2231. Configuration changes for HA namenode. (suresh)
|
||
|
||
HDFS-2418. Change ConfiguredFailoverProxyProvider to take advantage of
|
||
HDFS-2231. (atm)
|
||
|
||
HDFS-2393. Mark appropriate methods of ClientProtocol with the idempotent
|
||
annotation. (atm)
|
||
|
||
HDFS-2523. Small NN fixes to include HAServiceProtocol and prevent NPE on
|
||
shutdown. (todd)
|
||
|
||
HDFS-2577. NN fails to start since it tries to start secret manager in
|
||
safemode. (todd)
|
||
|
||
HDFS-2582. Scope dfs.ha.namenodes config by nameservice (todd)
|
||
|
||
HDFS-2591. MiniDFSCluster support to mix and match federation with HA (todd)
|
||
|
||
HDFS-1975. Support for sharing the namenode state from active to standby.
|
||
(jitendra, atm, todd)
|
||
|
||
HDFS-1971. Send block report from datanode to both active and standby
|
||
namenodes. (sanjay, todd via suresh)
|
||
|
||
HDFS-2616. Change DatanodeProtocol#sendHeartbeat() to return HeartbeatResponse.
|
||
(suresh)
|
||
|
||
HDFS-2622. Fix TestDFSUpgrade in HA branch. (todd)
|
||
|
||
HDFS-2612. Handle refreshNameNodes in federated HA clusters (todd)
|
||
|
||
HDFS-2623. Add test case for hot standby capability (todd)
|
||
|
||
HDFS-2626. BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized
|
||
(todd)
|
||
|
||
HDFS-2624. ConfiguredFailoverProxyProvider doesn't correctly stop
|
||
ProtocolTranslators (todd)
|
||
|
||
HDFS-2625. TestDfsOverAvroRpc failing after introduction of HeartbeatResponse
|
||
type (todd)
|
||
|
||
HDFS-2627. Determine DN's view of which NN is active based on heartbeat
|
||
responses (todd)
|
||
|
||
HDFS-2634. Standby needs to ingest latest edit logs before transitioning to
|
||
active (todd)
|
||
|
||
HDFS-2671. NN should throw StandbyException in response to RPCs in STANDBY
|
||
state (todd)
|
||
|
||
HDFS-2680. DFSClient should construct failover proxy with exponential backoff
|
||
(todd)
|
||
|
||
HDFS-2683. Authority-based lookup of proxy provider fails if path becomes
|
||
canonicalized (todd)
|
||
|
||
HDFS-2689. HA: BookKeeperEditLogInputStream doesn't implement isInProgress()
|
||
(atm)
|
||
|
||
HDFS-2602. NN should log newly-allocated blocks without losing BlockInfo (atm)
|
||
|
||
HDFS-2667. Fix transition from active to standby (todd)
|
||
|
||
HDFS-2684. Fix up some failing unit tests on HA branch (todd)
|
||
|
||
HDFS-2679. Add interface to query current state to HAServiceProtocol (eli via
|
||
todd)
|
||
|
||
HDFS-2677. Web UI should indicate the NN state. (eli via todd)
|
||
|
||
HDFS-2678. When a FailoverProxyProvider is used, DFSClient should not retry
|
||
connection ten times before failing over (atm via todd)
|
||
|
||
HDFS-2682. When a FailoverProxyProvider is used, Client should not retry for 45
|
||
times if it is timing out to connect to server. (Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-2693. Fix synchronization issues around state transition (todd)
|
||
|
||
HDFS-1972. Fencing mechanism for block invalidations and replications (todd)
|
||
|
||
HDFS-2714. Fix test cases which use standalone FSNamesystems (todd)
|
||
|
||
HDFS-2692. Fix bugs related to failover from/into safe mode. (todd)
|
||
|
||
HDFS-2716. Configuration needs to allow different dfs.http.addresses for each
|
||
HA NN (todd)
|
||
|
||
HDFS-2720. Fix MiniDFSCluster HA support to work properly on Windows. (Uma
|
||
Maheswara Rao G via todd)
|
||
|
||
HDFS-2291. Allow the StandbyNode to make checkpoints in an HA setup. (todd)
|
||
|
||
HDFS-2709. Appropriately handle error conditions in EditLogTailer (atm via
|
||
todd)
|
||
|
||
HDFS-2730. Refactor shared HA-related test code into HATestUtil class (todd)
|
||
|
||
HDFS-2762. Fix TestCheckpoint timing out on HA branch. (Uma Maheswara Rao G via
|
||
todd)
|
||
|
||
HDFS-2724. NN web UI can throw NPE after startup, before standby state is
|
||
entered. (todd)
|
||
|
||
HDFS-2753. Fix standby getting stuck in safemode when blocks are written while
|
||
SBN is down. (Hari Mankude and todd via todd)
|
||
|
||
HDFS-2773. Reading edit logs from an earlier version should not leave blocks in
|
||
under-construction state. (todd)
|
||
|
||
HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing
|
||
intermittently. (todd)
|
||
|
||
HDFS-2766. Test for case where standby partially reads log and then performs
|
||
checkpoint. (atm)
|
||
|
||
HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress streams
|
||
even when non-in-progress are requested. (atm)
|
||
|
||
HDFS-2789. TestHAAdmin.testFailover is failing (eli)
|
||
|
||
HDFS-2747. Entering safe mode after starting SBN can NPE. (Uma Maheswara Rao G
|
||
via todd)
|
||
|
||
HDFS-2772. On transition to active, standby should not swallow ELIE. (atm)
|
||
|
||
HDFS-2767. ConfiguredFailoverProxyProvider should support NameNodeProtocol.
|
||
(Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-2795. Standby NN takes a long time to recover from a dead DN starting up.
|
||
(todd)
|
||
|
||
HDFS-2592. Balancer support for HA namenodes. (Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-2367. Enable the configuration of multiple HA cluster addresses. (atm)
|
||
|
||
HDFS-2812. When becoming active, the NN should treat all leases as freshly
|
||
renewed. (todd)
|
||
|
||
HDFS-2737. Automatically trigger log rolls periodically on the active NN. (todd
|
||
and atm)
|
||
|
||
HDFS-2820. Add a simple sanity check for HA config (todd)
|
||
|
||
HDFS-2688. Add tests for quota tracking in an HA cluster. (todd)
|
||
|
||
HDFS-2804. Should not mark blocks under-replicated when exiting safemode (todd)
|
||
|
||
HDFS-2807. Service level authorizartion for HAServiceProtocol. (jitendra)
|
||
|
||
HDFS-2809. Add test to verify that delegation tokens are honored after
|
||
failover. (jitendra and atm)
|
||
|
||
HDFS-2838. NPE in FSNamesystem when in safe mode. (Gregory Chanan via eli)
|
||
|
||
HDFS-2805. Add a test for a federated cluster with HA NNs. (Brandon Li via
|
||
jitendra)
|
||
|
||
HDFS-2841. HAAdmin does not work if security is enabled. (atm)
|
||
|
||
HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas
|
||
immediately upon pipeline creation. (todd)
|
||
|
||
HDFS-2824. Fix failover when prior NN died just after creating an edit log
|
||
segment. (atm via todd)
|
||
|
||
HDFS-2853. HA: NN fails to start if the shared edits dir is marked required
|
||
(atm via eli)
|
||
|
||
HDFS-2845. SBN should not allow browsing of the file system via web UI. (Bikas
|
||
Saha via atm)
|
||
|
||
HDFS-2742. HA: observed dataloss in replication stress test. (todd via eli)
|
||
|
||
HDFS-2870. Fix log level for block debug info in processMisReplicatedBlocks
|
||
(todd)
|
||
|
||
HDFS-2859. LOCAL_ADDRESS_MATCHER.match has NPE when called from
|
||
DFSUtil.getSuffixIDs when the host is incorrect (Bikas Saha via todd)
|
||
|
||
HDFS-2861. checkpointing should verify that the dfs.http.address has been
|
||
configured to a non-loopback for peer NN (todd)
|
||
|
||
HDFS-2860. TestDFSRollback#testRollback is failing. (atm)
|
||
|
||
HDFS-2769. HA: When HA is enabled with a shared edits dir, that dir should be
|
||
marked required. (atm via eli)
|
||
|
||
HDFS-2863. Failures observed if dfs.edits.dir and shared.edits.dir have same
|
||
directories. (Bikas Saha via atm)
|
||
|
||
HDFS-2874. Edit log should log to shared dirs before local dirs. (todd)
|
||
|
||
HDFS-2890. DFSUtil#getSuffixIDs should skip unset configurations. (atm)
|
||
|
||
HDFS-2792. Make fsck work. (atm)
|
||
|
||
HDFS-2808. HA: haadmin should use namenode ids. (eli)
|
||
|
||
HDFS-2819. Document new HA-related configs in hdfs-default.xml. (eli)
|
||
|
||
HDFS-2752. HA: exit if multiple shared dirs are configured. (eli)
|
||
|
||
HDFS-2894. HA: automatically determine the nameservice Id if only one
|
||
nameservice is configured. (eli)
|
||
|
||
HDFS-2733. Document HA configuration and CLI. (atm)
|
||
|
||
HDFS-2794. Active NN may purge edit log files before standby NN has a chance to
|
||
read them (todd)
|
||
|
||
HDFS-2901. Improvements for SBN web UI - not show under-replicated/missing
|
||
blocks. (Brandon Li via jitendra)
|
||
|
||
HDFS-2905. HA: Standby NN NPE when shared edits dir is deleted. (Bikas Saha via
|
||
jitendra)
|
||
|
||
HDFS-2579. Starting delegation token manager during safemode fails. (todd)
|
||
|
||
HDFS-2510. Add HA-related metrics. (atm)
|
||
|
||
HDFS-2924. Standby checkpointing fails to authenticate in secure cluster.
|
||
(todd)
|
||
|
||
HDFS-2915. HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race
|
||
condition. (Bikas Saha via jitendra)
|
||
|
||
HDFS-2912. Namenode not shutting down when shared edits dir is inaccessible.
|
||
(Bikas Saha via atm)
|
||
|
||
HDFS-2917. HA: haadmin should not work if run by regular user (eli)
|
||
|
||
HDFS-2939. TestHAStateTransitions fails on Windows. (Uma Maheswara Rao G via
|
||
atm)
|
||
|
||
HDFS-2947. On startup NN throws an NPE in the metrics system. (atm)
|
||
|
||
HDFS-2942. TestActiveStandbyElectorRealZK fails if build dir does not exist.
|
||
(atm)
|
||
|
||
HDFS-2948. NN throws NPE during shutdown if it fails to startup (todd)
|
||
|
||
HDFS-2909. HA: Inaccessible shared edits dir not getting removed from FSImage
|
||
storage dirs upon error. (Bikas Saha via jitendra)
|
||
|
||
HDFS-2934. Allow configs to be scoped to all NNs in the nameservice. (todd)
|
||
|
||
HDFS-2935. Shared edits dir property should be suffixed with nameservice and
|
||
namenodeID (todd)
|
||
|
||
HDFS-2928. ConfiguredFailoverProxyProvider should not create a NameNode proxy
|
||
with an underlying retry proxy. (Uma Maheswara Rao G via atm)
|
||
|
||
HDFS-2955. IllegalStateException during standby startup in getCurSegmentTxId.
|
||
(Hari Mankude via atm)
|
||
|
||
HDFS-2937. TestDFSHAAdmin needs tests with MiniDFSCluster. (Brandon Li via
|
||
suresh)
|
||
|
||
HDFS-2586. Add protobuf service and implementation for HAServiceProtocol.
|
||
(suresh via atm)
|
||
|
||
HDFS-2952. NN should not start with upgrade option or with a pending an
|
||
unfinalized upgrade. (atm)
|
||
|
||
HDFS-2974. MiniDFSCluster does not delete standby NN name dirs during format.
|
||
(atm)
|
||
|
||
HDFS-2929. Stress test and fixes for block synchronization (todd)
|
||
|
||
HDFS-2972. Small optimization building incremental block report (todd)
|
||
|
||
HDFS-2973. Re-enable NO_ACK optimization for block deletion. (todd)
|
||
|
||
HDFS-2922. HA: close out operation categories (eli)
|
||
|
||
HDFS-2993. HA: BackupNode#checkOperation should permit CHECKPOINT operations
|
||
(eli)
|
||
|
||
HDFS-2904. Client support for getting delegation tokens. (todd)
|
||
|
||
HDFS-3013. HA: NameNode format doesn't pick up
|
||
dfs.namenode.name.dir.NameServiceId configuration (Mingjie Lai via todd)
|
||
|
||
HDFS-3019. Fix silent failure of TestEditLogJournalFailures (todd)
|
||
|
||
HDFS-2958. Sweep for remaining proxy construction which doesn't go through
|
||
failover path. (atm)
|
||
|
||
HDFS-2920. fix remaining TODO items. (atm and todd)
|
||
|
||
HDFS-3027. Implement a simple NN health check. (atm)
|
||
|
||
HDFS-3023. Optimize entries in edits log for persistBlocks call. (todd)
|
||
|
||
HDFS-2979. Balancer should use logical uri for creating failover proxy with HA
|
||
enabled. (atm)
|
||
|
||
HDFS-3035. Fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS (todd)
|
||
|
||
HDFS-3039. Address findbugs and javadoc warnings on branch. (todd via atm)
|
||
|
||
Release 0.23.10 - UNRELEASED
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-5010. Reduce the frequency of getCurrentUser() calls from namenode
|
||
(kihwal)
|
||
|
||
HDFS-5346. Avoid unnecessary call to getNumLiveDataNodes() for each block
|
||
during IBR processing (Ravi Prakash via kihwal)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4998. TestUnderReplicatedBlocks fails intermittently (kihwal)
|
||
|
||
HDFS-4329. DFSShell issues with directories with spaces in name (Cristina
|
||
L. Abad via jeagles)
|
||
|
||
HDFS-5526. Datanode cannot roll back to previous layout version (kihwal)
|
||
|
||
HDFS-5557. Write pipeline recovery for the last packet in the block may
|
||
cause rejection of valid replicas. (kihwal)
|
||
|
||
HDFS-5558. LeaseManager monitor thread can crash if the last block is
|
||
complete but another block is not. (kihwal)
|
||
|
||
Release 0.23.9 - 2013-07-08
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4867. metaSave NPEs when there are invalid blocks in repl queue.
|
||
(Plamen Jeliazkov and Ravi Prakash via shv)
|
||
|
||
HDFS-4862. SafeModeInfo.isManual() returns true when resources are low even
|
||
if it wasn't entered into manually (Ravi Prakash via kihwal)
|
||
|
||
HDFS-4832. Namenode doesn't change the number of missing blocks in
|
||
safemode when DNs rejoin or leave (Ravi Prakash via kihwal)
|
||
|
||
HDFS-4878. On Remove Block, block is not removed from neededReplications
|
||
queue. (Tao Luo via shv)
|
||
|
||
HDFS-4205. fsck fails with symlinks. (jlowe)
|
||
|
||
Release 0.23.8 - 2013-06-05
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-4714. Log short messages in Namenode RPC server for exceptions
|
||
meant for clients. (kihwal)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4477. Secondary namenode may retain old tokens (daryn via kihwal)
|
||
|
||
HDFS-4699. TestPipelinesFailover#testPipelineRecoveryStress fails
|
||
sporadically (Chris Nauroth via kihwal)
|
||
|
||
HDFS-4805. Webhdfs client is fragile to token renewal errors
|
||
(daryn via kihwal)
|
||
|
||
HDFS-3875. Issue handling checksum errors in write pipeline. (kihwal)
|
||
|
||
HDFS-4807. createSocketForPipeline() should not include timeout extension
|
||
on connect. (Cristina L. Abad via kihwal)
|
||
|
||
Release 0.23.7 - 2013-04-08
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-4532. RPC call queue may fill due to current user lookup (daryn)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4288. NN accepts incremental BR as IBR in safemode (daryn via kihwal)
|
||
|
||
HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit
|
||
(kihwal)
|
||
|
||
HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails
|
||
in the middle (kihwal via daryn)
|
||
|
||
HDFS-4542. Webhdfs doesn't support secure proxy users (Daryn Sharp via
|
||
kihwal)
|
||
|
||
HDFS-4560. Webhdfs cannot use tokens obtained by another user (daryn)
|
||
|
||
HDFS-4566. Webdhfs token cancelation should use authentication (daryn)
|
||
|
||
HDFS-4567. Webhdfs does not need a token for token operations (daryn via
|
||
kihwal)
|
||
|
||
HDFS-4577. Webhdfs operations should declare if authentication is required
|
||
(daryn via kihwal)
|
||
|
||
HDFS-3344. Unreliable corrupt blocks counting in TestProcessCorruptBlocks
|
||
(kihwal)
|
||
|
||
HDFS-3367. WebHDFS doesn't use the logged in user when opening
|
||
connections (daryn)
|
||
|
||
HDFS-4581. checkDiskError should not be called on network errors (Rohit
|
||
Kochar via kihwal)
|
||
|
||
HDFS-4649. Webhdfs cannot list large directories (daryn via kihwal)
|
||
|
||
HDFS-4548. Webhdfs doesn't renegotiate SPNEGO token (daryn via kihwal)
|
||
|
||
Release 0.23.6 - 2013-02-06
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-4247. saveNamespace should be tolerant of dangling lease (daryn)
|
||
|
||
HDFS-4248. Renaming directories may incorrectly remove the paths in leases
|
||
under the tree. (daryn via szetszwo)
|
||
|
||
HDFS-4385. Maven RAT plugin is not checking all source files (tgraves)
|
||
|
||
HDFS-4426. Secondary namenode shuts down immediately after startup.
|
||
(Arpit Agarwal via suresh)
|
||
|
||
Release 0.23.5 - 2012-11-28
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-4080. Add a separate logger for block state change logs to enable turning
|
||
off those logs. (Kihwal Lee via suresh)
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-4075. Reduce recommissioning overhead (Kihwal Lee via daryn)
|
||
|
||
HDFS-3990. NN's health report has severe performance problems (daryn)
|
||
|
||
HDFS-4181. LeaseManager tries to double remove and prints extra messages
|
||
(Kihwal Lee via daryn)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-3829. TestHftpURLTimeouts fails intermittently with JDK7 (Trevor
|
||
Robinson via tgraves)
|
||
|
||
HDFS-3824. TestHftpDelegationToken fails intermittently with JDK7 (Trevor
|
||
Robinson via tgraves)
|
||
|
||
HDFS-3224. Bug in check for DN re-registration with different storage ID
|
||
(jlowe)
|
||
|
||
HDFS-4090. getFileChecksum() result incompatible when called against
|
||
zero-byte files. (Kihwal Lee via daryn)
|
||
|
||
HDFS-4172. namenode does not URI-encode parameters when building URI for
|
||
datanode request (Derek Dagit via bobby)
|
||
|
||
HDFS-4182. SecondaryNameNode leaks NameCache entries (bobby)
|
||
|
||
HDFS-4186. logSync() is called with the write lock held while releasing
|
||
lease (Kihwal Lee via daryn)
|
||
|
||
Release 0.23.4
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-3831. Failure to renew tokens due to test-sources left in classpath
|
||
(jlowe via bobby)
|
||
|
||
Release 0.23.3
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. (Ravi
|
||
Prakash via szetszwo)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-3166. Add timeout to Hftp connections. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . (Kihwal
|
||
Lee via szetszwo)
|
||
|
||
HDFS-2652. Add support for host-based delegation tokens. (Daryn Sharp via
|
||
szetszwo)
|
||
|
||
HDFS-3308. Uses canonical URI to select delegation tokens in HftpFileSystem
|
||
and WebHdfsFileSystem. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the
|
||
delegation tokens have to use secure URI. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it
|
||
hangs on transfers >2 GB. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3321. Fix safe mode turn off tip message. (Ravi Prakash via szetszwo)
|
||
|
||
HDFS-3334. Fix ByteRangeInputStream stream leakage. (Daryn Sharp via
|
||
szetszwo)
|
||
|
||
HDFS-3331. In namenode, check superuser privilege for setBalancerBandwidth
|
||
and acquire the write lock for finalizeUpgrade. (szetszwo)
|
||
|
||
HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is
|
||
racy. (atm)
|
||
|
||
HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of
|
||
StreamingOutput, otherwise, it will fail to transfer large files.
|
||
(szetszwo)
|
||
|
||
HDFS-3646. LeaseRenewer can hold reference to inactive DFSClient
|
||
instances forever. (Kihwal Lee via daryn)
|
||
|
||
HDFS-3696. Set chunked streaming mode in WebHdfsFileSystem write operations
|
||
to get around a Java library bug causing OutOfMemoryError. (szetszwo)
|
||
|
||
HDFS-3553. Hftp proxy tokens are broken (daryn)
|
||
|
||
HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner
|
||
thread (Kihwal Lee via daryn)
|
||
|
||
HDFS-3861. Deadlock in DFSClient (Kihwal Lee via daryn)
|
||
|
||
HDFS-3873. Hftp assumes security is disabled if token fetch fails (daryn)
|
||
|
||
HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn)
|
||
|
||
HDFS-3890. filecontext mkdirs doesn't apply umask as expected
|
||
(Tom Graves via daryn)
|
||
|
||
Release 0.23.2 - UNRELEASED
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-2887. FSVolume, is a part of FSDatasetInterface implementation, should
|
||
not be referred outside FSDataset. A new FSVolumeInterface is defined.
|
||
The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also
|
||
updated. (szetszwo)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-2943. Expose last checkpoint time and transaction stats as JMX
|
||
metrics. (atm)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-2931. Switch DataNode's BlockVolumeChoosingPolicy to private-audience.
|
||
(harsh via szetszwo)
|
||
|
||
HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. (Brandon Li
|
||
via jitendra)
|
||
|
||
HDFS-2725. hdfs script usage information is missing the information
|
||
about "dfs" command (Prashant Sharma via stevel)
|
||
|
||
HDFS-2907. Add a conf property dfs.datanode.fsdataset.factory to make
|
||
FSDataset in Datanode pluggable. (szetszwo)
|
||
|
||
HDFS-2985. Improve logging when replicas are marked as corrupt. (todd)
|
||
|
||
HDFS-3098. Update and add tests for HADOOP-8173. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3104. Add tests for HADOOP-8175. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-3066. Cap space usage of default log4j rolling policy.
|
||
(Patrick Hunt via eli)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-3024. Improve performance of stringification in addStoredBlock (todd)
|
||
|
||
BUG FIXES
|
||
HDFS-2923. Namenode IPC handler count uses the wrong configuration key
|
||
(todd)
|
||
|
||
HDFS-2764. TestBackupNode is racy. (atm)
|
||
|
||
HDFS-2869. Fix an error in the webhdfs docs for the mkdir op (harsh)
|
||
|
||
HDFS-776. Fix exception handling in Balancer. (Uma Maheswara Rao G
|
||
via szetszwo)
|
||
|
||
HDFS-2815. Namenode sometimes oes not come out of safemode during
|
||
NN crash + restart. (Uma Maheswara Rao via suresh)
|
||
|
||
HDFS-2950. Secondary NN HTTPS address should be listed as a
|
||
NAMESERVICE_SPECIFIC_KEY. (todd)
|
||
|
||
HDFS-2525. Race between BlockPoolSliceScanner and append. (Brandon Li
|
||
via jitendra)
|
||
|
||
HDFS-2938. Recursive delete of a large directory make namenode
|
||
unresponsive. (Hari Mankude via suresh)
|
||
|
||
HDFS-2969. ExtendedBlock.equals is incorrectly implemented (todd)
|
||
|
||
HDFS-2944. Typo in hdfs-default.xml causes
|
||
dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly
|
||
disabled. (atm)
|
||
|
||
HDFS-2981. In hdfs-default.xml, the default value of
|
||
dfs.client.block.write.replace-datanode-on-failure.enable should be true.
|
||
(szetszwo)
|
||
|
||
HDFS-3008. Negative caching of local addrs doesn't work. (eli)
|
||
|
||
HDFS-3006. In WebHDFS, when the return body is empty, set the Content-Type
|
||
to application/octet-stream instead of application/json. (szetszwo)
|
||
|
||
HDFS-2991. Fix case where OP_ADD would not be logged in append(). (todd)
|
||
|
||
HDFS-3012. Exception while renewing delegation token. (Bobby Evans via
|
||
jitendra)
|
||
|
||
HDFS-3032. Change DFSClient.renewLease() so that it only retries up to the
|
||
lease soft-limit. (Kihwal Lee via szetszwo)
|
||
|
||
HDFS-2038. Update TestHDFSCLI to handle relative paths with globs.
|
||
(Kihwal Lee via szetszwo)
|
||
|
||
HDFS-3101. Cannot read empty file using WebHDFS. (szetszwo)
|
||
|
||
HDFS-3160. httpfs should exec catalina instead of forking it.
|
||
(Roman Shaposhnik via eli)
|
||
|
||
Release 0.23.1 - 2012-02-17
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-2316. [umbrella] WebHDFS: a complete FileSystem implementation for
|
||
accessing HDFS over HTTP (szetszwo)
|
||
|
||
HDFS-2594. Support getDelegationTokens and createSymlink in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2545. Change WebHDFS to support multiple namenodes in federation.
|
||
(szetszwo)
|
||
|
||
HDFS-2178. Contributing Hoop to HDFS, replacement for HDFS proxy
|
||
with read/write capabilities. (tucu)
|
||
|
||
|
||
IMPROVEMENTS
|
||
HDFS-2560. Refactor BPOfferService to be a static inner class (todd)
|
||
|
||
HDFS-2544. Hadoop scripts unconditionally source
|
||
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
|
||
|
||
HDFS-2543. HADOOP_PREFIX cannot be overridden. (Bruno Mahé via tomwhite)
|
||
|
||
HDFS-2562. Refactor DN configuration variables out of DataNode class
|
||
(todd)
|
||
|
||
HDFS-2563. Some cleanup in BPOfferService. (todd)
|
||
|
||
HDFS-2568. Use a set to manage child sockets in XceiverServer.
|
||
(harsh via eli)
|
||
|
||
HDFS-2454. Move maxXceiverCount check to before starting the
|
||
thread in dataXceiver. (harsh via eli)
|
||
|
||
HDFS-2570. Add descriptions for dfs.*.https.address in hdfs-default.xml.
|
||
(eli)
|
||
|
||
HDFS-2536. Remove unused imports. (harsh via eli)
|
||
|
||
HDFS-2566. Move BPOfferService to be a non-inner class. (todd)
|
||
|
||
HDFS-2552. Add Forrest doc for WebHDFS REST API. (szetszwo)
|
||
|
||
HDFS-2587. Add apt doc for WebHDFS REST API. (szetszwo)
|
||
|
||
HDFS-2604. Add a log message to show if WebHDFS is enabled and a
|
||
configuration section in the forrest doc. (szetszwo)
|
||
|
||
HDFS-2511. Add dev script to generate HDFS protobufs. (tucu)
|
||
|
||
HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. (eli)
|
||
|
||
HDFS-2675. Reduce warning verbosity when double-closing edit logs
|
||
(todd)
|
||
|
||
HDFS-2335. DataNodeCluster and NNStorage always pull fresh entropy.
|
||
(Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-2574. Remove references to some deprecated properties in conf
|
||
templates and defaults files. (Joe Crobak via harsh)
|
||
|
||
HDFS-2722. HttpFs should not be using an int for block size. (harsh)
|
||
|
||
HDFS-2710. Add HDFS tests related to HADOOP-7933. (Siddarth Seth via
|
||
suresh)
|
||
|
||
HDFS-2349. Corruption detected during block transfers between DNs
|
||
should log a WARN instead of INFO. (harsh)
|
||
|
||
HDFS-2729. Update BlockManager's comments regarding the invalid block
|
||
set (harsh)
|
||
|
||
HDFS-2726. Fix a logging issue under DFSClient's createBlockOutputStream
|
||
method (harsh)
|
||
|
||
HDFS-554. Use System.arraycopy in BlockInfo.ensureCapacity. (harsh)
|
||
|
||
HDFS-1314. Make dfs.blocksize accept size-indicating prefixes.
|
||
(Sho Shimauchi via harsh)
|
||
|
||
HDFS-69. Improve the 'dfsadmin' commandline help. (harsh)
|
||
|
||
HDFS-2788. HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code. (eli)
|
||
|
||
HDFS-362. FSEditLog should not writes long and short as UTF8, and should
|
||
not use ArrayWritable for writing non-array items. (Uma Maheswara Rao G
|
||
via szetszwo)
|
||
|
||
HDFS-2803. Add logging to LeaseRenewer for better lease expiration debugging.
|
||
(Jimmy Xiang via todd)
|
||
|
||
HDFS-2817. Combine the two TestSafeMode test suites. (todd)
|
||
|
||
HDFS-2818. Fix a missing space issue in HDFS webapps' title tags. (Devaraj K via harsh)
|
||
|
||
HDFS-2397. Undeprecate SecondaryNameNode. (eli)
|
||
|
||
HDFS-2814. NamenodeMXBean does not account for svn revision in the version
|
||
information. (Hitesh Shah via jitendra)
|
||
|
||
HDFS-2784. Update hftp and hdfs for host-based token support.
|
||
(Kihwal Lee via jitendra)
|
||
|
||
HDFS-2785. Update webhdfs and httpfs for host-based token support.
|
||
(Robert Joseph Evans via jitendra)
|
||
|
||
HDFS-2868. Expose xceiver counts via the DataNode MXBean. (harsh)
|
||
|
||
HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal
|
||
Lee via jitendra)
|
||
|
||
HDFS-208. name node should warn if only one dir is listed in dfs.name.dir.
|
||
(Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-3139. Minor Datanode logging improvement. (eli)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-2130. Switch default checksum to CRC32C. (todd)
|
||
|
||
HDFS-2533. Remove needless synchronization on some FSDataSet methods.
|
||
(todd)
|
||
|
||
HDFS-2129. Simplify BlockReader to not inherit from FSInputChecker.
|
||
(todd)
|
||
|
||
HDFS-2246. Enable reading a block directly from local file system
|
||
for a client on the same node as the block file. (Andrew Purtell,
|
||
Suresh Srinivas and Jitendra Nath Pandey via szetszwo)
|
||
|
||
HDFS-2825. Add test hook to turn off the writer preferring its local
|
||
DN. (todd)
|
||
|
||
HDFS-2826. Add test case for HDFS-1476 (safemode can initialize
|
||
replication queues before exiting) (todd)
|
||
|
||
HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION
|
||
from FSDataset. (szetszwo)
|
||
|
||
HDFS-2879. Change FSDataset to package private. (szetszwo)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-2541. For a sufficiently large value of blocks, the DN Scanner
|
||
may request a random number with a negative seed value. (harsh via eli)
|
||
|
||
HDFS-2502. hdfs-default.xml should include dfs.name.dir.restore.
|
||
(harsh via eli)
|
||
|
||
HDFS-2567. When 0 DNs are available, show a proper error when
|
||
trying to browse DFS via web UI. (harsh via eli)
|
||
|
||
HDFS-2575. DFSTestUtil may create empty files (todd)
|
||
|
||
HDFS-2588. hdfs jsp pages missing DOCTYPE. (Dave Vronay via mattf)
|
||
|
||
HDFS-2590. Fix the missing links in the WebHDFS forrest doc. (szetszwo)
|
||
|
||
HDFS-2596. TestDirectoryScanner doesn't test parallel scans. (eli)
|
||
|
||
HDFS-2606. webhdfs client filesystem impl must set the content-type
|
||
header for create/append. (tucu)
|
||
|
||
HDFS-2614. hadoop dist tarball is missing hdfs headers. (tucu)
|
||
|
||
HDFS-2653. DFSClient should cache whether addrs are non-local when
|
||
short-circuiting is enabled. (eli)
|
||
|
||
HDFS-2649. eclipse:eclipse build fails for hadoop-hdfs-httpfs.
|
||
(Jason Lowe via eli)
|
||
|
||
HDFS-2640. Javadoc generation hangs. (tomwhite)
|
||
|
||
HDFS-2553. Fix BlockPoolSliceScanner spinning in a tight loop (Uma
|
||
Maheswara Rao G via todd)
|
||
|
||
HDFS-2658. HttpFS introduced 70 javadoc warnings. (tucu)
|
||
|
||
HDFS-2706. Use configuration for blockInvalidateLimit if it is set.
|
||
(szetszwo)
|
||
|
||
HDFS-2646. Hadoop HttpFS introduced 4 findbug warnings. (tucu)
|
||
|
||
HDFS-2657. TestHttpFSServer and TestServerWebApp are failing on trunk. (tucu)
|
||
|
||
HDFS-2705. HttpFS server should check that upload requests have correct
|
||
content-type. (tucu)
|
||
|
||
HDFS-2707. HttpFS should read the hadoop-auth secret from a file instead
|
||
inline from the configuration. (tucu)
|
||
|
||
HDFS-2790. FSNamesystem.setTimes throws exception with wrong
|
||
configuration name in the message. (Arpit Gupta via eli)
|
||
|
||
HDFS-2810. Leases not getting renewed properly by clients (todd)
|
||
|
||
HDFS-2751. Datanode may incorrectly drop OS cache behind reads
|
||
even for short reads. (todd)
|
||
|
||
HDFS-2816. Fix missing license header in httpfs findbugsExcludeFile.xml.
|
||
(hitesh via tucu)
|
||
|
||
HDFS-2822. processMisReplicatedBlock incorrectly identifies
|
||
under-construction blocks as under-replicated. (todd)
|
||
|
||
HDFS-442. dfsthroughput in test jar throws NPE (harsh)
|
||
|
||
HDFS-2836. HttpFSServer still has 2 javadoc warnings in trunk.
|
||
(revans2 via tucu)
|
||
|
||
HDFS-2837. mvn javadoc:javadoc not seeing LimitedPrivate class.
|
||
(revans2 via tucu)
|
||
|
||
HDFS-2840. TestHostnameFilter should work with localhost or
|
||
localhost.localdomain (tucu)
|
||
|
||
HDFS-2791. If block report races with closing of file, replica is
|
||
incorrectly marked corrupt. (todd)
|
||
|
||
HDFS-2827. When the parent of a directory is the root, renaming the
|
||
directory results in leases updated incorrectly. (Uma Maheswara Rao G
|
||
via szetszwo)
|
||
|
||
HDFS-2835. Fix findbugs and javadoc issue with GetConf.java.
|
||
(suresh)
|
||
|
||
HDFS-2889. getNumCurrentReplicas is package private but should be public on
|
||
0.23 (see HDFS-2408). (Gregory Chanan via atm)
|
||
|
||
HDFS-2893. The start/stop scripts don't start/stop the 2NN when
|
||
using the default configuration. (eli)
|
||
|
||
Release 0.23.0 - 2011-11-01
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-1526. Dfs client name for a map/reduce task should be unique
|
||
among threads. (hairong)
|
||
|
||
HDFS-1536. Improve HDFS WebUI. (hairong)
|
||
|
||
HDFS-2210. Remove hdfsproxy. (eli)
|
||
|
||
HDFS-1073. Redesign the NameNode's storage layout for image checkpoints
|
||
and edit logs to introduce transaction IDs and be more robust.
|
||
Please see HDFS-1073 section below for breakout of individual patches.
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-1359. Add BlockPoolID to Block. (suresh)
|
||
|
||
HDFS-1365. Federation: propose ClusterID and BlockPoolID format
|
||
(Tanping via boryas)
|
||
|
||
HDFS-1394. Federation: modify -format option for namenode to generated
|
||
new blockpool id and accept newcluster (boryas)
|
||
|
||
HDFS-1400. Federation: DataTransferProtocol uses ExtendedBlockPool to
|
||
include BlockPoolID in the protocol. (suresh)
|
||
|
||
HDFS-1428. Federation : add cluster ID and block pool ID into
|
||
Name node web UI(Tanping via boryas)
|
||
|
||
HDFS-1450. Federation: Introduce block pool ID into FSDatasetInterface.
|
||
(suresh)
|
||
|
||
HDFS-1632. Federation: data node storage structure changes and
|
||
introduce block pool storage. (Tanping via suresh)
|
||
|
||
HDFS-1634. Federation: Convert single threaded DataNode into
|
||
per BlockPool thread model.(boryas)
|
||
|
||
HDFS-1637. Federation: FSDataset in Datanode should be created after
|
||
initial handshake with namenode. (boryas and jitendra)
|
||
|
||
HDFS-1653. Federation: Block received message from datanode sends invalid
|
||
DatanodeRegistration. (Tanping via suresh)
|
||
|
||
HDFS-1645. Federation: DatanodeCommond.Finalize needs to include
|
||
BlockPoolId. (suresh)
|
||
|
||
HDFS-1638. Federation: DataNode.handleDiskError needs to inform
|
||
ALL namenodes if a disk failed (boryas)
|
||
|
||
HDFS-1647. Federation: Multiple namenode configuration. (jitendra)
|
||
|
||
HDFS-1639. Federation: Add block pool management to FSDataset. (suresh)
|
||
|
||
HDFS-1648. Federation: Only DataStorage must be locked using in_use.lock
|
||
and no locks must be associated with BlockPoolStorage. (Tanping via suresh)
|
||
|
||
HDFS-1641. Federation: Datanode fields that are no longer used should
|
||
be removed (boryas)
|
||
|
||
HDFS-1642. Federation: add Datanode.getDNRegistration(String bpid)
|
||
method (boryas)
|
||
|
||
HDFS-1643. Federation: remove namenode argument from DataNode
|
||
constructor (boryas)
|
||
|
||
HDFS-1657. Federation: Tests that corrupt block files fail due to changed
|
||
file path in federation. (suresh)
|
||
|
||
HDFS-1661. Federation: Remove unnecessary TODO:FEDERATION comments.
|
||
(jitendra)
|
||
|
||
HDFS-1660. Federation: Datanode doesn't start with two namenodes (boryas)
|
||
|
||
HDFS-1650. Federation: TestReplication fails. (Tanping via suresh)
|
||
|
||
HDFS-1651. Federation: Tests fail due to null pointer exception in
|
||
Datnode#shutdown() method. (Tanping via suresh)
|
||
|
||
HDFS-1649. Federation: Datanode command to refresh namenode list at
|
||
the datanode. (jitendra)
|
||
|
||
HDFS-1646. Federation: MiniDFSClsuter#waitActive() waits for ever
|
||
with the introduction of BPOfferService in datanode. (suresh)
|
||
|
||
HDFS-1659. Federation: BPOfferService exits after one iteration
|
||
incorrectly. (Tanping via suresh)
|
||
|
||
HDFS-1654. Federation: Fix TestDFSUpgrade and TestDFSRollback failures.
|
||
(suresh)
|
||
|
||
HDFS-1668. Federation: Datanodes sends block pool usage information
|
||
to the namenode in heartbeat. (suresh)
|
||
|
||
HDFS-1669. Federation: Fix TestHftpFileSystem failure. (suresh)
|
||
|
||
HDFS-1670. Federation: remove dnRegistration from Datanode (boryas)
|
||
|
||
HDFS-1662. Federation: fix unit test case, TestCheckpoint
|
||
and TestDataNodeMXBean (tanping via boryas)
|
||
|
||
HDFS-1671. Federation: shutdown in DataNode should be able to
|
||
shutdown individual BP threads as well as the whole DN (boryas).
|
||
|
||
HDFS-1663. Federation: Rename getPoolId() everywhere to
|
||
getBlockPoolId() (tanping via boryas)
|
||
|
||
HDFS-1652. FederationL Add support for multiple namenodes in
|
||
MiniDFSCluster. (suresh)
|
||
|
||
HDFS-1672. Federation: refactor stopDatanode(name) to work
|
||
with multiple Block Pools (boryas)
|
||
|
||
HDFS-1687. Federation: DirectoryScanner changes for
|
||
federation (Matt Foley via boryas)
|
||
|
||
HDFS-1626. Make BLOCK_INVALIDATE_LIMIT configurable. (szetszwo)
|
||
|
||
HDFS-1655. Federation: DatablockScanner should scan blocks for
|
||
all the block pools. (jitendra)
|
||
|
||
HDFS-1664. Federation: Add block pool storage usage to Namenode WebUI.
|
||
(Tanping via suresh)
|
||
|
||
HDFS-1674. Federation: Rename BlockPool class to BlockPoolSlice.
|
||
(jghoman, Tanping via suresh)
|
||
|
||
HDFS-1673. Federation: Datanode changes to track block token secret per
|
||
namenode. (suresh)
|
||
|
||
HDFS-1677. Federation: Fix TestFsck and TestListCorruptFileBlocks
|
||
failures. (Tanping via suresh)
|
||
|
||
HDFS-1678. Federation: Remove unnecessary #getBlockpool()
|
||
for NameNodeMXBean in FSNameSystem. (Tanping via Suresh)
|
||
|
||
HDFS-1688. Federation: Fix failures in fault injection tests,
|
||
TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. (suresh)
|
||
|
||
HDFS-1696. Federation: when build version doesn't match -
|
||
datanode should wait (keep connecting) untill NN comes up
|
||
with the right version (boryas)
|
||
|
||
HDFS-1681. Balancer: support per pool and per node policies. (szetszwo)
|
||
|
||
HDFS-1695. Federation: Fix testOIV and TestDatanodeUtils
|
||
(jhoman and tanping via boryas)
|
||
|
||
HDFS-1699. Federation: Fix failure of TestBlockReport.
|
||
(Matt Foley via suresh)
|
||
|
||
HDFS-1698. Federation: TestOverReplicatedBlocks and TestWriteToReplica
|
||
failing. (jhoman and jitendra)
|
||
|
||
HDFS-1701. Federation: Fix TestHeartbeathandling.
|
||
(Erik Steffl and Tanping Wang via suresh)
|
||
|
||
HDFS-1693. Federation: Fix TestDFSStorageStateRecovery failure. (suresh)
|
||
|
||
HDFS-1694. Federation: SimulatedFSDataset changes to work with
|
||
federation and multiple block pools. (suresh)
|
||
|
||
HDFS-1689. Federation: Configuration for namenodes. (suresh and jitendra)
|
||
|
||
HDFS-1682. Change Balancer CLI for multiple namenodes and balancing
|
||
policy. (szetszwo)
|
||
|
||
HDFS-1697. Federation: fix TestBlockRecovery (boryas)
|
||
|
||
HDFS-1702. Federation: fix TestBackupNode and TestRefreshNamendoes
|
||
failures. (suresh)
|
||
|
||
HDFS-1706. Federation: TestFileAppend2, TestFileAppend3 and
|
||
TestBlockTokenWithDFS failing. (jitendra)
|
||
|
||
HDFS-1704. Federation: Add a tool that lists namenodes, secondary and
|
||
backup from configuration file. (suresh)
|
||
|
||
HDFS-1711. Federation: create method for updating machine name in
|
||
DataNode.java (boryas)
|
||
|
||
HDFS-1712. Federation: when looking up datanode we should use machineNmae
|
||
(in testOverReplicatedBlocks) (boryas)
|
||
|
||
HDFS-1709. Federation: Error "nnaddr url param is null" when clicking on a
|
||
node from NN Live Node Link. (jitendra)
|
||
|
||
HDFS-1714. Federation: refactor upgrade object in DataNode (boryas)
|
||
|
||
HDFS-1715. Federation: warning/error not generated when datanode sees
|
||
inconsistent/different Cluster ID between namenodes (boryas)
|
||
|
||
HDFS-1715. Federation: warning/error not generated when datanode sees
|
||
inconsistent/different Cluster ID between namenodes (boryas)
|
||
|
||
HDFS-1716. Federation: Add decommission tests for federated namenodes.
|
||
(suresh)
|
||
|
||
HDFS-1713. Federation: Prevent DataBlockScanner from running in tight loop.
|
||
(jitendra)
|
||
|
||
HDFS-1721. Federation: Configuration for principal names should not be
|
||
namenode specific. (jitendra)
|
||
|
||
HDFS-1717. Federation: FSDataset volumeMap access is not synchronized
|
||
correctly. (suresh)
|
||
|
||
HDFS-1722. Federation: Add flag to MiniDFSCluser to differentiate between
|
||
federation and non-federation modes. (boryas via suresh)
|
||
|
||
HDFS-1718. Federation: MiniDFSCluster#waitActive() bug causes some tests
|
||
to fail. (suresh)
|
||
|
||
HDFS-1719. Federation: Fix TestDFSRemove that fails intermittently.
|
||
(suresh)
|
||
|
||
HDFS-1720. Federation: FSVolumeSet volumes is not synchronized correctly.
|
||
(suresh)
|
||
|
||
HDFS-1700. Federation: fsck needs to work with federation changes.
|
||
(Matt Foley via suresh)
|
||
|
||
HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem.
|
||
(Patrick Kling via hairong)
|
||
|
||
HDFS-1448. Add a new tool Offline Edits Viewer (oev). (Erik Steffl
|
||
via szetszwo)
|
||
|
||
HDFS-1735. Federation: merge FSImage change in federation to
|
||
FSImage+NNStorage refactoring in trunk. (suresh)
|
||
|
||
HDFS-1737. Federation: Update the layout version for federation
|
||
changes. (suresh)
|
||
|
||
HDFS-1744. Federation: Add new layout version to offline image viewer
|
||
and edits viewer. (suresh)
|
||
|
||
HDFS-1745. Federation: Fix fault injection test failures. (suresh)
|
||
|
||
HDFS-1746. Federation: TestFileAppend3 fails intermittently. (jitendra)
|
||
|
||
HDFS-1703. Improve start/stop scripts and add decommission tool for
|
||
federation. (Tanping Wang, Erik Steffl via suresh)
|
||
|
||
HDFS-1749. Federation: TestListCorruptFileBlocks failing in federation
|
||
branch. (jitendra)
|
||
|
||
HDFS-1754. Federation: testFsck fails. (boryas)
|
||
|
||
HDFS-1755. Federation: The BPOfferService must always connect to namenode
|
||
as the login user. (jitendra)
|
||
|
||
HDFS-1675. Support transferring RBW between datanodes. (szetszwo)
|
||
|
||
HDFS-1791. Federation: Add command to delete block pool directories
|
||
from a datanode. (jitendra)
|
||
|
||
HDFS-1761. Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK,
|
||
for transferring RBW/Finalized with acknowledgement and without using RPC.
|
||
(szetszwo)
|
||
|
||
HDFS-1813. Federation: Authentication using BlockToken in RPC to datanode
|
||
fails. (jitendra)
|
||
|
||
HDFS-1630. Support fsedits checksum. (hairong)
|
||
|
||
HDFS-1606. Provide a stronger data guarantee in the write pipeline by
|
||
adding a new datanode when an existing datanode failed. (szetszwo)
|
||
|
||
HDFS-1442. Api to get delegation token in Hdfs class. (jitendra)
|
||
|
||
HDFS-1070. Speedup namenode image loading and saving by storing only
|
||
local file names. (hairong)
|
||
|
||
HDFS-1751. Intrinsic limits for HDFS files, directories (daryn via boryas).
|
||
|
||
HDFS-1873. Federation: Add cluster management web console.
|
||
(Tanping Wang via suresh)
|
||
|
||
HDFS-1911 HDFS tests for the newly added viewfs
|
||
|
||
HDFS-1814. Add "hdfs groups" command to query the server-side groups
|
||
resolved for a user. (Aaron T. Myers via todd)
|
||
|
||
HDFS-1914. Federation: namenode storage directories must be configurable
|
||
specific to name service. (suresh)
|
||
|
||
HDFS-1963. Create RPM and Debian packages for HDFS. Changes deployment
|
||
layout to be consistent across the binary tgz, rpm, and deb.
|
||
(Eric Yang via omalley)
|
||
|
||
HDFS-2058. Change Data Transfer wire protocol to use protocol buffers.
|
||
(todd)
|
||
|
||
HDFS-2055. Add hflush support to libhdfs. (Travis Crawford via eli)
|
||
|
||
HDFS-2083. Query JMX statistics over http via JMXJsonServlet. (tanping)
|
||
|
||
HDFS-2156. Make hdfs and mapreduce rpm only depend on the same major
|
||
version for common and hdfs. (eyang via omalley)
|
||
|
||
HDFS-2202. Add a new DFSAdmin command to set balancer bandwidth of
|
||
datanodes without restarting. (Eric Payne via szetszwo)
|
||
|
||
HDFS-2284. Add a new FileSystem, webhdfs://, for supporting write Http
|
||
access to HDFS. (szetszwo)
|
||
|
||
HDFS-2317. Support read access to HDFS in WebHDFS. (szetszwo)
|
||
|
||
HDFS-2338. Add configuration option to enable/disable WebHDFS.
|
||
(jitendra via szetszwo)
|
||
|
||
HDFS-2318. Provide authentication to WebHDFS using SPNEGO and delegation
|
||
tokens. (szetszwo)
|
||
|
||
HDFS-2340. Support getFileBlockLocations and getDelegationToken in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2348. Support getContentSummary and getFileChecksum in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2385. Support renew and cancel delegation tokens in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2539. Support doAs and GETHOMEDIRECTORY in WebHDFS.
|
||
(szetszwo)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-1875. MiniDFSCluster hard-codes dfs.datanode.address to localhost
|
||
(Eric Payne via mattf)
|
||
|
||
HDFS-2019. Fix all the places where Java method File.list is used with
|
||
FileUtil.list API (Bharath Mundlapudi via mattf)
|
||
|
||
HDFS-1934. Fix NullPointerException when certain File APIs return null
|
||
(Bharath Mundlapudi via mattf)
|
||
|
||
HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel)
|
||
|
||
HDFS-1628. Display full path in AccessControlException. (John George
|
||
via szetszwo)
|
||
|
||
HDFS-1707. Federation: Failure in browsing data on new namenodes.
|
||
(jitendra)
|
||
|
||
HDFS-1683. Test Balancer with multiple NameNodes. (szetszwo)
|
||
|
||
HDFS-1547. Improve decommission mechanism. (suresh)
|
||
|
||
HDFS-2143. Federation: In cluster web console, add link to namenode page
|
||
that displays live and dead datanodes. (Ravi Prakash via suresh)
|
||
|
||
HDFS-1588. Remove hardcoded strings for configuration keys, "dfs.hosts"
|
||
and "dfs.hosts.exlude". (Erik Steffl via suresh)
|
||
|
||
HDFS-1481. NameNode should validate fsimage before rolling. (hairong)
|
||
|
||
HDFS-1506. Refactor fsimage loading code. (hairong)
|
||
|
||
HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API
|
||
(HDFS portion) (Patrick Kling via hairong)
|
||
|
||
HDFS-1476. listCorruptFileBlocks should be functional while the
|
||
name node is in safe mode. (Patrick Kling via hairong)
|
||
|
||
HDFS-1534. Fix some incorrect logs in FSDirectory. (eli)
|
||
|
||
HDFS-1539. A config option for the datanode to fsycn a block file
|
||
when block is completely written. (dhruba)
|
||
|
||
HDFS-1335. HDFS side change of HADDOP-6904: RPC compatibility. (hairong)
|
||
|
||
HDFS-1557. Separate Storage from FSImage. (Ivan Kelly via jitendra)
|
||
|
||
HDFS-560 Enhancements/tuning to hadoop-hdfs/build.xml
|
||
|
||
HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen
|
||
nodes in the output array. (szetszwo)
|
||
|
||
HDFS-1731. Allow using a file to exclude certain tests from build (todd)
|
||
|
||
HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell.
|
||
(Daryn Sharp via szetszwo)
|
||
|
||
HDFS-780. Revive TestFuseDFS. (eli)
|
||
|
||
HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink().
|
||
(Matt Foley via jghoman)
|
||
|
||
HDFS-1763. Replace hard-coded option strings with variables from
|
||
DFSConfigKeys. (eli)
|
||
|
||
HDFS-1541. Not marking datanodes dead when namenode in safemode.
|
||
(hairong)
|
||
|
||
HDFS-1120. Make DataNode's block-to-device placement policy pluggable
|
||
(Harsh J Chouraria via todd)
|
||
|
||
HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used
|
||
multiple times for determining whether the source is a client or a
|
||
datanode. (szetszwo)
|
||
|
||
HDFS-1789. Refactor frequently used codes from DFSOutputStream and
|
||
DataXceiver. (szetszwo)
|
||
|
||
HDFS-1767. Namenode ignores non-initial block report from datanodes
|
||
when in safemode during startup. (Matt Foley via suresh)
|
||
|
||
HDFS-1817. Move pipeline_Fi_[39-51] from TestFiDataTransferProtocol
|
||
to TestFiPipelineClose. (szetszwo)
|
||
|
||
HDFS-1760. In FSDirectory.getFullPathName(..), it is better to return "/"
|
||
for root directory instead of an empty string. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-1833. Reduce repeated string constructions and unnecessary fields,
|
||
and fix comments in BlockReceiver.PacketResponder. (szetszwo)
|
||
|
||
HDFS-1486. Generalize CLITest structure and interfaces to faciliate
|
||
upstream adoption (e.g. for web testing). (cos)
|
||
|
||
HDFS-1844. Move "fs -help" shell command tests from HDFS to COMMOM; see
|
||
also HADOOP-7230. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-1840. In DFSClient, terminate the lease renewing thread when all files
|
||
being written are closed for a grace period, and start a new thread when
|
||
new files are opened for write. (szetszwo)
|
||
|
||
HDFS-1854. make failure message more useful in
|
||
DFSTestUtil.waitReplication(). (Matt Foley via eli)
|
||
|
||
HDFS-1562. Add rack policy tests. (eli)
|
||
|
||
HDFS-1856. TestDatanodeBlockScanner waits forever, errs without giving
|
||
information. (Matt Foley via eli)
|
||
|
||
HDFS-1295. Improve namenode restart times by short-circuiting the
|
||
first block reports from datanodes. (Matt Foley via suresh)
|
||
Corrected merge error in DataNode.java. (Matt Foley)
|
||
|
||
HDFS-1843. Discover file not found early for file append.
|
||
(Bharath Mundlapudi via jitendra)
|
||
|
||
HDFS-1862. Improve test reliability of HDFS-1594. (Aaron T. Myers via eli)
|
||
|
||
HDFS-1846. Preallocate edit log with OP_INVALID instead of zero bytes
|
||
to ensure blocks are actually allocated. (Aaron T. Myers via todd)
|
||
|
||
HDFS-1741. Provide a minimal pom file to allow integration of HDFS into Sonar
|
||
analysis (cos)
|
||
|
||
HDFS-1870. Move and rename DFSClient.LeaseChecker to a seperated class
|
||
LeaseRenewer. (szetszwo)
|
||
|
||
HDFS-1866. Document dfs.datanode.max.transfer.threads in hdfs-default.xml
|
||
(Harsh J Chouraria via todd)
|
||
|
||
HDFS-1890. Improve the name, class and value type of the map
|
||
LeaseRenewer.pendingCreates. (szetszwo)
|
||
|
||
HDFS-1865. Share LeaseRenewer among DFSClients so that there is only a
|
||
LeaseRenewer thread per namenode per user. (szetszwo)
|
||
|
||
HDFS-1906. Remove logging exception stack trace in client logs when one of
|
||
the datanode targets to read from is not reachable. (suresh)
|
||
|
||
HDFS-1378. Edit log replay should track and report file offsets in case of
|
||
errors. (Aaron T. Myers and Todd Lipcon via todd)
|
||
|
||
HDFS-1917. Separate hdfs jars from common in ivy configuration. (Eric Yang
|
||
via szetszwo)
|
||
|
||
HDFS-1899. GenericTestUtils.formatNamenode should be moved to DFSTestUtil
|
||
(Ted Yu via todd)
|
||
|
||
HDFS-1117. Metrics 2.0 HDFS instrumentation. (Luke Lu via suresh)
|
||
|
||
HDFS-1946. HDFS part of HADOOP-7291. (eli)
|
||
|
||
HDFS-1945. Removed the deprecated fields in DataTransferProtocol.
|
||
(szetszwo)
|
||
|
||
HDFS-1730. Use DaemonFactory from common and delete it from HDFS.
|
||
(Tanping via suresh)
|
||
|
||
HDFS-1573. Add useful tracing information to Lease Renewer thread names
|
||
(todd)
|
||
|
||
HDFS-1939. In ivy.xml, test conf should not extend common conf.
|
||
(Eric Yang via szetszwo)
|
||
|
||
HDFS-1332 Include more information in exceptions and debug messages
|
||
when BlockPlacementPolicy cannot be satisfied. (Ted Yu via szetszwo)
|
||
|
||
HDFS-1958. Confirmation should be more lenient of user input when
|
||
formatting the NameNode. (todd)
|
||
|
||
HDFS-1905. Improve namenode -format command by not making -clusterId
|
||
parameter mandatory. (Bharath Mundlapudi via suresh)
|
||
|
||
HDFS-1877. Add a new test for concurrent read and write. (CW Chung
|
||
via szetszwo)
|
||
|
||
HDFS-1959. Better error message for missing namenode directory. (eli)
|
||
|
||
HDFS-1996. ivy: hdfs test jar should be independent to common test jar.
|
||
(Eric Yang via szetszwo)
|
||
|
||
HDFS-1812. TestHDFSCLI should clean up cluster in teardown method.
|
||
(Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-1884. Improve TestDFSStorageStateRecovery to properly throw in the
|
||
case of errors. (Aaron T. Myers via todd)
|
||
|
||
HDFS-1727. fsck command should display command usage if user passes any
|
||
illegal argument. (Sravan Kumar via todd)
|
||
|
||
HDFS-1636. If dfs.name.dir points to an empty dir, namenode format
|
||
shouldn't require confirmation. (Harsh J Chouraria via todd)
|
||
|
||
HDFS-1966. Encapsulate individual DataTransferProtocol op headers.
|
||
(szetszwo)
|
||
|
||
HDFS-2024. Format TestWriteRead source codes. (CW Chung via szetszwo)
|
||
|
||
HDFS-1968. Enhance TestWriteRead to support position/sequential read,
|
||
append, truncate and verbose options. (CW Chung via szetszwo)
|
||
|
||
HDFS-1986. Add option to get http/https address from
|
||
DFSUtil#getInfoServer(). (Tanping via suresh)
|
||
|
||
HDFS-2029. In TestWriteRead, check visible length immediately after
|
||
openning the file and fix code style. (John George via szetszwo)
|
||
|
||
HDFS-2040. Only build libhdfs if a flag is passed. (eli)
|
||
|
||
HDFS-1586. Add InterfaceAudience and InterfaceStability annotations to
|
||
MiniDFSCluster. (suresh)
|
||
|
||
HDFS-2003. Separate FSEditLog reading logic from edit log memory state
|
||
building logic. (Ivan Kelly via todd)
|
||
|
||
HDFS-2066. Create a package and individual class files for
|
||
DataTransferProtocol. (szetszwo)
|
||
|
||
HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects.
|
||
(todd, nigel)
|
||
|
||
HDFS-2046. Force entropy to come from non-true random for tests. (todd)
|
||
|
||
HDFS-2073. Add @Override annotation to NameNode. (suresh)
|
||
|
||
HDFS-420. Fuse-dfs should cache fs handles. (Brian Bockelman and eli)
|
||
|
||
HDFS-1568. Improve the log messages in DataXceiver. (Joey Echeverria via
|
||
szetszwo)
|
||
|
||
HDFS-2100. Improve TestStorageRestore. (atm)
|
||
|
||
HDFS-2092. Remove some object references to Configuration in DFSClient.
|
||
(Bharath Mundlapudi via szetszwo)
|
||
|
||
HDFS-2087. Declare methods in DataTransferProtocol interface, and change
|
||
Sender and Receiver to implement the interface. (szetszwo)
|
||
|
||
HDFS-1723. quota errors messages should use the same scale. (Jim Plush via
|
||
atm)
|
||
|
||
HDFS-2110. StreamFile and ByteRangeInputStream cleanup. (eli)
|
||
|
||
HDFS-2107. Move block management code from o.a.h.h.s.namenode to a new
|
||
package o.a.h.h.s.blockmanagement. (szetszwo)
|
||
|
||
HDFS-2109. Store uMask as member variable to DFSClient.Conf. (Bharath
|
||
Mundlapudi via szetszwo)
|
||
|
||
HDFS-2111. Add tests for ensuring that the DN will start with a few bad
|
||
data directories. (Harsh J Chouraria via todd)
|
||
|
||
HDFS-2134. Move DecommissionManager to the blockmanagement package.
|
||
(szetszwo)
|
||
|
||
HDFS-1977. Stop using StringUtils.stringifyException().
|
||
(Bharath Mundlapudi via jitendra)
|
||
|
||
HDFS-2131. Add new tests for the -overwrite/-f option in put and
|
||
copyFromLocal by HADOOP-7361. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2140. Move Host2NodesMap to the blockmanagement package. (szetszwo)
|
||
|
||
HDFS-2154. In TestDFSShell, use TEST_ROOT_DIR and fix some deprecated
|
||
warnings. (szetszwo)
|
||
|
||
HDFS-2153. Move DFSClientAdapter to test and fix some javac warnings in
|
||
OfflineEditsViewerHelper. (szetszwo)
|
||
|
||
HDFS-2159. Deprecate DistributedFileSystem.getClient() and fixed the
|
||
deprecated warnings in DFSAdmin. (szetszwo)
|
||
|
||
HDFS-2157. Improve header comment in o.a.h.hdfs.server.namenode.NameNode.
|
||
(atm via eli)
|
||
|
||
HDFS-2147. Move cluster network topology to block management and fix some
|
||
javac warnings. (szetszwo)
|
||
|
||
HDFS-2141. Remove NameNode roles Active and Standby (they become
|
||
states of the namenode). (suresh)
|
||
|
||
HDFS-2161. Move createNamenode(..), createClientDatanodeProtocolProxy(..)
|
||
and Random object creation to DFSUtil; move DFSClient.stringifyToken(..)
|
||
to DelegationTokenIdentifier. (szetszwo)
|
||
|
||
HDFS-1774. Small optimization to FSDataset. (Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-2167. Move dnsToSwitchMapping and hostsReader from FSNamesystem to
|
||
DatanodeManager. (szetszwo)
|
||
|
||
HDFS-2116. Use Mokito in TestStreamFile and TestByteRangeInputStream.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-2112. Move ReplicationMonitor to block management. (Uma Maheswara
|
||
Rao G via szetszwo)
|
||
|
||
HDFS-1739. Add available volume size to the error message when datanode
|
||
throws DiskOutOfSpaceException. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2144. If SNN shuts down during initialization it does not log the
|
||
cause. (Ravi Prakash via atm)
|
||
|
||
HDFS-2180. Refactor NameNode HTTP server into new class. (todd)
|
||
|
||
HDFS-2198. Remove hardcoded configuration keys. (suresh)
|
||
|
||
HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp
|
||
implementations. (Ivan Kelly via todd)
|
||
|
||
HDFS-2191. Move datanodeMap from FSNamesystem to DatanodeManager.
|
||
(szetszwo)
|
||
|
||
HDFS-2200. Change FSNamesystem.LOG to package private. (szetszwo)
|
||
|
||
HDFS-2195. Refactor StorageDirectory to not be an non-static inner class.
|
||
(todd via eli)
|
||
|
||
HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams.
|
||
(todd via eli)
|
||
|
||
HDFS-2199. Move blockTokenSecretManager from FSNamesystem to BlockManager.
|
||
(Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2187. Make EditLogInputStream act like an iterator over FSEditLogOps
|
||
(Ivan Kelly and todd via todd)
|
||
|
||
HDFS-2225. Refactor edit log file management so it's not in classes
|
||
which should be generic to the type of edit log storage. (Ivan Kelly
|
||
via todd)
|
||
|
||
HDFS-2108. Move datanode heartbeat handling from namenode package to
|
||
blockmanagement package. (szetszwo)
|
||
|
||
HDFS-2226. Clean up counting of operations in FSEditLogLoader (todd)
|
||
|
||
HDFS-2228. Move block and datanode code from FSNamesystem to
|
||
BlockManager and DatanodeManager. (szetszwo)
|
||
|
||
HDFS-2238. In NamenodeFsck.toString(), uses StringBuilder.(..) instead of
|
||
string concatenation. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2230. ivy to resolve/retrieve latest common-tests jar published by
|
||
hadoop common maven build. (gkesavan)
|
||
|
||
HDFS-2227. getRemoteEditLogManifest should pull its information from
|
||
FileJournalManager during checkpoint process (Ivan Kelly and Todd Lipcon
|
||
via todd)
|
||
|
||
HDFS-2239. Reduce access levels of the fields and methods in FSNamesystem.
|
||
(szetszwo)
|
||
|
||
HDFS-2241. Remove implementing FSConstants interface to just get the
|
||
constants from the interface. (suresh)
|
||
|
||
HDFS-2237. Change UnderReplicatedBlocks from public to package private.
|
||
(szetszwo)
|
||
|
||
HDFS-2233. Add WebUI tests with URI reserved chars. (eli)
|
||
|
||
HDFS-2265. Remove unnecessary BlockTokenSecretManager fields/methods from
|
||
BlockManager. (szetszwo)
|
||
|
||
HDFS-2260. Refactor BlockReader into an interface and implementation.
|
||
(todd)
|
||
|
||
HDFS-2096. Mavenization of hadoop-hdfs (Alejandro Abdelnur via tomwhite)
|
||
|
||
HDFS-2273. Refactor BlockManager.recentInvalidateSets to a new class.
|
||
(szetszwo)
|
||
|
||
HDFS-2266. Add Namesystem and SafeMode interfaces to avoid directly
|
||
referring to FSNamesystem in BlockManager. (szetszwo)
|
||
|
||
HDFS-1217. Change some NameNode methods from public to package private.
|
||
(Laxman via szetszwo)
|
||
|
||
HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants ->
|
||
HdfsConstants. (Harsh J Chouraria via atm)
|
||
|
||
HDFS-2197. Refactor RPC call implementations out of NameNode class (todd)
|
||
|
||
HDFS-2332. Add test for HADOOP-7629 (using an immutable FsPermission
|
||
object as an RPC parameter fails). (todd)
|
||
|
||
HDFS-2363. Move datanodes size printing from FSNamesystem.metasave(..)
|
||
to BlockManager. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2209. Make MiniDFS easier to embed in other apps. (stevel)
|
||
|
||
HDFS-2205. Log message for failed connection to datanode is not
|
||
followed by a success message. (Ravi Prakash via stevel)
|
||
|
||
HDFS-2401. Running a set of methods in a Single Test Class.
|
||
(Jonathan Eagles via mahadev)
|
||
|
||
HDFS-2471. Add federation documentation. (suresh)
|
||
|
||
HDFS-2485. Improve code layout and constants in UnderReplicatedBlocks
|
||
(stevel)
|
||
|
||
HDFS-2356. Support case insensitive query parameter names in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to
|
||
hdfs-site.xml. (szetszwo)
|
||
|
||
HDFS-2395. Add a root element in the JSON responses of WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2427. Change the default permission in WebHDFS to 755 and add range
|
||
check/validation for all parameters. (szetszwo)
|
||
|
||
HDFS-2501. Add version prefix and root methods to WebHDFS. (szetszwo)
|
||
|
||
HDFS-1869. mkdirs should use the supplied permission for all of the created
|
||
directories. (Daryn Sharp via szetszwo)
|
||
|
||
HDFS-2355. Federation: enable using the same configuration file across
|
||
all the nodes in the cluster. (suresh)
|
||
|
||
HDFS-2371. Refactor BlockSender.java for better readability. (suresh)
|
||
|
||
HDFS-2493. Remove reference to FSNamesystem in blockmanagement classes.
|
||
(szetszwo)
|
||
|
||
HDFS-2294. Download of commons-daemon TAR should not be under target (tucu)
|
||
|
||
HDFS-2322. the build fails in Windows because commons-daemon TAR cannot be
|
||
fetched. (tucu)
|
||
|
||
HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on
|
||
directories. (Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2512. Add textual error message to data transfer protocol responses
|
||
(todd)
|
||
|
||
HDFS-2521. Remove custom checksum headers from data transfer protocol
|
||
(todd)
|
||
|
||
HDFS-2308. NamenodeProtocol.endCheckpoint is vestigial and can be removed.
|
||
(eli)
|
||
|
||
HDFS-2507. Allow saveNamespace operations to be canceled. (todd)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-1458. Improve checkpoint performance by avoiding unnecessary image
|
||
downloads and loading. (hairong)
|
||
|
||
HDFS-1601. Pipeline ACKs are sent as lots of tiny TCP packets (todd)
|
||
|
||
HDFS-1826. NameNode should save image to name directories in parallel
|
||
during upgrade. (Matt Foley via hairong)
|
||
|
||
HDFS-2030. Improve usability of namenode -upgrade command.
|
||
(Bharath Mundlapudi via suresh)
|
||
|
||
HDFS-2056. Update fetchdt usage. (Tanping Wang via jitendra)
|
||
|
||
HDFS-2118. Couple dfs data dir improvements. (eli)
|
||
|
||
HDFS-2500. Avoid file system operations in BPOfferService thread while
|
||
processing deletes. (todd)
|
||
|
||
HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. (todd)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-2344. Fix the TestOfflineEditsViewer test failure in 0.23 branch.
|
||
(Uma Maheswara Rao G via mattf)
|
||
|
||
HDFS-2347. Fix checkpointTxnCount's comment about editlog size.
|
||
(Uma Maheswara Rao G via mattf)
|
||
|
||
HDFS-2011. Removal and restoration of storage directories on checkpointing
|
||
failure doesn't work properly. (Ravi Prakash via mattf)
|
||
|
||
HDFS-1955. FSImage.doUpgrade() was made too fault-tolerant by HDFS-1826.
|
||
(mattf)
|
||
|
||
HDFS-2061. Two minor bugs in BlockManager block report processing. (mattf)
|
||
|
||
HDFS-1449. Fix test failures - ExtendedBlock must return
|
||
block file name in #getBlockName(). (suresh)
|
||
|
||
HDFS-1680. Fix TestBalancer. (szetszwo)
|
||
|
||
HDFS-1705. Balancer command throws NullPointerException. (suresh via
|
||
szetszwo)
|
||
|
||
HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1585. Fix build after HDFS-1547 (todd)
|
||
|
||
HDFS-1684. Balancer cannot start with with multiple namenodes. (szetszwo)
|
||
|
||
HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos)
|
||
|
||
HDFS-1360. TestBlockRecovery should bind ephemeral ports.
|
||
(Todd Lipcon via hairong)
|
||
|
||
HDFS-1551. Fix pom templates dependency list (gkesavan)
|
||
|
||
HDFS-1509. A savenamespace command writes the fsimage and edits into
|
||
all configured directories. (dhruba)
|
||
|
||
HDFS-1540. Make Datanode handle errors from RPC calls to namenode
|
||
more elegantly. (dhruba)
|
||
|
||
HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba)
|
||
|
||
HDFS-863. Potential deadlock in TestOverReplicatedBlocks.
|
||
(Ken Goodhope via jghoman)
|
||
|
||
HDFS-1607. Fix referenced to misspelled method name getProtocolSigature
|
||
(todd)
|
||
|
||
HDFS-1610. Fix TestClientProtocolWithDelegationToken and TestBlockToken
|
||
on trunk after HADOOP-6904 (todd)
|
||
|
||
HDFS-1600. Fix release audit warnings on trunk. (todd)
|
||
|
||
HDFS-1691. Remove a duplicated static initializer for reading default
|
||
configurations in DFSck. (Alexey Diomin via szetszwo)
|
||
|
||
HDFS-1748. Balancer utilization classification is incomplete. (szetszwo)
|
||
|
||
HDFS-1738. change hdfs jmxget to return an empty string instead of
|
||
null when an attribute value is not available (tanping vi boryas)
|
||
|
||
HDFS-1757. Don't compile fuse-dfs by default. (eli)
|
||
|
||
HDFS-1770. TestFiRename fails due to invalid block size. (eli)
|
||
|
||
HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd)
|
||
|
||
HDFS-1611. Fix up some log messages in DFSClient and MBean registration
|
||
(Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-1543. Reduce dev. cycle time by moving system testing artifacts from
|
||
default build and push to maven for HDFS (Luke Lu via cos)
|
||
|
||
HDFS-1818. TestHDFSCLI is failing on trunk after HADOOP-7202.
|
||
(Aaron T. Myers via todd)
|
||
|
||
HDFS-1828. TestBlocksWithNotEnoughRacks intermittently fails assert.
|
||
(Matt Foley via eli)
|
||
|
||
HDFS-1824. delay instantiation of file system object until it is
|
||
needed (linked to HADOOP-7207) (boryas)
|
||
|
||
HDFS-1831. Fix append bug in FileContext and implement CreateFlag
|
||
check (related to HADOOP-7223). (suresh)
|
||
|
||
HDFS-1594. When the disk becomes full Namenode is getting shutdown and
|
||
not able to recover. (Aaron T. Myers via eli)
|
||
|
||
HDFS-1822. Handle editlog opcode conflict with 0.20.203 during upgrade,
|
||
by throwing an error to indicate the editlog needs to be empty.
|
||
(suresh)
|
||
|
||
HDFS-1808. TestBalancer waits forever, errs without giving information.
|
||
(Matt Foley via eli)
|
||
|
||
HDFS-1829. TestNodeCount waits forever, errs without giving information.
|
||
(Matt Foley via eli)
|
||
|
||
HDFS-1860. when renewing/canceling DelegationToken over http we need to
|
||
pass exception information back to the caller.(boryas)
|
||
|
||
HDFS-1871. Mapreduce build fails due to MiniDFSCluster change from
|
||
HDFS-1052. (suresh)
|
||
|
||
HDFS-1876. One MiniDFSCluster constructor ignores numDataNodes parameter
|
||
(todd)
|
||
|
||
HDFS-1773. Do not show decommissioned datanodes, which are not in both
|
||
include and exclude lists, on web and JMX interfaces.
|
||
(Tanping Wang via szetszwo)
|
||
|
||
HDFS-1888. MiniDFSCluster#corruptBlockOnDatanodes() access must be
|
||
public. (suresh)
|
||
|
||
HDFS-1889. incorrect path in start/stop dfs script. (John George via eli)
|
||
|
||
HDFS-1891. Disable IPV6 for junit tests to fix TestBackupNode failure.
|
||
(suresh)
|
||
|
||
HDFS-1898. Tests failing on trunk due to use of NameNode.format.
|
||
(todd via eli)
|
||
|
||
HDFS-1902. Fix setrep path display for TestHDFSCLI. (Daryn Sharp
|
||
via szetszwo)
|
||
|
||
HDFS-1827. Fix timeout problem in TestBlockReplacement. (Matt Foley
|
||
via szetszwo)
|
||
|
||
HDFS-1908. Fix a NullPointerException in fi.DataTransferTestUtil.
|
||
(szetszwo)
|
||
|
||
HDFS-1912. Update tests for FsShell standardized error messages.
|
||
(Daryn Sharp via szetszwo)
|
||
|
||
HDFS-1903. Fix path display for rm/rmr in TestHDFSCLI and TestDFSShell.
|
||
(Daryn Sharp via szetszwo)
|
||
|
||
HDFS-1627. Fix NullPointerException in Secondary NameNode. (hairong)
|
||
|
||
HDFS-1928. Fix path display for touchz in TestHDFSCLI.
|
||
(Daryn Sharp via todd)
|
||
|
||
HDFS-1938. Fix ivy-retrieve-hdfs dependence in build.xml and aop.xml.
|
||
(Eric Yang via szetszwo)
|
||
|
||
HDFS-1929. TestEditLogFileOutputStream fails if running on same host as NN
|
||
(Aaron T. Myers via todd)
|
||
|
||
HDFS-1933. Update TestDFSShell for improved "test" shell command. (Daryn
|
||
Sharp via todd)
|
||
|
||
HDFS-1931. Update TestDFSShell for improved "du" shell command. (Daryn
|
||
Sharp via todd)
|
||
|
||
HDFS-1439. HDFS Federation: Fix compilation error in TestFiHftp. (suresh)
|
||
|
||
HDFS-1881. Federation: after taking snapshot the current directory
|
||
of datanode is empty. (Tanping Wang via suresh)
|
||
|
||
HDFS-1927. Fix a bug which causes ip=null in NameNode audit log.
|
||
(John George via szetszwo)
|
||
|
||
HDFS-1953. Federation: Change name node mxbean name in cluster web
|
||
console. (Tanping Wang via suresh)
|
||
|
||
HDFS-1922. Fix recurring failure of TestJMXGet (Luke Lu via todd)
|
||
|
||
HDFS-1371. One bad node can incorrectly flag many files as corrupt.
|
||
(Tanping Wang via jitendra)
|
||
|
||
HDFS-1943. Fail to start datanode while start-dfs.sh is executed by
|
||
root user. (Wei Yongjun via jghoman)
|
||
|
||
HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and
|
||
TestDFSShell. (Daryn Sharp via todd)
|
||
|
||
HDFS-1999. Tests use deprecated configs. (Aaron T. Myers via eli)
|
||
|
||
HDFS-1592. Datanode startup doesn't honor volumes.tolerated.
|
||
(Bharath Mundlapudi via jitendra)
|
||
|
||
HDFS-1920. libhdfs does not build for ARM processors.
|
||
(Trevor Robinson via eli)
|
||
|
||
HDFS-1936. Layout version change from HDFS-1822 causes upgrade failure.
|
||
(suresh)
|
||
|
||
HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder.
|
||
(John George via szetszwo)
|
||
|
||
HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode
|
||
as a singleton. (suresh via todd)
|
||
|
||
HDFS-2022. ant binary should build libhdfs. (Eric Yang via eli)
|
||
|
||
HDFS-2014. Change HDFS scripts to work in developer enviroment post
|
||
RPM packaging changes. (Eric Yang via suresh)
|
||
|
||
HDFS-1995. Federation: Minor bug fixes and modification cluster web UI.
|
||
(Tanping Wang via suresh)
|
||
|
||
HDFS-1907. Fix position read for reading still-being-written file in
|
||
DFSInputStream. (John George via szetszwo)
|
||
|
||
HDFS-1923. In TestFiDataTransferProtocol2, reduce random sleep time period
|
||
and increase the number of datanodes. (szetszwo)
|
||
|
||
HDFS-1149. Lease reassignment should be persisted to the edit log.
|
||
(Aaron T. Myers via todd)
|
||
|
||
HDFS-1998. Federation: Make refresh-namenodes.sh refresh all the
|
||
namenode. (Tanping Wang via suresh)
|
||
|
||
HDFS-2041. OP_CONCAT_DELETE doesn't properly restore modification time
|
||
of the concatenated file when edit logs are replayed. (todd)
|
||
|
||
HDFS-2063. libhdfs test is broken. (Eric Yang via eli)
|
||
|
||
HDFS-2067. Bump DATA_TRANSFER_VERSION constant in trunk after introduction
|
||
of protocol buffers in the protocol. (szetszwo via todd)
|
||
|
||
HDFS-2069. Incorrect default trash interval value in the docs.
|
||
(Harsh J Chouraria via eli)
|
||
|
||
HDFS-1942. Datanode must exist when all the block pool service threads
|
||
exit. (Bharath Mundlapudi via suresh)
|
||
|
||
HDFS-1656. Fixes an issue to do with fetching of delegation tokens in
|
||
HftpFileSystem. Contributed by Kan Zhang.
|
||
|
||
HDFS-1692. In secure mode, Datanode process doesn't exit when disks
|
||
fail. (Bharath Mundlapudi via suresh)
|
||
|
||
HDFS-1734. 'Chunk size to view' option is not working in Name Node UI.
|
||
(Uma Maheswara Rao G via jitendra)
|
||
|
||
HDFS-2086. If the include hosts list contains host names, after restarting
|
||
namenode, data nodes registration is denied. Contributed by Tanping Wang.
|
||
|
||
HDFS-2082. SecondaryNameNode web interface doesn't show the right info. (atm)
|
||
|
||
HDFS-1321. If service port and main port are the same, there is no clear
|
||
log message explaining the issue. (Jim Plush via atm)
|
||
|
||
HDFS-1381. HDFS javadocs hard-code references to dfs.namenode.name.dir and
|
||
dfs.datanode.data.dir parameters (Jim Plush via atm)
|
||
|
||
HDFS-2053. Bug in INodeDirectory#computeContentSummary warning.
|
||
(Michael Noll via eli)
|
||
|
||
HDFS-1990. Fix resource leaks in BlockReceiver.close(). (Uma Maheswara
|
||
Rao G via szetszwo)
|
||
|
||
HDFS-2034. Length in DFSInputStream.getBlockRange(..) becomes -ve when
|
||
reading only from a currently being written block. (John George via
|
||
szetszwo)
|
||
|
||
HDFS-2132. Potential resource leak in EditLogFileOutputStream.close. (atm)
|
||
|
||
HDFS-2120. on reconnect, DN can connect to NN even with different source
|
||
versions. (John George via atm)
|
||
|
||
HDFS-2152. TestWriteConfigurationToDFS causing the random failures. (Uma
|
||
Maheswara Rao G via atm)
|
||
|
||
HDFS-2114. re-commission of a decommissioned node does not delete
|
||
excess replicas. (John George via mattf)
|
||
|
||
HDFS-1776. Bug in Concat code. (Bharath Mundlapudi via Dmytro Molkov)
|
||
|
||
HDFS-2196. Make ant build system work with hadoop-common JAR generated
|
||
by Maven. (Alejandro Abdelnur via tomwhite)
|
||
|
||
HDFS-2245. Fix a NullPointerException in BlockManager.chooseTarget(..).
|
||
(szetszwo)
|
||
|
||
HDFS-2229. Fix a deadlock in namenode by enforcing lock acquisition
|
||
ordering. (szetszwo)
|
||
|
||
HDFS-2235. Encode servlet paths. (eli)
|
||
|
||
HDFS-2186. DN volume failures on startup are not counted. (eli)
|
||
|
||
HDFS-2240. Fix a deadlock in LeaseRenewer by enforcing lock acquisition
|
||
ordering. (szetszwo)
|
||
|
||
HDFS-73. DFSOutputStream does not close all the sockets.
|
||
(Uma Maheswara Rao G via eli)
|
||
|
||
HDFS-1257. Fix a race condition on BlockManager.recentInvalidateSets.
|
||
(Eric Payne via szetszwo)
|
||
|
||
HDFS-2267. DataXceiver thread name incorrect while waiting on op during
|
||
keepalive. (todd)
|
||
|
||
HDFS-1480. All replicas of a block can end up on the same rack when
|
||
some datanodes are decommissioning. (todd)
|
||
|
||
HDFS-2286. DataXceiverServer logs AsynchronousCloseException at shutdown
|
||
(todd)
|
||
|
||
HDFS-2289. Ensure jsvc is bundled with the HDFS distribution artifact.
|
||
(Alejandro Abdelnur via acmurthy)
|
||
|
||
HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd)
|
||
|
||
HDFS-2323. start-dfs.sh script fails for tarball install (tomwhite)
|
||
|
||
HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon
|
||
execution order of test methods (Laxman via atm)
|
||
|
||
HDFS-2345. TestLeaseRecovery2 fails on 0.23 branch (Uma Maheswara Rao G
|
||
via atm)
|
||
|
||
HDFS-2412. Add backwards-compatibility layer for renamed FSConstants
|
||
class (todd)
|
||
|
||
HDFS-2414. Fix TestDFSRollback to avoid spurious failures. (todd)
|
||
|
||
HDFS-2422. The NN should tolerate the same number of low-resource volumes
|
||
as failed volumes (atm)
|
||
|
||
HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation
|
||
tokens. (omalley)
|
||
|
||
HDFS-2331. Fix WebHdfsFileSystem compilation problems for a bug in JDK
|
||
version < 1.6.0_26. (Abhijit Suresh Shingate via szetszwo)
|
||
|
||
HDFS-2333. Change DFSOutputStream back to package private, otherwise,
|
||
there are two SC_START_IN_CTOR findbugs warnings. (szetszwo)
|
||
|
||
HDFS-2366. Initialize WebHdfsFileSystem.ugi in object construction.
|
||
(szetszwo)
|
||
|
||
HDFS-2361. hftp is broken, fixed username checks in JspHelper. (jitendra)
|
||
|
||
HDFS-2403. NamenodeWebHdfsMethods.generateDelegationToken(..) does not use
|
||
the renewer parameter. (szetszwo)
|
||
|
||
HDFS-2409. _HOST in dfs.web.authentication.kerberos.principal. (jitendra)
|
||
|
||
HDFS-2404. WebHDFS liststatus json response is not correct. (suresh)
|
||
|
||
HDFS-2441. Remove the Content-Type set by HttpServer.QuotingInputFilter in
|
||
WebHDFS responses. (szetszwo)
|
||
|
||
HDFS-2428. Convert com.sun.jersey.api.ParamException$QueryParamException
|
||
to IllegalArgumentException and response it as http BAD_REQUEST in WebHDFS.
|
||
(szetszwo)
|
||
|
||
HDFS-2424. Added a root element "HdfsFileStatuses" for the response
|
||
of WebHDFS listStatus. (szetszwo)
|
||
|
||
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra)
|
||
|
||
HDFS-2439. Fix NullPointerException in WebHDFS when opening a non-existing
|
||
file or creating a file without specifying the replication parameter.
|
||
(szetszwo)
|
||
|
||
HDFS-2453. Fix http response code for partial content in WebHDFS, added
|
||
getDefaultBlockSize() and getDefaultReplication() in WebHdfsFileSystem
|
||
and cleared content type in ExceptionHandler. (szetszwo)
|
||
|
||
HDFS-2411. The the auth to local mappings are not being respected, with
|
||
WebHDFS enabled. (jitendra)
|
||
|
||
HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods.
|
||
(Uma Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2298. Fix TestDfsOverAvroRpc by changing ClientProtocol to
|
||
not include multiple methods of the same name. (cutting)
|
||
|
||
HDFS-2432. WebHDFS: response FORBIDDEN when setReplication on non-files;
|
||
clear umask before creating a flie; throw IllegalArgumentException if
|
||
setOwner with both owner and group empty; throw FileNotFoundException if
|
||
getFileStatus on non-existing files; fix bugs in getBlockLocations; and
|
||
changed getFileChecksum json response root to "FileChecksum". (szetszwo)
|
||
|
||
HDFS-2065. Add null checks in DFSClient.getFileChecksum(..). (Uma
|
||
Maheswara Rao G via szetszwo)
|
||
|
||
HDFS-2522. Disable TestDfsOverAvroRpc test. (suresh)
|
||
|
||
HDFS-2416. distcp with a WebHDFS uri on a secure cluster fails. (jitendra)
|
||
|
||
HDFS-2527. WebHDFS: remove the use of "Range" header in Open; use ugi
|
||
username if renewer parameter is null in GetDelegationToken; response OK
|
||
when setting replication for non-files; rename GETFILEBLOCKLOCATIONS to
|
||
GET_BLOCK_LOCATIONS and state that it is a private unstable API; replace
|
||
isDirectory and isSymlink with enum {FILE, DIRECTORY, SYMLINK} in
|
||
HdfsFileStatus JSON object. (szetszwo)
|
||
|
||
HDFS-2528. WebHDFS: set delegation kind to WEBHDFS and add a HDFS token
|
||
when http requests are redirected to datanode. (szetszwo)
|
||
|
||
HDFS-2540. WebHDFS: change "Expect: 100-continue" to two-step write; change
|
||
"HdfsFileStatus" and "localName" respectively to "FileStatus" and
|
||
"pathSuffix" in JSON response. (szetszwo)
|
||
|
||
BREAKDOWN OF HDFS-1073 SUBTASKS
|
||
|
||
HDFS-1521. Persist transaction ID on disk between NN restarts.
|
||
(Ivan Kelly and Todd Lipcon via todd)
|
||
|
||
HDFS-1538. Refactor more startup and image loading code out of FSImage.
|
||
(todd)
|
||
|
||
HDFS-1729. Add code to detect valid length of an edits file. (todd)
|
||
|
||
HDFS-1793. Add code to inspect a storage directory with txid-based
|
||
filenames (todd)
|
||
|
||
HDFS-1794. Add code to list which edit logs are available on a remote NN
|
||
(todd)
|
||
|
||
HDFS-1858. Add state management variables to FSEditLog (Ivan Kelly and Todd
|
||
Lipcon via todd)
|
||
|
||
HDFS-1859. Add some convenience functions to iterate over edit log streams
|
||
(Ivan Kelly and Todd Lipcon via todd)
|
||
|
||
HDFS-1894. Add constants for LAYOUT_VERSIONs in edits log branch (todd)
|
||
|
||
HDFS-1892. Fix EditLogFileInputStream.getValidLength to be aware of
|
||
OP_INVALID filler (todd)
|
||
|
||
HDFS-1799. Refactor log rolling and filename management out of FSEditLog
|
||
(Ivan Kelly and Todd Lipcon via todd)
|
||
|
||
HDFS-1801. Remove use of timestamps to identify checkpoints and logs (todd)
|
||
|
||
HDFS-1930. TestDFSUpgrade failing in HDFS-1073 branch (todd)
|
||
|
||
HDFS-1800. Extend image checksumming to function with multiple fsimage
|
||
files per directory. (todd)
|
||
|
||
HDFS-1725. Set storage directories only at FSImage construction (Ivan Kelly
|
||
via todd)
|
||
|
||
HDFS-1926. Remove references to StorageDirectory from JournalManager
|
||
interface (Ivan Kelly via todd)
|
||
|
||
HDFS-1893. Change edit logs and images to be named based on txid (todd)
|
||
|
||
HDFS-1985. Clean up image transfer servlet (todd)
|
||
|
||
HDFS-1984. Enable multiple secondary namenodes to run simultaneously (todd)
|
||
|
||
HDFS-1987. Re-enable TestCheckpoint.testSecondaryImageDownload which was
|
||
not running previously. (todd)
|
||
|
||
HDFS-1993. TestCheckpoint needs to clean up between cases (todd)
|
||
|
||
HDFS-1992. Remove vestiges of NNStorageListener. (todd)
|
||
|
||
HDFS-1991. Some refactoring of Secondary NameNode to be able to share more
|
||
code with the BackupNode or CheckpointNode. (todd)
|
||
|
||
HDFS-1994. Fix race conditions when running two rapidly checkpointing
|
||
Secondary NameNodes. (todd)
|
||
|
||
HDFS-2001. Remove use of previous.checkpoint and lastcheckpoint.tmp
|
||
directories (todd)
|
||
|
||
HDFS-2015. Remove checkpointTxId from VERSION file. (todd)
|
||
|
||
HDFS-2016. Add infrastructure to remove or archive old and unneeded storage
|
||
files within the name directories. (todd)
|
||
|
||
HDFS-2047. Improve TestNamespace and TestEditLog in HDFS-1073 branch.
|
||
(todd)
|
||
|
||
HDFS-2048. Add upgrade tests and fix upgrade from 0.22 with corrupt image.
|
||
(todd)
|
||
|
||
HDFS-2027. Image inspector should return finalized logs before unfinalized
|
||
logs. (todd)
|
||
|
||
HDFS-2074. Determine edit log validity by truly reading and validating
|
||
transactions. (todd)
|
||
|
||
HDFS-2085. Finalize in-progress edit logs at startup. (todd)
|
||
|
||
HDFS-2026. SecondaryNameNode should properly handle the case where the
|
||
NameNode is reformatted. (todd)
|
||
|
||
HDFS-2077. Address checkpoint upload when one of the storage dirs is failed
|
||
(todd)
|
||
|
||
HDFS-2078. NameNode should not clear directory when restoring removed
|
||
storage. (todd)
|
||
|
||
HDFS-2088. Move edits log archiving logic into FSEditLog/JournalManager
|
||
(todd)
|
||
|
||
HDFS-2093. Handle case where an entirely empty log is left during NN crash
|
||
(todd)
|
||
|
||
HDFS-2102. Zero-pad edits filename to make them lexically sortable. (Ivan
|
||
Kelly via todd)
|
||
|
||
HDFS-2010. Fix NameNode to exit if all edit streams become inaccessible.
|
||
(atm via todd)
|
||
|
||
HDFS-2123. Checkpoint interval should be based on txn count, not size.
|
||
(todd)
|
||
|
||
HDFS-1979. Fix backupnode for new edits/image layout. (todd)
|
||
|
||
HDFS-2101. Fix remaining unit tests for new storage filenames. (todd)
|
||
|
||
HDFS-2133. Address remaining TODOs and pre-merge cleanup on HDFS-1073
|
||
branch. (todd)
|
||
|
||
HDFS-1780. Reduce need to rewrite FSImage on startup. (todd)
|
||
|
||
HDFS-2104. Add a flag to the 2NN to format its checkpoint dirs on startup.
|
||
(todd)
|
||
|
||
HDFS-2135. Fix regression of HDFS-1955 in HDFS-1073 branch. (todd)
|
||
|
||
HDFS-2160. Fix CreateEditsLog test tool in HDFS-1073 branch. (todd)
|
||
|
||
HDFS-2168. Reenable TestEditLog.testFailedOpen and fix exposed bug. (todd)
|
||
|
||
HDFS-2169. Clean up TestCheckpoint and remove TODOs (todd)
|
||
|
||
HDFS-2170. Address remaining TODOs in HDFS-1073 branch. (todd)
|
||
|
||
HDFS-2172. Address findbugs and javadoc warnings in HDFS-1073 branch.
|
||
(todd)
|
||
|
||
HDFS-2445. Ensure failed tests exit with proper error code. (Jonathan
|
||
Eagles via acmurthy)
|
||
|
||
Release 0.22.1 - Unreleased
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
NEW FEATURES
|
||
|
||
IMPROVEMENTS
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-2718. Optimize OP_ADD in edits loading. (shv)
|
||
|
||
HDFS-2886. CreateEditLogs should generate a realistic edit log. (shv)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-2877. If locking of a storage dir fails, it will remove the other
|
||
NN's lock file on exit. (todd)
|
||
|
||
Release 0.22.0 - 2011-11-29
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-1825. Remove thriftfs contrib. (nigel via eli)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-992. Re-factor block access token implementation to conform to the
|
||
generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman)
|
||
|
||
HDFS-599. Allow NameNode to have a seprate port for service requests from
|
||
client requests. (Dmytro Molkov via hairong)
|
||
|
||
HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584.
|
||
(jghoman and Kan Zhang via jghoman)
|
||
|
||
HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman)
|
||
|
||
HDFS-1006. getImage/putImage http requests should be https for the case
|
||
of security enabled. (borya and jghoman via jghoman)
|
||
|
||
HDFS-1033. In secure clusters, NN and SNN should verify that the remote
|
||
principal during image and edits transfer. (jghoman)
|
||
|
||
HDFS-1023. Allow http server to start as regular principal if https
|
||
principal not defined. (jghoman)
|
||
|
||
HDFS-1150. Verify datanodes' identities to clients in secure clusters.
|
||
(jghoman)
|
||
|
||
HDFS-1330. Make RPCs to DataNodes timeout. (hairong)
|
||
Added additional unit tests per HADOOP-6889. (John George via mattf)
|
||
|
||
HDFS-202. HDFS support of listLocatedStatus introduced in HADOOP-6870.
|
||
HDFS piggyback block locations to each file status when listing a
|
||
directory. (hairong)
|
||
|
||
HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv)
|
||
|
||
HDFS-1435. Provide an option to store fsimage compressed. (hairong)
|
||
|
||
HDFS-903. Support fsimage validation through MD5 checksum. (hairong)
|
||
|
||
HDFS-1457. Provide an option to throttle image transmission between
|
||
pimary and secondary NameNodes. (Yifei Lu and hairong via hairong)
|
||
|
||
HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos)
|
||
|
||
HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457.
|
||
(eli)
|
||
|
||
HDFS-895. Allow hflush/sync to occur in parallel with new writes
|
||
to the file. (Todd Lipcon via hairong)
|
||
|
||
HDFS-528. Add ability for safemode to wait for a minimum number of
|
||
live datanodes (Todd Lipcon via eli)
|
||
|
||
HDFS-1753. Resource Leak in StreamFile. (Uma Maheswara Rao G via eli)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-1304. Add a new unit test for HftpFileSystem.open(..). (szetszwo)
|
||
|
||
HDFS-1096. fix for prev. commit. (boryas)
|
||
|
||
HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group
|
||
mappings (boryas)
|
||
|
||
HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas)
|
||
|
||
HDFS-1132. Refactor TestFileStatus (Eli Collins via cos)
|
||
|
||
HDFS-1163. normalize property names for JT/NN kerberos principal
|
||
names in configuration (from HADOOP 6633) (boryas)
|
||
|
||
HDFS-1003. authorization checks for inter-server protocol
|
||
(based on HADOOP-6600) (boryas)
|
||
|
||
HDFS-1061. Memory footprint optimization for INodeFile object.
|
||
(Bharath Mundlapudi via jghoman)
|
||
|
||
HDFS-1079. Throw exceptions as specified by the AbstractFileSystem
|
||
in HDFS implemenation and protocols. (suresh)
|
||
|
||
HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong)
|
||
|
||
HDFS-1119. Introduce a GSet interface to BlocksMap. (szetszwo)
|
||
|
||
HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman)
|
||
|
||
HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm.
|
||
(Jeff Ames via jghoman)
|
||
|
||
HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java.
|
||
(Jeff Ames via jghoman)
|
||
|
||
HDFS-1190. Remove unused getNamenode() method from DataNode.
|
||
(Jeff Ames via jghoman)
|
||
|
||
HDFS-1110. Reuses objects for commonly used file names in namenode to
|
||
reduce the heap usage. (suresh)
|
||
|
||
HDFS-752. Add interfaces classification to to HDFS source code. (suresh)
|
||
|
||
HDFS-947. An Hftp read request is redirected to a datanode that has
|
||
the most replicas of the blocks in the file. (Dmytro Molkov via dhruba)
|
||
|
||
HDFS-1272. Fixes to take care of the changes in HADOOP-6845.
|
||
(Jitendra Pandey via ddas)
|
||
|
||
HDFS-1298 - Add support in HDFS for new statistics added in FileSystem
|
||
to track the file system operations. (suresh)
|
||
|
||
HDFS-1201. The HDFS component for HADOOP-6632.
|
||
(Kan Zhang & Jitendra Pandey via ddas)
|
||
|
||
HDFS-1307 Add start time, end time and total time taken for FSCK to
|
||
FSCK report (suresh)
|
||
|
||
HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861.
|
||
(Jitendra Pandey & Owen O'Malley via ddas)
|
||
|
||
HDFS-1315. Add fsck event to audit log and remove other audit log events
|
||
corresponding to FSCK listStatus and open calls. (suresh)
|
||
|
||
HDFS-1178. The NameNode servlets should not use RPC to connect to the
|
||
NameNode. (Kan Zhang via jghoman)
|
||
|
||
HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can
|
||
be used to control who can view the default hdfs servlets. (ddas)
|
||
|
||
HDFS-1297. Fix some comments. (Jeff Ames via jghoman)
|
||
|
||
HDFS-330. Datanode Web UIs should provide robots.txt.
|
||
(Allen Wittenauer via jghoman)
|
||
|
||
HDFS-881. Refactor DataNode Packet header into DataTransferProtocol.
|
||
(Todd Lipcon via jghoman)
|
||
|
||
HDFS-1036. docs for fetchdt
|
||
|
||
HDFS-1318. Add JMX interface for read access to namenode and datanode
|
||
web UI information. (Tanping Wang via suresh).
|
||
|
||
HDFS-1356. Provide information as to whether or not security is
|
||
enabled on web interface for NameNode (boryas)
|
||
|
||
HDFS-1205. FSDatasetAsyncDiskService should name its threads.
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv)
|
||
|
||
HDFS-1395. Add @Override to FSDataset methods that implement
|
||
FSDatasetInterface methods. (suresh)
|
||
|
||
HDFS-1383. Improve the error messages when using hftp://. (szetszwo)
|
||
|
||
HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba)
|
||
|
||
HDFS-1407. Change DataTransferProtocol methods to use Block instead
|
||
of individual elements of Block. (suresh)
|
||
|
||
HDFS-1417. Add @Override to SimulatedFSDataset methods that implement
|
||
FSDatasetInterface methods. (suresh)
|
||
|
||
HDFS-1426. Remove unused method BlockInfo#listCount. (hairong)
|
||
|
||
HDFS-1472. Allow programmatic access to fsck output.
|
||
(Ramkumar Vadali via dhruba)
|
||
|
||
HADOOP-7007. Update the hudson-test-patch ant target to work with the
|
||
latest test-patch.sh script (gkesavan)
|
||
|
||
HDFS-1462. Refactor edit log loading to a separate class from edit log writing.
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv)
|
||
|
||
HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel)
|
||
|
||
HDFS-1408. Herriot NN and DN clients should vend statistics. (cos)
|
||
|
||
HDFS-1491 Update Hdfs to match the change of methods from protected to public
|
||
in AbstractFileSystem (Hadoop-6903) (sanjay)
|
||
|
||
HDFS-1160. Improve some FSDataset warnings and comments. (eli)
|
||
|
||
HDFS-556. Provide info on failed volumes in the web ui. (eli)
|
||
|
||
HDFS-697. Enable asserts for tests by default. (eli)
|
||
|
||
HDFS-1187. Modify fetchdt to allow renewing and canceling token.
|
||
(Owen O'Malley and Kan Zhang via jghoman)
|
||
|
||
HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli)
|
||
|
||
HDFS-455. Make NN and DN handle in a intuitive way comma-separated
|
||
configuration strings. (Michele Catasta via eli)
|
||
|
||
HDFS-1071. savenamespace should write the fsimage to all configured
|
||
fs.name.dir in parallel (Dmytro Molkov via jghoman)
|
||
|
||
HDFS-1055. Improve thread naming for DataXceivers.
|
||
(Todd Lipcon and Ramkumar Vadali via eli).
|
||
|
||
HDFS-718. Configuration parameter to prevent accidental formatting of
|
||
HDFS filesystem. (Andrew Ryan via jghoman)
|
||
|
||
HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon
|
||
via hairong)
|
||
|
||
HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding
|
||
blocks are corrupt. (Patrick Kling via hairong)
|
||
|
||
HDFS-259. Remove intentionally corrupt 0.13 directory layout creation.
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1513. Fix a number of warnings. (eli)
|
||
|
||
HDFS-1473. Refactor storage management into separate classes than fsimage
|
||
file reading/writing. (Todd Lipcon via eli)
|
||
|
||
HDFS-1582. Remove auto-generated native build files. (rvs via eli)
|
||
|
||
HDFS-1456. Provide builder for constructing instances of MiniDFSCluster.
|
||
(jghoman)
|
||
|
||
HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value.
|
||
(eli)
|
||
|
||
HDFS-1052. HDFS Federation - Merge of umbrella jira changes from
|
||
HDFS-1052 branch into trunk.
|
||
|
||
HDFS-1835. DataNode should not depend on SHA1PRNG secure random generator
|
||
to generate a storage ID. (John Carrino via todd)
|
||
|
||
HDFS-1947. DFSClient should use mapreduce.task.attempt.id. (eli)
|
||
|
||
HDFS-1957. Add documentation for HFTP. (Ari Rabkin via todd)
|
||
|
||
HDFS-1454. Update the documentation to reflect that clients don't write
|
||
blocks to local disk before copying to HDFS. (Harsh J Chouraria via todd)
|
||
|
||
HDFS-1980. Move build/webapps deeper in the build directory heirarchy
|
||
to aid eclipse users. (todd)
|
||
|
||
HDFS-1619. Remove AC_TYPE* from the libhdfs. (Roman Shaposhnik via eli)
|
||
|
||
HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to
|
||
trigger lease recovery' (stack)
|
||
|
||
HDFS-1954. Improved corrupt files warning on NameNode web UI.
|
||
(Patrick Hunt via shv)
|
||
|
||
HDFS-1409. BackupNode registration throwing
|
||
UnsupportedActionException("register") instead of "journal".
|
||
(Ching-Shen Chen via shv)
|
||
|
||
HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures
|
||
encountered during transferToFully() (Kihwal Lee via stack)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv)
|
||
|
||
HDFS-1081. Performance regression in
|
||
DistributedFileSystem::getFileBlockLocations in secure systems (jghoman)
|
||
|
||
HDFS-1114. Implement LightWeightGSet for BlocksMap in order to reduce
|
||
NameNode memory footprint. (szetszwo)
|
||
|
||
HDFS-1320. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
|
||
(Erik Steffl via szetszwo)
|
||
|
||
HDFS-1368. Add a block counter to DatanodeDescriptor. (hairong)
|
||
|
||
HDFS-1434. Refactor Datanode#startDataNode method into smaller methods.
|
||
(suresh)
|
||
|
||
HDFS-941. The DFS client should cache and reuse open sockets to datanodes
|
||
while performing reads. (bc Wong and Todd Lipcon via todd)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-1039. Adding test for JspHelper.getUGI(jnp via boryas)
|
||
|
||
HDFS-1019. Incorrect default values for delegation tokens in
|
||
hdfs-default.xml (jnp via boryas)
|
||
|
||
HDFS-1039. Service should be set in the token in JspHelper.getUGI(jnp via boryas)
|
||
|
||
HDFS-1038. FIX. A test missed in a previous commit for this JIRA. (boryas)
|
||
|
||
HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only
|
||
if security is enabled. (jnp via boryas)
|
||
|
||
HDFS-1044. Cannot submit mapreduce job from secure client to
|
||
unsecure sever (boryas)
|
||
|
||
HDFS-1021. specify correct server principal for RefreshAuthorizationPolicyProtocol
|
||
and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612) (boryas)
|
||
|
||
HDFS-970. fsync fsimage to disk before closing fsimage file.
|
||
(Todd Lipcon via dhruba)
|
||
|
||
HDFS-1027. Update copyright year to 2010. (Ravi Phulari via jghoman)
|
||
|
||
HDFS-1080. SecondaryNameNode image transfer should use the defined http
|
||
address rather than local ip address. (jghoman)
|
||
|
||
HDFS-1198. Resolving cross-realm principals. (Jitendra Pandey via jghoman)
|
||
|
||
HDFS-1118. Fix socketleak on DFSClient. (Zheng Shao via dhruba)
|
||
|
||
HDFS-1192. refreshSuperUserGroupsConfiguration should use server side
|
||
configuration for the refresh (for HADOOP-6815) (boryas)
|
||
|
||
HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas)
|
||
|
||
HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather
|
||
than using createRemoteUser() (jnp via boryas)
|
||
|
||
HDFS-1250. Namenode should reject block reports and block received
|
||
requests from dead datanodes (suresh)
|
||
|
||
HDFS-1145. When NameNode is shutdown it does not try to exit
|
||
safemode anymore. (dhruba)
|
||
|
||
HDFS-1202. DataBlockScanner throws NPE when updated before
|
||
initialized. (Todd Lipcon via dhruba)
|
||
|
||
HDFS-882. Datanode logs the hostname and port its listening on.
|
||
(Steve Loughran via dhruba)
|
||
|
||
HDFS-1238. ant eclipse-files has drifted again, (jghoman)
|
||
|
||
HDFS-1045. In secure clusters, re-login is necessary for https
|
||
clients before opening connections. (jghoman)
|
||
|
||
HDFS-1289. Datanode secure mode is broken. (Kan Zhang via jghoman)
|
||
|
||
HDFS-1007. HFTP needs to be updated to use delegation tokens (boryas)
|
||
|
||
HDFS-1085. HFTP read may fail silently on the client side if there is an
|
||
exception on the server side. (szetszwo)
|
||
|
||
HDFS-1308. job conf key for the services name of DelegationToken for HFTP
|
||
url is constructed incorrectly in HFTPFileSystem (boryas)
|
||
|
||
HDFS-1319. Fix location of re-login for secondary namenode from HDFS-999.
|
||
(jghoman)
|
||
|
||
HDFS-1317. Remove the FILEPATH_PATTERN from hdfsproxy.AuthorizationFilter.
|
||
(Rohini Palaniswamy via szetszwo)
|
||
|
||
HDFS-912. sed in build.xml on Solaris fails. (Allen Wittenauer via jghoman)
|
||
|
||
HDFS-1296. using delegation token over hftp for long running
|
||
clients (boryas)
|
||
|
||
HDFS-1334. open in HftpFileSystem does not add delegation tokens to the url.
|
||
(Jitendra Pandey via jghoman)
|
||
|
||
HDFS-1301. TestHDFSProxy need to use server side conf for ProxyUser
|
||
stuff. (boryas)
|
||
|
||
HDFS-1340. When security is turned off, there is a potential XSS attack.
|
||
This patch fixes it by removing delegationtoken string from the URL,
|
||
before returning a response to the client. (Jitendra Pandey via ddas)
|
||
|
||
HDFS-1347. TestDelegationToken uses mortbay.log for logging (boryas)
|
||
|
||
HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's
|
||
bindings (cos)
|
||
|
||
HDFS-1349. Remove empty java files. (Eli Collins)
|
||
|
||
HDFS-1340. A null delegation token is appended to the url if security
|
||
is disabled when browsing filesystem. (boryas)
|
||
|
||
HDFS-1352. Fix jsvc.location. (Eli Collins via jghoman)
|
||
|
||
HDFS-1284. TestBlockToken fails. (Kan Zhang via jghoman)
|
||
|
||
HDFS-1355. ant veryclean (clean-cache) doesn't clean enough.
|
||
(Luke Lu via jghoman)
|
||
|
||
HDFS-1353. Remove most of getBlockLocation optimization. (jghoman)
|
||
|
||
HDFS-1369. Invalid javadoc reference in FSDatasetMBean.java (Eli Collins)
|
||
|
||
HDFS-829. hdfsJniHelper.c: #include <error.h> is not portable.
|
||
(Allen Wittenauer via jghoman)
|
||
|
||
HDFS-1310. The ClientDatanodeProtocol proxy should be stopped in
|
||
DFSInputStream.readBlockLength(..). (sam rash via szetszwo)
|
||
|
||
HDFS-1357. HFTP traffic served by DataNode shouldn't use service port
|
||
on NameNode. (Kan Zhang via jghoman)
|
||
|
||
HDFS-1419. HDFS Federation: Three test cases need minor modification after
|
||
the new block id change (Tanping Wang via suresh)
|
||
|
||
HDFS-96. HDFS supports blocks larger than 2 GB.
|
||
(Patrick Kling via dhruba)
|
||
|
||
HDFS-1433. Fix test failures - TestPread and TestFileLimit. (suresh)
|
||
|
||
HDFS-1364. Makes long running HFTP-based applications do relogins
|
||
if necessary. (Jitendra Pandey via ddas)
|
||
|
||
HDFS-1399. Distinct minicluster services (e.g. NN and JT) overwrite each
|
||
other's service policies. (Aaron T. Myers via tomwhite)
|
||
|
||
HDFS-1440. Fix TestComputeInvalidateWork failure. (suresh)
|
||
|
||
HDFS-1498. FSDirectory#unprotectedConcat calls setModificationTime
|
||
on a file. (eli)
|
||
|
||
HDFS-1625. Ignore disk space values in TestDataNodeMXBean. (szetszwo)
|
||
|
||
HDFS-1850. DN should transmit absolute failed volume count rather than
|
||
increments to the NN. (eli)
|
||
|
||
HDFS-671. Documentation change for updated configuration keys.
|
||
(tomwhite via eli)
|
||
|
||
HDFS-1544. Ivy resolve force mode should be turned off by default.
|
||
(Luke Lu via tomwhite)
|
||
|
||
HDFS-1615. seek() on closed DFS input stream throws NullPointerException
|
||
(Scott Carey via todd)
|
||
|
||
HDFS-1897. Documentation refers to removed option dfs.network.script
|
||
(Andrew Whang via todd)
|
||
|
||
HDFS-1621. Fix references to hadoop-common-${version} in build.xml
|
||
(Jolly Chen via todd)
|
||
|
||
HDFS-1505. saveNamespace appears to succeed even if all directories fail
|
||
to save. (Aaron T. Myers via todd)
|
||
|
||
HDFS-1921. saveNamespace can cause NN to be unable to come up on restart
|
||
(Matt Foley via todd)
|
||
|
||
HDFS-1925. SafeModeInfo should use the correct constant instead of a
|
||
hard-coded value for its default. (Joey Echeverria via todd)
|
||
|
||
HDFS-1575. Viewing block from web UI is broken. (Aaron T. Myers via todd)
|
||
|
||
HDFS-1932. Ensure that HDFS configuration deprecations are set up in every
|
||
spot that HDFS configurations are loaded. (Jolly Chen via todd)
|
||
|
||
HDFS-1952. FSEditLog.open() appears to succeed even if all EDITS
|
||
directories fail. (Andrew Wang via todd)
|
||
|
||
HDFS-1965. IPCs done using block token-based tickets can't reuse
|
||
connections (todd)
|
||
|
||
HDFS-1978. All but first option in LIBHDFS_OPTS is ignored. (eli)
|
||
|
||
HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper
|
||
(Aaron T. Myers via todd)
|
||
|
||
HDFS-1997. Image transfer process misreports client side exceptions.
|
||
(todd via eli)
|
||
|
||
HDFS-2000. Missing deprecation for io.bytes.per.checksum.
|
||
(Aaron T. Myers vie eli)
|
||
|
||
HDFS-977. DataNode.createInterDataNodeProtocolProxy() guards a log
|
||
at the wrong level. (Harsh J Chouraria via todd)
|
||
|
||
HDFS-1969. Running rollback on new-version namenode destroys the
|
||
namespace. (todd)
|
||
|
||
HDFS-2039. TestNameNodeMetrics uses a bad test root path, preventing it
|
||
from running inside Eclipse. (todd)
|
||
|
||
HDFS-988. saveNamespace race can corrupt the edits log. (eli)
|
||
|
||
HDFS-2071. Use of isConnected() in DataXceiver is invalid. (Kihwal Lee
|
||
via todd)
|
||
|
||
HDFS-1981. NameNode does not saveNamespace() when editsNew is empty.
|
||
(Uma Maheswara Rao G via shv)
|
||
|
||
HDFS-2232. Generalize regular expressions in TestHDFSCLI.
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-2290. Block with corrupt replica is not getting replicated.
|
||
(Benoy Antony via shv)
|
||
|
||
HDFS-2452. OutOfMemoryError in DataXceiverServer takes down the DataNode
|
||
(Uma Maheswara Rao via cos)
|
||
|
||
HDFS-2002. Incorrect computation of needed blocks in getTurnOffTip().
|
||
(Plamen Jeliazkov via shv)
|
||
|
||
HDFS-2514. Link resolution bug for intermediate symlinks with
|
||
relative targets. (eli)
|
||
|
||
Release 0.21.1 - Unreleased
|
||
|
||
HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli)
|
||
|
||
HDFS-874. TestHDFSFileContextMainOperations fails on weirdly
|
||
configured DNS hosts. (Todd Lipcon via eli)
|
||
|
||
HDFS-1507. TestAbandonBlock should abandon a block. (eli)
|
||
|
||
HDFS-1487. FSDirectory.removeBlock() should update diskspace count
|
||
of the block owner node (Zhong Wang via eli).
|
||
|
||
HDFS-1467. Append pipeline never succeeds with more than one replica.
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1167. New property for local conf directory in system-test-hdfs.xml
|
||
file. (Vinay Thota via cos)
|
||
|
||
HDFS-1503. TestSaveNamespace fails. (Todd Lipcon via cos)
|
||
|
||
HDFS-1524. Image loader should make sure to read every byte in image file.
|
||
(hairong)
|
||
|
||
HDFS-1523. TestLargeBlock is failing on trunk. (cos)
|
||
|
||
HDFS-1502. TestBlockRecovery triggers NPE in assert. (hairong via cos)
|
||
|
||
HDFS-1532. Exclude Findbugs warning in FSImageFormat$Saver. (Todd Lipcon
|
||
via cos)
|
||
|
||
HDFS-1527. SocketOutputStream.transferToFully fails for blocks >= 2GB on
|
||
32 bit JVM. (Patrick Kling via cos)
|
||
|
||
HDFS-1531. Clean up stack traces due to duplicate MXBean registration.
|
||
(Todd Lipcon via cos)
|
||
|
||
HDFS-613. TestBalancer and TestBlockTokenWithDFS fail Balancer assert.
|
||
(Todd Lipcon via cos)
|
||
|
||
HDFS-1511. 98 Release Audit warnings on trunk and branch-0.22.
|
||
(jghoman)
|
||
|
||
HDFS-1560. dfs.data.dir permissions should default to 700.
|
||
(Todd Lipcon via eli)
|
||
|
||
HDFS-1550. NPE when listing a file with no location. (hairong)
|
||
|
||
HDFS-1542. Add test for HADOOP-7082, a deadlock writing Configuration to
|
||
HDFS. (todd)
|
||
|
||
HDFS-1504. FSImageSaver should catch all exceptions, not just IOE. (todd)
|
||
|
||
HDFS-884. DataNode throws IOException if all data directories are
|
||
unavailable. (Steve Loughran and shv)
|
||
|
||
HDFS-1591. HDFS part of HADOOP-6642. (Chris Douglas, Po Cheung via shv)
|
||
|
||
HDFS-900. Corrupt replicas are not processed correctly in block report (shv)
|
||
|
||
HDFS-1529. Incorrect handling of interrupts in waitForAckedSeqno can cause
|
||
deadlock (todd)
|
||
|
||
HDFS-1597. Batched edit log syncs can reset synctxid and throw assertions
|
||
(todd)
|
||
|
||
HDFS-1602. Fix HADOOP-4885 for it is doesn't work as expected. (boryas)
|
||
|
||
HDFS-1618. configure files that are generated as part of the released
|
||
tarball need to have executable bit set (Roman Shaposhnik via cos)
|
||
|
||
HDFS-981. test-contrib fails due to test-cactus failure (cos)
|
||
|
||
HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv
|
||
CHECKSUM_OK. (bc Wong via eli)
|
||
|
||
HDFS-1781. Fix the path for jsvc in bin/hdfs. (John George via szetszwo)
|
||
|
||
HDFS-1782. Fix an NPE in FSNamesystem.startFileInternal(..).
|
||
(John George via szetszwo)
|
||
|
||
HDFS-1821. Fix username resolution in NameNode.createSymlink(..) and
|
||
FSDirectory.addSymlink(..). (John George via szetszwo)
|
||
|
||
HDFS-1806. TestBlockReport.blockReport_08() and _09() are timing-dependent
|
||
and likely to fail on fast servers. (Matt Foley via eli)
|
||
|
||
HDFS-1845. Symlink comes up as directory after namenode restart.
|
||
(John George via eli)
|
||
|
||
HDFS-1666. Disable failing hdfsproxy test TestAuthorizationFilter (todd)
|
||
|
||
HDFS-1823. start-dfs.sh script fails if HADOOP_HOME is not set.
|
||
(tomwhite via eli)
|
||
|
||
Release 0.21.1 - Unreleased
|
||
|
||
HDFS-1411. Correct backup node startup command in hdfs user guide.
|
||
(Ching-Shen Chen via shv)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv)
|
||
|
||
HDFS-1413. Fix broken links to HDFS Wiki. (shv)
|
||
|
||
HDFS-1420. Clover build doesn't generate per-test coverage (cos)
|
||
|
||
HDFS-1444. Test related code of build.xml is error-prone and needs to be
|
||
re-aligned. (cos)
|
||
|
||
HDFS-1343. Instrumented build should be concentrated in one build area (cos)
|
||
|
||
HDFS-1452. ant compile-contrib is broken (cos)
|
||
|
||
HDFS-1474. ant binary-system is broken (cos)
|
||
|
||
HDFS-1292. Allow artifacts to be published to the staging Apache Nexus
|
||
Maven Repository. (Giridharan Kesavan via tomwhite)
|
||
|
||
HDFS-1552. Remove java5 dependencies from build. (cos)
|
||
|
||
HDFS-1189. Quota counts missed between clear quota and set quota.
|
||
(John George via szetszwo)
|
||
|
||
HDFS-1665. Balancer misuses dfs.heartbeat.interval as milliseconds.
|
||
(szetszwo)
|
||
|
||
HDFS-1728. SecondaryNameNode.checkpointSize is in bytes but not in MB.
|
||
(szetszwo)
|
||
|
||
HDFS-1206. TestFiHFlush fails intermittently. (cos)
|
||
|
||
HDFS-1548. Fault-injection tests are executed multiple times if invoked
|
||
with run-test-hdfs-fault-inject target (cos)
|
||
|
||
HDFS-1552. Remove java5 dependencies from build. (cos)
|
||
|
||
HDFS-996. JUnit tests should never depend on anything in conf (cos)
|
||
|
||
HDFS-1612. Update HDFS design documentation for append, quota, symlink,
|
||
block placement and checkpoint/backup node features. (Joe Crobak
|
||
via szetszwo)
|
||
|
||
|
||
HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.*
|
||
in documentations. (Harsh J Chouraria via szetszwo)
|
||
|
||
HDFS-1786. Some cli test cases expect a "null" message
|
||
(Uma Maheswara Rao G via todd)
|
||
|
||
HDFS-1855. TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy()
|
||
part 2 fails in two different ways. (Matt Foley via eli)
|
||
|
||
Release 0.21.0 - 2010-08-13
|
||
|
||
INCOMPATIBLE CHANGES
|
||
|
||
HDFS-538. Per the contract elucidated in HADOOP-6201, throw
|
||
FileNotFoundException from FileSystem::listStatus rather than returning
|
||
null. (Jakob Homan via cdouglas)
|
||
|
||
HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException
|
||
instead of FileNotFoundException. (Boris Shkolnik via suresh)
|
||
|
||
HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong)
|
||
|
||
HDFS-576. Block report includes under-construction replicas. (shv)
|
||
|
||
HDFS-636. SafeMode counts complete blocks only. (shv)
|
||
|
||
HDFS-644. Lease recovery, concurrency support. (shv)
|
||
|
||
HDFS-570. Get last block length from a data-node when opening a file
|
||
being written to. (Tsz Wo (Nicholas), SZE via shv)
|
||
|
||
HDFS-657. Remove unused legacy data-node protocol methods. (shv)
|
||
|
||
HDFS-658. Block recovery for primary data-node. (shv)
|
||
|
||
HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv)
|
||
|
||
HDFS-512. Block.equals() and compareTo() compare blocks based
|
||
only on block Ids, ignoring generation stamps. (shv)
|
||
|
||
HDFS-873. Configuration specifies data-node storage directories as URIs.
|
||
(shv)
|
||
|
||
HDFS-905. Use the new UserGroupInformation from HDFS-6299.
|
||
(jghoman via omalley)
|
||
|
||
HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv)
|
||
|
||
HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the
|
||
error message on the screen when cat a directory or a
|
||
non-existent file. (hairong)
|
||
|
||
NEW FEATURES
|
||
|
||
HDFS-1134. Large-scale Automated Framework. (cos)
|
||
|
||
HDFS-436. Introduce AspectJ framework for HDFS code and tests.
|
||
(Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas)
|
||
|
||
HDFS-459. Introduce Job History Log Analyzer. (shv)
|
||
|
||
HDFS-461. Tool to analyze file size distribution in HDFS. (shv)
|
||
|
||
HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt
|
||
blocks/replicas information. (Bill Zeller via szetszwo)
|
||
|
||
HDFS-578. Add support for new FileSystem method for clients to get server
|
||
defaults. (Kan Zhang via suresh)
|
||
|
||
HDFS-595. umask settings in configuration may now use octal or symbolic
|
||
instead of decimal. (Jakob Homan via suresh)
|
||
|
||
HADOOP-6234. Updated hadoop-core and test jars to propagate new option
|
||
dfs.umaskmode in configuration. (Jakob Homan via suresh)
|
||
|
||
HDFS-235. Add support for byte ranges in HftpFileSystem to serve
|
||
range of bytes from a file. (Bill Zeller via suresh)
|
||
|
||
HDFS-385. Add support for an experimental API that allows a module external
|
||
to HDFS to specify how HDFS blocks should be placed. (dhruba)
|
||
|
||
HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext
|
||
file system application interface. (Sanjay Radia via suresh).
|
||
|
||
HDFS-567. Add block forensics contrib tool to print history of corrupt and
|
||
missing blocks from the HDFS logs.
|
||
(Bill Zeller, Jitendra Nath Pandey via suresh).
|
||
|
||
HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo)
|
||
|
||
HDFS-536. Support hflush at DFSClient. (hairong)
|
||
|
||
HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica
|
||
states while writing. (shv)
|
||
|
||
HDFS-565. Introduce block committing logic during new block allocation
|
||
and file close. (shv)
|
||
|
||
HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the
|
||
support of dfs writes/hflush. It also updates a replica's bytes received,
|
||
bytes on disk, and bytes acked after receiving a packet. (hairong)
|
||
|
||
HDFS-585. Datanode should serve up to visible length of a replica for read
|
||
requests. (szetszwo)
|
||
|
||
HDFS-604. Block report processing for append. (shv)
|
||
|
||
HDFS-619. Support replica recovery initialization in datanode for the new
|
||
append design. (szetszwo)
|
||
|
||
HDFS-592. Allow clients to fetch a new generation stamp from NameNode for
|
||
pipeline recovery. (hairong)
|
||
|
||
HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup
|
||
for append. (hairong)
|
||
|
||
HDFS-627. Support replica update in data-node.
|
||
(Tsz Wo (Nicholas), SZE and Hairong Kuang via shv)
|
||
|
||
HDFS-642. Support pipeline close and close error recovery. (hairong)
|
||
|
||
HDFS-631. Rename configuration keys towards API standardization and
|
||
backward compatibility. (Jitendra Nath Pandey via suresh)
|
||
|
||
HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins)
|
||
|
||
HDFS-731. Support new Syncable interface in HDFS. (hairong)
|
||
|
||
HDFS-702. Add HDFS implementation of AbstractFileSystem.
|
||
(Sanjay Radio via suresh)
|
||
|
||
HDFS-758. Add decommissioning status page to Namenode Web UI.
|
||
(Jitendra Nath Pandey via suresh)
|
||
|
||
HDFS-814. Add an api to get the visible length of a DFSDataInputStream.
|
||
(szetszwo)
|
||
|
||
HDFS-654. Add support new atomic rename functionality in HDFS for
|
||
supporting rename in FileContext. (suresh)
|
||
|
||
HDFS-222. Support for concatenating of files into a single file
|
||
without copying. (Boris Shkolnik via hairong)
|
||
|
||
HDFS-933. Adds Delegation token based authentication in the NameNode.
|
||
(Kan Zhang via ddas)
|
||
|
||
HDFS-935. Adds a real user component in Delegation token.
|
||
(Jitendra Nath Pandey via ddas)
|
||
|
||
HDFS-245. Adds a symlink implementation to HDFS. This complements the new
|
||
symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia)
|
||
|
||
HDFS-1009. Support Kerberos authorization in HDFSProxy. (Srikanth
|
||
Sundarrajan via szetszwo)
|
||
|
||
HDFS-1091. Implement listStatus that returns an iterator of FileStatus.
|
||
(hairong)
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-381. Remove blocks from DataNode maps when corresponding file
|
||
is deleted. (Suresh Srinivas via rangadi)
|
||
|
||
HDFS-377. Separate codes which implement DataTransferProtocol.
|
||
(szetszwo)
|
||
|
||
HDFS-396. NameNode image and edits directories are specified as URIs.
|
||
(Luca Telloli via rangadi)
|
||
|
||
HDFS-444. Allow to change probability levels dynamically in the fault
|
||
injection framework. (Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv)
|
||
|
||
HADOOP-6106. Updated hadoop-core and test jars from hudson trunk
|
||
build #12. (Giridharan Kesavan)
|
||
|
||
HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode.
|
||
(Jitendra Nath Pandey via szetszwo)
|
||
|
||
HDFS-278. HDFS Outputstream close does not hang forever. (dhruba)
|
||
|
||
HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode.
|
||
(Jitendra Nath Pandey via szetszwo)
|
||
|
||
HDFS-475. Add new ant targets for fault injection jars and tests.
|
||
(Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan
|
||
via szetszwo)
|
||
|
||
HDFS-493. Change build.xml so that the fault-injected tests are executed
|
||
only by the run-test-*-fault-inject targets. (Konstantin Boudnik via
|
||
szetszwo)
|
||
|
||
HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv)
|
||
|
||
HADOOP-6160. Fix releaseaudit target to run on specific directories.
|
||
(gkesavan)
|
||
|
||
HDFS-501. Use enum to define the constants in DataTransferProtocol.
|
||
(szetszwo)
|
||
|
||
HDFS-508. Factor out BlockInfo from BlocksMap. (shv)
|
||
|
||
HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo.
|
||
(Jakob Homan & Hairong Kuang via shv)
|
||
|
||
HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol.
|
||
(Jakob Homan via shv)
|
||
|
||
HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller
|
||
via szetszwo)
|
||
|
||
HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo)
|
||
|
||
HDFS-511. Remove redundant block searches in BlockManager. (shv)
|
||
|
||
HDFS-504. Update the modification time of a file when the file
|
||
is closed. (Chun Zhang via dhruba)
|
||
|
||
HDFS-498. Add development guide and documentation for the fault injection
|
||
framework. (Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo)
|
||
|
||
HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches
|
||
in BlockManager. (shv)
|
||
|
||
HDFS-530. Refactor TestFileAppend* to remove code duplication.
|
||
(Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo)
|
||
|
||
HDFS-409. Add more access token tests. (Kan Zhang via szetszwo)
|
||
|
||
HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv)
|
||
|
||
HDFS-457. Do not shutdown datanode if some, but not all, volumes fail.
|
||
(Boris Shkolnik via szetszwo)
|
||
|
||
HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong)
|
||
|
||
HDFS-539. Refactor fault injeciton pipeline test util for future reuse.
|
||
(Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new
|
||
tests. (szetszwo)
|
||
|
||
HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..).
|
||
(szetszwo)
|
||
|
||
HDFS-581. Introduce an iterator over blocks in the block report array.(shv)
|
||
|
||
HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which
|
||
allows an execution of non-FI tests in FI-enable environment. (Konstantin
|
||
Boudnik via szetszwo)
|
||
|
||
HDFS-173. Namenode will not block until a large directory deletion
|
||
completes. It allows other operations when the deletion is in progress.
|
||
(suresh)
|
||
|
||
HDFS-551. Create new functional test for a block report. (Konstantin
|
||
Boudnik via hairong)
|
||
|
||
HDFS-288. Redundant computation in hashCode() implementation.
|
||
(szetszwo via tomwhite)
|
||
|
||
HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible.
|
||
(Brian Bockelman via tomwhite)
|
||
|
||
HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design
|
||
document. (Zhiyong Zhang via cdouglas)
|
||
|
||
HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo)
|
||
|
||
HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo)
|
||
|
||
HDFS-574. Split the documentation between the subprojects.
|
||
(Corinne Chandel via omalley)
|
||
|
||
HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite)
|
||
|
||
HDFS-641. Move all of the components that depend on map/reduce to
|
||
map/reduce. (omalley)
|
||
|
||
HDFS-509. Redesign DataNode volumeMap to include all types of Replicas.
|
||
(hairong)
|
||
|
||
HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from
|
||
un-closed file. (szetszwo)
|
||
|
||
HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary,
|
||
writeToRBW, ad append. (hairong)
|
||
|
||
HDFS-603. Add a new interface, Replica, which is going to replace the use
|
||
of Block in datanode. (szetszwo)
|
||
|
||
HDFS-589. Change block write protocol to support pipeline recovery.
|
||
(hairong)
|
||
|
||
HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv)
|
||
|
||
HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin
|
||
Boudnik via szetszwo)
|
||
|
||
HDFS-662. Unnecessary info message from DFSClient. (hairong)
|
||
|
||
HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik
|
||
via szetszwo)
|
||
|
||
HDFS-688. Add configuration resources to DFSAdmin. (shv)
|
||
|
||
HDFS-29. Validate the consistency of the lengths of replica and its file
|
||
in replica recovery. (szetszwo)
|
||
|
||
HDFS-680. Add new access method to a copy of a block's replica. (shv)
|
||
|
||
HDFS-704. Unify build property names to facilitate cross-projects
|
||
modifications (cos)
|
||
|
||
HDFS-705. Create an adapter to access some of package-private methods of
|
||
DataNode from tests (cos)
|
||
|
||
HDFS-710. Add actions with constraints to the pipeline fault injection
|
||
tests and change SleepAction to support uniform random sleeping over an
|
||
interval. (szetszwo)
|
||
|
||
HDFS-713. Need to properly check the type of the test class from an aspect
|
||
(cos)
|
||
|
||
HDFS-716. Define a pointcut for pipeline close and add a few fault
|
||
injection tests to simulate out of memory problem. (szetszwo)
|
||
|
||
HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow
|
||
datanodes and disk errors. (szetszwo)
|
||
|
||
HDFS-616. Create functional tests for new design of the block report. (cos)
|
||
|
||
HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos)
|
||
|
||
HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode
|
||
and out-of-memory problem for pipeline close ack. (szetszwo)
|
||
|
||
HDFS-728. Create a comprehensive functional test for append. (hairong)
|
||
|
||
HDFS-736. commitBlockSynchronization() updates block GS and length
|
||
in-place. (shv)
|
||
|
||
HADOOP-5107. Use Maven ant tasks to publish the subproject jars.
|
||
(Giridharan Kesavan via omalley)
|
||
|
||
HDFS-521. Create new tests for pipeline (cos)
|
||
|
||
HDFS-764. Places the Block Access token implementation in hdfs project.
|
||
(Kan Zhang via ddas)
|
||
|
||
HDFS-787. Upgrade some libraries to be consistent with common and
|
||
mapreduce. (omalley)
|
||
|
||
HDFS-519. Create new tests for lease recovery (cos)
|
||
|
||
HDFS-804. New unit tests for concurrent lease recovery (cos)
|
||
|
||
HDFS-813. Enable the append test in TestReadWhileWriting. (szetszwo)
|
||
|
||
HDFS-145. Cleanup inconsistent block length handling code in
|
||
FSNameSystem#addStoredBlock. (hairong)
|
||
|
||
HDFS-127. Reset failure count in DFSClient for each block acquiring
|
||
operation. (Igor Bolotin via szetszwo)
|
||
|
||
HDFS-520. Create new tests for block recovery. (hairong)
|
||
|
||
HDFS-1067. Create block recovery tests that handle errors. (hairong)
|
||
|
||
HDFS-1107. Turn on append by default. (shv)
|
||
|
||
HDFS-968. Use StringBuilder instead of StringBuffer for better
|
||
performance. (Kay Kay via suresh)
|
||
|
||
HDFS-703. Replace current fault injection implementation with one
|
||
from (cos)
|
||
|
||
HDFS-754. Reduce ivy console output to observable level (cos)
|
||
|
||
HDFS-832. HDFS side of HADOOP-6222. (cos)
|
||
|
||
HDFS-840. Change tests to use FileContext test helper introduced in
|
||
HADOOP-6394. (Jitendra Nath Pandey via suresh)
|
||
|
||
HDFS-685. Use the user-to-groups mapping service in the NameNode.
|
||
(boryas, acmurthy)
|
||
|
||
HDFS-755. Read multiple checksum chunks at once in DFSInputStream.
|
||
(Todd Lipcon via tomwhite)
|
||
|
||
HDFS-786. Implement getContentSummary in HftpFileSystem.
|
||
(Tsz Wo (Nicholas), SZE via cdouglas)
|
||
|
||
HDFS-587. Add support for specifying queue name in mapreduce tests.
|
||
(Erik Steffl via suresh)
|
||
|
||
HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley)
|
||
|
||
HDFS-800. The last block of a file under construction may change to the
|
||
COMPLETE state in response to getAdditionalBlock or completeFileInternal.
|
||
(hairong)
|
||
|
||
HDFS-899. Delegation Token Implementation
|
||
and corresponding changes in Namenode and DFS Api to issue,
|
||
renew and cancel delegation tokens. (jnp via boryas)
|
||
|
||
HDFS-844. Log the filename when file locking fails. (tomwhite)
|
||
|
||
HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient.
|
||
(Todd Lipcon via tomwhite)
|
||
|
||
HDFS-949. Move DelegationToken into Common so that it can be used by
|
||
MapReduce. (omalley)
|
||
|
||
HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched.
|
||
(Kay Kay via szetszwo)
|
||
|
||
HDFS-986. Delegation token renewing and cancelling should provide
|
||
meaningful exceptions when there are failures instead of returning
|
||
false. (omalley)
|
||
|
||
HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley)
|
||
|
||
HDFS-991. Allow authentication to the web ui via a delegation token.
|
||
(omalley)
|
||
|
||
HDFS-994. Allow fetching of delegation token from NameNode for hftp.
|
||
(Jakob Homan via acmurthy)
|
||
|
||
HDFS-998. Quote blocks streamed through jsps. (cdouglas)
|
||
|
||
HDFS-729. NameNode API to list files that have missing blocks.
|
||
(Rodrigo Schmidt via dhruba)
|
||
|
||
HDFS-850. The WebUI display more details about namenode memory usage.
|
||
(Dmytro Molkov via dhruba)
|
||
|
||
HDFS-826. The DFSOutputStream has a API that returns the number of
|
||
active datanode(s) in the current pipeline. (dhruba)
|
||
|
||
HDFS-985. HDFS should issue multiple RPCs for listing a large
|
||
directory. (hairong)
|
||
|
||
HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of
|
||
server-side user group resolution. (shv)
|
||
|
||
HDFS-892. Optionally use Avro reflection for Namenode RPC. This
|
||
is not a complete implementation yet, but rather a starting point.
|
||
(cutting)
|
||
|
||
HDFS-854. Datanode should scan devices in parallel to generate
|
||
block report. (Dmytro Molkov via jhoman)
|
||
|
||
HDFS-1032. fsck has an option to list corrupt files.
|
||
(Andre Oriai via dhruba)
|
||
|
||
HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file.
|
||
(Dmytro Molkov via dhruba)
|
||
|
||
HDFS-1011. hdfsproxy: Improve log messages by restoring the previous
|
||
thread name. (Srikanth Sundarrajan via szetszwo)
|
||
|
||
HDFS-997. Allow datanode storage directory permissions to be configurable.
|
||
(Luke Lu via cdouglas)
|
||
|
||
HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to
|
||
simple unqualified path. (Srikanth Sundarrajan via szetszwo)
|
||
|
||
HDFS-933. Namenode should issue a delegation token only for kerberos
|
||
authenticated clients.(jnp via boryas)
|
||
|
||
HDFS-1087. Modify audit log to use a StringBuilder rather than a Formatter.
|
||
(cdouglas)
|
||
|
||
HDFS-1083. Update TestHDFSCLI not to expect exception class name
|
||
in error messages. (suresh)
|
||
|
||
HDFS-1099. Add test for umask backward compatibility. (suresh)
|
||
|
||
HDFS-1092. Use logging rather than System.err in MiniDFSCluster.
|
||
(Kay Kay via jghoman)
|
||
|
||
HDFS-1047. Install/deploy source jars to Maven repo.
|
||
(Patrick Angeles via jghoman)
|
||
|
||
HDFS-666. Unit test for FsShell -text. (cdouglas via jghoman)
|
||
|
||
HDFS-1054. Remove unnecessary sleep after failure in nextBlockOutputStream.
|
||
(Todd Lipcon via jghoman)
|
||
|
||
HDFS-921. Convert TestDFSClientRetries::testNotYetReplicatedErrors
|
||
to Mockito. (jghoman)
|
||
|
||
HDFS-1100. Override unwrapException in TestFcHdfsSymlink to test
|
||
symlink API conformance. (Eli Collins via suresh).
|
||
|
||
HDFS-1089. Remove uses of FileContext#isFile, isDirectory, and exists.
|
||
(Eli Collins via hairong)
|
||
|
||
HDFS-1028. Efficient splitting of path components reduces the time
|
||
to load in fsimage by 20%. (Dmytro Molkov via dhruba)
|
||
|
||
HDFS-1109. HFTP supports filenames that contains the character "+".
|
||
(Dmytro Molkov via dhruba)
|
||
|
||
HDFS-853. The HDFS webUI displays the balanced-ness of the cluster.
|
||
(Dmytro Molkov via dhruba)
|
||
|
||
HDFS-1126. Change HDFS to depend on Hadoop 'common' artifacts instead
|
||
of 'core'. (tomwhite)
|
||
|
||
HDFS-995. Replace usage of FileStatus#isDir(). (Eli Collins via
|
||
tomwhite)
|
||
|
||
HDFS-1161. Make DN minimum valid volumes configurable.
|
||
(Eli Collins via tomwhite)
|
||
|
||
HDFS-1181. Move configuration and script files post split. (tomwhite)
|
||
|
||
HDFS-1170. Add more assertions to TestLargeDirectoryDelete.
|
||
(Steve Loughran via tomwhite)
|
||
|
||
HDFS-1199. Extract a subset of tests for smoke (DOA) validation. (cos)
|
||
|
||
HDFS-1174. New properties for suspend and resume process. (Vinay Thota via
|
||
cos)
|
||
|
||
HDFS-1277. [Herriot] New property for multi user list. (Vinay Thota via
|
||
cos)
|
||
|
||
HDFS-806. Add new unit tests to the 10-mins 'run-commit-test' target (cos)
|
||
|
||
OPTIMIZATIONS
|
||
|
||
HDFS-946. NameNode should not return full path name when lisitng a
|
||
diretory or getting the status of a file. (hairong)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-76. Better error message to users when commands fail because of
|
||
lack of quota. Allow quota to be set even if the limit is lower than
|
||
current consumption. (Boris Shkolnik via rangadi)
|
||
|
||
HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under
|
||
Hadoop (Owen O'Malley)
|
||
|
||
HADOOP-6096. Fix Eclipse project and classpath files following project
|
||
split. (tomwhite)
|
||
|
||
HDFS-195. Handle expired tokens when write pipeline is reestablished.
|
||
(Kan Zhang via rangadi)
|
||
|
||
HDFS-181. Validate src path in FSNamesystem.getFileInfo(..). (Todd
|
||
Lipcon via szetszwo)
|
||
|
||
HDFS-441. Remove TestFTPFileSystem. (szetszwo)
|
||
|
||
HDFS-440. Fix javadoc broken links in DFSClient. (szetszwo)
|
||
|
||
HDFS-480. Fix a typo in the jar name in build.xml.
|
||
(Konstantin Shvachko via gkesavan)
|
||
|
||
HDFS-438. Check for NULL before invoking GenericArgumentParser in
|
||
DataNode. (Raghu Angadi)
|
||
|
||
HDFS-415. BlockReceiver hangs in case of certain runtime exceptions.
|
||
(Konstantin Boudnik via rangadi)
|
||
|
||
HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv)
|
||
|
||
HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan
|
||
via szetszwo)
|
||
|
||
HDFS-445. pread() does not pick up changes to block locations.
|
||
(Kan Zhang via rangadi)
|
||
|
||
HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for
|
||
FSImage). (Suresh Srinivas via rangadi)
|
||
|
||
HDFS-484. Fix bin-package and package target to package jar files.
|
||
(gkesavan)
|
||
|
||
HDFS-490. Eliminate the deprecated warnings introduced by H-5438.
|
||
(He Yongqiang via szetszwo)
|
||
|
||
HDFS-119. Fix a bug in logSync(), which causes NameNode block forever.
|
||
(Suresh Srinivas via shv)
|
||
|
||
HDFS-534. Include avro in ivy. (szetszwo)
|
||
|
||
HDFS-532. Allow applications to know that a read request failed
|
||
because block is missing. (dhruba)
|
||
|
||
HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol.
|
||
(Kan Zhang via szetszwo)
|
||
|
||
HDFS-553. BlockSender reports wrong failed position in ChecksumException.
|
||
(hairong)
|
||
|
||
HDFS-568. Set mapred.job.tracker.retire.jobs to false in
|
||
src/test/mapred-site.xml for mapreduce tests to run. (Amareshwari
|
||
Sriramadasu via szetszwo)
|
||
|
||
HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong)
|
||
|
||
HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails.
|
||
(Jitendra Nath Pandey via hairong)
|
||
|
||
HADOOP-6243. Fixed a NullPointerException in handling deprecated keys.
|
||
(Sreekanth Ramakrishnan via yhemanth)
|
||
|
||
HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr
|
||
target. (Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas()
|
||
(shv)
|
||
|
||
HDFS-601. TestBlockReport obtains data directories directly from
|
||
MiniHDFSCluster. (Konstantin Boudnik via shv)
|
||
|
||
HDFS-614. TestDatanodeBlockScanner obtains data directories directly from
|
||
MiniHDFSCluster. (shv)
|
||
|
||
HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset. (szetszwo)
|
||
|
||
HDFS-622. checkMinReplication should count live nodes only. (shv)
|
||
|
||
HDFS-629. Remove ReplicationTargetChooser.java along with fixing
|
||
import warnings generated by Eclipse. (dhruba)
|
||
|
||
HDFS-637. DataNode sends a Success ack when block write fails. (hairong)
|
||
|
||
HDFS-640. Fixed TestHDFSFileContextMainOperations.java build failure. (suresh)
|
||
|
||
HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice
|
||
sometimes fails with CloseByInterruptException. (hairong)
|
||
|
||
HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv)
|
||
|
||
HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas
|
||
when handling detached replicas. (hairong)
|
||
|
||
HDFS-659. If the the last block is not complete, update its length with
|
||
one of its replica's length stored in datanode. (szetszwo)
|
||
|
||
HDFS-649. Check null pointers for DataTransferTest. (Konstantin Boudnik
|
||
via szetszwo)
|
||
|
||
HDFS-661. DataNode upgrade fails on non-existant current directory.
|
||
(hairong)
|
||
|
||
HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in
|
||
FSDatasetAspects. (Konstantin Boudnik via szetszwo)
|
||
|
||
HDFS-665. TestFileAppend2 sometimes hangs. (hairong)
|
||
|
||
HDFS-676. Fix NPE in FSDataset.updateReplicaUnderRecovery() (shv)
|
||
|
||
HDFS-673. BlockReceiver#PacketResponder should not remove a packet from
|
||
the ack queue before its ack is sent. (hairong)
|
||
|
||
HDFS-682. Fix bugs in TestBlockUnderConstruction. (szetszwo)
|
||
|
||
HDFS-668. TestFileAppend3#TC7 sometimes hangs. (hairong)
|
||
|
||
HDFS-679. Appending to a partial chunk incorrectly assumes the
|
||
first packet fills up the partial chunk. (hairong)
|
||
|
||
HDFS-722. Fix callCreateBlockWriteStream pointcut in FSDatasetAspects.
|
||
(szetszwo)
|
||
|
||
HDFS-690. TestAppend2#testComplexAppend failed on "Too many open files".
|
||
(hairong)
|
||
|
||
HDFS-725. Support the build error fix for HADOOP-6327. (Sanjay Radia via
|
||
szetszwo)
|
||
|
||
HDFS-625. Fix NullPointerException thrown from ListPathServlet. (suresh)
|
||
|
||
HDFS-735. TestReadWhileWriting has wrong line termination symbols (cos)
|
||
|
||
HDFS-691. Fix an overflow error in DFSClient.DFSInputStream.available().
|
||
(szetszwo)
|
||
|
||
HDFS-733. TestBlockReport fails intermittently. (cos)
|
||
|
||
HDFS-774. Intermittent race condition in TestFiPipelines (cos)
|
||
|
||
HDFS-741. TestHFlush test doesn't seek() past previously written part of
|
||
the file (cos, szetszwo)
|
||
|
||
HDFS-706. Intermittent failures in TestFiHFlush (cos)
|
||
|
||
HDFS-646. Fix test-patch failure by adding test-contrib ant target.
|
||
(gkesavan)
|
||
|
||
HDFS-791. Build is broken after HDFS-787 patch has been applied (cos)
|
||
|
||
HDFS-792. TestHDFSCLI is failing. (Todd Lipcon via cos)
|
||
|
||
HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented.
|
||
(Suresh)
|
||
|
||
HDFS-192. Fix TestBackupNode failures. (shv)
|
||
|
||
HDFS-797. TestHDFSCLI much slower after HDFS-265 merge. (Todd Lipcon via cos)
|
||
|
||
HDFS-824. Stop lease checker in TestReadWhileWriting. (szetszwo)
|
||
|
||
HDFS-823. CheckPointer should use addInternalServlet for image-fetching
|
||
servlet (jghoman)
|
||
|
||
HDFS-456. Fix URI generation for windows file paths. (shv)
|
||
|
||
HDFS-812. FSNamesystem#internalReleaseLease throws NullPointerException on
|
||
a single-block file's lease recovery. (cos)
|
||
|
||
HDFS-724. Pipeline hangs if one of the block receiver is not responsive.
|
||
(hairong)
|
||
|
||
HDFS-564. Adding pipeline tests 17-35. (hairong)
|
||
|
||
HDFS-849. TestFiDataTransferProtocol2#pipeline_Fi_18 sometimes fails.
|
||
(hairong)
|
||
|
||
HDFS-762. Balancer causes Null Pointer Exception.
|
||
(Cristian Ivascu via dhruba)
|
||
|
||
HDFS-868. Fix link to Hadoop Upgrade Wiki. (Chris A. Mattmann via shv)
|
||
|
||
HDFS-880. TestNNLeaseRecovery fails on windows (cos, shv)
|
||
|
||
HDFS-699. Primary datanode should compare replicas' on disk lengths.
|
||
(hairong)
|
||
|
||
HDFS-897. Fix a bug related to generation stamp comparison in
|
||
ReplicasMap. (suresh)
|
||
|
||
HDFS-793. Data node should receive the whole packet ack message before it
|
||
constructs and sends its own ack message for the packet. (hairong)
|
||
|
||
HDFS-101. DFS write pipeline: DFSClient sometimes does not detect second
|
||
datanode failure. (hairong)
|
||
|
||
HDFS-822. Appends to already-finalized blocks can rename across volumes.
|
||
(hairong)
|
||
|
||
HDFS-1046. Fix Tomcat version in hdfsproxy/build.xml. (Srikanth
|
||
Sundarrajan via szetszwo)
|
||
|
||
HDFS-1072. Fix TestReadWhileWriting failure. (Erik Steffl via shv)
|
||
|
||
HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java
|
||
to include it in tests run by ant target run-test-hdfs-fault-inject.
|
||
(suresh)
|
||
|
||
HDFS-695. RaidNode should read in configuration from hdfs-site.xml.
|
||
(dhruba)
|
||
|
||
HDFS-726. Eclipse .classpath template has outdated jar files and is
|
||
missing some new ones. (cos)
|
||
|
||
HDFS-750. Fix build failure due to TestRename. (suresh)
|
||
|
||
HDFS-712. Move libhdfs from mapreduce subproject to hdfs subproject.
|
||
(Eli Collins via dhruba)
|
||
|
||
HDFS-757. Enable Unit test for HDFS Raid. (dhruba)
|
||
|
||
HDFS-611. Prevent DataNode heartbeat times from increasing even when
|
||
the DataNode has many blocks to delete. (Zheng Shao via dhruba)
|
||
|
||
HDFS-751. Fix TestCrcCorruption to pick up the correct datablocks to
|
||
corrupt. (dhruba)
|
||
|
||
HDFS-763. Fix slightly misleading report from DataBlockScanner
|
||
about corrupted scans. (dhruba)
|
||
|
||
HDFS-727. bug setting block size hdfsOpenFile (Eli Collins via cos)
|
||
|
||
HDFS-756. libhdfs unit tests do not run. (Eli Collins via cos)
|
||
|
||
HDFS-783. libhdfs tests brakes code coverage runs with Clover (cos)
|
||
|
||
HDFS-785. Add Apache license to several namenode unit tests.
|
||
(Ravi Phulari via jghoman)
|
||
|
||
HDFS-802. Update Eclipse configuration to match changes to Ivy
|
||
configuration (Edwin Chan via cos)
|
||
|
||
HDFS-423. Unbreak FUSE build and fuse_dfs_wrapper.sh (Eli Collins via cos)
|
||
|
||
HDFS-825. Build fails to pull latest hadoop-core-* artifacts (cos)
|
||
|
||
HDFS-94. The Heap Size printed in the NameNode WebUI is accurate.
|
||
(Dmytro Molkov via dhruba)
|
||
|
||
HDFS-767. An improved retry policy when the DFSClient is unable to fetch a
|
||
block from the datanode. (Ning Zhang via dhruba)
|
||
|
||
HDFS-775. FSDataset calls getCapacity() twice. (stevel)
|
||
|
||
HDFS-885. Datanode toString() NPEs on null dnRegistration. (stevel)
|
||
|
||
HDFS-877. Client-driven block verification not functioning. (Todd
|
||
Lipcon via hairong)
|
||
|
||
HDFS-630. In DFSOutputStream.nextBlockOutputStream(), the client can
|
||
exclude specific datanodes when locating the next block.
|
||
(Cosmin Lehene via Stack)
|
||
|
||
HDFS-922. Remove unnecessary semicolon added by HDFS-877 that causes
|
||
problems for Eclipse compilation. (jghoman)
|
||
|
||
HDFS-927 DFSInputStream retries too many times for new block locations
|
||
(Todd Lipcon via Stack)
|
||
|
||
HDFS-938. Replace calls to UGI.getUserName() with UGI.getShortUserName()
|
||
(jghoman)
|
||
|
||
HDFS-894. DatanodeID.ipcPort is not updated when existing node
|
||
re-registers. (Todd Lipcon via tomwhite)
|
||
|
||
HDFS-965. Split TestDelegationToken in to two parts and fix configuration
|
||
to allow proxy users in the test. (Jitendra Pandey via omalley)
|
||
|
||
HDFS-999. Secondary namenode should login using kerberos if security is
|
||
configured (boryas)
|
||
|
||
HDFS-856. Hardcoded replication level for new files in fuse-dfs.
|
||
(Brian Bockelman via tomwhite)
|
||
|
||
HDFS-857. Incorrect type for fuse-dfs capacity can cause "df" to return
|
||
negative values on 32-bit machines. (Brian Bockelman via tomwhite)
|
||
|
||
HDFS-858. Incorrect return codes for fuse-dfs. (Brian Bockelman via
|
||
tomwhite)
|
||
|
||
HDFS-859. fuse-dfs utime behavior causes issues with tar.
|
||
(Brian Bockelman via tomwhite)
|
||
|
||
HDFS-861. fuse-dfs does not support O_RDWR. (Brian Bockelman via tomwhite)
|
||
|
||
HDFS-961. dfs_readdir incorrectly parses paths. (Eli Collins via tomwhite)
|
||
|
||
HDFS-1015. Fix intermittent failure in TestSecurityTokenEditLog.
|
||
(Jitendra Nath Pandey via suresh)
|
||
|
||
HDFS-939. libhdfs test is broken. (Eli Collins via tomwhite)
|
||
|
||
HDFS-1074. hdfsproxy: Fix bugs in TestProxyUtil. (Srikanth Sundarrajan
|
||
via szetszwo)
|
||
|
||
HDFS-481. hdfsproxy: Bug Fixes + HdfsProxy to use proxy user to
|
||
impresonate the real user. (Srikanth Sundarrajan via szetszwo)
|
||
|
||
HDFS-482. Move HsftpFileSystem's ssl.client.do.not.authenticate.server
|
||
configuration setting to ssl-client.xml. (Srikanth Sundarrajan via
|
||
szetszwo)
|
||
|
||
HDFS-1010. hdfsproxy: Retrieve groups from UnixUserGroupInformation
|
||
instead of LdapEntry. (Srikanth Sundarrajan via szetszwo)
|
||
|
||
HDFS-466. hdfs_write infinite loop when dfs fails and cannot write
|
||
files > 2 GB. (Pete Wyckoff via tomwhite)
|
||
|
||
HDFS-651. HDFS Docs - fix listing of docs in the doc menu.
|
||
(Corinne Chandel via tomwhite)
|
||
|
||
HDFS-1014. Error in reading delegation tokens from edit logs.
|
||
(Jitendra Nath Pandey via jhoman)
|
||
|
||
HDFS-1088. Prevent renaming a symbolik link to its target.
|
||
(Eli Collins via suresh)
|
||
|
||
HDFS-966. NameNode does not recovers lease when it is in safemode.
|
||
(dhruba)
|
||
|
||
HDFS-833. Datanode shutdown should log problems with Storage.unlockAll()
|
||
(Steve Loughran via dhruba)
|
||
|
||
HDFS-1101. TestDiskError.testLocalDirs() fails. (cdouglas via jghoman)
|
||
|
||
HDFS-1031. Enhance the webUi to list a few of the corrupted files in HDFS.
|
||
(Andre Orian via dhruba)
|
||
|
||
HDFS-1078. Create static and dynamic versions of libhdfs.
|
||
(Sam Rash via dhruba)
|
||
|
||
HDFS-1104. Fsck triggers full GC on NameNode. (hairong)
|
||
|
||
HDFS-1141. Closing a file is successful only if the client still has a
|
||
valid lease. (Todd Lipcon via dhruba)
|
||
|
||
HDFS-1138. Prevent erroneous updation of modification time of a directory
|
||
when fsimage loads. (Dmytro Molkov via dhruba)
|
||
|
||
HDFS-1000. Updates libhdfs to the new API for UGI (ddas)
|
||
|
||
HDFS-609. Create a file with the append flag does not work in HDFS.
|
||
(tomwhite)
|
||
|
||
HDFS-1255. Fix failing test-libhdfs.sh test. (tomwhite)
|
||
|
||
HDFS-1256. libhdfs is missing from the tarball. (tomwhite)
|
||
|
||
HDFS-1057. Concurrent readers hit ChecksumExceptions if following a
|
||
writer to very end of file. (sam rash via hairong)
|
||
|
||
HDFS-1212. Harmonize HDFS JAR library versions with Common. (tomwhite)
|
||
|
||
HDFS-1159. clean-cache target removes wrong ivy cache (cos)
|
||
|
||
HDFS-1193. -mvn-system-deploy target is broken which inturn fails the
|
||
mvn-deploy task leading to unstable mapreduce build (Giridharan
|
||
Kesavan via cos)
|
||
|
||
HDFS-1299. 'compile-fault-inject' never should be called directly. (cos)
|
||
|
||
HDFS-1311. Running tests with 'testcase' cause triple execution of the
|
||
same test case (Cos)
|
||
|
||
HDFS-1267. fuse-dfs does not compile. (Devaraj Das via tomwhite)
|
||
|
||
HDFS-1598. Directory listing on hftp:// does not show .*.crc files.
|
||
(szetszwo)
|
||
|
||
HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName()
|
||
to get file name since it may return an empty string. (szetszwo)
|
||
|
||
Release 0.20.3 - Unreleased
|
||
|
||
IMPROVEMENTS
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to
|
||
the first datanode fails. (szetszwo)
|
||
|
||
HDFS-909. Wait until edits syncing is finishes before purging edits.
|
||
(Todd Lipcon via shv)
|
||
|
||
HDFS-1258. Clearing namespace quota on "/" corrupts fs image.
|
||
(Aaron T. Myers via szetszwo)
|
||
|
||
HDFS-1406. TestCLI fails on Ubuntu with default /etc/hosts. (cos)
|
||
|
||
Release 0.20.203.0 - 2011-5-11
|
||
|
||
IMPROVEMENTS
|
||
|
||
HADOOP-7259. Contrib modules should include the build.properties from
|
||
the enclosing hadoop directory. (omalley)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-132. Fix namenode to not report files deleted metrics for deletions
|
||
done while replaying edits during startup. (suresh & shv)
|
||
|
||
HDFS-955. New implementation of saveNamespace() to avoid loss of edits
|
||
when name-node fails during saving. (shv)
|
||
|
||
Release 0.20.2 - 2009-09-01
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-737. Add full path name of the file to the block information and
|
||
summary of total number of files, blocks, live and deadnodes to
|
||
metasave output. (Jitendra Nath Pandey via suresh)
|
||
|
||
HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry
|
||
via cos)
|
||
|
||
HDFS-907. Add tests for getBlockLocations and totalLoad metrics.
|
||
(Ravi Phulari via cos)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-686. NullPointerException is thrown while merging edit log and image.
|
||
(hairong)
|
||
|
||
HDFS-677. Rename failure when both source and destination quota exceeds
|
||
results in deletion of source. (suresh)
|
||
|
||
HDFS-709. Fix TestDFSShell failure due to rename bug introduced by
|
||
HDFS-677. (suresh)
|
||
|
||
HDFS-579. Fix DfsTask to follow the semantics of 0.19, regarding non-zero
|
||
return values as failures. (Christian Kunz via cdouglas)
|
||
|
||
HDFS-723. Fix deadlock in DFSClient#DFSOutputStream. (hairong)
|
||
|
||
HDFS-596. Fix memory leak in hdfsFreeFileInfo() for libhdfs.
|
||
(Zhang Bingjun via dhruba)
|
||
|
||
HDFS-185. Disallow chown, chgrp, chmod, setQuota, and setSpaceQuota when
|
||
name-node is in safemode. (Ravi Phulari via shv)
|
||
|
||
HDFS-187. Initialize secondary namenode http address in TestStartup.
|
||
(Todd Lipcon via szetszwo)
|
||
|
||
HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh)
|
||
|
||
HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli)
|
||
|
||
Release 0.20.1 - 2009-09-01
|
||
|
||
IMPROVEMENTS
|
||
|
||
HDFS-438. Improve help message for space quota command. (Raghu Angadi)
|
||
|
||
BUG FIXES
|
||
|
||
HDFS-167. Fix a bug in DFSClient that caused infinite retries on write.
|
||
(Bill Zeller via szetszwo)
|
||
|
||
HDFS-527. Remove/deprecate unnecessary DFSClient constructors. (szetszwo)
|
||
|
||
HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread
|
||
safe. (Suresh Srinivas and cdouglas)
|
||
|
||
HDFS-761. Fix failure to process rename operation from edits log due to
|
||
quota verification. (suresh)
|