From 1bc454615e916e66b132eb688a4bc8e76dcb0820 Mon Sep 17 00:00:00 2001 From: Andrew Wang Date: Thu, 3 Mar 2016 21:02:31 -0800 Subject: [PATCH] HADOOP-11792. Remove all of the CHANGES.txt files. --- .../hadoop-common/CHANGES.txt | 18364 ---------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 14611 ------------ hadoop-mapreduce-project/CHANGES.txt | 8906 -------- hadoop-yarn-project/CHANGES.txt | 6905 ------ 4 files changed, 48786 deletions(-) delete mode 100644 hadoop-common-project/hadoop-common/CHANGES.txt delete mode 100644 hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt delete mode 100644 hadoop-mapreduce-project/CHANGES.txt delete mode 100644 hadoop-yarn-project/CHANGES.txt diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt deleted file mode 100644 index 78ac5d78b90..00000000000 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ /dev/null @@ -1,18364 +0,0 @@ -Hadoop Change Log - -Trunk (Unreleased) - - INCOMPATIBLE CHANGES - - HADOOP-8124. Remove the deprecated FSDataOutputStream constructor, - FSDataOutputStream.sync() and Syncable.sync(). (szetszwo) - - HADOOP-10474 Move o.a.h.record to hadoop-streaming. (wheat9) - - HADOOP-9902. Shell script rewrite (aw) - - HADOOP-10950. rework heap management vars (John Smith via aw) - - HADOOP-6857. FsShell should report raw disk usage including replication - factor. (Byron Wong via shv) - - HADOOP-11657. Align the output of `hadoop fs -du` to be more Unix-like. - (aajisaka) - - HADOOP-11553. Formalize the shell API (aw) - - HADOOP-11731. Rework the changelog and releasenotes (aw) - - HADOOP-11627. Remove io.native.lib.available. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11698. Remove DistCpV1 and Logalyzer. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11356. Removed deprecated o.a.h.fs.permission.AccessControlException. - (Li Lu via wheat9) - - HADOOP-10787 Rename/remove non-HADOOP_*, etc from the shell scripts. - (aw via vvasudev) - - HADOOP-12294. Throw an Exception when fs.permissions.umask-mode is - misconfigured (Chang Li vha wheat9) - - NEW FEATURES - - HADOOP-6590. Add a username check for hadoop sub-commands (John Smith via - aw) - - HADOOP-11353. Add support for .hadooprc (aw) - - HADOOP-9044. add FindClass main class to provide classpath checking - of installations (Steve Loughran via aw) - - HADOOP-11485. Pluggable shell integration (aw) - - HADOOP-11554. Expose HadoopKerberosName as a hadoop subcommand (aw) - - HADOOP-11565. Add --slaves shell option (aw) - - HADOOP-9642. Configuration to resolve environment variables via - ${env.VARIABLE} references (Kengo Seki via aw) - - HADOOP-7947. Validate XMLs if a relevant tool is available, when using - scripts (Kengo Seki via aw) - - HADOOP-10854. unit tests for the shell scripts (aw) - - HADOOP-10949. metrics2 sink plugin for Apache Kafka (Babak Behzad via aw) - - IMPROVEMENTS - - HADOOP-11203. Allow ditscp to accept bandwitdh in fraction MegaBytes - (Raju Bairishetti via amareshwari) - - HADOOP-8017. Configure hadoop-main pom to get rid of M2E plugin execution - not covered (Eric Charles via bobby) - - HADOOP-8015. ChRootFileSystem should extend FilterFileSystem - (Daryn Sharp via bobby) - - HADOOP-7595. Upgrade dependency to Avro 1.5.3. (Alejandro Abdelnur via atm) - - HADOOP-8078. Add capability to turn on security in unit tests. (Jaimin - Jetly via jitendra) - - HADOOP-7757. Test file reference count is at least 3x actual value (Jon - Eagles via bobby) - - HADOOP-8147. test-patch should run tests with -fn to avoid masking test - failures (Robert Evans via tgraves) - - HADOOP-8290. Remove remaining references to hadoop.native.lib (harsh) - - HADOOP-8308. Support cross-project Jenkins builds. (tomwhite) - - HADOOP-8297. Writable javadocs don't carry default constructor (harsh) - - HADOOP-8360. empty-configuration.xml fails xml validation - (Radim Kolar via harsh) - - HADOOP-7659. fs -getmerge isn't guaranteed to work well over non-HDFS - filesystems (harsh) - - HADOOP-8434. Add tests for Configuration setter methods. - (Madhukara Phatak via suresh) - - HADOOP-8523. test-patch.sh doesn't validate patches before building - (Jack Dintruff via jeagles) - - HADOOP-8719. Workaround for kerberos-related log errors upon running any - hadoop command on OSX. (Jianbin Wei via harsh) - - HADOOP-8588. SerializationFactory shouldn't throw a - NullPointerException if the serializations list is empty. - (Sho Shimauchi via harsh) - - HADOOP-7930. Kerberos relogin interval in UserGroupInformation - should be configurable (Robert Kanter via harsh) - - HADOOP-8838. Colorize the test-patch output sent to JIRA (Harsh J via - bobby) - - HADOOP-8840. Fix the test-patch colorizer to cover all sorts of +1 lines. - (Harsh J via bobby) - - HADOOP-8864. Addendum to HADOOP-8840: Add a coloring case for +0 results - too. (harsh) - - HADOOP-8910. Add examples to GlobExpander#expand method. (suresh) - - HADOOP-8920. Add more javadoc to metrics2 related classes. (suresh) - - HADOOP-8776. Provide an option in test-patch that can enable/disable - compiling native code. (Chris Nauroth via suresh) - - HADOOP-6616. Improve documentation for rack awareness. (Adam Faris via - jghoman) - - HADOOP-9075. FileContext#FSLinkResolver should be made static. - (Arpit Agarwal via suresh) - - HADOOP-9277. Improve javadoc for FileContext. (Andrew Wang via suresh) - - HADOOP-9258 Add stricter tests to FileSystemContractTestBase (stevel) - - HADOOP-9486. Promoted Windows and Shell related utils from YARN to Hadoop - Common. (Chris Nauroth via vinodkv) - - HADOOP-8844. Add a plaintext fs -text test-case. - (Akira AJISAKA via harsh) - - HADOOP-9432 Add support for markdown .md files in site documentation - (stevel) - - HADOOP-9186. test-patch.sh should report build failure to JIRA. - (Binglin Chang via Colin Patrick McCabe) - - HADOOP-9833 move slf4j to version 1.7.5 (Kousuke Saruta via stevel) - - HADOOP-10325. Improve jenkins javadoc warnings from test-patch.sh (cmccabe) - - HADOOP-10342. Add a new method to UGI to use a Kerberos login subject to - build a new UGI. (Larry McCay via omalley) - - HADOOP-10485. Remove dead classes in hadoop-streaming. (wheat9) - - HADOOP-11013. CLASSPATH handling should be consolidated, debuggable (aw) - - HADOOP-11041. VersionInfo specifies subversion (Tsuyoshi OZAWA via aw) - - HADOOP-11092. hadoop shell commands should print usage if not given a - a class (aw) - - HADOOP-11231. Remove dead code in ServletUtil. (Li Lu via wheat9) - - HADOOP-11025. hadoop-daemons.sh should just call hdfs directly (Masatake - Iwasaki via aw) - - HADOOP-11150. hadoop command should show the reason on failure by - invalid COMMAND or CLASSNAME (Masatake Iwasaki via aw) - - HADOOP-11208. Replace "daemon" with better name in script subcommands (aw) - - HADOOP-10926. Improve smart-apply-patch.sh to apply binary diffs (cmccabe) - - HADOOP-11081. Document hadoop properties expected to be set by the shell - code in *-env.sh (aw) - - HADOOP-11352 Clean up test-patch.sh to disable "+1 contrib tests" - (Akira AJISAKA via stevel) - - HADOOP-10788. Rewrite kms to use new shell framework (John Smith via aw) - - HADOOP-11058. Missing HADOOP_CONF_DIR generates strange results - (Masatake Iwasaki via aw) - - HADOOP-11460. Deprecate shell vars (John Smith via aw) - - HADOOP-11346. Rewrite sls/rumen to use new shell framework (John Smith - via aw) - - HADOOP-6964. Allow compact property description in xml (Kengo Seki - via aw) - - HADOOP-11575. Daemon log documentation is misleading - (Naganarasimha G R via aw) - - HADOOP-11596. Allow smart-apply-patch.sh to add new files in binary git - patches (raviprak) - - HADOOP-11593. Convert site documentation from apt to markdown (stragglers) - (Masatake Iwasaki via aw) - - HADOOP-11673. Skip using JUnit Assume in TestCodec. (Brahma Reddy Battula - via cdouglas) - - HADOOP-10115. Exclude duplicate jars in hadoop package under different - component's lib (Vinayakumar B via aw) - - HADOOP-11524. hadoop_do_classpath_subcommand throws a shellcheck warning. - (cnauroth) - - HADOOP-11781. fix race conditions and add URL support to - smart-apply-patch.sh (Raymie Stata via aw) - - HADOOP-11850. Typos in hadoop-common java docs. (Surendra Singh Lilhore - via jghoman) - - HADOOP-11852. Disable symlinks in trunk. (Andrew Wang) - - HADOOP-11857. Fix CommandFormat#commandFormat java doc annotation. - (J.Andreina via jghoman) - - HADOOP-11813. releasedocmaker.py should use today's date instead of - unreleased (Darrell Taylor via aw) - - HADOOP-11590. Update sbin commands and documentation to use new --slaves - option (aw) - - HADOOP-10993. Dump java command line to *.out file - (Kengo Seki via vinayakumarb) - - HADOOP-12016. Typo in FileSystem::listStatusIterator - (Arthur Vigil via jghoman) - - HADOOP-11142. Remove hdfs dfs reference from file system shell - documentation (Kengo Seki via aw) - - HADOOP-12149. copy all of test-patch BINDIR prior to re-exec (aw) - - HADOOP-10979. Auto-entries in hadoop_usage (aw) - - HADOOP-12249. pull argument parsing into a function (aw) - - HADOOP-12436. GlobPattern regex library has performance issues with - wildcard characters (Matthew Paduano via aw) - - HADOOP-12385. Include nested stack trace in SaslRpcClient.getServerToken() - (stevel) - - HADOOP-12133. Add schemas to Maven Assembly XMLs (Gábor Lipták via aw) - - HADOOP-12541. make re2j dependency consistent (Matthew Paduano via aw) - - HADOOP-11887. Introduce Intel ISA-L erasure coding library for native - erasure encoding support (Kai Zheng via Colin P. McCabe) - - HADOOP-11828. Implement the Hitchhiker erasure coding algorithm. - (Jack Liuquan via zhz) - - BUG FIXES - - HADOOP-12617. SPNEGO authentication request to non-default realm gets - default realm name inserted in target server principal. (mattf) - - HADOOP-11473. test-patch says "-1 overall" even when all checks are +1 - (Jason Lowe via raviprak) - - HADOOP-9451. Fault single-layer config if node group topology is enabled. - (Junping Du via llu) - - HADOOP-8419. Fixed GzipCode NPE reset for IBM JDK. (Yu Li via eyang) - - HADOOP-8177. MBeans shouldn't try to register when it fails to create - MBeanName. (Devaraj K via umamahesh) - - HADOOP-8018. Hudson auto test for HDFS has started throwing javadoc - (Jon Eagles via bobby) - - HADOOP-7610. Fix for hadoop debian package (Eric Yang via gkesavan) - - HADOOP-7641. Add Apache License to template config files. - (Eric Yang via atm) - - HADOOP-7621. alfredo config should be in a file not readable by users - (Alejandro Abdelnur via atm) - - HADOOP-7669 Fix newly introduced release audit warning. - (Uma Maheswara Rao G via stevel) - - HADOOP-6220. HttpServer wraps InterruptedExceptions by IOExceptions - if interrupted in startup (stevel) - - HADOOP-7703. Improved exception handling of shutting down web server. - (Devaraj K via Eric Yang) - - HADOOP-7704. Reduce number of object created by JMXJsonServlet. - (Devaraj K via Eric Yang) - - HADOOP-7769. TestJMXJsonServlet is failing. (tomwhite) - - HADOOP-7770. ViewFS getFileChecksum throws FileNotFoundException for files in - /tmp and /user. (Ravi Prakash via jitendra) - - HADOOP-8141. Add method to SecurityUtil to init krb5 cipher suites. - (todd) - - HADOOP-7788. Add simple HealthMonitor class to watch an HAService (todd) - - HADOOP-8312. testpatch.sh should provide a simpler way to see which - warnings changed (bobby) - - HADOOP-8339. jenkins complaining about 16 javadoc warnings - (Tom White and Robert Evans via tgraves) - - HADOOP-8354. test-patch findbugs may fail if a dependent module is changed - (Tom White and Robert Evans) - - HADOOP-8375. test-patch should stop immediately once it has found - compilation errors (bobby) - - HADOOP-8395. Text shell command unnecessarily demands that a - SequenceFile's key class be WritableComparable (harsh) - - HADOOP-8413. test-patch.sh gives out the wrong links for - newPatchFindbugsWarnings (Colin Patrick McCabe via bobby) - - HADOOP-6871. When the value of a configuration key is set to its - unresolved form, it causes the IllegalStateException in - Configuration.get() stating that substitution depth is too large. - (Arvind Prabhakar via harsh) - - HADOOP-8548. test-patch.sh shows an incorrect link in Jekins builds - (Kihwal Lee via bobby) - - HADOOP-8110. Fix trash checkpoint collisions (Jason Lowe via daryn) - - HADOOP-8584. test-patch.sh should not immediately exit when no - tests are added or modified. (Colin Patrick McCabe via eli) - - HADOOP-8521. Port StreamInputFormat to new Map Reduce API (madhukara - phatak via bobby) - - HADOOP-8593. Add missed @Override annotations in Metric/Metrics2 package. - (Brandon Li via suresh) - - HADOOP-8623. hadoop jar command should respect HADOOP_OPTS. - (Steven Willis via suresh) - - HADOOP-8684. Deadlock between WritableComparator and WritableComparable. - (Jing Zhao via suresh) - - HADOOP-8786. HttpServer continues to start even if AuthenticationFilter - fails to init (todd) - - HADOOP-8767. Secondary namenode is started on slave nodes instead of - master nodes. (Giovanni Delussu via suresh) - - HADOOP-8821. Fix findbugs warning related to concatenating string in a - for loop in Configuration#dumpDeprecatedKeys(). (suresh) - - HADOOP-7256. Resource leak during failure scenario of closing - of resources. (Ramkrishna S. Vasudevan via harsh) - - HADOOP-8813. Add InterfaceAudience and InterfaceStability annotations - to RPC Server and Client classes. (Brandon Li via suresh) - - HADOOP-8386. hadoop script doesn't work if 'cd' prints to stdout - (default behavior in some bash setups (esp. Ubuntu)) - (Chiristopher Berner and Andy Isaacson via harsh) - - HADOOP-8839. test-patch's -1 on @author tag presence doesn't cause - a -1 to the overall result (harsh) - - HADOOP-8918. test-patch.sh is parsing modified files wrong. - (Raja Aluri via suresh) - - HADOOP-8974. TestDFVariations fails on Windows. (Chris Nauroth via suresh) - - HADOOP-9037. Bug in test-patch.sh and precommit build process (Kihwal Lee - via jlowe) - - HADOOP-9131. Turn off TestLocalFileSystem#testListStatusWithColons on - Windows. (Chris Nauroth via suresh) - - HADOOP-9139. improve killKdc.sh (Ivan A. Veselovsky via bobby) - - HADOOP-9202. test-patch.sh fails during mvn eclipse:eclipse if patch adds - a new module to the build (Chris Nauroth via bobby) - - HADOOP-9249. hadoop-maven-plugins version-info goal causes build failure - when running with Clover. (Chris Nauroth via suresh) - - HADOOP-9393. TestRPC fails with JDK7. (Andrew Wang via atm) - - HADOOP-9394. Port findHangingTest.sh from HBase to Hadoop. (Andrew Wang - via atm) - - HADOOP-9431 TestSecurityUtil#testLocalHostNameForNullOrWild on systems where hostname - contains capital letters (Chris Nauroth via sanjay) - - HADOOP-9261 S3n filesystem can move a directory under itself -and so lose data - (fixed in HADOOP-9258) (stevel) - - HADOOP-9265 S3 blockstore filesystem breaks part of the Filesystem contract - (fixed in HADOOP-9258) (stevel) - - HADOOP-9433 TestLocalFileSystem#testHasFileDescriptor leaks file handle - (Chris Nauroth via sanjay) - - HADOOP-9583. test-patch gives +1 despite build failure when running tests. - (jlowe via kihwal) - - HADOOP-9847. TestGlobPath symlink tests fail to cleanup properly. - (cmccabe via wang) - - HADOOP-9740. Fix FsShell '-text' command to be able to read Avro - files stored in HDFS and other filesystems. (Allan Yan via cutting) - - HADOOP-10044 Improve the javadoc of rpc code (sanjay Radia) - - HADOOP-10625. Trim configuration names when putting/getting them - to properties. (Wangda Tan via xgong) - - HADOOP-10831. UserProvider is not thread safe. (Benoy Antony via umamahesh) - - HADOOP-10834. Typo in CredentialShell usage. (Benoy Antony via umamahesh) - - HADOOP-11002. shell escapes are incompatible with previous releases (aw) - - HADOOP-10996. Stop violence in the *_HOME (aw) - - HADOOP-11033. shell scripts ignore JAVA_HOME on OS X. (aw) - - HADOOP-11052. hadoop_verify_secure_prereq's results aren't checked - in bin/hdfs (aw) - - HADOOP-11055. non-daemon pid files are missing (aw) - - HADOOP-11022. User replaced functions get lost 2-3 levels deep (e.g., - sbin) (aw) - - HADOOP-11284. Fix variable name mismatches in hadoop-functions.sh (Masatake - Iwasaki via aw) - - HADOOP-11298. slaves.sh and stop-all.sh are missing slashes (aw) - - HADOOP-11296. hadoop-daemons.sh throws 'host1: bash: host3: - command not found...' (vinayakumarb) - - HADOOP-11380. Restore Rack Awareness documenation (aw) - - HADOOP-11397. Can't override HADOOP_IDENT_STRING (Kengo Seki via aw) - - HADOOP-10908. Common needs updates for shell rewrite (aw) - - HADOOP-11559. Add links to RackAwareness and InterfaceClassification - to site index (Masatake Iwasaki via aw) - - HADOOP-11580. Remove SingleNodeSetup.md from trunk (aajisaka) - - HADOOP-11583. Fix syntax error in SecureMode.md (Masatake Iwasaki via aw) - - HADOOP-11437. Remove the version and author information from distcp's - README file (Brahma Reddy Battula via aw) - - HADOOP-11585. Fix formatting in Tracing.md (Masatake Iwasaki via aw) - - HADOOP-11625. Minor fixes to command manual & SLA doc (aw) - - HADOOP-11637. bash location hard-coded in shell scripts (aw) - - HADOOP-10774. Update KerberosTestUtils for hadoop-auth tests when using - IBM Java (sangamesh via aw) - - HADOOP-11653. shellprofiles should require .sh extension - (Brahma Reddy Battula via aw) - - HADOOP-11668. hadoop-daemons.sh bw compat broke with --slaves change - (Vinayakumar B via aw) - - HADOOP-11703. git should ignore .DS_Store files on Mac OS X (Abin Shahab - via aw) - - HADOOP-11797. releasedocmaker.py needs to put ASF headers on output (aw) - - HADOOP-11865. Incorrect path mentioned in document for accessing script - files (J.Andreina via aw) - - HADOOP-11775. Fix Javadoc typos in hadoop-openstack module (Yanjun Wang - via aw) - - HADOOP-9891. CLIMiniCluster instructions fail with MiniYarnCluster - ClassNotFoundException (Darrell Taylor via aw) - - HADOOP-11406. xargs -P is not portable (Kengo Seki via aw) - - HADOOP-11983. HADOOP_USER_CLASSPATH_FIRST works the opposite of what it is - supposed to do (Sangjin Lee via aw) - - HADOOP-12022. fix site -Pdocs -Pdist in hadoop-project-dist; cleanout - remaining forrest bits (aw) - - HADOOP-9905. remove dependency of zookeeper for hadoop-client (vinayakumarb) - - HADOOP-11347. RawLocalFileSystem#mkdir and create should honor umask (Varun - Saxena via Colin P. McCabe) - - HADOOP-12107. long running apps may have a huge number of StatisticsData - instances under FileSystem (Sangjin Lee via Ming Ma) - - HADOOP-11762. Enable swift distcp to secure HDFS (Chen He via aw) - - HADOOP-12009. Clarify FileSystem.listStatus() sorting order & fix - FileSystemContractBaseTest:testListStatus. (J.Andreina via jghoman) - - HADOOP-12244. recover broken rebase during precommit (aw) - - HADOOP-11942. Add links to SLGUserGuide to site index. - (Masatake Iwasaki via xyao) - - HADOOP-12364. Deleting pid file after stop is causing the daemons to - keep restarting (Siqi Li via aw) - - HADOOP-12515. Mockito dependency is missing in hadoop-kafka module. - (Kai Zheng via aajisaka) - - HADOOP-12494. fetchdt stores the token based on token kind instead - of token service (HeeSoo Kim via aw) - - HADOOP-12553. [JDK8] Fix javadoc error caused by illegal tag. (aajisaka) - - HADOOP-12638. UnsatisfiedLinkError while checking ISA-L in checknative - command. (Kai Sasaki via Colin P. McCabe) - - HADOOP-12615. Fix NPE in MiniKMS.start(). (Wei-Chiu Chuang via zhz) - - HADOOP-12647. Add /usr/lib default for ISAL library and fix missing .c file - in CMakeLists (Kai Zheng via cmccabe) - - HADOOP-12070. Some of the bin/hadoop subcommands are not available - on Windows. (Kengo Seki via Arpit Agarwal) - - OPTIMIZATIONS - - HADOOP-7761. Improve the performance of raw comparisons. (todd) - - HADOOP-8589. ViewFs tests fail when tests and home dirs are nested (sanjay Radia) - - BREAKDOWN OF HADOOP-11264 SUBTASKS AND RELATED JIRAS (Common part of HDFS-7285) - - HADOOP-11514. Raw Erasure Coder API for concrete encoding and decoding - (Kai Zheng via umamahesh) - - HADOOP-11534. Minor improvements for raw erasure coders - ( Kai Zheng via vinayakumarb ) - - HADOOP-11541. Raw XOR coder - ( Kai Zheng ) - - HADOOP-11542. Raw Reed-Solomon coder in pure Java. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11643. Define EC schema API for ErasureCodec. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11646. Erasure Coder API for encoding and decoding of block group - ( Kai Zheng via vinayakumarb ) - - HADOOP-11705. Make erasure coder configurable. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11706. Refine a little bit erasure coder API. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11707. Add factory to create raw erasure coder. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11647. Reed-Solomon ErasureCoder. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11782 Correct two thrown messages in ECSchema class. Contributed by Xinwei Qin - ( Xinwei Qin via Kai Zheng ) - - HADOOP-11740. Combine erasure encoder and decoder interfaces (Zhe Zhang) - - HADOOP-11805 Better to rename some raw erasure coders. Contributed by Kai Zheng - ( Kai Zheng ) - - HADOOP-11645. Erasure Codec API covering the essential aspects for an erasure code - ( Kai Zheng via vinayakumarb ) - - HADOOP-11818. Minor improvements for erasurecode classes. (Rakesh R via Kai Zheng) - - HADOOP-11841. Remove unused ecschema-def.xml files. (szetszwo) - - HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng via Zhe Zhang) - - HADOOP-11920. Refactor some codes for erasure coders. (Kai Zheng via Zhe Zhang) - - HADOOP-11566. Add tests and fix for erasure coders to recover erased parity - units. (Kai Zheng via Zhe Zhang) - - HADOOP-11938. Enhance ByteBuffer version encode/decode API of raw erasure - coder. (Kai Zheng via Zhe Zhang) - - HADOOP-12013. Generate fixed data to perform erasure coder test. (Kai Zheng) - - HADOOP-12029. Remove chunkSize from ECSchema as its not required for coders - (vinayakumarb) - - HADOOP-11847. Enhance raw coder allowing to read least required inputs in decoding. - (Kai Zheng) - - HADOOP-12011. Allow to dump verbose information to ease debugging in raw erasure coders - (Kai Zheng) - - HADOOP-12065. Using more meaningful keys in EC schema. (Kai Zheng) - - HADOOP-12060. Fix ByteBuffer usage for raw erasure coders. (Kai Zheng via - jing9) - - HADOOP-11921. Enhance tests for erasure coders. (Kai Zheng) - - HADOOP-12327. Initialize output buffers with ZERO bytes in erasure coder. - (Kai Zheng via waltersu4549) - - HADOOP-12047. Indicate preference not to affect input buffers during - coding in erasure coder. (Kai Zheng via waltersu4549) - - HADOOP-12544. Erasure Coding: create dummy raw coder to isolate performance - issues in testing. (Rui Li via zhz) - - HADOOP-12685. Input buffer position after encode/decode not consistent - between different kinds of buffers. (Rui Li via zhz) - - HADOOP-12757. Findbug compilation fails for 'Kafka Library support'. - (aajisaka) - - HADOOP-12041. Implement another Reed-Solomon coder in pure Java. - (Kai Zheng via zhz) - - HADOOP-12808. Rename the RS coder from HDFS-RAID as legacy. - (Rui Li via zhz) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HADOOP-12702. Add an HDFS metrics sink. (Daniel Templeton via kasha) - - IMPROVEMENTS - - HADOOP-12321. Make JvmPauseMonitor an AbstractService. - (Sunil G via Stevel) - - HADOOP-12625. Add a config to disable the /logs endpoints (rkanter) - - HADOOP-12566. Add NullGroupMapping. (Daniel Templeton via kasha) - - HADOOP-12663. Remove Hard-Coded Values From FileSystem.java. - (BELUGA BEHR via stevel) - - HADOOP-12653. Use SO_REUSEADDR to avoid getting "Address already in use" - when using kerberos and attempting to bind to any port on the local IP - address (cmccabe) - - HADOOP-12683. Add number of samples in last interval in snapshot of - MutableStat. (Vikram Srivastava via kasha) - - HADOOP-8887. Use a Maven plugin to build the native code using CMake - (cmccabe) - - HADOOP-12713. Disable spurious checkstyle checks. (wang) - - HADOOP-12662. The build should fail if a -Dbundle option fails (Kai Zheng - via cmccabe) - - HADOOP-12749. Create a threadpoolexecutor that overrides afterExecute to - log uncaught exceptions/errors. (Sidharta Seethana via vvasudev) - - HADOOP-12710. Remove dependency on commons-httpclient for TestHttpServerLogs - (Wei-Chiu Chuang via iwasakims) - - HADOOP-12817. Enable TLS v1.1 and 1.2 (rkanter) - - HADOOP-12841. Update s3-related properties in core-default.xml. (Wei-Chiu Chuang via lei) - - HADOOP-10321. TestCompositeService should cover all enumerations of - adding a service to a parent service. (Ray Chiang via kasha) - - HADOOP-12859. Disable hiding field style checks in class setters. (Kai Zheng via wang) - - BUG FIXES - - HADOOP-12605. Fix intermittent failure of TestIPC.testIpcWithReaderQueuing - (iwasakims) - - HADOOP-12655. TestHttpServer.testBindAddress bind port range is wider - than expected. (Wei-Chiu Chuang via stevel) - - HADOOP-12613. TestFind.processArguments occasionally fails. - (Wei-Chiu Chuang via stevel) - - HADOOP-12597. In kms-site.xml configuration - "hadoop.security.keystore.JavaKeyStoreProvider.password" - should be updated with new name. (Surendra Singh Lilhore via stevel) - - HADOOP-12573. TestRPC.testClientBackOff failing. - (Xiao Chen via stevel) - - HDFS-9517. Fix missing @Test annotation on - TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) - - HADOOP-12712. Fix some cmake plugin and native build warnings (cmccabe) - - HADOOP-12714. Fix hadoop-mapreduce-client-nativetask unit test which fails - because it is not able to open the "glibc bug spill" file. (cmccabe) - - HADOOP-12829. StatisticsDataReferenceCleaner swallows interrupt exceptions - (Gregory Chanan via cmccabe) - -Release 2.8.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - HADOOP-11746. rewrite test-patch.sh (aw) - - HADOOP-12416. Trash messages should be handled by Logger instead of being - delivered on System.out. (Mingliang Liu via aajisaka) - - HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) - - NEW FEATURES - - HADOOP-11226. Add a configuration to set ipc.Client's traffic class with - IPTOS_LOWDELAY|IPTOS_RELIABILITY. (Gopal V via ozawa) - - HADOOP-9477. Add posixGroups support for LDAP groups mapping service. - (Dapeng Sun via Yongjun Zhang) - - HADOOP-11843. Make setting up the build environment easier. - (Niels Basjes via cnauroth) - - HADOOP-11949. Add user-provided plugins to test-patch (Sean Busbey via aw) - - HADOOP-8934. Shell command ls should include sort options (Jonathan Allen - via aw) - - HADOOP-10971. Add -C flag to make `hadoop fs -ls` print filenames only. - (Kengo Seki via aajisaka) - - HADOOP-5732. Add SFTP FileSystem. (Ramtin Boustani and Inigo Goiri via - cdouglas) - - HADOOP-12360. Create StatsD metrics2 sink. (Dave Marion via stevel) - - HADOOP-10300. Allowed deferred sending of call responses. (Daryn Sharp via - yliu) - - HADOOP-12657. Add a option to skip newline on empty files with getMerge -nl. - (Kanaka Kumar Avvaru via aajisaka) - - HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) - - HADOOP-12691. Add CSRF Filter for REST APIs to Hadoop Common. - (Larry McCay via cnauroth) - - HADOOP-12635. Adding Append API support for WASB. (Dushyanth via cnauroth) - - HADOOP-12426. Add Entry point for Kerberos health check - (Steve Loughran via cnauroth) - - HADOOP-12548. Read s3a creds from a Credential Provider. - (Larry McCay via cnauroth) - - IMPROVEMENTS - - HADOOP-12831. LocalFS/FSOutputSummer NPEs in constructor if bytes per checksum - set to 0 (Mingliang Liu via gtcarrera9) - - HADOOP-12458. Retries is typoed to spell Retires in parts of - hadoop-yarn and hadoop-common - (Neelesh Srinivas Salian via harsh) - - HADOOP-12271. Hadoop Jar Error Should Be More Explanatory - (Josh Elser via harsh) - - HADOOP-6842. "hadoop fs -text" does not give a useful text representation - of MapWritable objects (Akira Ajisaka via bobby) - - HADOOP-11719. [Fsshell] Remove bin/hadoop reference from - GenericOptionsParser default help text. - (Brahma Reddy Battula via harsh) - - HADOOP-11692. Improve authentication failure WARN message to avoid user - confusion. (Yongjun Zhang) - - HADOOP-11659. o.a.h.fs.FileSystem.Cache#remove should use a single hash map - lookup. (Brahma Reddy Battula via aajisaka) - - HADOOP-11709. Time.NANOSECONDS_PER_MILLISECOND - use class-level final - constant instead of method variable (Ajith S via ozawa) - - HADOOP-11447. Add a more meaningful toString method to SampleStat and - MutableStat. (kasha) - - HADOOP-11737. mockito's version in hadoop-nfs’ pom.xml shouldn't be - specified. (Kengo Seki via ozawa) - - HADOOP-11741. Add LOG.isDebugEnabled() guard for some LOG.debug(). - (Walter Su via ozawa) - - HADOOP-11660. Add support for hardware crc of HDFS checksums on ARM aarch64 - architecture (Edward Nevill via Colin P. McCabe) - - HADOOP-9805. Refactor RawLocalFileSystem#rename for improved testability. - (Jean-Pierre Matsumoto via cnauroth) - - HADOOP-11717. Support JWT tokens for web single sign on to the Hadoop - servers. (Larry McCay via omalley) - - HADOOP-11814. Reformat hadoop-annotations, o.a.h.classification.tools. - (Li Lu via wheat9) - - HADOOP-7713. dfs -count -q should label output column (Jonathan Allen - via aw) - - HADOOP-11819. HttpServerFunctionalTest#prepareTestWebapp should create web - app directory if it does not exist. (Rohith via vinayakumarb) - - HADOOP-11827. Speed-up distcp buildListing() using threadpool - (Zoran Dimitrijevic via raviprak) - - HADOOP-10597. RPC Server signals backoff to clients when all request - queues are full. (Ming Ma via Arpit Agarwal) - - HADOOP-11357. Print information of the build enviornment in test-patch.sh - (aw) - - HADOOP-11328. ZKFailoverController does not log Exception when doRun raises - errors. (Tianyin Xu via ozawa) - - HADOOP-11120. hadoop fs -rmr gives wrong advice. (Juliet Hougland via wang) - - HADOOP-11911. test-patch should allow configuration of default branch - (Sean Busbey via aw) - - HADOOP-11917. test-patch.sh should work with ${BASEDIR}/patchprocess - setups (aw) - - HADOOP-11925. backport trunk's smart-apply-patch.sh to branch-2 (aw) - - HADOOP-11906. test-patch.sh should use 'file' command for patch - determinism (Sean Busbey via aw) - - HADOOP-11950. Add cli option to test-patch to set the project-under-test - (Sean Busbey via aw) - - HADOOP-11948. test-patch's issue matching regex should be configurable. - (Sean Busbey via aw) - - HADOOP-9723. Improve error message when hadoop archive output path already - exists. (Jean-Baptiste Onofré and Yongjun Zhang via aajisak) - - HADOOP-11713. ViewFileSystem should support snapshot methods. - (Rakesh R via cnauroth) - - HADOOP-11960. Enable Azure-Storage Client Side logging. - (Dushyanth via cnauroth) - - HADOOP-11939. Deprecate DistCpV1 and Logalyzer. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11884. test-patch.sh should pull the real findbugs version - (Kengo Seki via aw) - - HADOOP-11944. add option to test-patch to avoid relocating patch process - directory (Sean Busbey via aw) - - HADOOP-1540. Support file exclusion list in distcp. (Rich Haase via jing9) - - HADOOP-11103. Clean up RemoteException (Sean Busbey via vinayakumarb) - - HADOOP-11995. Make jetty version configurable from the maven command line. - (Sriharsha Devineni via wheat9) - - HADOOP-10366. Add whitespaces between classes for values in - core-default.xml to fit better in browser. - (kanaka kumar avvaru via aajisaka) - - HADOOP-11594. Improve the readability of site index of documentation. - (Masatake Iwasaki via aajisaka) - - HADOOP-12030. test-patch should only report on newly introduced - findbugs warnings. (Sean Busbey via aw) - - HADOOP-11894. Bump the version of Apache HTrace to 3.2.0-incubating - (Masatake Iwasaki via Colin P. McCabe) - - HADOOP-12043. Display warning if defaultFs is not set when running fs - commands. (Lei Xu via wang) - - HADOOP-12037. Fix wrong classname in example configuration of hadoop-auth - documentation. (Masatake Iwasaki via wang) - - HADOOP-12059. S3Credentials should support use of CredentialProvider. - (Sean Busbey via wang) - - HADOOP-12056. Use DirectoryStream in DiskChecker#checkDirs to detect - errors when listing a directory. (Zhihai Xu via wang) - - HADOOP-12055. Deprecate usage of NativeIO#link. (Andrew Wang via cnauroth) - - HADOOP-11971. Move test utilities for tracing from hadoop-hdfs to - hadoop-common. (Masatake Iwasaki via aajisaka) - - HADOOP-11965. determine-flaky-tests needs a summary mode. - (Yufei Gu via Yongjun Zhang) - - HADOOP-11958. MetricsSystemImpl fails to show backtrace when an error - occurs (Jason Lowe via jeagles) - - HADOOP-12158. Improve error message in - TestCryptoStreamsWithOpensslAesCtrCryptoCodec when OpenSSL is not - installed. (wang) - - HADOOP-12124. Add HTrace support for FsShell (cmccabe) - - HADOOP-12171. Shorten overly-long htrace span names for server (cmccabe) - - HADOOP-12045. Enable LocalFileSystem#setTimes to change atime. - (Kazuho Fujii via cnauroth) - - HADOOP-11974. Fix FIONREAD #include on Solaris (Alan Burlison via Colin P. - McCabe) - - HADOOP-12193. Rename Touchz.java to Touch.java. (wang) - - HADOOP-12195. Add annotation to package-info.java file to workaround - MCOMPILER-205. (wang) - - HADOOP-12201. Add tracing to FileSystem#createFileSystem and Globber#glob - (cmccabe) - - HADOOP-12180. Move ResourceCalculatorPlugin from YARN to Common. - (Chris Douglas via kasha) - - HADOOP-12210. Collect network usage on the node (Robert Grandl via cdouglas) - - HADOOP-12211. Collect disk usage on the node (Robert Grandl via cdouglas) - - HADOOP-12153. ByteBufferReadable doesn't declare @InterfaceAudience and - @InterfaceStability. (Brahma Reddy Battula via ozawa) - - HADOOP-11893. Mark org.apache.hadoop.security.token.Token as - @InterfaceAudience.Public. (Brahma Reddy Battula via stevel) - - HADOOP-12081. Fix UserGroupInformation.java to support 64-bit zLinux. - (aajisaka) - - HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. - (vinayakumarb) - - HADOOP-12184. Remove unused Linux-specific constants in NativeIO (Martin - Walsh via Colin P. McCabe) - - HADOOP-12161. Add getStoragePolicy API to the FileSystem interface. - (Brahma Reddy Battula via Arpit Agarwal) - - HADOOP-12189. Improve CallQueueManager#swapQueue to make queue elements - drop nearly impossible. (Zhihai Xu via wang) - - HADOOP-12259. Utility to Dynamic port allocation (brahmareddy via rkanter) - - HADOOP-12170. hadoop-common's JNIFlags.cmake is redundant and can be - removed (Alan Burlison via Colin P. McCabe) - - HADOOP-11807. add a lint mode to releasedocmaker (ramtin via aw) - - HADOOP-12183. Annotate the HTrace span created by FsShell with the - command-line arguments passed by the user (Masatake Iwasaki via Colin P. - McCabe) - - HADOOP-12318. Expose underlying LDAP exceptions in SaslPlainServer. (Mike - Yoder via atm) - - HADOOP-9891. CLIMiniCluster instructions fail with MiniYarnCluster - ClassNotFoundException (Darrell Taylor via aw) - - HADOOP-12295. Improve NetworkTopology#InnerNode#remove logic. (yliu) - - HADOOP-12050. Enable MaxInactiveInterval for hadoop http auth token - (hzlu via benoyantony) - - HADOOP-12325. RPC Metrics : Add the ability track and log slow RPCs. - (Anu Engineer via xyao) - - HADOOP-12368. Mark ViewFileSystemBaseTest and ViewFsBaseTest as abstract. - (wang) - - HADOOP-12367. Move TestFileUtil's test resources to resources folder. - (wang via yliu) - - HADOOP-12369. Point hadoop-project/pom.xml java.security.krb5.conf - within target folder. (wang) - - HADOOP-12358. Add -safely flag to rm to prompt when deleting many files. - (xyao via wang) - - HADOOP-12384. Add "-direct" flag option for fs copy so that user can choose - not to create "._COPYING_" file (J.Andreina via vinayakumarb) - - HADOOP-12324. Better exception reporting in SaslPlainServer. - (Mike Yoder via stevel) - - HADOOP-12404. Disable caching for JarURLConnection to avoid sharing - JarFile with other users when loading resource from URL in Configuration - class. (zxu) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - HADOOP-12446. Undeprecate createNonRecursive() (Ted Yu via kihwal) - - HADOOP-12452. Fix tracing documention reflecting the update to htrace-4 - (Masatake Iwasaki via Colin P. McCabe) - - HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and - writes (Dushyanth via cnauroth) - - HADOOP-11104. org.apache.hadoop.metrics2.lib.MetricsRegistry needs numerical - parameter checking. (Ray Chiang via aajisaka) - - HADOOP-12344. Improve validateSocketPathSecurity0 error message (Casey - Brotherton via Colin P. McCabe) - - HADOOP-10775. Shell operations to fail with meaningful errors on windows if - winutils.exe not found. (stevel) - - HADOOP-12481. JWTRedirectAuthenticationHandler doesn't Retain Original Query - String (Larry McCay via cnauroth) - - HADOOP-12460. Add overwrite option for 'get' shell command - (Jagadesh Kiran N via vinayakumarb) - - HADOOP-10406. TestIPC.testIpcWithReaderQueuing may fail. (Xiao Chen via wang) - - HADOOP-9692. Improving log message when SequenceFile reader throws - EOFException on zero-length file. (Zhe Zhang and Chu Tong via ozawa) - - HADOOP-7266. Deprecate metrics v1. (Akira AJISAKA via ozawa) - - HADOOP-12472. Make GenericTestUtils.assertExceptionContains robust. - (Steve Loughran via jing9) - - HADOOP-12514. Make static fields in GenericTestUtils for - assertExceptionContains() package-private and final. - (Mingliang Liu via stevel) - - HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate - Azure Storage account keys for service integration tests. (cnauroth) - - HADOOP-11685. StorageException complaining " no lease ID" during HBase - distributed log splitting (Duo Xu via cnauroth) - - HADOOP-12040. Adjust inputs order for the decode API in raw erasure coder. - (Kai Zheng via yliu) - - HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor - and Aaron Fabbri via lei) - - HADOOP-12562. Make hadoop dockerfile usable by Yetus. - (Allen Wittenauer via wheat9) - - HADOOP-12568. Update core-default.xml to describe posixGroups support. - (Wei-Chiu Chuang via aajisaka) - - HADOOP-12564. Upgrade JUnit3 TestCase to JUnit 4 in - org.apache.hadoop.io package. (Dustin Cote via ozawa) - - HADOOP-12582. Using BytesWritable's getLength() and getBytes() instead - of get() and getSize(). (Akira AJISAKA via ozawa) - - HADOOP-12575. Add build instruction for docker toolbox instead of - boot2docker(Kai Sasaki via ozawa) - - HADOOP-10035. Cleanup TestFilterFileSystem. (Suresh Srinivas via wheat9) - - HADOOP-10555. Add offset support to MurmurHash. - (Sergey Shelukhin via wheat9) - - HADOOP-10465. Fix use of generics within SortedMapWritable. - (Bertrand Dechoux via wheat9) - - HADOOP-12600. FileContext and AbstractFileSystem should be annotated as a - Stable interface. (cnauroth) - - HADOOP-12618. Fix NPE in TestSequenceFile. (Brahma Reddy Battula via umamahesh) - - HADOOP-12639. Improve JavaDoc for getTrimmedStrings. (BELUGA BEHR via umamahesh) - - HADOOP-12570. HDFS Secure Mode Documentation updates. - (Arpit Agarwal via cnauroth) - - HADOOP-12686. Update FileSystemShell documentation to mention the meaning - of each columns of fs -du. (Daisuke Kobayashi via aajisaka) - - HADOOP-12658. Clear javadoc and check style issues around DomainSocket - (Kai Zheng via umamahesh) - - HADOOP-12604. Exception may be swallowed in KMSClientProvider. - (Yongjun Zhang) - - HADOOP-12696. Add tests for S3Filesystem Contract (Matt Paduano via raviprak) - - HADOOP-12755. Fix typo in defaultFS warning message. (wang) - - HADOOP-12292. Make use of DeleteObjects optional. - (Thomas Demoor via stevel) - - HADOOP-12759. RollingFileSystemSink should eagerly rotate directories. - (Daniel Templeton via wang) - - HADOOP-12752. Improve diagnostics/use of envvar/sysprop credential - propagation (Steve Loughran via cnauroth) - - HADOOP-12764. Increase default value of KMX maxHttpHeaderSize and make it - configurable. (zhz) - - HADOOP-12788. OpensslAesCtrCryptoCodec should log which random number generator is used. - (Wei-Chiu Chuang via umamahesh) - - HADOOP-12699. TestKMS#testKMSProvider intermittently fails during - 'test rollover draining'. (Xiao Chen via wang) - - HADOOP-12776. Remove getaclstatus call for non-acl commands in getfacl. - (Brahma Reddy Battula via vinayakumarb) - - HADOOP-11031. Design Document for Credential Provider API. - (Larry McCay via cnauroth) - - HADOOP-12828. Print user when services are started. - (Wei-Chiu Chuang via Yongjun Zhang) - - HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. - (Masatake Iwasaki via cnauroth) - - HADOOP-12668. Support excluding weak Ciphers in HttpServer2 through - ssl-server.conf. (Vijay Singh via zhz) - - HADOOP-12555. WASB to read credentials from a credential provider. - (Larry McCay via cnauroth) - - HADOOP-12535. Run FileSystem contract tests with hadoop-azure. - (Madhumita Chakraborty via cnauroth) - - HADOOP-12824. Collect network and disk usage on the node running Windows. - (Inigo Goiri via xyao) - - HADOOP-12825. Log slow name resolutions. - (Sidharta Seethana via stevel) - - HADOOP-12622. Improve the loggings in RetryPolicies and RetryInvocationHandler. - (Junping Du via jianhe) - - HADOOP-12793. Write a new group mapping service guide - (Wei-Chiu Chuang via iwasakims) - - OPTIMIZATIONS - - HADOOP-11785. Reduce the number of listStatus operation in distcp - buildListing (Zoran Dimitrijevic via Colin P. McCabe) - - HADOOP-11970. Replace uses of ThreadLocal with JDK7 - ThreadLocalRandom. (Sean Busbey via Colin P. McCabe) - - HADOOP-11772. RPC Invoker relies on static ClientCache which has - synchronized(this) blocks. (wheat9) - - HADOOP-11242. Record the time of calling in tracing span of - IPC server. (Masatake Iwasaki via aajisaka) - - HADOOP-11885. hadoop-dist dist-layout-stitching.sh does not work with dash. - (wang) - - HADOOP-12104. Migrate Hadoop Pipes native build to new CMake framework - (alanburlison via cmccabe) - - HADOOP-12036. Consolidate all of the cmake extensions in one directory - (alanburlison via cmccabe) - - HADOOP-12112. Make hadoop-common-project Native code -Wall-clean - (alanburlison via cmccabe) - - HADOOP-12172. FsShell mkdir -p makes an unnecessary check for the existence - of the parent. (cnauroth) - - HADOOP-12194. Support for incremental generation in the protoc plugin. - (wang) - - HADOOP-11878. FileContext#fixRelativePart should check for not null for a - more informative exception. (Brahma Reddy Battula via kasha) - - HADOOP-11984. Enable parallel JUnit tests in pre-commit. - (Chris Nauroth via vinayakumarb) - - HADOOP-12450. UserGroupInformation should not log at WARN level if no groups - are found. (Elliott Clark via stevel) - - HADOOP-12758. Extend CSRF Filter with UserAgent Checks - (Larry McCay via cnauroth) - - HADOOP-10865. Add a Crc32 chunked verification benchmark for both directly - and non-directly buffer cases. (szetszwo) - - HADOOP-12853. Change WASB documentation regarding page blob support. - (Madhumita Chakraborty via cnauroth) - - BUG FIXES - - HADOOP-12352. Delay in checkpointing Trash can leave trash for 2 intervals - before deleting (Casey Brotherton via harsh) - - HADOOP-11568. Description on usage of classpath in hadoop command is - incomplete. ( Archana T via vinayakumarb ) - - HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of - jclass to GetStaticObjectField. (Hui Zheng via cnauroth) - - HADOOP-11724. DistCp throws NPE when the target directory is root. - (Lei Eddy Xu via Yongjun Zhang) - - HADOOP-11760. Fix typo of javadoc in DistCp. (Brahma Reddy Battula via - ozawa). - - HADOOP-11800. Clean up some test methods in TestCodec.java. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11811. Fix typos in hadoop-project/pom.xml and TestAccessControlList. - (Brahma Reddy Battula via ozawa) - - HADOOP-11704. DelegationTokenAuthenticationFilter must pass ipaddress - instead of hostname to ProxyUsers#authorize (Anubhav Dhoot via asuresh) - - HADOOP-11846 TestCertificateUtil.testCorruptPEM failing on Jenkins JDK8. - (Larry McCay via stevel) - - HADOOP-11859. PseudoAuthenticationHandler fails with httpcomponents v4.4. - (Eugene Koifman via jitendra) - - HADOOP-11848. Incorrect arguments to sizeof in DomainSocket.c - (Malcolm Kavalsky via Colin P. McCabe) - - HADOOP-11861. test-patch.sh rewrite addendum patch. - (Allen Wittenauer via cnauroth) - - HADOOP-11864. JWTRedirectAuthenticationHandler breaks java8 javadocs. - (Larry McCay via stevel) - - HADOOP-11876. Refactor code to make it more readable, minor - maybePrintStats bug (Zoran Dimitrijevic via raviprak) - - HADOOP-11870. [JDK8] AuthenticationFilter, CertificateUtil, - SignerSecretProviders, KeyAuthorizationKeyProvider Javadoc issues (rkanter) - - HADOOP-11881. test-patch.sh javac result is wildly wrong (Kengo Seki via - aw) - - HADOOP-11821. Fix findbugs warnings in hadoop-sls. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11866. increase readability and reliability of checkstyle, - shellcheck, and whitespace reports (aw) - - HADOOP-11889. Make checkstyle runnable from root project - (Gera Shegalov via jeagles) - - HADOOP-11900. Add failIfNoTests=false to hadoop-build-tools pom. (gera) - - HADOOP-11916. TestStringUtils#testLowerAndUpperStrings failed on MAC - due to a JVM bug. (Ming Ma via ozawa) - - HADOOP-11904. test-patch.sh goes into an infinite loop on non-maven - builds (aw) - - HADOOP-11926. test-patch.sh mv does wrong math (aw) - - HADOOP-11912. Extra configuration key used in TraceUtils should respect - prefix (Masatake Iwasaki via Colin P. McCabe) - - HADOOP-10387. Misspelling of threshold in log4j.properties for tests in - hadoop-common-project. (Brahma Reddy Battula via aajisaka) - - HADOOP-11936. Dockerfile references a removed image (aw) - - HADOOP-11922. Misspelling of threshold in log4j.properties for tests - in hadoop-tools (Gabor Liptak via vinayakumarb) - - HADOOP-11877. SnappyDecompressor's Logger class name is wrong - (surendra singh lilhore via vinayakumarb) - - HADOOP-10356. Corrections in winutils/chmod.c. (Rene Nyffenegger via - Arpit Agarwal) - - HADOOP-7165. listLocatedStatus(path, filter) is not redefined in FilterFs - (Hairong Kuang via jlowe) - - HADOOP-9729. The example code of org.apache.hadoop.util.Tool is incorrect - (hellojinjie via jlowe) - - HADOOP-11928. Test-patch check for @author tags incorrectly flags - removal of @author tags (Kengo Seki via aw) - - HADOOP-11951. test-patch should give better info about failures to handle - dev-support updates without resetrepo option (Sean Busbey via aw) - - HADOOP-11947. test-patch should return early from determine-issue when - run in jenkins mode. (Sean Busbey via aw) - - HADOOP-11962. Sasl message with MD5 challenge text shouldn't be LOG out - even in debug level. (Junping Du via wheat9) - - HADOOP-8174. Remove confusing comment in Path#isAbsolute() - (Suresh Srinivas via vinayakumarb) - - HADOOP-11988. Fix typo in the document for hadoop fs -find. - (Kengo Seki via aajisaka) - - HADOOP-10582. Fix the test case for copying to non-existent dir in - TestFsShellCopy. (Kousuke Saruta via aajisaka) - - HADOOP-11581. Multithreaded correctness Warnings - #org.apache.hadoop.fs.shell.Ls (Brahma Reddy Battula via aw) - - HADOOP-11963. Metrics documentation for FSNamesystem misspells - PendingDataNodeMessageCount. (Anu Engineer via cnauroth) - - HADOOP-12000. cannot use --java-home in test-patch (aw) - - HADOOP-12014. hadoop-config.cmd displays a wrong error message. - (Kengo Seki via aajisaka) - - HADOOP-11955. Fix a typo in the cluster setup doc. - (Yanjun Wang via aajisaka) - - HADOOP-11743. maven doesn't clean all the site files - (ramtin via vinayakumarb) - - HADOOP-11927. Fix "undefined reference to dlopen" error when compiling - libhadooppipes (Xianyin Xin via Colin P. McCabe) - - HADOOP-8751. NPE in Token.toString() when Token is constructed using null - identifier. (kanaka kumar avvaru via aajisaka) - - HADOOP-12004. test-patch breaks with reexec in certain situations (Sean - Busbey via aw) - - HADOOP-12035. shellcheck plugin displays a wrong version potentially - (Kengo Seki via aw) - - HDFS-8429. Avoid stuck threads if there is an error in DomainSocketWatcher - that stops the thread. (zhouyingchao via cmccabe) - - HADOOP-11930. test-patch in offline mode should tell maven to be in - offline mode (Sean Busbey via aw) - - HADOOP-11959. WASB should configure client side socket timeout in storage - client blob request options. (Ivan Mitic via cnauroth) - - HADOOP-12042. Users may see TrashPolicy if hdfs dfs -rm is run - (Andreina J via vinayakumarb) - - HADOOP-11991. test-patch.sh isn't re-executed even if smart-apply-patch.sh - is modified. (Kengo Seki via aajisaka) - - HADOOP-12018. smart-apply-patch.sh fails if the patch edits CR+LF files - and is created by 'git diff --no-prefix'. (Kengo Seki via aajisaka) - - HADOOP-12019. update BUILDING.txt to include python for 'mvn site' - in windows (vinayakumarb) - - HADOOP-11994. smart-apply-patch wrongly assumes that git is infallible. - (Kengo Seki via Arpit Agarwal) - - HADOOP-11924. Tolerate JDK-8047340-related exceptions in - Shell#isSetSidAvailable preventing class init. (Tsuyoshi Ozawa via gera) - - HADOOP-12052 IPC client downgrades all exception types to IOE, breaks - callers trying to use them. (Brahma Reddy Battula via stevel) - - HADOOP-12054. RPC client should not retry for InvalidToken exceptions. - (Varun Saxena via Arpit Agarwal) - - HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on - EOF. (Ivan Mitic via cnauroth) - - HADOOP-7817. RawLocalFileSystem.append() should give FSDataOutputStream - with accurate .getPos() (kanaka kumar avvaru via vinayakumarb) - - HADOOP-12074. in Shell.java#runCommand() rethrow InterruptedException as - InterruptedIOException (Lavkesh Lahngir via vinayakumarb) - - HADOOP-12001. Fixed LdapGroupsMapping to include configurable Posix UID and - GID attributes during the search. (Patrick White via vinodkv) - - HADOOP-12095. org.apache.hadoop.fs.shell.TestCount fails. - (Brahma Reddy Battula via aajisaka) - - HADOOP-12076. Incomplete Cache Mechanism in CredentialProvider API. - (Larry McCay via cnauroth) - - HADOOP-12119. hadoop fs -expunge does not work for federated namespace - (J.Andreina via vinayakumarb) - - HADOOP-12089. StorageException complaining " no lease ID" when updating - FolderLastModifiedTime in WASB. (Duo Xu via cnauroth) - - HADOOP-12154. FileSystem#getUsed() returns the file length only from root '/' - (J.Andreina via vinayakumarb) - - HADOOP-10798. globStatus() should always return a sorted list of files - (cmccabe) - - HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil - and fix for HA namespaces (rchiang via rkanter) - - HADOOP-12116. Fix unrecommended syntax usages in hadoop/hdfs/yarn script for - cygwin in branch-2. (Li Lu via cnauroth) - - HADOOP-12164. Fix TestMove and TestFsShellReturnCode failed to get command - name using reflection. (Lei (Eddy) Xu) - - HADOOP-12173. NetworkTopology::add calls toString always. - (Inigo Goiri via cdouglas) - - HADOOP-12185. NetworkTopology is not efficient adding/getting/removing - nodes. (Inigo Goiri via cdouglas) - - HADOOP-12117. Potential NPE from Configuration#loadProperty with - allowNullValueProperties set. (zhihai xu via vinayakumarb) - - HADOOP-12200. TestCryptoStreamsWithOpensslAesCtrCryptoCodec should be - skipped in non-native profile. (Masatake Iwasaki via aajisaka) - - HADOOP-10615. FileInputStream in JenkinsHash#main() is never closed. - (Chen He via ozawa) - - HADOOP-12240. Fix tests requiring native library to be skipped in non-native - profile. (Masatake Iwasaki via ozawa) - - HADOOP-12235 hadoop-openstack junit & mockito dependencies should be - "provided". (Ted Yu via stevel) - - HADOOP-12209 Comparable type should be in FileStatus. - (Yong Zhang via stevel) - - HADOOP-12088. KMSClientProvider uses equalsIgnoreCase("application/json"). - (Brahma Reddy Battula via stevel) - - HADOOP-11797. releasedocmaker.py needs to put ASF headers on output (aw) - - HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal - tags. (Akira AJISAKA via stevel). - - HADOOP-12386. RetryPolicies.RETRY_FOREVER should be able to specify a - retry interval. (Sunil G via wangda) - - HADOOP-8436. NPE In getLocalPathForWrite ( path, conf ) when the - required context item is not configured - (Brahma Reddy Battula via harsh) - - HADOOP-12252. LocalDirAllocator should not throw NPE with empty string - configuration. (Zhihai Xu) - - HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. - (Lei (Eddy) Xu via cnauroth) - - HADOOP-12440. TestRPC#testRPCServerShutdown did not produce the desired - thread states before shutting down. (Xiao Chen via mingma) - - HADOOP-12447. Clean up some htrace integration issues (cmccabe) - - HADOOP-12448. TestTextCommand: use mkdirs rather than mkdir to create test - directory. (Contributed by Colin Patrick McCabe and Chris Nauroth) - - HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). - (Rahul Palamuttam and Kanaka Kumar Avvaru via aajisaka) - - HADOOP-8437. getLocalPathForWrite should throw IOException for invalid - paths. (Brahma Reddy Battula via zxu) - - HADOOP-11098. [JDK8] Max Non Heap Memory default changed between JDK7 - and 8. (ozawa) - - HADOOP-12284. UserGroupInformation doAs can throw misleading exception - (Aaron Dosset via stevel) - - HADOOP-12449. TestDNS and TestNetUtils failing if no network. (stevel) - - HADOOP-11515. Upgrade jsch lib to jsch-0.1.51 to avoid problems running - on java7. (stevel and ozawa) - - HADOOP-12478. Shell.getWinUtilsPath() has been renamed - Shell.getWinutilsPath(). (stevel) - - HADOOP-12475. Replace guava Cache with ConcurrentHashMap for caching - Connection in ipc Client (Walter Su via sjlee) - - HADOOP-12479. ProtocMojo does not log the reason for a protoc compilation - failure. (cnauroth) - - HADOOP-11628. SPNEGO auth does not work with CNAMEs in JDK8. - (Daryn Sharp via stevel). - - HADOOP-10941. Proxy user verification NPEs if remote host is unresolvable. - (Benoy Antony via stevel). - - HADOOP-12483. Maintain wrapped SASL ordering for postponed IPC responses. - (Daryn Sharp via yliu) - - HADOOP-12513. Dockerfile lacks initial 'apt-get update'. - (Akihiro Suda via ozawa) - - HADOOP-12457. [JDK8] Fix a failure of compiling common by javadoc. - (Akira AJISAKA via ozawa) - - HADOOP-12178. NPE during handling of SASL setup if problem with SASL - resolver class. (Steve Loughran via zxu) - - HADOOP-12519. hadoop-azure tests should avoid creating a metrics - configuration file in the module root directory. (cnauroth) - - HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. - (Dushyanth via cnauroth) - - HADOOP-12508. delete fails with exception when lease is held on blob. - (Gaurav Kanade via cnauroth) - - HADOOP-12542. TestDNS fails on Windows after HADOOP-12437. (cnauroth) - - HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics - fails intermittently due to assumption that a lease error will be thrown. - (Gaurav Kanade via cnauroth) - - HADOOP-12560: Fix sprintf warnings in {{DomainSocket.c}} introduced by - HADOOP-12344 (Mingliang Liu via Colin P. McCabe) - - HADOOP-12567: NPE in SaslRpcServer (Sergey Shelukhin via cnauroth) - - HADOOP-8419. Fixed GzipCode NPE reset for IBM JDK. (Yu Li via eyang) - - HADOOP-11149. TestZKFailoverController times out. (Steve Loughran - via ozawa) - - HADOOP-9822. Create constant MAX_CAPACITY in RetryCache rather than - hard-coding 16 in RetryCache constructor. (Tsuyoshi Ozawa via wheat9) - - HADOOP-11954. Solaris does not support RLIMIT_MEMLOCK as in Linux - (Alan Burlison via aw) - - HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. - (Xin Wang via aajisaka) - - HADOOP-12468. Partial group resolution failure should not result in user - lockout. (Wei-Chiu Chuang via Yongjun Zhang) - - HADOOP-12656. MiniKdc throws "address in use" BindException. - (Wei-Chiu Chuang via Arpit Agarwal) - - HADOOP-12051. ProtobufRpcEngine.invoke() should use Exception.toString() - over getMessage() in logging/span events. (Varun Saxena via stevel) - - HADOOP-12017. Hadoop archives command should use configurable replication - factor when closing (Bibin A Chundatt via vinayakumarb) - - HADOOP-12239. StorageException complaining " no lease ID" when updating - FolderLastModifiedTime in WASB. (Duo Xu via cnauroth) - - HADOOP-12245. References to misspelled REMAINING_QUATA in - FileSystemShell.md. (Gabor Liptak via aajisaka) - - HADOOP-12175. FsShell must load SpanReceierHost to support tracing - (Masatake Iwasaki via Colin P. McCabe) - - HADOOP-10945. 4-digit octal umask permissions throws a parse error (Chang - Li via jlowe) - - HADOOP-7824. NativeIO.java flags and identifiers must be set correctly for - each platform, not hardcoded to their Linux values (Martin Walsh via Colin - P. McCabe) - - HADOOP-12268. AbstractContractAppendTest#testRenameFileBeingAppended - misses rename operation. (Zhihai Xu) - - HADOOP-12274. Remove direct download link from BULIDING.txt. - (Caleb Severn via aajisaka) - - HADOOP-12302. Fix native compilation on Windows after HADOOP-7824 - (Vinayakumar B via Colin P. McCabe) - - HADOOP-12258. Need translate java.nio.file.NoSuchFileException to - FileNotFoundException to avoid regression. (Zhihai Xu via cnauroth) - - HADOOP-12322. typos in rpcmetrics.java. (Anu Engineer via - Arpit Agarwal) - - HADOOP-12317. Applications fail on NM restart on some linux distro - because NM container recovery declares AM container as LOST - (adhoot via rkanter) - - HADOOP-12347. Fix mismatch parameter name in javadocs of - AuthToken#setMaxInactives (xyao) - - HADOOP-12362. Set hadoop.tmp.dir and hadoop.log.dir in pom. - (Charlie Helin via wang) - - HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter - error message. (Wei Yan via ozawa) - - HADOOP-12388. Fix components' version information in the web page - 'About the Cluster'. (Jun Gong via zxu) - - HADOOP-12407. Test failing: hadoop.ipc.TestSaslRPC. (stevel) - - HADOOP-12417. TestWebDelegationToken failing with port in use. - (Mingliang Liu via wheat9) - - HADOOP-12438. Reset RawLocalFileSystem.useDeprecatedFileStatus in - TestLocalFileSystem. (Chris Nauroth via wheat9) - - HADOOP-12437. Allow SecurityUtil to lookup alternate hostnames. - (Arpit Agarwal) - - HADOOP-12442. Display help if the command option to 'hdfs dfs' is not valid - (nijel via vinayakumarb) - - HADOOP-12441. Fixed shell-kill command behaviour to work correctly on some - Linux distributions after HADOOP-12317. (Wangda Tan via vinodkv) - - HADOOP-12474. MiniKMS should use random ports for Jetty server by default. - (Mingliang Liu via wheat9) - - HADOOP-12418. TestRPC.testRPCInterruptedSimple fails intermittently. - (kihwal) - - HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass - Azure Storage Throttling after retries. (Gaurav Kanade via cnauroth) - - HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race - Condition Scenarios. (Gaurav Kanade via cnauroth) - - HADOOP-11901. BytesWritable fails to support 2G chunks due to integer - overflow. (Reynold Xin via wheat9) - - HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory. - (Vijay Singh via wheat9) - - HADOOP-12467. Respect user-defined JAVA_LIBRARY_PATH in Windows Hadoop - scripts. (Radhey Shah via wheat9) - - HADOOP-12181. Fix intermittent test failure of TestZKSignerSecretProvider. - (Masatake Iwasaki via wheat9) - - HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. - (Varun Saxena via wheat9) - - HADOOP-10068. Improve log4j regex in testFindContainingJar. - (Robert Rati via wheat9) - - HADOOP-11677. Add cookie flags for logs and static contexts. - (nijel via wheat9) - - HADOOP-12313. NPE in JvmPauseMonitor when calling stop() before start(). - (Gabor Liptak via wheat9) - - HADOOP-12609. Fix intermittent failure of TestDecayRpcScheduler. - (Masatake Iwasaki via Arpit Agarwal) - - HADOOP-10729. Add tests for PB RPC in case version mismatch of client and - server. (Junping Du via wheat9) - - HADOOP-12636. Prevent ServiceLoader failure init for unused FileSystems. - (Inigo Goiri via cdouglas) - - HADOOP-12681. start-build-env.sh fails in branch-2. - (Kengo Seki via aajisaka) - - HADOOP-12559. KMS connection failures should trigger TGT renewal. - (Zhe Zhang via xyao) - - HADOOP-12682. Fix TestKMS#testKMSRestart* failure. - (Wei-Chiu Chuang via xyao) - - HADOOP-12689. S3 filesystem operations stopped working correctly - (Matt Paduano via raviprak) - - HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to - address case of potential data loss due to partial copy - (Gaurav Kanade via cnauroth) - - HADOOP-12675. Fix description about retention period in usage of expunge - command. (Masatake Iwasaki via stevel) - - HADOOP-12678. Handle empty rename pending metadata file during atomic rename - in redo path. (Madhumita Chakraborty via cnauroth) - - HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API - (Dushyanth via cnauroth) - - HADOOP-12584. Disable browsing the static directory in HttpServer2. - (Robert Kanter via aajisaka) - - HADOOP-12603. TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir - occasionally fail. (Wei-Chiu Chuang via cnauroth) - - HADOOP-12590. TestCompressorDecompressor failing without stack traces - (John Zhuge via stevel) - - HADOOP-12700. Remove unused import in TestCompressorDecompressor.java. - (John Zhuge via aajisaka) - - HADOOP-12356. Fix computing CPU usage statistics on Windows. - (Inigo Goiri via wangda) - - HADOOP-12423. Handle failure of registering shutdownhook by - ShutdownHookManager in static block (Abhishek Agarwal via vinayakumarb) - - HADOOP-7161. Remove unnecessary oro package from dependency management - section. (Sean Busbey via aajisaka) - - HADOOP-12659. Incorrect usage of config parameters in token manager of - KMS. (Mingliang Liu via xyao) - - HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly - documented as required. (Kengo Seki via aajisaka) - - HADOOP-12731. Remove useless boxing/unboxing code. - (Kousuke Saruta via aajisaka) - - HADOOP-12718. Incorrect error message by fs -put local dir without - permission. (John Zhuge via Yongjun Zhang) - - HADOOP-12735. core-default.xml misspells - hadoop.workaround.non.threadsafe.getpwuid (Ray Chiang via cmccabe) - - HADOOP-12771. Fix typo in JvmPauseMonitor#getNumGcWarnThreadholdExceeded. - (Xiaobing Zhou via xyao) - - HADOOP-12766. The default value of - "hadoop.workaround.non.threadsafe.getpwuid" is different between - core-default.xml and NativeIO.java (Akira AJISAKA via cmccabe) - - HADOOP-12795. KMS does not log detailed stack trace for unexpected errors. - (cnauroth) - - HADOOP-12780. During atomic rename handle crash when one directory has been - renamed but not file under it. (Madhumita Chakraborty via cnauroth) - - HADOOP-12878. KMS SPNEGO sequence does not work with WEBHDFS. (xyao) - - HADOOP-12716. KerberosAuthenticator#doSpnegoSequence use incorrect class to - determine isKeyTab in JDK8. (Xiaoyu Yao via cnauroth) - - HADOOP-12849. TestSymlinkLocalFSFileSystem fails intermittently. - (Mingliang Liu via cnauroth) - - HADOOP-12846. Credential Provider Recursive Dependencies. - (Larry McCay via cnauroth) - - HADOOP-12813. Migrate TestRPC and related codes to rebase on - ProtobufRpcEngine. (Kai Zheng via wheat9) - - HADOOP-12851. S3AFileSystem Uptake of - ProviderUtils.excludeIncompatibleCredentialProviders. - (Larry McCay via cnauroth) - - HADOOP-12843. Fix findbugs warnings in hadoop-common (branch-2). (aajisaka) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-12413. AccessControlList should avoid calling getGroupNames in - isUserInList with empty groups. (Zhihai Xu via cnauroth) - - HADOOP-12794. Support additional compression levels for GzipCodec. - (Ravi Mutyala via junping_du) - - OPTIMIZATIONS - - HADOOP-12810. FileSystem#listLocatedStatus causes unnecessary RPC calls - (Ryan Blue via vinayakumarb) - - BUG FIXES - - HADOOP-12296. when setnetgrent returns 0 in linux, exception should be - thrown (Chang Li via jlowe) - - HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, - ImpersonationProvider, DefaultImpersonationProvider, and DistCp. - (Arshad Mohammad via aajisaka) - - HADOOP-12374. Updated expunge command description. - (WeiWei Yang via eyang) - - HADOOP-12348. MetricsSystemImpl creates MetricsSourceAdapter with wrong - time unit parameter. (zxu via rkanter) - - HADOOP-12482. Race condition in JMX cache update. (Tony Wu via lei) - - HADOOP-12565. Replace DSA with RSA for SSH key type in SingleCluster.md. - (Mingliang Liu via aajisaka) - - HADOOP-12602. TestMetricsSystemImpl#testQSize occasionally fails. - (Masatake Iwasaki via aajisaka) - - HADOOP-12588. Fix intermittent test failure of TestGangliaMetrics. - (Masatake Iwasaki via aajisaka) - - HADOOP-12107. long running apps may have a huge number of StatisticsData - instances under FileSystem (Sangjin Lee via Ming Ma) - - HADOOP-12706. TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp - times out occasionally (Sangjin Lee and Colin Patrick McCabe via jlowe) - - HADOOP-12761. incremental maven build is not really incremental (sjlee) - - HADOOP-12772. NetworkTopologyWithNodeGroup.getNodeGroup() can loop - infinitely for invalid 'loc' values (Kuhu Shukla via kihwal) - - HADOOP-12792. TestUserGroupInformation#testGetServerSideGroups fails in - chroot (Eric Badger via jlowe) - - HADOOP-8818. Use equals instead == in MD5MD5CRC32FileChecksum - and TFileDumper. (Brandon Li via suresh) - - HADOOP-9121. InodeTree.java has redundant check for vName while - throwing exception. (Arup Malakar via suresh) - - HADOOP-12870. Fix typo admininistration in CommandsManual.md. - (John Zhuge via aajisaka) - - HADOOP-12871. Fix dead link to NativeLibraries.html in CommandsManual.md. - (Brahma Reddy Battula via aajisaka) - - HADOOP-12872. Fix formatting in ServiceLevelAuth.md. - (Brahma Reddy Battula via aajisaka) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-12232. Upgrade Tomcat dependency to 6.0.44. (cnauroth) - - HADOOP-5323. Trash documentation should describe its directory structure and - configurations. (Weiwei Yang via ozawa) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-12186. ActiveStandbyElector shouldn't call monitorLockNodeAsync - multiple times (zhihai xu via vinayakumarb) - - HADOOP-12191. Bzip2Factory is not thread safe. (Brahma Reddy Battula - via ozawa) - - HDFS-8767. RawLocalFileSystem.listStatus() returns null for UNIX pipefile. - (kanaka kumar avvaru via wheat9) - - HADOOP-12304. Applications using FileContext fail with the default file - system configured to be wasb/s3/etc. (cnauroth) - - HADOOP-12061. Incorrect command in single cluster setup document. - (Kengo Seki via aajisaka) - - HADOOP-12359. hadoop fs -getmerge doc is wrong. - (Jagadesh Kiran N via aajisaka) - - HADOOP-10365. BufferedOutputStream in FileUtil#unpackEntries() should be - closed in finally block. (Kiran Kumar M R and Sanghyun Yun via ozawa) - - HADOOP-12213. Interrupted exception can occur when Client#stop is called. - (Kuhu Shukla via ozawa) - - HADOOP-12465. Incorrect javadoc in WritableUtils.java. - (Jagadesh Kiran N via aajisaka) - - HADOOP-12464. Interrupted client may try to fail-over and retry (kihwal) - - HADOOP-12415. Fixed pom files to correctly include compile-time dependency on - netty. (Tom Zeng via cos) - - HADOOP-12786. "hadoop key" command usage is not documented. - (Xiao Chen via aajisaka) - -Release 2.7.1 - 2015-07-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-12103. Small refactoring of DelegationTokenAuthenticationFilter to - allow code sharing. (Yongjun Zhang) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-11868. Invalid user logins trigger large backtraces in server log - (Chang Li via jlowe) - - HADOOP-11872. "hadoop dfs" command prints message about using "yarn jar" on - Windows(branch-2 only) (Varun Vasudev via cnauroth) - - HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) - - HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when - native code is not loaded. (Zhijie Shen via ozawa) - - HADOOP-11663. Remove description about Java 6 from docs. - (Masatake Iwasaki via aajisaka) - - HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed - through hadoop-daemon.sh. (cnauroth) - - HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get - created with ACLs. (Gregory Chanan via asuresh) - - HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. - (Kazuho Fujii via aajisaka) - - HADOOP-12078. The default retry policy does not handle RetriableException - correctly. (Arpit Agarwal) - - HADOOP-12100. ImmutableFsPermission should not override applyUmask since - that method doesn't modify the FsPermission (Bibin A Chundatt via Colin P. - McCabe) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - HADOOP-10530 Make hadoop build on Java7+ only (stevel) - - NEW FEATURES - - HADOOP-10987. Provide an iterator-based listing API for FileSystem (kihwal) - - HADOOP-7984. Add hadoop --loglevel option to change log level. - (Akira AJISAKA via cnauroth) - - HADOOP-9629. Support Windows Azure Storage - Blob as a file system in Hadoop. - (Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, - Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, - Alexander Stojanovic, Brian Swan, and Min Wei via cnauroth) - - HADOOP-10728. Metrics system for Windows Azure Storage Filesystem. - (Dexter Bradshaw, Mostafa Elhemali, Xi Fang, Johannes Klein, David Lao, - Mike Liddell, Chuan Liu, Lengning Liu, Ivan Mitic, Michael Rys, - Alexander Stojanovich, Brian Swan, and Min Wei via cnauroth) - - HADOOP-8989. hadoop fs -find feature (Jonathan Allen via aw) - - HADOOP-11490. Expose truncate API via FileSystem and shell command. - (Milan Desai via shv) - - HADOOP-11045. Introducing a tool to detect flaky tests of hadoop jenkins testing - job. (Yongjun Zhang and Todd Lipcon via ozawa) - - HADOOP-11510. Expose truncate API via FileContext. (yliu) - - HADOOP-11569. Provide Merge API for MapFile to merge multiple similar MapFiles - to one MapFile. (Vinayakumar B via ozawa) - - IMPROVEMENTS - - HADOOP-11483. HardLink.java should use the jdk7 createLink method (aajisaka) - - HADOOP-11156. DelegateToFileSystem should implement - getFsStatus(final Path f). (Zhihai Xu via wang) - - HADOOP-11172. Improve error message in Shell#runCommand on OutOfMemoryError. - (Yongjun Zhang via wang) - - HADOOP-10748. HttpServer2 should not load JspServlet. (wheat9) - - HADOOP-10847. Remove the usage of sun.security.x509.* in testing code. - (Pascal Oliva via wheat9) - - HADOOP-10563. Remove the dependency of jsp in trunk. (wheat9) - - HADOOP-11291. Log the cause of SASL connection failures. - (Stephen Chu via cnauroth) - - HADOOP-11173. Improve error messages for some KeyShell commands. (wang) - - HADOOP-11257: Update "hadoop jar" documentation to warn against using it - for launching yarn jars (iwasakims via cmccabe) - - HADOOP-11341. KMS support for whitelist key ACLs. (Arun Suresh via wang) - - HADOOP-11301. [optionally] update jmx cache to drop old metrics - (Maysam Yabandeh via stack) - - HADOOP-11313. Adding a document about NativeLibraryChecker. - (Tsuyoshi OZAWA via cnauroth) - - HADOOP-11287. Simplify UGI#reloginFromKeytab for Java 7+. - (Li Lu via wheat9) - - HADOOP-10476) Bumping the findbugs version to 3.0.0. (wheat9) - - HADOOP-11410. Make the rpath of libhadoop.so configurable (cmccabe) - - HADOOP-11416. Move ChunkedArrayList into hadoop-common (cmccabe) - - HADOOP-10840. Fix OutOfMemoryError caused by metrics system in Azure File - System. (Shanyu Zhao via cnauroth) - - HADOOP-11248. Add hadoop configuration to disable Azure Filesystem metrics - collection. (Shanyu Zhao via cnauroth) - - HADOOP-11421. Add IOUtils#listDirectory (cmccabe) - - HADOOP-11427. ChunkedArrayList: fix removal via iterator and implement get - (cmccabe) - - HADOOP-11430. Add GenericTestUtils#disableLog, GenericTestUtils#setLogLevel - (cmccabe) - - HADOOP-11422. Check CryptoCodec is AES-CTR for Crypto input/output stream - (Yi Liu via Colin P. McCabe) - - HADOOP-11213. Typos in html pages: SecureMode and EncryptedShuffle. - (Wei Yan via kasha) - - HADOOP-11395. Add site documentation for Azure Storage FileSystem - integration. (Chris Nauroth via Arpit Agarwal) - - HDFS-7555. Remove the support of unmanaged connectors in HttpServer2. - (wheat9) - - HADOOP-11399. Java Configuration file and .xml files should be - automatically cross-compared (rchiang via rkanter) - - HADOOP-11455. KMS and Credential CLI should request confirmation for - deletion by default. (Charles Lamb via yliu) - - HADOOP-11390 Metrics 2 ganglia provider to include hostname in - unresolved address problems. (Varun Saxena via stevel) - - HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch - (ozawa) - - HADOOP-11464. Reinstate support for launching Hadoop processes on Windows - using Cygwin. (cnauroth) - - HADOOP-9992. Modify the NN loadGenerator to optionally run as a MapReduce job - (Akshay Radia via brandonli) - - HADOOP-11465. Fix findbugs warnings in hadoop-gridmix. (Varun Saxena via - Arpit Agarwal) - - HADOOP-11481. ClassCastException while using a key created by keytool to - create encryption zone. (Charles Lamb via Colin P. Mccabe) - - HADOOP-8757. Metrics should disallow names with invalid characters - (rchiang via rkanter) - - HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel) - - HADOOP-11171 Enable using a proxy server to connect to S3a. - (Thomas Demoor via stevel) - - HADOOP-11489 Dropping dependency on io.netty from hadoop-nfs' pom.xml - (Ted Yu via ozawa) - - HADOOP-11419 Improve hadoop-maven-plugins. (Herve Boutemy via stevel) - - HADOOP-11450. Cleanup DistCpV1 not to use deprecated methods and fix - javadocs. (Varun Saxena via ozawa) - - HADOOP-4297. Enable Java assertions when running tests. - (Tsz Wo Nicholas Sze via wheat9) - - HADOOP-10626. Limit Returning Attributes for LDAP search. (Jason Hubbard - via atm) - - HADOOP-11317. Increment SLF4J version to 1.7.10. (Tim Robertson via ozawa) - - HADOOP-10525. Remove DRFA.MaxBackupIndex config from log4j.properties - (aajisaka) - - HADOOP-10574. Bump the maven plugin versions too -moving the numbers into - properties. (aajisaka) - - HADOOP-11441. Hadoop-azure: Change few methods scope to public. - (Shashank Khandelwal via cnauroth) - - HADOOP-9137. Support connection limiting in IPC server (kihwal) - - HADOOP-11498. Bump the version of HTrace to 3.1.0-incubating (Masatake - Iwasaki via Colin P. McCabe) - - HADOOP-11442. hadoop-azure: Create test jar. - (Shashank Khandelwal via cnauroth) - - HADOOP-11544. Remove unused configuration keys for tracing. (Masatake - Iwasaki via aajisaka) - - HADOOP-11492. Bump up curator version to 2.7.1. (Arun Suresh and - Karthik Kambatla via kasha) - - HADOOP-11463 Replace method-local TransferManager object with - S3AFileSystem#transfers. (Ted Yu via stevel) - - HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. - (Thomas Demoor via stevel) - - HADOOP-11579. Documentation for truncate. (shv) - - HADOOP-10140. Specification of HADOOP_CONF_DIR via the environment in - hadoop_config.cmd. (Kiran Kumar M R via cnauroth) - - HADOOP-11543. Improve help message for hadoop/yarn command. (Brahma - Reddy Battula via ozawa). - - HADOOP-11586. Update use of Iterator to Iterable in - AbstractMetricsContext.java. (Ray Chiang via aajisaka) - - HADOOP-9869. Configuration.getSocketAddr()/getEnum() should use - getTrimmed(). (Tsuyoshi Ozawa via aajisaka) - - HADOOP-11589. NetUtils.createSocketAddr should trim the input URI. - (Rakesh R via ozawa) - - HADOOP-11600. Fix up source codes to be compiled with Guava 17.0. (ozawa) - - HADOOP-11522. Update S3A Documentation. (Thomas Demoor via stevel) - - HADOOP-11521. Make connection timeout configurable in s3a. - (Thomas Demoor via stevel) - - HADOOP-11440. Use "test.build.data" instead of "build.test.dir" for testing - in ClientBaseWithFixes. (Kengo Seki via aajisaka) - - HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel) - - HADOOP-11495. Convert site documentation from apt to markdown - (Masatake Iwasaki via aw) - - HADOOP-11632. Cleanup Find.java to remove SupressWarnings annotations. - (Akira Ajisaka via ozawa) - - HADOOP-10976. moving the source code of hadoop-tools docs to the - directory under hadoop-tools (Masatake Iwasaki via aw) - - HADOOP-11658. Externalize io.compression.codecs property. - (Kai Zheng via aajisaka) - - HADOOP-11648. Set DomainSocketWatcher thread name explicitly. - (Liang Xie via ozawa) - - HADOOP-11642. Upgrade azure sdk version from 0.6.0 to 2.0.0. - (Shashank Khandelwal and Ivan Mitic via cnauroth) - - HADOOP-11714. Add more trace log4j messages to SpanReceiverHost (cmccabe) - - HADOOP-8059. Add javadoc to InterfaceAudience and InterfaceStability. - (Brandon Li via suresh) - - HADOOP-9329. document native build dependencies in BUILDING.txt (Vijay Bhat - via Colin P. McCabe) - - HADOOP-10670. Allow AuthenticationFilters to load secret from signature - secret files. (Kai Zheng via wheat9) - - HADOOP-11801. Update BUILDING.txt for Ubuntu. (Gabor Liptak via - Arpit Agarwal) - - HADOOP-9737. JarFinder#getJar should delete the jar file upon destruction - of the JVM (jbonofre via rkanter) - - OPTIMIZATIONS - - HADOOP-11323. WritableComparator#compare keeps reference to byte array. - (Wilfred Spiegelenburg via wang) - - HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw, - Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth) - - HADOOP-11188. hadoop-azure: automatically expand page blobs when they become - full. (Eric Hanson via cnauroth) - - HADOOP-11620. Add support for load balancing across a group of KMS for HA. - (Arun Suresh via wang) - - HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel) - - BUG FIXES - - HADOOP-11512. Use getTrimmedStrings when reading serialization keys - (Ryan P via harsh) - - HADOOP-11488. Difference in default connection timeout for S3A FS - (Daisuke Kobayashi via harsh) - - HADOOP-11256. Some site docs have inconsistent appearance (Masatake - Iwasaki via aw) - - HADOOP-11318. Update the document for hadoop fs -stat (aajisaka) - - HADOOP 11400. GraphiteSink does not reconnect to Graphite after 'broken pipe' - (Kamil Gorlo via raviprak) - - HADOOP-11236. NFS: Fix javadoc warning in RpcProgram.java (Abhiraj Butala via harsh) - - HADOOP-11166. Remove ulimit from test-patch.sh. (wang) - - HDFS-7227. Fix findbugs warning about NP_DEREFERENCE_OF_READLINE_VALUE in - SpanReceiverHost (cmccabe) - - HADOOP-11186. documentation should talk about - hadoop.htrace.spanreceiver.classes, not hadoop.trace.spanreceiver.classes (cmccabe) - - HADOOP-11165. TestUTF8 fails when run against java 8. - (Stephen Chu via cnauroth) - - HADOOP-10717. HttpServer2 should load jsp DTD from local jars instead of - going remote. (Dapeng Sun via wheat9) - - HADOOP-11268. Update BUILDING.txt to remove the workaround for tools.jar. - (Li Lu via wheat9) - - HADOOP-11230. Add missing dependency of bouncycastle for kms, httpfs, hdfs, MR - and YARN. (Robert Kanter via wheat9) - - HADOOP-11269. Add java 8 profile for hadoop-annotations. (Li Lu via wheat9) - - HADOOP-11271. Use Time.monotonicNow() in Shell.java instead of Time.now() - (vinayakumarb) - - HADOOP-11266. Remove no longer supported activation properties for packaging - from pom. (Masatake Iwasaki via wheat9) - - HADOOP-11267. TestSecurityUtil fails when run with JDK8 because of empty - principal names. (Stephen Chu via wheat9) - - HADOOP-10714. AmazonS3Client.deleteObjects() need to be limited to 1000 - entries per call. (Juan Yu via atm) - - HADOOP-11272. Allow ZKSignerSecretProvider and - ZKDelegationTokenSecretManager to use the same curator client. (Arun Suresh via atm) - - HADOOP-11187 NameNode - KMS communication fails after a long period of - inactivity. (Arun Suresh via atm) - - HADOOP-11289. Fix typo in RpcUtil log message. (Charles Lamb via wheat9) - - HADOOP-11294. Nfs3FileAttributes should not change the values of rdev, - nlink and size in the constructor. (Brandon Li via wheat9) - - HADOOP-11157. ZKDelegationTokenSecretManager never shuts down - listenerThreadPool. (Arun Suresh via atm) - - HADOOP-11311. Restrict uppercase key names from being created with JCEKS. - (wang) - - HADOOP-11309. System class pattern package.Foo should match - package.Foo$Bar, too (Gera Shegalov via jlowe) - - HADOOP-11312. Fix unit tests to not use uppercase key names. (wang) - - HADOOP-11201. Hadoop Archives should support globs resolving to files. - (Gera Shegalov via cnauroth) - - HADOOP-11322. key based ACL check in KMS always check KeyOpType.MANAGEMENT - even actual KeyOpType is not MANAGEMENT. (Dian Fu via yliu) - - HADOOP-11300. KMS startup scripts must not display the keystore / - truststore passwords. (Arun Suresh via wang) - - HADOOP-11337. KeyAuthorizationKeyProvider access checks need to be done - atomically. (Dian Fu via wang) - - HADOOP-11344. KMS kms-config.sh sets a default value for the keystore - password even in non-ssl setup. (Arun Suresh via wang) - - HADOOP-11342. KMS key ACL should ignore ALL operation for default key ACL - and whitelist key ACL. (Dian Fu via wang) - - HADOOP-11332. KerberosAuthenticator#doSpnegoSequence should check if - kerberos TGT is available in the subject. (Dian Fu via atm) - - HADOOP-11348. Remove unused variable from CMake error message for finding - openssl (Dian Fu via Colin P. McCabe) - - HADOOP-11355. When accessing data in HDFS and the key has been deleted, - a Null Pointer Exception is shown. (Arun Suresh via wang) - - HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. - (Ted Yu via jing9) - - HADOOP-11329. Add JAVA_LIBRARY_PATH to KMS startup options. (Arun Suresh via wang) - - HADOOP-11363 Hadoop maven surefire-plugin uses must set heap size. (stevel) - - HADOOP-10134 [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc - comments. (apurtell via stevel) - - HADOOP-11367. Fix warnings from findbugs 3.0 in hadoop-streaming. (Li Lu via wheat9) - - HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, - non-core directories. (Li Lu via wheat9) - - HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. - (Li Lu via wheat9) - - HADOOP-11273. TestMiniKdc failure: login options not compatible with IBM - JDK. (Gao Zhong Liang via wheat9) - - HADOOP-11379. Fix new findbugs warnings in hadoop-auth*. (Li Lu via wheat9) - - HADOOP-11378. Fix new findbugs warnings in hadoop-kms. (Li Lu via wheat9) - - HADOOP-11349. RawLocalFileSystem leaks file descriptor while creating a - file if creat succeeds but chmod fails. (Varun Saxena via Colin P. McCabe) - - HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, - hadoop-azure, and hadoop-openstack. (Li Lu via wheat9) - - HADOOP-10482. Fix various findbugs warnings in hadoop-common. (wheat9) - - HADOOP-11388. Remove deprecated o.a.h.metrics.file.FileContext. - (Li Lu via wheat9) - - HADOOP-11386. Replace \n by %n in format hadoop-common format strings. - (Li Lu via wheat9) - - HADOOP-11211. mapreduce.job.classloader.system.classes semantics should be - be order-independent. (Yitong Zhou via gera) - - HADOOP-11389. Clean up byte to string encoding issues in hadoop-common. - (wheat9) - - HADOOP-11394. hadoop-aws documentation missing. (cnauroth) - - HADOOP-11396. Provide navigation in the site documentation linking to the - Hadoop Compatible File Systems. (cnauroth) - - HADOOP-11412 POMs mention "The Apache Software License" rather than - "Apache License". (Herve Boutemy via stevel) - - HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user - has Full Control permissions. (cnauroth) - - HADOOP-11420. Use latest maven-site-plugin and replace link to svn with - link to git. (Herve Boutemy via wheat9) - - HADOOP-10689. InputStream is not closed in - AzureNativeFileSystemStore#retrieve(). (Chen He via cnauroth) - - HADOOP-10690. Lack of synchronization on access to InputStream in - NativeAzureFileSystem#NativeAzureFsInputStream#close(). - (Chen He via cnauroth) - - HADOOP-11358. Tests for encryption/decryption with IV calculation - overflow. (yliu) - - HADOOP-11125. Remove redundant tests in TestOsSecureRandom. - (Masanori Oyama via wheat9) - - HADOOP-11385. Prevent cross site scripting attack on JMXJSONServlet. - (wheat9) - - HADOOP-11409. FileContext.getFileContext can stack overflow if default fs - misconfigured (Gera Shegalov via jlowe) - - HADOOP-11428. Remove obsolete reference to Cygwin in BUILDING.txt. - (Arpit Agarwal via wheat9) - - HADOOP-11431. clean up redundant maven-site-plugin configuration. - (Herve Boutemy via wheat9) - - HADOOP-11429. Findbugs warnings in hadoop extras. - (Varun Saxena via wheat9) - - HADOOP-11414. FileBasedIPList#readLines() can leak file descriptors. - (ozawa) - - HADOOP-11283. SequenceFile.Writer can leak file descriptors in - DistCpV1#setup(). (Varun Saxena via ozawa) - - HADOOP-11448. Fix findbugs warnings in FileBasedIPList. (ozawa) - - HADOOP-11039. ByteBufferReadable API doc is inconsistent with the - implementations. (Yi Liu via Colin P. McCabe) - - HADOOP-11446. S3AOutputStream should use shared thread pool to - avoid OutOfMemoryError. (Ted Yu via stevel) - - HADOOP-11459. Fix recent findbugs in ActiveStandbyElector, NetUtils - and ShellBasedIdMapping (vinayakumarb) - - HADOOP-11445. Bzip2Codec: Data block is skipped when position of newly - created stream is equal to start of split (Ankit Kamboj via jlowe) - - HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform. - (Ayappan via cnauroth) - - HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore - retrieveBlock(). (Ted Yu via stevel) - - HADOOP-10668. TestZKFailoverControllerStress#testExpireBackAndForth - occasionally fails. (Ming Ma via cnauroth) - - HADOOP-11327. BloomFilter#not() omits the last bit, resulting in an - incorrect filter (Eric Payne via jlowe) - - HADOOP-11209. Configuration#updatingResource/finalParameters are not - thread-safe. (Varun Saxena via ozawa) - - HADOOP-11500. InputStream is left unclosed in ApplicationClassLoader. - (Ted Yu via ozawa) - - HADOOP-11008. Remove duplicated description about proxy-user in site - documents (Masatake Iwasaki via aw) - - HADOOP-11493. Fix some typos in kms-acls.xml description. - (Charles Lamb via aajisaka) - - HADOOP-11507 Hadoop RPC Authentication problem with different user locale. - (Talat UYARER via stevel) - - HADOOP-11499. Check of executorThreadsStarted in - ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe) - - HADOOP-6221 RPC Client operations cannot be interrupted. (stevel) - - HADOOP-11509. Change parsing sequence in GenericOptionsParser to parse -D - parameters before -files. (xgong) - - HADOOP-11469. KMS should skip default.key.acl and whitelist.key.acl when - loading key acl. (Dian Fu via yliu) - - HADOOP-11316. "mvn package -Pdist,docs -DskipTests -Dtar" fails because - of non-ascii characters. (ozawa) - - HADOOP-9907. Webapp http://hostname:port/metrics link is not working. - (aajisaka) - - HADOOP-11403. Avoid using sys_errlist on Solaris, which lacks support for it - (Malcolm Kavalsky via Colin P. McCabe) - - HADOOP-11523. StorageException complaining " no lease ID" when updating - FolderLastModifiedTime in WASB. (Duo Xu via cnauroth) - - HADOOP-11432. Fix SymlinkBaseTest#testCreateLinkUsingPartQualPath2. - (Liang Xie via gera) - - HADOOP-10181. GangliaContext does not work with multicast ganglia setup. - (Andrew Johnson via cnauroth) - - HADOOP-11529. Fix findbugs warnings in hadoop-archives. - (Masatake Iwasaki via wheat9) - - HADOOP-11546. Checkstyle failing: Unable to instantiate - DoubleCheckedLockingCheck. (ozawa) - - HADOOP-11548. checknative should display a nicer error message when openssl - support is not compiled in. (Anu Engineer via cnauroth) - - HADOOP-11547. hadoop-common native compilation fails on Windows due to - missing support for __attribute__ declaration. (cnauroth) - - HADOOP-11549. flaky test detection tool failed to handle special control - characters in test result. (Yongjun Zhang via aajisaka) - - HADOOP-10062. race condition in MetricsSystemImpl#publishMetricsNow that - causes incorrect results. (Sangjin Lee via junping_du) - - HADOOP-11526. Memory leak in Bzip2Compressor and Bzip2Decompressor. - (Anu Engineer via cnauroth) - - HADOOP-11535 TableMapping related tests failed due to 'successful' - resolving of invalid test hostname. (Kai Zheng via stevel) - - HADOOP-10953. NetworkTopology#add calls NetworkTopology#toString without - holding the netlock (Liang Xie via Colin P. McCabe) - - HADOOP-11587. TestMapFile#testMainMethodMapFile creates test files in - hadoop-common project root. (Xiaoyu Yao via wheat9) - - HADOOP-11467. KerberosAuthenticator can connect to a non-secure cluster. - (yzhangal via rkanter) - - HADOOP-11000. HAServiceProtocol's health state is incorrectly transitioned - to SERVICE_NOT_RESPONDING (Ming Ma via vinayakumarb) - - HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of - the object from S3. (Dan Hecht via stevel). - - HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when - IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa) - - HADOOP-11545. ArrayIndexOutOfBoundsException is thrown with "hadoop - credential list -provider". (Brahma Reddy Battula via aajisaka) - - HADOOP-11595. Add default implementation for AbstractFileSystem#truncate. - (yliu) - - HADOOP-9087. Queue size metric for metric sinks isn't actually maintained - (Akira AJISAKA via jlowe) - - HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. - (rkanter) - - HADOOP-11584 s3a file block size set to 0 in getFileStatus. - (Brahma Reddy Battula via stevel) - - HADOOP-10478. Fix new findbugs warnings in hadoop-maven-plugins. - (Li Lu via wheat9) - - HADOOP-8642. Document that io.native.lib.available only controls native bz2 - and zlib compression codecs. (aajisaka) - - HADOOP-11619. FTPFileSystem should override getDefaultPort. - (Brahma Reddy Battula via gera) - - HADOOP-11480. Typo in hadoop-aws/index.md uses wrong scheme for - test.fs.s3.name. (Ted Yu via aajisaka) - - HADOOP-11629. WASB filesystem should not start BandwidthGaugeUpdater if - fs.azure.skip.metrics set to true. (Shanyu Zhao via cnauroth) - - HADOOP-9922. hadoop windows native build will fail in 32 bit machine. - (Kiran Kumar M R via cnauroth) - - HADOOP-11634. Description of webhdfs' principal/keytab should switch places - each other. (Brahma Reddy Battula via ozawa) - - HADOOP-11615. Update ServiceLevelAuth.md for YARN. - (Brahma Reddy Battula via aajisaka) - - HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it - to wrapped FS. (gera) - - HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) - - HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) - - HADOOP-11686. MiniKDC cannot change ORG_NAME or ORG_DOMAIN. - (Duo Zhang via wheat9) - - HADOOP-11618. DelegateToFileSystem erroneously uses default FS's port in - constructor. (Brahma Reddy Battula via gera) - - HADOOP-11693. Azure Storage FileSystem rename operations are throttled too - aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) - - HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre - Matsumoto via ozawa) - - HADOOP-11638. OpensslSecureRandom.c pthreads_thread_id should support FreeBSD - and Solaris in addition to Linux. (Kiran Kumar M R via cnauroth) - - HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal - tags in hadoop-tools. (Akira AJISAKA via ozawa) - - HADOOP-11722. Some Instances of Services using - ZKDelegationTokenSecretManager go down when old token cannot be deleted. - (Arun Suresh via atm) - - HADOOP-10703. HttpServer2 creates multiple authentication filters. - (Benoy Antony via wheat9) - - HADOOP-11729. Fix link to cgroups doc in site.xml. (Masatake Iwasaki via - ozawa) - - HADOOP-11609. Correct credential commands info in - CommandsManual.html#credential. (Varun Saxena via ozawa) - - HADOOP-11014. Potential resource leak in JavaKeyStoreProvider due to - unclosed stream. (ozawa) - - HADOOP-11738. Fix a link of Protocol Buffers 2.5 for download in BUILDING.txt. - (ozawa) - - HADOOP-11748. The secrets of auth cookies should not be specified in - configuration in clear text. (Li Lu and Haohui Mai via wheat9) - - HADOOP-11691. X86 build of libwinutils is broken. - (Kiran Kumar M R via cnauroth) - - HADOOP-11639. Clean up Windows native code compilation warnings related to - Windows Secure Container Executor. (Remus Rusanu via cnauroth) - - HADOOP-11761. Fix findbugs warnings in org.apache.hadoop.security - .authentication. (Li Lu via wheat9) - - HADOOP-11754. RM fails to start in non-secure mode due to authentication - filter failure. (wheat9) - - HADOOP-11787. OpensslSecureRandom.c pthread_threadid_np usage signature is - wrong on 32-bit Mac. (Kiran Kumar M R via cnauroth) - - HADOOP-11757. NFS gateway should shutdown when it can't start UDP or TCP - server (brandonli) - - HADOOP-11377. Added Null.java without which jdiff completely flops. (Tsuyoshi - Ozawa via vinodkv) - - HADOOP-11776. Fixed the broken JDiff support in Hadoop 2. (Li Lu via vinodkv) - - HADOOP-11796. Skip TestShellBasedIdMapping.testStaticMapUpdate on Windows. - (Xiaoyu Yao via cnauroth) - - HADOOP-11815. HttpServer2 should destroy SignerSecretProvider when it - stops. (Rohith via wheat9) - - HADOOP-11837. AuthenticationFilter should destroy SignerSecretProvider in - Tomcat deployments. (Bowen Zhang via wheat9) - -Release 2.6.5 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-12805. Annotate CanUnbuffer with @InterfaceAudience.Public - (Ted Yu via vinayakumarb) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-12773. HBase classes fail to load with client/job classloader - enabled (sjlee) - - HADOOP-12589. Fix intermittent test failure of TestCopyPreserveFlag - (iwasakims) - -Release 2.6.4 - 2016-02-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-11252. RPC client does not time out by default. - (Wilfred Spiegelenburg and Masatake Iwasaki via aajisaka) - - HADOOP-12107. long running apps may have a huge number of StatisticsData - instances under FileSystem (Sangjin Lee via Ming Ma) - - HADOOP-12706. TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp - times out occasionally (Sangjin Lee and Colin Patrick McCabe via jlowe) - - HADOOP-12736. TestTimedOutTestsListener#testThreadDumpAndDeadlocks - sometimes times out. (Xiao Chen via aajisaka) - - HADOOP-12715. TestValueQueue#testgetAtMostPolicyALL fails intermittently. - (Xiao Chen via waltersu4549) - -Release 2.6.3 - 2015-12-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-12413. AccessControlList should avoid calling getGroupNames in - isUserInList with empty groups. (Zhihai Xu via cnauroth) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-12526. there are duplicate dependency definitions in pom's (sjlee) - - HADOOP-12577. Bumped up commons-collections version to 3.2.2 to address a - security flaw. (Wei-Chiu Chuang via vinodkv) - -Release 2.6.2 - 2015-10-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.6.1 - 2015-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-7139. Allow appending to existing SequenceFiles - (kanaka kumar avvaru via vinayakumarb) - - HADOOP-12280. Skip unit tests based on maven profile rather than - NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) - - OPTIMIZATIONS - - HADOOP-11238. Update the NameNode's Group Cache in the background when - possible (Chris Li via Colin P. McCabe) - - HADOOP-11506. Configuration variable expansion regex expensive for long - values. (Gera Shegalov via gera) - - BUG FIXES - - HADOOP-11466: FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC - architecture because it is slower there (Suman Somasundar via Colin P. - McCabe) - - HADOOP-10786. Fix UGI#reloginFromKeytab on Java 8. (Stephen Chu via wheat9) - - HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification - pipe is full (zhaoyunjiong via cmccabe) - - HADOOP-11343. Overflow is not properly handled in caclulating final iv for - AES CTR. (Jerry Chen via wang) - - HADOOP-11368. Fix SSLFactory truststore reloader thread leak in - KMSClientProvider. (Arun Suresh via wang) - - HADOOP-11350. The size of header buffer of HttpServer is too small when - HTTPS is enabled. (Benoy Antony via wheat9) - - HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy - user. Contributed by Arun Suresh. - - HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is - full. (Ming Ma via kihwal) - - HADOOP-11604. Prevent ConcurrentModificationException while closing domain - sockets during shutdown of DomainSocketWatcher thread. (cnauroth) - - HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream - should be non static. (Sean Busbey via yliu) - - HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt - synchronization. (Sean Busbey via yliu) - - HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up - split calculation (gera) - - HADOOP-11730. Regression: s3n read failure recovery broken. - (Takenori Sato via stevel) - - HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - - HADOOP-11491. HarFs incorrectly declared as requiring an authority. - (Brahma Reddy Battula via gera) - - HADOOP-8151. Error handling in snappy decompressor throws invalid - exceptions. (Matt Foley via harsh) - - HADOOP-11932. MetricsSinkAdapter may hang when being stopped. - (Brahma Reddy Battula via jianhe) - - HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes - infinite loop. (Larry McCay via cnauroth) - -Release 2.6.0 - 2014-11-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HADOOP-10433. Key Management Server based on KeyProvider API. (tucu) - - HADOOP-10893. isolated classloader on the client side (Sangjin Lee via - jlowe) - - HADOOP-10400. Incorporate new S3A FileSystem implementation. (Jordan - Mendelson and Dave Wang via atm) - - IMPROVEMENTS - - HADOOP-10808. Remove unused native code for munlock. (cnauroth) - - HADOOP-10815. Implement Windows equivalent of mlock. (cnauroth) - - HADOOP-7664. Remove warmings when overriding final parameter configuration - if the override value is same as the final parameter value. - (Ravi Prakash via suresh) - - HADOOP-10673. Update rpc metrics when the call throws an exception. (Ming Ma - via jing9) - - HADOOP-10845. Add common tests for ACLs in combination with viewfs. - (Stephen Chu via cnauroth) - - HADOOP-10839. Add unregisterSource() to MetricsSystem API. - (Shanyu Zhao via cnauroth) - - HADOOP-10607. Create an API to separate credentials/password storage - from applications (Larry McCay via omalley) - - HADOOP-10732. Fix locking in credential update. (Ted Yu via omalley) - - HADOOP-10733. Fix potential null dereference in CredShell. (Ted Yu via - omalley) - - HADOOP-10610. Upgrade S3n s3.fs.buffer.dir to support multi directories. - (Ted Malaska via atm) - - HADOOP-10817. ProxyUsers configuration should support configurable - prefixes. (tucu) - - HADOOP-10755. Support negative caching of user-group mapping. - (Lei Xu via wang) - - HADOOP-10855. Allow Text to be read with a known Length. (todd) - - HADOOP-10887. Add XAttrs to ViewFs and make XAttrs + ViewFileSystem - internal dir behavior consistent. (Stephen Chu via wang) - - HADOOP-10882. Move DirectBufferPool into common util. (todd) - - HADOOP-8069. Enable TCP_NODELAY by default for IPC. (Todd Lipcon via - Arpit Agarwal) - - HADOOP-10902. Deletion of directories with snapshots will not output - reason for trash move failure. (Stephen Chu via wang) - - HADOOP-10900. CredentialShell args should use single-dash style. (wang) - - HADOOP-10903. Enhance hadoop classpath command to expand wildcards or write - classpath into jar manifest. (cnauroth) - - HADOOP-10791. AuthenticationFilter should support externalizing the - secret for signing and provide rotation support. (rkanter via tucu) - - HADOOP-10771. Refactor HTTP delegation support out of httpfs to common. - (tucu) - - HADOOP-10835. Implement HTTP proxyuser support in HTTP authentication - client/server libraries. (tucu) - - HADOOP-10820. Throw an exception in GenericOptionsParser when passed - an empty Path. (Alex Holmes and Zhihai Xu via wang) - - HADOOP-10281. Create a scheduler, which assigns schedulables a priority - level. (Chris Li via Arpit Agarwal) - - HADOOP-8944. Shell command fs -count should include human readable option - (Jonathan Allen via aw) - - HADOOP-10231. Add some components in Native Libraries document (Akira - AJISAKA via aw) - - HADOOP-10650. Add ability to specify a reverse ACL (black list) of users - and groups. (Benoy Antony via Arpit Agarwal) - - HADOOP-10335. An ip whilelist based implementation to resolve Sasl - properties per connection. (Benoy Antony via Arpit Agarwal) - - HADOOP-10975. org.apache.hadoop.util.DataChecksum should support calculating - checksums in native code (James Thomas via Colin Patrick McCabe) - - HADOOP-10201. Add listing to KeyProvider API. (Larry McCay via omalley) - - HADOOP-10177. Create CLI tools for managing keys. (Larry McCay via omalley) - - HADOOP-10432. Refactor SSLFactory to expose static method to determine - HostnameVerifier. (tucu) - - HADOOP-10429. KeyStores should have methods to generate the materials - themselves, KeyShell should use them. (tucu) - - HADOOP-10427. KeyProvider implementations should be thread safe. (tucu) - - HADOOP-10428. JavaKeyStoreProvider should accept keystore password via - configuration falling back to ENV VAR. (tucu) - - HADOOP-10430. KeyProvider Metadata should have an optional description, - there should be a method to retrieve the metadata from all keys. (tucu) - - HADOOP-10431. Change visibility of KeyStore.Options getter methods to - public. (tucu) - - HADOOP-10534. KeyProvider getKeysMetadata should take a list of names - rather than returning all keys. (omalley) - - HADOOP-10719. Add generateEncryptedKey and decryptEncryptedKey - methods to KeyProvider. (asuresh via tucu) - - HADOOP-10817. ProxyUsers configuration should support configurable - prefixes. (tucu) - - HADOOP-10881. Clarify usage of encryption and encrypted encryption - key in KeyProviderCryptoExtension. (wang) - - HADOOP-10770. KMS add delegation token support. (tucu) - - HADOOP-10698. KMS, add proxyuser support. (tucu) - - HADOOP-8896. Javadoc points to Wrong Reader and Writer classes - in SequenceFile (Ray Chiang via aw) - - HADOOP-10998. Fix bash tab completion code to work (Jim Hester via aw) - - HADOOP-10880. Move HTTP delegation tokens out of URL querystring to - a header. (tucu) - - HADOOP-11005. Fix HTTP content type for ReconfigurationServlet. - (Lei Xu via wang) - - HADOOP-10814. Update Tomcat version used by HttpFS and KMS to latest - 6.x version. (rkanter via tucu) - - HADOOP-10994. KeyProviderCryptoExtension should use CryptoCodec for - generation/decryption of keys. (tucu) - - HADOOP-11021. Configurable replication factor in the hadoop archive - command. (Zhe Zhang via wang) - - HADOOP-11030. Define a variable jackson.version instead of using constant - at multiple places. (Juan Yu via kasha) - - HADOOP-10990. Add missed NFSv3 request and response classes (brandonli) - - HADOOP-10863. KMS should have a blacklist for decrypting EEKs. - (asuresh via tucu) - - HADOOP-11054. Add a KeyProvider instantiation based on a URI. (tucu) - - HADOOP-11015. Http server/client utils to propagate and recreate - Exceptions from server to client. (tucu) - - HADOOP-11060. Create a CryptoCodec test that verifies interoperability - between the JCE and OpenSSL implementations. (hitliuyi via tucu) - - HADOOP-11070. Create MiniKMS for testing. (tucu) - - HADOOP-11057. checknative command to probe for winutils.exe on windows. - (Xiaoyu Yao via cnauroth) - - HADOOP-10758. KMS: add ACLs on per key basis. (tucu) - - HADOOP-9540. Expose the InMemoryS3 and S3N FilesystemStores implementations - for Unit testing. (Hari via stevel) - - HADOOP-10373 create tools/hadoop-amazon for aws/EMR support (stevel) - - HADOOP-11074. Move s3-related FS connector code to hadoop-aws (David S. - Wang via Colin Patrick McCabe) - - HADOOP-11091. Eliminate old configuration parameter names from s3a (David - S. Wang via Colin Patrick McCabe) - - HADOOP-10868. AuthenticationFilter should support externalizing the - secret for signing and provide rotation support. (rkanter via tucu) - - HADOOP-10922. User documentation for CredentialShell. (Larry McCay via wang) - - HADOOP-11016. KMS should support signing cookies with zookeeper secret - manager. (tucu) - - HADOOP-11106. Document considerations of HAR and Encryption. (clamb via wang) - - HADOOP-10970. Cleanup KMS configuration keys. (wang) - - HADOOP-11017. KMS delegation token secret manager should be able to use - zookeeper as store. (asuresh via tucu) - - HADOOP-11009. Add Timestamp Preservation to DistCp (Gary Steelman via aw) - - HADOOP-11101. How about inputstream close statement from catch block to - finally block in FileContext#copy() ( skrho via vinayakumarb ) - - HADOOP-8808. Update FsShell documentation to mention deprecation of some of - the commands, and mention alternatives (Akira AJISAKA via aw) - - HADOOP-10954. Adding site documents of hadoop-tools (Masatake Iwasaki - via aw) - - HADOOP-10731. Remove @date JavaDoc comment in ProgramDriver class (Henry - Saputra via aw) - - HADOOP-11153. Make number of KMS threads configurable. (wang) - - HADOOP-11007. Reinstate building of ant tasks support. (jlowe via kihwal) - - HADOOP-11178. Fix findbugs exclude file. (Arun Suresh via wang) - - HADOOP-11174. Delegation token for KMS should only be got once if it - already exists. (Yi Liu via wang) - - HADOOP-11184. Update Hadoop's lz4 to version r123. (cmccabe) - - HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all - sub-classes of AbstractDelegationTokenIdentifier. (zjshen) - - HADOOP-11207. Enhanced common DelegationTokenAuthenticationHandler to support - proxy-users on Delegation-token management operations. (Zhijie Shen via - vinodkv) - - HADOOP-11216. Improve Openssl library finding. (cmccabe via yliu) - - HADOOP-11254. Changed visibility of AccessControlList to be public for - consumption by ecosystem. (Zhijie Shen via vinodkv) - - HADOOP-11286. Copied LimitInputStream from guava-0.14 to hadoop to avoid - issues with newer versions of guava in applications. (Christopher Tubbs - via acmurthy) - - OPTIMIZATIONS - - HADOOP-10838. Byte array native checksumming. (James Thomas via todd) - - HADOOP-10696. Add optional attributes to KeyProvider Options and Metadata. - (tucu) - - HADOOP-10695. KMSClientProvider should respect a configurable timeout. - (yoderme via tucu) - - HADOOP-10757. KeyProvider KeyVersion should provide the key name. - (asuresh via tucu) - - HADOOP-10769. Create KeyProvider extension to handle delegation tokens. - (Arun Suresh via atm) - - HADOOP-10812. Delegate KeyProviderExtension#toString to underlying - KeyProvider. (wang) - - HADOOP-10736. Add key attributes to the key shell. (Mike Yoder via wang) - - HADOOP-10824. Refactor KMSACLs to avoid locking. (Benoy Antony via umamahesh) - - HADOOP-10841. EncryptedKeyVersion should have a key name property. - (asuresh via tucu) - - HADOOP-10842. CryptoExtension generateEncryptedKey method should - receive the key name. (asuresh via tucu) - - HADOOP-10750. KMSKeyProviderCache should be in hadoop-common. - (asuresh via tucu) - - HADOOP-10720. KMS: Implement generateEncryptedKey and decryptEncryptedKey - in the REST API. (asuresh via tucu) - - HADOOP-10891. Add EncryptedKeyVersion factory method to - KeyProviderCryptoExtension. (wang) - - HADOOP-10756. KMS audit log should consolidate successful similar requests. - (asuresh via tucu) - - HADOOP-10793. KeyShell args should use single-dash style. (wang) - - HADOOP-10936. Change default KeyProvider bitlength to 128. (wang) - - HADOOP-10224. JavaKeyStoreProvider has to protect against corrupting - underlying store. (asuresh via tucu) - - HADOOP-10282. Create a FairCallQueue: a multi-level call queue which - schedules incoming calls and multiplexes outgoing calls. (Chris Li via - Arpit Agarwal) - - HADOOP-10833. Remove unused cache in UserProvider. (Benoy Antony) - - HADOOP-11112. TestKMSWithZK does not use KEY_PROVIDER_URI. (tucu via wang) - - HADOOP-11111 MiniKDC to use locale EN_US for case conversions. (stevel) - - HADOOP-10681. Remove unnecessary synchronization from Snappy & Zlib - codecs. (Gopal Vijayaraghavan via acmurthy) - - HADOOP-11194. Ignore .keep files. (kasha) - - HADOOP-11195. Move Id-Name mapping in NFS to the hadoop-common area for - better maintenance (Yongjun Zhang via brandonli) - - HADOOP-11247. Fix a couple javac warnings in NFS. (Brandon Li via wheat9) - - BUG FIXES - - HADOOP-11182. GraphiteSink emits wrong timestamps (Sascha Coenen via raviprak) - - HADOOP-10781. Unportable getgrouplist() usage breaks FreeBSD (Dmitry - Sivachenko via Colin Patrick McCabe) - - HADOOP-10507. FsShell setfacl can throw ArrayIndexOutOfBoundsException when - no perm is specified. (Stephen Chu and Sathish Gurram via cnauroth) - - HADOOP-10780. hadoop_user_info_alloc fails on FreeBSD due to incorrect - sysconf use (Dmitry Sivachenko via Colin Patrick McCabe) - - HADOOP-10810. Clean up native code compilation warnings. (cnauroth) - - HADOOP-9921. daemon scripts should remove pid file on stop call after stop - or process is found not running ( vinayakumarb ) - - HADOOP-10591. Compression codecs must used pooled direct buffers or - deallocate direct buffers when stream is closed (cmccabe) - - HADOOP-10857. Native Libraries Guide doen't mention a dependency on - openssl-development package (ozawa via cmccabe) - - HADOOP-10866. RawLocalFileSystem fails to read symlink targets via the stat - command when the format of the stat command uses non-curly quotes (yzhang - via cmccabe) - - HADOOP-10830. Missing lock in JavaKeyStoreProvider.createCredentialEntry. - (Benoy Antony via umamahesh) - - HADOOP-10928. Incorrect usage on `hadoop credential list`. - (Josh Elser via wang) - - HADOOP-10927. Fix CredentialShell help behavior and error codes. - (Josh Elser via wang) - - HADOOP-10933. FileBasedKeyStoresFactory Should use Configuration.getPassword - for SSL Passwords. (lmccay via tucu) - - HADOOP-10759. Remove hardcoded JAVA_HEAP_MAX. (Sam Liu via Eric Yang) - - HADOOP-10905. LdapGroupsMapping Should use configuration.getPassword for SSL - and LDAP Passwords. (lmccay via brandonli) - - HADOOP-10931 compile error on tools/hadoop-openstack (xukun via stevel) - - HADOOP-10929. Typo in Configuration.getPasswordFromCredentialProviders - (lmccay via brandonli) - - HADOOP-10402. Configuration.getValByRegex does not substitute for - variables. (Robert Kanter via kasha) - - HADOOP-10851. NetgroupCache does not remove group memberships. (Benoy - Antony via Arpit Agarwal) - - HADOOP-10962. Flags for posix_fadvise are not valid in some architectures - (David Villegas via Colin Patrick McCabe) - - HADOOP-10966. Hadoop Common native compilation broken in windows. - (David Villegas via Arpit Agarwal) - - HADOOP-10843. TestGridmixRecord unit tests failure on PowerPC (Jinghui Wang - via Colin Patrick McCabe) - - HADOOP-10121. Fix javadoc spelling for HadoopArchives#writeTopLevelDirs - (Akira AJISAKA via aw) - - HADOOP-10964. Small fix for NetworkTopologyWithNodeGroup#sortByDistance. - (Yi Liu via wang) - - HADOOP-10059. RPC authentication and authorization metrics overflow to - negative values on busy clusters (Tsuyoshi OZAWA and Akira AJISAKA - via jlowe) - - HADOOP-10973. Native Libraries Guide contains format error. (Peter Klavins - via Arpit Agarwal) - - HADOOP-10972. Native Libraries Guide contains mis-spelt build line (Peter - Klavins via aw) - - HADOOP-10873. Fix dead link in Configuration javadoc (Akira AJISAKA - via aw) - - HADOOP-10968. hadoop native build fails to detect java_libarch on - ppc64le (Dinar Valeev via Colin Patrick McCabe) - - HADOOP-10141. Create KeyProvider API to separate encryption key storage - from the applications. (omalley) - - HADOOP-10237. JavaKeyStoreProvider needs to set keystore permissions - correctly. (Larry McCay via omalley) - - HADOOP-10244. TestKeyShell improperly tests the results of delete (Larry - McCay via omalley) - - HADOOP-10583. bin/hadoop key throws NPE with no args and assorted other fixups. (clamb via tucu) - - HADOOP-10586. KeyShell doesn't allow setting Options via CLI. (clamb via tucu) - - HADOOP-10645. TestKMS fails because race condition writing acl files. (tucu) - - HADOOP-10611. KMS, keyVersion name should not be assumed to be - keyName@versionNumber. (tucu) - - HADOOP-10816. KeyShell returns -1 on error to the shell, should be 1. - (Mike Yoder via wang) - - HADOOP-10826. Iteration on KeyProviderFactory.serviceLoader is - thread-unsafe. (benoyantony viat tucu) - - HADOOP-10920. site plugin couldn't parse hadoop-kms index.apt.vm. - (Akira Ajisaka via wang) - - HADOOP-10937. Need to set version name correctly before decrypting EEK. - (Arun Suresh via wang) - - HADOOP-10918. JMXJsonServlet fails when used within Tomcat. (tucu) - - HADOOP-10939. Fix TestKeyProviderFactory testcases to use default 128 bit - length keys. (Arun Suresh via wang) - - HADOOP-10862. Miscellaneous trivial corrections to KMS classes. - (asuresh via tucu) - - HADOOP-10967. Improve DefaultCryptoExtension#generateEncryptedKey - performance. (hitliuyi via tucu) - - HADOOP-10488. TestKeyProviderFactory fails randomly. (tucu) - - HADOOP-10989. Work around buggy getgrouplist() implementations on Linux that - return 0 on failure. (cnauroth) - - HADOOP-8815. RandomDatum needs to override hashCode(). - (Brandon Li via suresh) - - HADOOP-11056. OsSecureRandom.setConf() might leak file descriptors (yzhang - via cmccabe) - - HADOOP-11040. Return value of read(ByteBuffer buf) in CryptoInputStream is - incorrect in some cases. (Yi Liu via wang) - - HADOOP-11140. hadoop-aws only need test-scoped dependency on - hadoop-common's tests jar. (Juan Yu via wang) - - HADOOP-1110. JavaKeystoreProvider should not report a key as created if it - was not flushed to the backing file. - - HADOOP-11130. NFS updateMaps OS check is reversed (brandonli) - - HADOOP-11113. Namenode not able to reconnect to KMS after KMS restart. - (Arun Suresh via wang) - - HADOOP-11160. Fix typo in nfs3 server duplicate entry reporting. - (Charles Lamb via wheat9) - - HADOOP-11151. Automatically refresh auth token and retry on auth failure. - (Arun Suresh via wang) - - HADOOP-11168. Remove duplicated entry "dfs.webhdfs.enabled" in the useri - doc. (Yi Liu via wheat9) - - HADOOP-11169. Fix DelegationTokenAuthenticatedURL to pass the connection - Configurator to the authenticator. (Arun Suresh via wang) - - HADOOP-10404. Some accesses to DomainSocketWatcher#closed are not protected - by the lock (cmccabe) - - HADOOP-11161. Expose close method in KeyProvider to give clients of - Provider implementations a hook to release resources. (Arun Suresh via atm) - - HADOOP-11133. Should trim the content of keystore password file for JavaKeyStoreProvider - (Yi Liu via umamahesh) - - HADOOP-11193. Fix uninitialized variables in NativeIO.c - (Xiaoyu Yao via wheat9) - - HADOOP-11176. KMSClientProvider authentication fails when both currentUgi - and loginUgi are a proxied user. (Arun Suresh via atm) - - HADOOP-11198. Fix typo in javadoc for FileSystem#listStatus(). - (Li Lu via wheat9) - - HADOOP-11253. Hadoop streaming test TestStreamXmlMultipleRecords fails on - Windows. (Varun Vasudev via wheat9) - - BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS - - HADOOP-10734. Implement high-performance secure random number sources. - (Yi Liu via Colin Patrick McCabe) - - HADOOP-10603. Crypto input and output streams implementing Hadoop stream - interfaces. (Yi Liu and Charles Lamb) - - HADOOP-10628. Javadoc and few code style improvement for Crypto - input and output streams. (Yi Liu via clamb) - - HADOOP-10632. Minor improvements to Crypto input and output streams. - (Yi Liu) - - HADOOP-10635. Add a method to CryptoCodec to generate SRNs for IV. (Yi Liu) - - HADOOP-10653. Add a new constructor for CryptoInputStream that - receives current position of wrapped stream. (Yi Liu) - - HADOOP-10662. NullPointerException in CryptoInputStream while wrapped - stream is not ByteBufferReadable. Add tests using normal stream. (Yi Liu) - - HADOOP-10713. Refactor CryptoCodec#generateSecureRandom to take a byte[]. - (wang via yliu) - - HADOOP-10693. Implementation of AES-CTR CryptoCodec using JNI to OpenSSL. - (Yi Liu via cmccabe) - - HADOOP-10803. Update OpensslCipher#getInstance to accept CipherSuite#name - format. (Yi Liu) - - HADOOP-10735. Fall back AesCtrCryptoCodec implementation from OpenSSL to - JCE if non native support. (Yi Liu) - - HADOOP-10870. Failed to load OpenSSL cipher error logs on systems with old - openssl versions (cmccabe) - - HADOOP-10853. Refactor get instance of CryptoCodec and support create via - algorithm/mode/padding. (Yi Liu) - - HADOOP-10919. Copy command should preserve raw.* namespace - extended attributes. (clamb) - - HDFS-6873. Constants in CommandWithDestination should be static. (clamb) - - HADOOP-10871. incorrect prototype in OpensslSecureRandom.c (cmccabe) - - HADOOP-10886. CryptoCodec#getCodecclasses throws NPE when configurations not - loaded. (umamahesh) - -- - - HADOOP-10911. hadoop.auth cookie after HADOOP-10710 still not proper - according to RFC2109. (gchanan via tucu) - - HADOOP-11036. Add build directory to .gitignore (Tsuyoshi OZAWA via aw) - - HADOOP-11012. hadoop fs -text of zero-length file causes EOFException - (Eric Payne via jlowe) - - HADOOP-11063. KMS cannot deploy on Windows, because class names are too long. - (cnauroth) - - HADOOP-11067. warning message 'ssl.client.truststore.location has not - been set' gets printed for hftp command. (Xiaoyu Yao via Arpit Agarwal) - - HADOOP-11069. KMSClientProvider should use getAuthenticationMethod() to - determine if in proxyuser mode or not. (tucu) - - HADOOP-11073. Credential Provider related Unit Tests Failure on Windows. - (Xiaoyu Yao via cnauroth) - - HADOOP-11071. KMSClientProvider should drain the local generated EEK cache - on key rollover. (tucu) - - HADOOP-10925. Compilation fails in native link0 function on Windows. - (cnauroth) - - HADOOP-11077. NPE if hosts not specified in ProxyUsers. (gchanan via tucu) - - HADOOP-9989. Bug introduced in HADOOP-9374, which parses the -tokenCacheFile - as binary file but set it to the configuration as JSON file. (zxu via tucu) - - HADOOP-11085. Excessive logging by org.apache.hadoop.util.Progress when - value is NaN (Mit Desai via jlowe) - - HADOOP-11083. After refactoring of HTTP proxyuser to common, doAs param is - case sensitive. (tucu) - - HADOOP-11088. Unittest TestKeyShell, TestCredShell and TestKMS assume UNIX - path separator for JECKS key store path. (Xiaoyu Yao via cnauroth) - - HADOOP-11096. KMS: KeyAuthorizationKeyProvider should verify the keyversion - belongs to the keyname on decrypt. (tucu) - - HADOOP-11097. kms docs say proxyusers, not proxyuser for config params. - (clamb via tucu) - - HADOOP-11062. CryptoCodec testcases requiring OpenSSL should be run - only if -Pnative is used. (asuresh via tucu) - - HADOOP-11099. KMS return HTTP UNAUTHORIZED 401 on ACL failure. (tucu) - - HADOOP-11105. MetricsSystemImpl could leak memory in registered callbacks. - (Chuan Liu via cnauroth) - - HADOOP-10982. KMS: Support for multiple Kerberos principals. (tucu) - - HADOOP-11109. Site build is broken. (Jian He via atm) - - HADOOP-10946. Fix a bunch of typos in log messages (Ray Chiang via aw) - - HADOOP-10131. NetWorkTopology#countNumOfAvailableNodes() is returning - wrong value if excluded nodes passed are not part of the cluster tree - (vinayakumarb) - - HADOOP-11064. UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due to - NativeCRC32 method changes. (cnauroth) - - HADOOP-11048. user/custom LogManager fails to load if the client - classloader is enabled (Sangjin Lee via jlowe) - - HADOOP-10552. Fix usage and example at FileSystemShell.apt.vm (Kenji - Kikushima via aw) - - HADOOP-11143 NetUtils.wrapException loses inner stack trace on BindException - (stevel) - - HADOOP-11049. javax package system class default is too broad (Sangjin Lee - via jlowe) - - HADOOP-11154. Update BUILDING.txt to state that CMake 3.0 or newer is - required on Mac. (cnauroth) - - HADOOP-11145. TestFairCallQueue fails. (Akira AJISAKA via cnauroth) - - HADOOP-11117 UGI HadoopLoginModule doesn't catch & wrap all - kerberos-related exceptions (stevel) - - HADOOP-11163. MetricsSystemImpl may miss a registered source. - (Chuan Liu via cnauroth) - - HADOOP-11179. Java untar should handle the case that the file entry comes - without its parent directory entry. (Craig Welch via zjshen) - - HADOOP-11175. Fix several issues of hadoop security configuration in user - doc. (Yi Liu via cnauroth) - - HADOOP-11122. Fix findbugs in ZK DelegationTokenSecretManagers. - (Arun Suresh via kasha) - - HADOOP-11228. Winutils task: unsecure path should not call - AddNodeManagerAndUserACEsToObject. (Remus Rusanu via jianhe) - - HADOOP-11170. ZKDelegationTokenSecretManager fails to renewToken created by - a peer. (Arun Suresh and Gregory Chanan via kasha) - - HADOOP-11217. Disable SSLv3 in KMS. (Robert Kanter via kasha) - - HADOOP-11068. Match hadoop.auth cookie format to jetty output. - (Gregory Chanan via cnauroth) - - HADOOP-11250. fix endmacro of set_find_shared_library_without_version in - CMakeLists (Yi Liu via Colin P. McCabe) - - HADOOP-11221. IdentityHashStore assumes System.identityHashCode() is - non-negative. (Jinghui Wang via szetszwo) - - HADOOP-11241. Fixed intermittent TestNMSimulator failure due to timing issue. - (Varun Vasudev via zjshen) - - HADOOP-11265. Credential and Key Shell Commands not available on Windows. - (Larry McCay via cnauroth) - - HADOOP-11280. TestWinUtils#testChmod fails after removal of - NO_PROPAGATE_INHERIT_ACE. (cnauroth) - - HADOOP-11282. Skip NFS TestShellBasedIdMapping tests that are irrelevant on - Windows. (cnauroth) - - HADOOP-9576. Changed NetUtils#wrapException to throw EOFException instead - of wrapping it as IOException. (Steve Loughran via jianhe) - - -Release 2.5.2 - 2014-11-19 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-11243. SSLFactory shouldn't allow SSLv3. (Wei Yan via kasha) - - HADOOP-11260. Patch up Jetty to disable SSLv3. (Mike Yoder via kasha) - - HADOOP-11307. create-release script should run git clean first (kasha) - -Release 2.5.1 - 2014-09-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-10956. Fix create-release script to include docs and necessary txt - files. (kasha) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-11001. Fix test-patch to work with the git repo. (kasha) - - HADOOP-11065. Rat check should exclude "**/build/**". (kasha) - -Release 2.5.0 - 2014-08-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HADOOP-10498. Add support for proxy server. (daryn) - - HADOOP-9704. Write metrics sink plugin for Hadoop/Graphite (Chu Tong, Alex Newman and Babak Behzad via raviprak) - - HADOOP-8943. Support multiple group mapping providers. (Kai Zheng via brandonli) - - HADOOP-9361 Strictly define the expected behavior of filesystem APIs and - write tests to verify compliance (stevel) - - IMPROVEMENTS - - HADOOP-10451. Remove unused field and imports from SaslRpcServer. - (Benoy Antony via jing9) - - HADOOP-10345. Sanitize the the inputs (groups and hosts) for the proxyuser - configuration. (Benoy Antony via jing9) - - HADOOP-10454. Provide FileContext version of har file system. (Kihwal Lee - via jeagles) - - HADOOP-10104. Update jackson to 1.9.13 (Akira Ajisaka via stevel) - - HADOOP-10503. Move junit up to v 4.11. (cnauroth) - - HADOOP-10535. Make the retry numbers in ActiveStandbyElector configurable. - (jing9) - - HADOOP-10322. Add ability to read principal names from a keytab. - (Benoy Antony and Daryn Sharp via kihwal) - - HADOOP-10549. MAX_SUBST and varPat should be final in Configuration.java. - (Gera Shegalov via cnauroth) - - HADOOP-10471. Reduce the visibility of constants in ProxyUsers. - (Benoy Antony via wheat9) - - HADOOP-10556. Add toLowerCase support to auth_to_local rules - for service name. (tucu) - - HADOOP-10467. Enable proxyuser specification to support list of users in - addition to list of groups (Benoy Antony via Arpit Agarwal) - - HADOOP-10158. SPNEGO should work with multiple interfaces/SPNs. - (daryn via kihwal) - - HADOOP-10566. Refactor proxyservers out of ProxyUsers. - (Benoy Antony via suresh) - - HADOOP-10572. Example NFS mount command must pass noacl as it isn't - supported by the server yet. (Harsh J via brandonli) - - HADOOP-10609. .gitignore should ignore .orig and .rej files. (kasha) - - HADOOP-10614. CBZip2InputStream is not threadsafe (Xiangrui Meng via - Sandy Ryza) - - HADOOP-10618. Remove SingleNodeSetup.apt.vm. (Akira Ajisaka via - Arpit Agarwal) - - HADOOP-9968. Makes ProxyUsers to work with NetGroups (Benoy Antony via - ddas) - - HADOOP-10448. Support pluggable mechanism to specify proxy user settings. - (Benoy Antony via Arpit Agarwal) - - HADOOP-9555. HA functionality that uses ZooKeeper may experience inadvertent - TCP RST and miss session expiration event due to bug in client connection - management. (cnauroth) - - HADOOP-10376. Refactor refresh*Protocols into a single generic - refreshConfigProtocol. (Chris Li via Arpit Agarwal) - - HADOOP-6350. Documenting Hadoop metrics. (Akira Ajisaka via Arpit Agarwal) - - HADOOP-10691. Improve the readability of 'hadoop fs -help'. - (Lei Xu via wang) - - HADOOP-10688. Expose thread-level FileSystem StatisticsData (Sandy Ryza) - - HADOOP-10657. Have RetryInvocationHandler log failover attempt at INFO - level. (Ming Ma via jing9) - - HADOOP-10666. Remove Copyright /d/d/d/d Apache Software Foundation from - the source files license header. (Henry Saputra via wang) - - HADOOP-10557. FsShell -cp -pa option for preserving extended ACLs. - (Akira Ajisaka via cnauroth) - - HADOOP-10279. Create multiplexer, a requirement for the fair queue. - (Chris Li via Arpit Agarwal) - - HADOOP-10659. Refactor AccessControlList to reuse utility functions - and to improve performance. (Benoy Antony via Arpit Agarwal) - - HADOOP-10665. Make Hadoop Authentication Handler loads case in-sensitive - (Benoy Antony via vinayakumarb) - - HADOOP-10652. Refactor Proxyusers to use AccessControlList. (Benoy - Antony via Arpit Agarwal) - - HADOOP-10747. Support configurable retries on SASL connection failures in - RPC client. (cnauroth) - - HADOOP-10754. Reenable several HA ZooKeeper-related tests on Windows. - (cnauroth) - - HADOOP-10565. Support IP ranges (CIDR) in proxyuser.hosts. (Benoy Antony - via Arpit Agarwal) - - HADOOP-10649. Allow overriding the default ACL for service authorization - (Benoy Antony via Arpit Agarwal) - - HADOOP-10767. Clean up unused code in Ls shell command. (cnauroth) - - HADOOP-9651 Filesystems to throw FileAlreadyExistsException in - createFile(path, overwrite=false) when the file exists (stevel) - - HADOOP-9495 Define behaviour of Seekable.seek(), write tests, - fix all hadoop implementations for compliance - - HADOOP-10312 Shell.ExitCodeException to have more useful toString (stevel) - - HADOOP-10782. Fix typo in DataChecksum class. (Jingguo Yao via suresh) - - HADOOP-10896. Update compatibility doc to capture visibility of - un-annotated classes/ methods. (kasha) - - OPTIMIZATIONS - - HADOOP-10674. Improve PureJavaCrc32 performance and use java.util.zip.CRC32 - for Java 7 and above. (szetszwo) - - BUG FIXES - - HADOOP-10378. Typo in help printed by hdfs dfs -help. - (Mit Desai via suresh) - - HADOOP-10418. SaslRpcClient should not assume that remote principals are in - the default_realm. (atm) - - HADOOP-10426. Declare CreateOpts.getOpt(..) with generic type argument, - removes unused FileContext.getFileStatus(..) and fixes various javac - warnings. (szetszwo) - - HADOOP-10414. Incorrect property name for RefreshUserMappingProtocol in - hadoop-policy.xml. (Joey Echeverria via atm) - - HADOOP-10459. distcp V2 doesn't preserve root dir's attributes when -p is - specified. (Yongjun Zhang via atm) - - HADOOP-10462. DF#getFilesystem is not parsing the command output. - (Akira AJISAKA via umamahesh) - - HADOOP-10468. TestMetricsSystemImpl.testMultiThreadedPublish fails - intermediately. (wheat9) - - HADOOP-10475. ConcurrentModificationException in - AbstractDelegationTokenSelector.selectToken(). (jing9) - - HADOOP-10350. BUILDING.txt should mention openssl dependency required - for hadoop-pipes (Vinayakumar B) - - HADOOP-10495. TestFileUtil fails on Windows due to bad permission - assertions. (cnauroth) - - HADOOP-10496. Metrics system FileSink can leak file descriptor. (cnauroth) - - HADOOP-10500. TestDoAsEffectiveUser fails on JDK7 due to failure to reset - proxy user configuration. (cnauroth) - - HADOOP-10499. Remove unused parameter from ProxyUsers.authorize(). - (Benoy Antony via cnauroth) - - HADOOP-9919. Update hadoop-metrics2.properties examples to Yarn. - (Akira AJISAKA via suresh) - - HADOOP-10526. Chance for Stream leakage in CompressorStream. (Rushabh - Shah via kihwal) - - HADOOP-10251. Both NameNodes could be in STANDBY State if SNN network is unstable - (Vinayakumar B via umamahesh) - - HADOOP-10531. hadoop-config.sh - bug in --hosts argument. - (Sebastien Barrier via wang) - - HADOOP-10539. Provide backward compatibility for ProxyUsers.authorize() - call. (Benoy Antony via cnauroth) - - HADOOP-10540. Datanode upgrade in Windows fails with hardlink error. - (Chris Nauroth and Arpit Agarwal) - - HADOOP-10508. RefreshCallQueue fails when authorization is enabled. - (Chris Li via wheat9) - - HADOOP-10547. Give SaslPropertiesResolver.getDefaultProperties() public - scope. (Benoy Antony via Arpit Agarwal) - - HADOOP-10543. RemoteException's unwrapRemoteException method failed for - PathIOException. (Yongjun Zhang via atm) - - HADOOP-10568. Add s3 server-side encryption. (David S. Wang via atm) - - HADOOP-10541. InputStream in MiniKdc#initKDCServer for minikdc.ldiff is not - closed. (Swarnim Kulkarni via cnauroth) - - HADOOP-10517. InputStream is not closed in two methods of JarFinder. - (Ted Yu via cnauroth) - - HADOOP-10581. TestUserGroupInformation#testGetServerSideGroups fails - because groups stored in Set and ArrayList are compared. - (Mit Desai via kihwal) - - HADOOP-10585. Retry polices ignore interrupted exceptions (Daryn Sharp via - jeagles) - - HADOOP-10401. ShellBasedUnixGroupsMapping#getGroups does not always return - primary group first (Akira AJISAKA via Colin Patrick McCabe) - - HADOOP-10489. UserGroupInformation#getTokens and UserGroupInformation - #addToken can lead to ConcurrentModificationException (Robert Kanter via atm) - - HADOOP-10602. Documentation has broken "Go Back" hyperlinks. - (Akira AJISAKA via cnauroth) - - HADOOP-10639. FileBasedKeyStoresFactory initialization is not using default - for SSL_REQUIRE_CLIENT_CERT_KEY. (tucu) - - HADOOP-10638. Updating hadoop-daemon.sh to work as expected when nfs is - started as a privileged user. (Manikandan Narayanaswamy via atm) - - HADOOP-10630. Possible race condition in RetryInvocationHandler. (jing9) - - HADOOP-10658. SSLFactory expects truststores being configured. (tucu via atm) - - HADOOP-10647. String Format Exception in SwiftNativeFileSystemStore.java. - (Gene Kim via stevel) - - HADOOP-9099. NetUtils.normalizeHostName fails on domains where - UnknownHost resolves to an IP address. (Ivan Mitic via suresh) - - HADOOP-10664. TestNetUtils.testNormalizeHostName fails. (atm) - - HADOOP-10656. The password keystore file is not picked by LDAP group mapping - (brandonli) - - HADOOP-10622. Shell.runCommand can deadlock (Gera Shegalov via jlowe) - - HADOOP-10686. Writables are not always configured. - (Abraham Elmahrek via kasha) - - HADOOP-10678. SecurityUtil has unnecessary synchronization on collection - used for only tests. (Benoy Antony via cnauroth) - - HADOOP-10683. Users authenticated with KERBEROS are recorded as being - authenticated with SIMPLE. (Benoy Antony via cnauroth) - - HADOOP-10702. KerberosAuthenticationHandler does not log the principal names - correctly. (Benoy Antony via cnauroth) - - HADOOP-10699. Fix build native library on mac osx (Binglin Chang via - jlowe) - - HADOOP-10660. GraphiteSink should implement Closeable (Chen He and Ted Yu via raviprak) - - HADOOP-10716. Cannot use more than 1 har filesystem. - (Rushabh Shah via cnauroth) - - HADOOP-9559. When metrics system is restarted MBean names get incorrectly - flagged as dupes. (Mostafa Elhemali and Mike Liddell via cnauroth) - - HADOOP-10746. TestSocketIOWithTimeout#testSocketIOWithTimeout fails on - Power PC. (Jinghui Wang via Arpit Agarwal) - - HADOOP-9705. FsShell cp -p does not preserve directory attibutes. - (Akira AJISAKA via cnauroth) - - HADOOP-10739. Renaming a file into a directory containing the same - filename results in a confusing I/O error (chang li via jlowe) - - HADOOP-10533 S3 input stream NPEs in MapReduce join (stevel) - - HADOOP-10419 BufferedFSInputStream NPEs on getPos() on a closed stream - (stevel) - - HADOOP-10801 dead link in site.xml (Akira AJISAKA via stevel) - - HADOOP-10590. ServiceAuthorizationManager is not threadsafe. (Benoy Antony via vinayakumarb) - - HADOOP-10711. Cleanup some extra dependencies from hadoop-auth. (rkanter via tucu) - - HADOOP-10479. Fix new findbugs warnings in hadoop-minikdc. - (Swarnim Kulkarni via wheat9) - - HADOOP-10715. Remove public GraphiteSink#setWriter (Babak Behzad via raviprak) - - HADOOP-10710. hadoop.auth cookie is not properly constructed according to - RFC2109. (Juan Yu via tucu) - - HADOOP-10864. Tool documentenation is broken. (Akira Ajisaka - via Arpit Agarwal) - - HADOOP-10872. TestPathData fails intermittently with "Mkdirs failed - to create d1". (Yongjun Zhang via Arpit Agarwal) - - HADOOP-10890. TestDFVariations.testMount fails intermittently. (Yongjun - Zhang via Arpit Agarwal) - - HADOOP-10894. Fix dead link in ToolRunner documentation. (Akira Ajisaka - via Arpit Agarwal) - - HADOOP-10910. Increase findbugs maxHeap size. (wang) - - BREAKDOWN OF HADOOP-10514 SUBTASKS AND RELATED JIRAS - - HADOOP-10520. Extended attributes definition and FileSystem APIs for - extended attributes. (Yi Liu via wang) - - HADOOP-10546. Javadoc and other small fixes for extended attributes in - hadoop-common. (Charles Lamb via wang) - - HADOOP-10521. FsShell commands for extended attributes. (Yi Liu via wang) - - HADOOP-10548. Improve FsShell xattr error handling and other fixes. (Charles Lamb via umamahesh) - - HADOOP-10567. Shift XAttr value encoding code out for reuse. (Yi Liu via umamahesh) - - HADOOP-10621. Remove CRLF for xattr value base64 encoding for better display.(Yi Liu via umamahesh) - - HADOOP-10575. Small fixes for XAttrCommands and test. (Yi Liu via umamahesh) - - HADOOP-10561. Copy command with preserve option should handle Xattrs. - (Yi Liu via cnauroth) - -Release 2.4.1 - 2014-06-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-10466. Lower the log level in UserGroupInformation. (Nicolas - Liochon via szetszwo) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-10455. When there is an exception, ipc.Server should first check - whether it is an terse exception. (szetszwo) - - HADOOP-10456. Bug in Configuration.java exposed by Spark - (ConcurrentModificationException). (Nishkam Ravi via cnauroth) - - HADOOP-10473. TestCallQueueManager should interrupt before counting calls. - (szetszwo) - - HADOOP-10490. TestMapFile and TestBloomMapFile leak file descriptors. - (cnauroth) - - HADOOP-10522. JniBasedUnixGroupMapping mishandles errors. (kihwal) - - HADOOP-10527. Fix incorrect return code and allow more retries on EINTR. - (kihwal) - - HADOOP-10612. NFS failed to refresh the user group id mapping table (brandonli) - - HADOOP-10562. Namenode exits on exception without printing stack trace - in AbstractDelegationTokenSecretManager. (Suresh Srinivas via Arpit - Agarwal) - -Release 2.4.0 - 2014-04-07 - - INCOMPATIBLE CHANGES - - HADOOP-8691. FsShell can print "Found xxx items" unnecessarily often. - (Daryn Sharp via wheat9) - - NEW FEATURES - - HADOOP-10184. Hadoop Common changes required to support HDFS ACLs. (See - breakdown of tasks below for features and contributors) - - IMPROVEMENTS - - HADOOP-10139. Update and improve the Single Cluster Setup document. - (Akira Ajisaka via Arpit Agarwal) - - HADOOP-10295. Allow distcp to automatically identify the checksum type of - source files and use it for the target. (jing9 and Laurent Goujon) - - HADOOP-10333. Fix grammatical error in overview.html document. - (René Nyffenegger via suresh) - - HADOOP-10343. Change info to debug log in LossyRetryInvocationHandler. (arpit) - - HADOOP-10348. Deprecate hadoop.ssl.configuration in branch-2, and remove - it in trunk. (Haohui Mai via jing9) - - HADOOP-9454. Support multipart uploads for s3native. (Jordan Mendelson and - Akira AJISAKA via atm) - - HADOOP-10374. InterfaceAudience annotations should have - RetentionPolicy.RUNTIME (Enis Soztutar via Arpit Agarwal) - - HADOOP-10278. Refactor to make CallQueue pluggable. (Chris Li via - Arpit Agarwal) - - HADOOP-10285. Admin interface to swap callqueue at runtime. (Chris Li via - Arpit Agarwal) - - HADOOP-10314. The ls command help still shows outdated 0.16 format. - (Rushabh S Shah via kihwal) - - HADOOP-10379. Protect authentication cookies with the HttpOnly and Secure - flags. (wheat9) - - HADOOP-10211. Enable RPC protocol to negotiate SASL-QOP values between - clients and servers. (Benoy Antony via Arpit Agarwal) - - HADOOP-10386. Log proxy hostname in various exceptions being thrown in a HA - setup. (wheat9) - - HADOOP-3679. Fixup assert ordering in unit tests to yield meaningful error - messages. (Jay Vyas via cdouglas) - - HADOOP-10221. Add a plugin to specify SaslProperties for RPC protocol - based on connection properties. (Benoy Antony and Daryn Sharp via - Arpit Agarwal) - - HADOOP-10015. UserGroupInformation prints out excessive warnings. - (Nicolas Liochon via szetszwo) - - HADOOP-10423. Clarify compatibility policy document for combination of - new client and old server. (Chris Nauroth via kasha) - - HADOOP-10280. Make Schedulables return a configurable identity of user - or group. (Chris Li via Arpit Agarwal) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-10320. Javadoc in InterfaceStability.java lacks final . - (René Nyffenegger via cnauroth) - - HADOOP-10085. CompositeService should allow adding services while being - inited. (Steve Loughran via kasha) - - HADOOP-10327. Trunk windows build broken after HDFS-5746. - (Vinay via cnauroth) - - HADOOP-10330. TestFrameDecoder fails if it cannot bind port 12345. - (Arpit Agarwal) - - HADOOP-10326. M/R jobs can not access S3 if Kerberos is enabled. (bc Wong - via atm) - - HADOOP-10338. Cannot get the FileStatus of the root inode from the new - Globber (cmccabe) - - HADOOP-10249. LdapGroupsMapping should trim ldap password read from file. - (Dilli Armugam via suresh) - - HADOOP-10346. Deadlock while logging tokens (jlowe) - - HADOOP-10328. loadGenerator exit code is not reliable. - (Haohui Mai via cnauroth) - - HADOOP-10355. Fix TestLoadGenerator#testLoadGenerator. (Haohui Mai via jing9) - - HADOOP-10070. RPC client doesn't use per-connection conf to determine - server's expected Kerberos principal name. (atm) - - HADOOP-10368. InputStream is not closed in VersionInfo ctor. - (Tsuyoshi OZAWA via szetszwo) - - HADOOP-10353. FsUrlStreamHandlerFactory is not thread safe. - (Tudor Scurtu via cnauroth) - - HADOOP-10393. Fix the javac warnings in hadoop-auth. (szetszwo) - - HADOOP-10395. TestCallQueueManager is flaky. (Arpit Agarwal) - - HADOOP-10394. TestAuthenticationFilter is flaky. (Arpit Agarwal) - - HADOOP-10337 ConcurrentModificationException from - MetricsDynamicMBeanBase.createMBeanInfo() (Liang Xie via stack) - - HADOOP-10407. Fix the javac warnings in org.apache.hadoop.ipc package. - (szetszwo) - - HADOOP-10191. Missing executable permission on viewfs internal dirs. - (Gera Shegalov via cnauroth) - - HADOOP-10422. Remove redundant logging of RPC retry attempts. (cnauroth) - - HADOOP-10425. LocalFileSystem.getContentSummary should not count crc files. - (szetszwo) - - HADOOP-10440. HarFsInputStream.read(byte[]) updates position incorrectly. - (guodongdong via szetszwo) - - HADOOP-10437. Fix the javac warnings in the conf and the util package. - (szetszwo) - - HADOOP-10441. Namenode metric "rpc.RetryCache/NameNodeRetryCache.CacheHit" - can't be correctly processed by Ganglia. (jing9) - - HADOOP-10449. Fix the javac warnings in the security package. (szetszwo) - - HADOOP-10450. Build zlib native code bindings in hadoop.dll for Windows. - (cnauroth) - - HADOOP-10301. AuthenticationFilter should return Forbidden for failed - authentication. (Daryn Sharp via jing9) - - HADOOP-9525. Add tests that validate winutils chmod behavior on folders - (ivanmi) - - BREAKDOWN OF HADOOP-10184 SUBTASKS AND RELATED JIRAS - - HADOOP-10185. FileSystem API for ACLs. (cnauroth) - - HADOOP-10186. Remove AclReadFlag and AclWriteFlag in FileSystem API. - (Haohui Mai via cnauroth) - - HADOOP-10187. FsShell CLI: add getfacl and setfacl with minimal support for - getting and setting ACLs. (Vinay via cnauroth) - - HADOOP-10192. FileSystem#getAclStatus has incorrect JavaDocs. (cnauroth) - - HADOOP-10220. Add ACL indicator bit to FsPermission. (cnauroth) - - HADOOP-10241. Clean up output of FsShell getfacl. (Chris Nauroth via wheat9) - - HADOOP-10213. Fix bugs parsing ACL spec in FsShell setfacl. - (Vinay via cnauroth) - - HADOOP-10277. setfacl -x fails to parse ACL spec if trying to remove the - mask entry. (Vinay via cnauroth) - - HADOOP-10270. getfacl does not display effective permissions of masked - entries. (cnauroth) - - HADOOP-10344. Fix TestAclCommands after merging HADOOP-10338 patch. - (cnauroth) - - HADOOP-10352. Recursive setfacl erroneously attempts to apply default ACL to - files. (cnauroth) - - HADOOP-10354. TestWebHDFS fails after merge of HDFS-4685 to trunk. (cnauroth) - - HADOOP-10361. Correct alignment in CLI output for ACLs. (cnauroth) - - HADOOP-10399. FileContext API for ACLs. (Vinayakumar B via cnauroth) - - HADOOP-10442. Group look-up can cause segmentation fault when certain - JNI-based mapping module is used. (Kihwal Lee via jeagles) - -Release 2.3.1 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.3.0 - 2014-02-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HADOOP-8545. Filesystem Implementation for OpenStack Swift - (Dmitry Mezhensky, David Dobbins, Stevel via stevel) - - IMPROVEMENTS - - HADOOP-10046. Print a log message when SSL is enabled. - (David S. Wang via wang) - - HADOOP-10079. log a warning message if group resolution takes too long. - (cmccabe) - - HADOOP-9623 Update jets3t dependency to 0.9.0. (Amandeep Khurana via Colin - Patrick McCabe) - - HADOOP-10132. RPC#stopProxy() should log the class of proxy when IllegalArgumentException - is encountered (Ted yu via umamahesh) - - HADOOP-10248. Property name should be included in the exception where property value - is null (Akira AJISAKA via umamahesh) - - HADOOP-10086. User document for authentication in secure cluster. - (Masatake Iwasaki via Arpit Agarwal) - - HADOOP-10274 Lower the logging level from ERROR to WARN for UGI.doAs method - (Takeshi Miao via stack) - - HADOOP-9784. Add a builder for HttpServer. (Junping Du via llu) - - HADOOP 9871. Fix intermittent findbugs warnings in DefaultMetricsSystem. - (Junping Du via llu) - - HADOOP-9319. Update bundled LZ4 source to r99. (Binglin Chang via llu) - - HADOOP-9241. DU refresh interval is not configurable (harsh) - - HADOOP-9417. Support for symlink resolution in LocalFileSystem / - RawLocalFileSystem. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-9703. org.apache.hadoop.ipc.Client leaks threads on stop. - (Tsuyoshi OZAWA vi Colin Patrick McCabe) - - HADOOP-9618. Add thread which detects GC pauses. - (Todd Lipcon via Colin Patrick McCabe) - - HADOOP-9435. Support building the JNI code against the IBM JVM. - (Tian Hong Wang via Colin Patrick McCabe) - - HADOOP-9848. Create a MiniKDC for use with security testing. - (ywskycn via tucu) - - HADOOP-9860. Remove class HackedKeytab and HackedKeytabEncoder from - hadoop-minikdc once jira DIRSERVER-1882 solved. (ywskycn via tucu) - - HADOOP-9866. convert hadoop-auth testcases requiring kerberos to - use minikdc. (ywskycn via tucu) - - HADOOP-9487 Deprecation warnings in Configuration should go to their - own log or otherwise be suppressible (Chu Tong via stevel) - - HADOOP-9889. Refresh the Krb5 configuration when creating a new kdc in - Hadoop-MiniKDC (Wei Yan via Sandy Ryza) - - HADOOP-9915. o.a.h.fs.Stat support on Mac OS X (Binglin Chang via Colin - Patrick McCabe) - - HADOOP-9998. Provide methods to clear only part of the DNSToSwitchMapping. - (Junping Du via Colin Patrick McCabe) - - HADOOP-10006. Compilation failure in trunk for - o.a.h.fs.swift.util.JSONUtil (Junping Du via stevel) - - HADOOP-9063. enhance unit-test coverage of class - org.apache.hadoop.fs.FileUtil (Ivan A. Veselovsky via jlowe) - - HADOOP-9254. Cover packages org.apache.hadoop.util.bloom, - org.apache.hadoop.util.hash (Vadim Bondarev via jlowe) - - HADOOP-9225. Cover package org.apache.hadoop.compress.Snappy (Vadim - Bondarev, Andrey Klochkov and Nathan Roberts via jlowe) - - HADOOP-9199. Cover package org.apache.hadoop.io with unit tests (Andrey - Klochkov via jeagles) - - HADOOP-9470. eliminate duplicate FQN tests in different Hadoop modules - (Ivan A. Veselovsky via daryn) - - HADOOP-9494. Excluded auto-generated and examples code from clover reports - (Andrey Klochkov via jeagles) - - HADOOP-9897. Add method to get path start position without drive specifier in - o.a.h.fs.Path. (Binglin Chang via cnauroth) - - HADOOP-9078. enhance unit-test coverage of class - org.apache.hadoop.fs.FileContext (Ivan A. Veselovsky via jeagles) - - HDFS-5276. FileSystem.Statistics should use thread-local counters to avoid - multi-threaded performance issues on read/write. (Colin Patrick McCabe) - - HADOOP-9291. enhance unit-test coverage of package o.a.h.metrics2 (Ivan A. - Veselovsky via jeagles) - - HADOOP-10064. Upgrade to maven antrun plugin version 1.7 (Arpit Agarwal via - jeagles) - - HADOOP-9594. Update apache commons math dependency (Timothy St. Clair via - stevel) - - HADOOP-10095. In CodecPool, synchronize pool and codecList separately in - order to reduce lock contention. (Nicolas Liochon via szetszwo) - - HADOOP-10067. Missing POM dependency on jsr305 (Robert Rati via stevel) - - HADOOP-10103. update commons-lang to 2.6 (Akira AJISAKA via stevel) - - HADOOP-10111. Allow DU to be initialized with an initial value (Kihwal Lee - via jeagles) - - HADOOP-10126. LightWeightGSet log message is confusing. (Vinay via suresh) - - HADOOP-10127. Add ipc.client.connect.retry.interval to control the frequency - of connection retries (Karthik Kambatla via Sandy Ryza) - - HADOOP-10102. Update commons IO from 2.1 to 2.4 (Akira Ajisaka via stevel) - - HADOOP-10168. fix javadoc of ReflectionUtils#copy. (Thejas Nair via suresh) - - HADOOP-10164. Allow UGI to login with a known Subject (bobby) - - HADOOP-10169. Remove the unnecessary synchronized in JvmMetrics class. - (Liang Xie via jing9) - - HADOOP-10198. DomainSocket: add support for socketpair. - (Colin Patrick McCabe via wang) - - HADOOP-10208. Remove duplicate initialization in StringUtils.getStringCollection. - (Benoy Antony via jing9) - - HADOOP-9420. Add percentile or max metric for rpcQueueTime, processing time. - (Liang Xie via wang) - - HADOOP-10143 replace WritableFactories's hashmap with ConcurrentHashMap - (Liang Xie via stack) - - HADOOP-9652. Allow RawLocalFs#getFileLinkStatus to fill in the link owner - and mode if requested. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-10305. Add "rpc.metrics.quantile.enable" and - "rpc.metrics.percentiles.intervals" to core-default.xml. - (Akira Ajisaka via wang) - - HADOOP-10317. Rename branch-2.3 release version from 2.4.0-SNAPSHOT - to 2.3.0-SNAPSHOT. (wang) - - HADOOP-10313. Script and jenkins job to produce Hadoop release artifacts. - (tucu) - - OPTIMIZATIONS - - HADOOP-10142. Avoid groups lookup for unprivileged users such as "dr.who" - (vinay via cmccabe) - - HADOOP-9748. Reduce blocking on UGI.ensureInitialized (daryn) - - HADOOP-10047. Add a direct-buffer based apis for compression. (Gopal V - via acmurthy) - - HADOOP-10172. Cache SASL server factories (daryn) - - HADOOP-10173. Remove UGI from DIGEST-MD5 SASL server creation (daryn via - kihwal) - - HADOOP-10228. FsPermission#fromShort() should cache FsAction.values(). - (Haohui Mai via cnauroth) - - BUG FIXES - - HADOOP-10028. Malformed ssl-server.xml.example. (Haohui Mai via jing9) - - HADOOP-10030. FsShell -put/copyFromLocal should support Windows local path. - (Chuan Liu via cnauroth) - - HADOOP-10031. FsShell -get/copyToLocal/moveFromLocal should support Windows - local path. (Chuan Liu via cnauroth) - - HADOOP-10039. Add Hive to the list of projects using - AbstractDelegationTokenSecretManager. (Haohui Mai via jing9) - - HADOOP-10040. hadoop.cmd in UNIX format and would not run by default on - Windows. (cnauroth) - - HADOOP-10055. FileSystemShell.apt.vm doc has typo "numRepicas". - (Akira Ajisaka via cnauroth) - - HADOOP-10072. TestNfsExports#testMultiMatchers fails due to non-deterministic - timing around cache expiry check. (cnauroth) - - HADOOP-9898. Set SO_KEEPALIVE on all our sockets. (todd via wang) - - HADOOP-9478. Fix race conditions during the initialization of Configuration - related to deprecatedKeyMap (cmccabe) - - HADOOP-9660. [WINDOWS] Powershell / cmd parses -Dkey=value from command line - as [-Dkey, value] which breaks GenericsOptionParser. - (Enis Soztutar via cnauroth) - - HADOOP-10078. KerberosAuthenticator always does SPNEGO. (rkanter via tucu) - - HADOOP-10110. hadoop-auth has a build break due to missing dependency. - (Chuan Liu via arp) - - HADOOP-9114. After defined the dfs.checksum.type as the NULL, write file and hflush will - through java.lang.ArrayIndexOutOfBoundsException (Sathish via umamahesh) - - HADOOP-10130. RawLocalFS::LocalFSFileInputStream.pread does not track - FS::Statistics (Binglin Chang via Colin Patrick McCabe) - - HDFS-5560. Trash configuration log statements prints incorrect units. - (Josh Elser via Andrew Wang) - - HADOOP-10081. Client.setupIOStreams can leak socket resources on exception - or error (Tsuyoshi OZAWA via jlowe) - - HADOOP-10087. UserGroupInformation.getGroupNames() fails to return primary - group first when JniBasedUnixGroupsMappingWithFallback is used (cmccabe) - - HADOOP-10175. Har files system authority should preserve userinfo. - (Chuan Liu via cnauroth) - - HADOOP-10090. Jobtracker metrics not updated properly after execution - of a mapreduce job. (ivanmi) - - HADOOP-10193. hadoop-auth's PseudoAuthenticationHandler can consume getInputStream. - (gchanan via tucu) - - HADOOP-10178. Configuration deprecation always emit "deprecated" warnings - when a new key is used. (Shanyu Zhao via cnauroth) - - HADOOP-10234. "hadoop.cmd jar" does not propagate exit code. (cnauroth) - - HADOOP-10240. Windows build instructions incorrectly state requirement of - protoc 2.4.1 instead of 2.5.0. (cnauroth) - - HADOOP-10167. Mark hadoop-common source as UTF-8 in Maven pom files / refactoring - (Mikhail Antonov via cos) - - HADOOP-9982. Fix dead links in hadoop site docs. (Akira Ajisaka via Arpit - Agarwal) - - HADOOP-10212. Incorrect compile command in Native Library document. - (Akira Ajisaka via Arpit Agarwal) - - HADOOP-9830. Fix typo at http://hadoop.apache.org/docs/current/ - (Kousuke Saruta via Arpit Agarwal) - - HADOOP-10255. Rename HttpServer to HttpServer2 to retain older - HttpServer in branch-2 for compatibility. (Haohui Mai via suresh) - - HADOOP-10291. TestSecurityUtil#testSocketAddrWithIP fails due to test - order dependency. (Mit Desai via Arpit Agarwal) - - HADOOP-9964. Fix deadlocks in TestHttpServer by synchronize - ReflectionUtils.printThreadInfo. (Junping Du via llu) - - HADOOP-9582. Non-existent file to "hadoop fs -conf" doesn't throw error - (Ashwin Shankar via jlowe) - - HADOOP-9817. FileSystem#globStatus and FileContext#globStatus need to work - with symlinks. (Colin Patrick McCabe via Andrew Wang) - - HADOOP-9875. TestDoAsEffectiveUser can fail on JDK 7. (Aaron T. Myers via - Colin Patrick McCabe) - - HADOOP-9865. FileContext#globStatus has a regression with respect to - relative path. (Chuan Lin via Colin Patrick McCabe) - - HADOOP-9909. org.apache.hadoop.fs.Stat should permit other LANG. - (Shinichi Yamashita via Andrew Wang) - - HADOOP-9908. Fix NPE when versioninfo properties file is missing (todd) - - HADOOP-9350. Hadoop not building against Java7 on OSX - (Robert Kanter via stevel) - - HADOOP-9929. Insufficient permissions for a path reported as file not found. - (Contributed by Colin Patrick McCabe) - - HADOOP-9791. Add a test case covering long paths for new FileUtil access - check methods (ivanmi) - - HADOOP-9981. globStatus should minimize its listStatus and getFileStatus - calls. (Contributed by Colin Patrick McCabe) - - HADOOP-9016. HarFsInputStream.skip(long) must never return negative value. - (Ivan A. Veselovsky via jeagles) - - HADOOP-10088. copy-nativedistlibs.sh needs to quote snappy lib dir. - (Raja Aluri via cnauroth) - - HADOOP-10093. hadoop-env.cmd sets HADOOP_CLIENT_OPTS with a max heap size - that is too small. (Shanyu Zhao via cnauroth) - - HADOOP-10094. NPE in GenericOptionsParser#preProcessForWindows(). - (Enis Soztutar via cnauroth) - - HADOOP-10100. MiniKDC shouldn't use apacheds-all artifact. (rkanter via tucu) - - HADOOP-10107. Server.getNumOpenConnections may throw NPE. (Kihwal Lee via - jing9) - - HADOOP-10135 writes to swift fs over partition size leave temp files and - empty output file (David Dobbins via stevel) - - HADOOP-10129. Distcp may succeed when it fails (daryn) - - HADOOP-10058. TestMetricsSystemImpl#testInitFirstVerifyStopInvokedImmediately - fails on trunk (Chen He via jeagles) - - HADOOP-8753. LocalDirAllocator throws "ArithmeticException: / by zero" when - there is no available space on configured local dir. (Benoy Antony via hitesh) - - HADOOP-10106. Incorrect thread name in RPC log messages. (Ming Ma via jing9) - - HADOOP-9611 mvn-rpmbuild against google-guice > 3.0 yields missing cglib - dependency (Timothy St. Clair via stevel) - - HADOOP-10171. TestRPC fails intermittently on jkd7 (Mit Desai via jeagles) - - HADOOP-10147 HDFS-5678 Upgrade to commons-logging 1.1.3 to avoid potential - deadlock in MiniDFSCluster (stevel) - - HADOOP-10207. TestUserGroupInformation#testLogin is flaky (jxiang via cmccabe) - - HADOOP-10214. Fix multithreaded correctness warnings in ActiveStandbyElector - (Liang Xie via kasha) - - HADOOP-10223. MiniKdc#main() should close the FileReader it creates. - (Ted Yu via tucu) - - HADOOP-10236. Fix typo in o.a.h.ipc.Client#checkResponse. (Akira Ajisaka - via suresh) - - HADOOP-10146. Workaround JDK7 Process fd close bug (daryn) - - HADOOP-10125. no need to process RPC request if the client connection - has been dropped (Ming Ma via brandonli) - - HADOOP-10235. Hadoop tarball has 2 versions of stax-api JARs. (tucu) - - HADOOP-10252. HttpServer can't start if hostname is not specified. (Jimmy - Xiang via atm) - - HADOOP-10203. Connection leak in - Jets3tNativeFileSystemStore#retrieveMetadata. (Andrei Savu via atm) - - HADOOP-10250. VersionUtil returns wrong value when comparing two versions. - (Yongjun Zhang via atm) - - HADOOP-10288. Explicit reference to Log4JLogger breaks non-log4j users - (todd) - - HADOOP-10310. SaslRpcServer should be initialized even when no secret - manager present. (atm) - - HADOOP-10311. Cleanup vendor names from the code base. (tucu) - - HADOOP-10273. Fix 'mvn site'. (Arpit Agarwal) - -Release 2.2.0 - 2013-10-13 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-9948. Add a config value to CLITestHelper to skip tests on Windows. - (Chuan Liu via cnauroth) - - HADOOP-9976. Different versions of avro and avro-maven-plugin (Karthik - Kambatla via Sandy Ryza) - - HADOOP-9758. Provide configuration option for FileSystem/FileContext - symlink resolution. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-8315. Support SASL-authenticated ZooKeeper in ActiveStandbyElector - (todd) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9776. HarFileSystem.listStatus() returns invalid authority if port - number is empty. (Shanyu Zhao via ivanmi) - - HADOOP-9761. ViewFileSystem#rename fails when using DistributedFileSystem. - (Andrew Wang via Colin Patrick McCabe) - - HADOOP-10003. HarFileSystem.listLocatedStatus() fails. - (Jason Dere and suresh via suresh) - - HADOOP-10017. Fix NPE in DFSClient#getDelegationToken when doing Distcp - from a secured cluster to an insecured cluster. (Haohui Mai via jing9) - -Release 2.1.1-beta - 2013-09-23 - - INCOMPATIBLE CHANGES - - HADOOP-9944. Fix RpcRequestHeaderProto.callId to be sint32 rather than - uint32 since ipc.Client.CONNECTION_CONTEXT_CALL_ID is signed (i.e. -3) - (acmurthy) - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-9910. proxy server start and stop documentation wrong - (Andre Kelpe via harsh) - - HADOOP-9446. Support Kerberos SPNEGO for IBM JDK. (Yu Gao via llu) - - HADOOP-9787. ShutdownHelper util to shutdown threads and threadpools. - (Karthik Kambatla via Sandy Ryza) - - HADOOP-9803. Add a generic type parameter to RetryInvocationHandler. - (szetszwo) - - HADOOP-9821. ClientId should have getMsb/getLsb methods. - (Tsuyoshi OZAWA via jing9) - - HADOOP-9672. Upgrade Avro dependency to 1.7.4. (sandy via kihwal) - - HADOOP-9789. Support server advertised kerberos principals (daryn) - - HADOOP-8814. Replace string equals "" by String#isEmpty(). - (Brandon Li via suresh) - - HADOOP-9802. Support Snappy codec on Windows. (cnauroth) - - HADOOP-9879. Move the version info of zookeeper dependencies to - hadoop-project/pom (Karthik Kambatla via Sandy Ryza) - - HADOOP-9886. Turn warning message in RetryInvocationHandler to debug (arpit) - - HADOOP-9906. Move HAZKUtil to o.a.h.util.ZKUtil and make inner-classes - public (Karthik Kambatla via Sandy Ryza) - - HADOOP-9918. Add addIfService to CompositeService (Karthik Kambatla via - Sandy Ryza) - - HADOOP-9945. HAServiceState should have a state for stopped services. - (Karthik Kambatla via atm) - - HADOOP-9962. in order to avoid dependency divergence within Hadoop itself - lets enable DependencyConvergence. (rvs via tucu) - - HADOOP-9669. Reduce the number of byte array creations and copies in - XDR data manipulation. (Haohui Mai via brandonli) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9916. Fix race in ipc.Client retry. (Binglin Chang via llu) - - HADOOP-9768. chown and chgrp reject users and groups with spaces on platforms - where spaces are otherwise acceptable. (cnauroth) - - HADOOP-9801. Configuration#writeXml uses platform defaulting encoding, which - may mishandle multi-byte characters. (cnauroth) - - HADOOP-9806 PortmapInterface should check if the procedure is out-of-range - (brandonli) - - HADOOP-9527. Add symlink support to LocalFileSystem on Windows. - (Arpit Agarwal via cnauroth) - - HADOOP-9831. Make checknative shell command accessible on Windows. (cnauroth) - - HADOOP-9675. use svn:eol-style native for html to prevent line ending - issues (Colin Patrick McCabe) - - HADOOP-9757. Har metadata cache can grow without limit (Cristina Abad via daryn) - - HADOOP-9857. Tests block and sometimes timeout on Windows due to invalid - entropy source. (cnauroth) - - HADOOP-9381. Document dfs cp -f option. (Keegan Witt, suresh via suresh) - - HADOOP-9868. Server must not advertise kerberos realm. (daryn via kihwal) - - HADOOP-9880. SASL changes from HADOOP-9421 breaks Secure HA NN. (daryn via - jing9) - - HADOOP-9887. globStatus does not correctly handle paths starting with a drive - spec on Windows. (Chuan Liu via cnauroth) - - HADOOP-9894. Race condition in Shell leads to logged error stream handling - exceptions (Arpit Agarwal) - - HADOOP-9774. RawLocalFileSystem.listStatus() return absolute paths when - input path is relative on Windows. (Shanyu Zhao via ivanmi) - - HADOOP-9924. FileUtil.createJarWithClassPath() does not generate relative - classpath correctly. (Shanyu Zhao via ivanmi) - - HADOOP-9932. Improper synchronization in RetryCache. (kihwal) - - HADOOP-9958. Add old constructor back to DelegationTokenInformation to - unbreak downstream builds. (Andrew Wang) - - HADOOP-9960. Upgrade Jersey version to 1.9. (Karthik Kambatla via atm) - - HADOOP-9557. hadoop-client excludes commons-httpclient. (Lohit Vijayarenu via - cnauroth) - - HADOOP-9961. versions of a few transitive dependencies diverged between hadoop - subprojects. (rvs via tucu) - - HADOOP-9977. Hadoop services won't start with different keypass and - keystorepass when https is enabled. (cnauroth) - - HADOOP-10005. No need to check INFO severity level is enabled or not. - (Jackie Chang via suresh) - -Release 2.1.0-beta - 2013-08-22 - - INCOMPATIBLE CHANGES - - HADOOP-8886. Remove KFS support. (eli) - - HADOOP-9163. [RPC v9] The rpc msg in ProtobufRpcEngine.proto should be moved out to - avoid an extra copy (Sanjay Radia) - - HADOOP-9151. [RPC v9] Include RPC error info in RpcResponseHeader instead of sending - it separately (sanjay Radia) - - HADOOP-9380. [RPC v9] Add totalLength to rpc response (sanjay Radia) - - HADOOP-9425. [RPC v9] Add error codes to rpc-response (sanjay Radia) - - HADOOP-9194. [RPC v9] RPC support for QoS. (Junping Du via llu) - - HADOOP-9630. [RPC v9] Remove IpcSerializationType. (Junping Du via llu) - - HADOOP-9421. [RPC v9] Convert SASL to use ProtoBuf and provide - negotiation capabilities (daryn) - - HADOOP-9688. Add globally unique Client ID to RPC requests. (suresh) - - HADOOP-9683. [RPC v9] Wrap IpcConnectionContext in RPC headers (daryn) - - HADOOP-9698. [RPC v9] Client must honor server's SASL negotiate response (daryn) - - HADOOP-9832. [RPC v9] Add RPC header to client ping (daryn) - - HADOOP-9820. [RPC v9] Wire protocol is insufficient to support multiplexing. (daryn via jitendra) - - NEW FEATURES - - HADOOP-9283. Add support for running the Hadoop client on AIX. (atm) - - HADOOP-8415. Add getDouble() and setDouble() in - org.apache.hadoop.conf.Configuration (Jan van der Lugt via harsh) - - HADOOP-9338. FsShell Copy Commands Should Optionally Preserve File - Attributes. (Nick White via atm) - - HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows - Azure environments. (See breakdown of tasks below for subtasks and - contributors) - - HADOOP-8469. Make NetworkTopology class pluggable. (Junping Du via - szetszwo) - - HADOOP-8470. Add NetworkTopologyWithNodeGroup, a 4-layer implementation - of NetworkTopology. (Junping Du via szetszwo) - - HADOOP-9763. Extends LightWeightGSet to support eviction of expired - elements. (Tsz Wo (Nicholas) SZE via jing9) - - HADOOP-9762. RetryCache utility for implementing RPC retries. - (Suresh Srinivas via jing9) - - HADOOP-9792. Retry the methods that are tagged @AtMostOnce along - with @Idempotent. (suresh) - - HADOOP-9509. Implement ONCRPC and XDR. (brandonli) - - HADOOP-9515. Add general interface for NFS and Mount. (brandonli) - - IMPROVEMENTS - - HADOOP-9164. Print paths of loaded native libraries in - NativeLibraryChecker. (Binglin Chang via llu) - - HADOOP-9253. Capture ulimit info in the logs at service start time. - (Arpit Gupta via suresh) - - HADOOP-8924. Add maven plugin alternative to shell script to save - package-info.java. (Chris Nauroth via suresh) - - HADOOP-9117. replace protoc ant plugin exec with a maven plugin. (tucu) - - HADOOP-9279. Document the need to build hadoop-maven-plugins for - eclipse and separate project builds. (Tsuyoshi Ozawa via suresh) - - HADOOP-9334. Upgrade netty version. (Nicolas Liochon via suresh) - - HADOOP-9343. Allow additional exceptions through the RPC layer. (sseth) - - HADOOP-9318. When exiting on a signal, print the signal name first. (Colin - Patrick McCabe via atm) - - HADOOP-9358. "Auth failed" log should include exception string (todd) - - HADOOP-9401. CodecPool: Add counters for number of (de)compressors - leased out. (kkambatl via tucu) - - HADOOP-9450. HADOOP_USER_CLASSPATH_FIRST is not honored; CLASSPATH - is PREpended instead of APpended. (Chris Nauroth and harsh via harsh) - - HADOOP-9503. Remove sleep between IPC client connect timeouts. - (Varun Sharma via szetszwo) - - HADOOP-9322. LdapGroupsMapping doesn't seem to set a timeout for - its directory search. (harsh) - - HADOOP-9523. Provide a generic IBM java vendor flag in PlatformName.java - to support non-Sun JREs. (Tian Hong Wang via suresh) - - HADOOP-9511. Adding support for additional input streams (FSDataInputStream - and RandomAccessFile) in SecureIOUtils so as to help YARN-578. (Omkar Vinit - Joshi via vinodkv) - - HADOOP-9560. metrics2#JvmMetrics should have max memory size of JVM. - (Tsuyoshi Ozawa via suresh) - - HADOOP-9140 Cleanup rpc PB protos (sanjay Radia) - - HADOOP-9218 Document the Rpc-wrappers used internally (sanjay Radia) - - HADOOP-9574. Added new methods in AbstractDelegationTokenSecretManager for - helping YARN ResourceManager to reuse code for RM restart. (Jian He via - vinodkv) - - HADOOP-7391 Document Interface Classification from HADOOP-5073 (sanjay Radia) - - HADOOP-9287. Parallel-testing hadoop-common (Andrey Klochkov via jlowe) - - HADOOP-9604. Javadoc of FSDataOutputStream is slightly inaccurate. (Jingguo - Yao via atm) - - HADOOP-9625. HADOOP_OPTS not picked up by hadoop command. - (Paul Han via arpit) - - HADOOP-9649. Promoted YARN service life-cycle libraries into Hadoop Common - for usage across all Hadoop projects. (Zhijie Shen via vinodkv) - - HADOOP-9517. Documented various aspects of compatibility for Apache - Hadoop. (Karthik Kambatla via acmurthy) - - HADOOP-8608. Add Configuration API for parsing time durations. (cdouglas) - - HADOOP-9619 Mark stability of .proto files (sanjay Radia) - - HADOOP-9676. Make maximum RPC buffer size configurable (Colin Patrick - McCabe) - - HADOOP-9691. RPC clients can generate call ID using AtomicInteger instead of - synchronizing on the Client instance. (cnauroth) - - HADOOP-9661. Allow metrics sources to be extended. (sandyr via tucu) - - HADOOP-9370. Write FSWrapper class to wrap FileSystem and FileContext for - better test coverage. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-9355. Abstract symlink tests to use either FileContext or - FileSystem. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-9673. NetworkTopology: when a node can't be added, print out its - location for diagnostic purposes. (Colin Patrick McCabe) - - HADOOP-9414. Refactor out FSLinkResolver and relevant helper methods. - (Andrew Wang via Colin Patrick McCabe) - - HADOOP-9416. Add new symlink resolution methods in FileSystem and - FileSystemLinkResolver. (Andrew Wang via Colin Patrick McCabe) - - HADOOP-9720. Rename Client#uuid to Client#clientId. (Arpit Agarwal via - suresh) - - HADOOP-9734. Common protobuf definitions for GetUserMappingsProtocol, - RefreshAuthorizationPolicyProtocol and RefreshUserMappingsProtocol (jlowe) - - HADOOP-9716. Rpc retries should use the same call ID as the original call. - (szetszwo) - - HADOOP-9717. Add retry attempt count to the RPC requests. (jing9) - - HADOOP-9751. Add clientId and retryCount to RpcResponseHeaderProto. - (szetszwo) - - HADOOP-9754. Remove unnecessary "throws IOException/InterruptedException", - and fix generic and other javac warnings. (szetszwo) - - HADOOP-9760. Move GSet and related classes to common from HDFS. - (suresh) - - HADOOP-9756. Remove the deprecated getServer(..) methods from RPC. - (Junping Du via szetszwo) - - HADOOP-9770. Make RetryCache#state non volatile. (suresh) - - HADOOP-9786. RetryInvocationHandler#isRpcInvocation should support - ProtocolTranslator. (suresh and jing9) - - OPTIMIZATIONS - - HADOOP-9150. Avoid unnecessary DNS resolution attempts for logical URIs - (todd) - - HADOOP-9845. Update protobuf to 2.5 from 2.4.x. (tucu) - - HADOOP-9872. Improve protoc version handling and detection. (tucu) - - BUG FIXES - - HADOOP-9294. GetGroupsTestBase fails on Windows. (Chris Nauroth via suresh) - - HADOOP-9305. Add support for running the Hadoop client on 64-bit AIX. (atm) - - HADOOP-9245. mvn clean without running mvn install before fails. - (Karthik Kambatla via suresh) - - HADOOP-9246 Execution phase for hadoop-maven-plugin should be - process-resources (Karthik Kambatla and Chris Nauroth via jlowe) - - HADOOP-9297. remove old record IO generation and tests. (tucu) - - HADOOP-9154. SortedMapWritable#putAll() doesn't add key/value classes to - the map. (Karthik Kambatla via tomwhite) - - HADOOP-9304. remove addition of avro genreated-sources dirs to build. (tucu) - - HADOOP-9267. hadoop -help, -h, --help should show usage instructions. - (Andrew Wang via atm) - - HADOOP-8569. CMakeLists.txt: define _GNU_SOURCE and _LARGEFILE_SOURCE. - (Colin Patrick McCabe via atm) - - HADOOP-9323. Fix typos in API documentation. (suresh) - - HADOOP-7487. DF should throw a more reasonable exception when mount cannot - be determined. (Andrew Wang via atm) - - HADOOP-8917. add LOCALE.US to toLowerCase in SecurityUtil.replacePattern. - (Arpit Gupta via suresh) - - HADOOP-9342. Remove jline from distribution. (thw via tucu) - - HADOOP-9230. TestUniformSizeInputFormat fails intermittently. - (kkambatl via tucu) - - HADOOP-9349. Confusing output when running hadoop version from one hadoop - installation when HADOOP_HOME points to another. (sandyr via tucu) - - HADOOP-9337. org.apache.hadoop.fs.DF.getMount() does not work on Mac OS. - (Ivan A. Veselovsky via atm) - - HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the - end. (Karthik Kambatla via atm) - - HADOOP-9379. capture the ulimit info after printing the log to the - console. (Arpit Gupta via suresh) - - HADOOP-9399. protoc maven plugin doesn't work on mvn 3.0.2 (todd) - - HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing - build issues. (Sangjin Lee via suresh) - - HADOOP-9405. TestGridmixSummary#testExecutionSummarizer is broken. (Andrew - Wang via atm) - - HADOOP-9430. TestSSLFactory fails on IBM JVM. (Amir Sanjar via suresh) - - HADOOP-9125. LdapGroupsMapping threw CommunicationException after some - idle time. (Kai Zheng via atm) - - HADOOP-9429. TestConfiguration fails with IBM JAVA. (Amir Sanjar via - suresh) - - HADOOP-9222. Cover package with org.apache.hadoop.io.lz4 unit tests (Vadim - Bondarev via jlowe) - - HADOOP-9233. Cover package org.apache.hadoop.io.compress.zlib with unit - tests (Vadim Bondarev via jlowe) - - HADOOP-9211. Set default max heap size in HADOOP_CLIENT_OPTS to 512m - in order to avoid OOME. (Plamen Jeliazkov via shv) - - HADOOP-9473. Typo in FileUtil copy() method. (Glen Mazza via suresh) - - HADOOP-9504. MetricsDynamicMBeanBase has concurrency issues in - createMBeanInfo (Liang Xie via jlowe) - - HADOOP-9455. HADOOP_CLIENT_OPTS appended twice causes JVM failures. - (Chris Nauroth via suresh) - - HADOOP-9550. Remove aspectj dependency. (kkambatl via tucu) - - HADOOP-9549. WebHdfsFileSystem hangs on close(). (daryn via kihwal) - - HADOOP-9485. No default value in the code for - hadoop.rpc.socket.factory.class.default. (Colin Patrick McCabe via atm) - - HADOOP-9459. ActiveStandbyElector can join election even before - Service HEALTHY, and results in null data at ActiveBreadCrumb. - (Vinay and todd via todd) - - HADOOP-9307. BufferedFSInputStream.read returns wrong results - after certain seeks. (todd) - - HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. - (tom and todd via todd) - - HADOOP-9563. Fix incompatibility introduced by HADOOP-9523. - (Tian Hong Wang via suresh) - - HADOOP-9566. Performing direct read using libhdfs sometimes raises SIGPIPE - (which in turn throws SIGABRT) causing client crashes. (Colin Patrick - McCabe via atm) - - HADOOP-9481. Broken conditional logic with HADOOP_SNAPPY_LIBRARY. (Vadim - Bondarev via atm) - - HADOOP-9593. stack trace printed at ERROR for all yarn clients without - hadoop.home set (stevel) - - HADOOP-8957. AbstractFileSystem#IsValidName should be overridden for - embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia) - - HADOOP-9607. Fixes in Javadoc build (Timothy St. Clair via cos) - - HADOOP-9605. Update junit dependency. (Timothy St. Clair via cos) - - HADOOP-9581. hadoop --config non-existent directory should result in error - (Ashwin Shankar via jlowe) - - HADOOP-9638. Parallel test changes caused invalid test path for several HDFS - tests on Windows (Andrey Klochkov via cnauroth) - - HADOOP-9632. TestShellCommandFencer will fail if there is a 'host' machine in - the network. (Chuan Liu via cnauroth) - - HADOOP-9624. TestFSMainOperationsLocalFileSystem failed when the Hadoop test - root path has "X" in its name. (Xi Fang via cnauroth) - - HADOOP-9439. JniBasedUnixGroupsMapping: fix some crash bugs (Colin Patrick - McCabe) - - HADOOP-9656. Gridmix unit tests fail on Windows and Linux. (Chuan Liu via - cnauroth) - - HADOOP-9707. Fix register lists for crc32c inline assembly. (todd via - kihwal) - - HADOOP-9738. TestDistCh fails. (jing9 via kihwal) - - HADOOP-9759. Add support for NativeCodeLoader#getLibraryName on Windows. - (Chuan Liu via cnauroth) - - HADOOP-9773. TestLightWeightCache should not set size limit to zero when - testing it. (szetszwo) - - HADOOP-9507. LocalFileSystem rename() is broken in some cases when - destination exists. (cnauroth) - - HADOOP-9816. RPC Sasl QOP is broken (daryn) - - HADOOP-9850. RPC kerberos errors don't trigger relogin. (daryn via kihwal) - - BREAKDOWN OF HADOOP-8562 SUBTASKS AND RELATED JIRAS - - HADOOP-8924. Hadoop Common creating package-info.java must not depend on - sh. (Chris Nauroth via suresh) - - HADOOP-8945. Merge winutils from branch-1-win to branch-trunk-win. - (Bikas Saha, Chuan Liu, Giridharan Kesavan, Ivan Mitic, and Steve Maine - ported by Chris Nauroth via suresh) - - HADOOP-8946. winutils: compile codebase during Maven build on - branch-trunk-win. (Chris Nauroth via suresh) - - HADOOP-8947. Merge FileUtil and Shell changes from branch-1-win to - branch-trunk-win to enable initial test pass. (Raja Aluri, Davio Lao, - Sumadhur Reddy Bolli, Ahmed El Baz, Kanna Karanam, Chuan Liu, - Ivan Mitic, Chris Nauroth, and Bikas Saha via suresh) - - HADOOP-8954. "stat" executable not found on Windows. (Bikas Saha, Ivan Mitic - ported by Chris Narouth via suresh) - - HADOOP-8959. TestUserGroupInformation fails on Windows due to "id" executable - not found. (Bikas Saha, Ivan Mitic, ported by Chris Narouth via suresh) - - HADOOP-8955. "chmod" executable not found on Windows. - (Chris Nauroth via suresh) - - HADOOP-8960. TestMetricsServlet fails on Windows. (Ivan Mitic via suresh) - - HADOOP-8961. GenericOptionsParser URI parsing failure on Windows. - (Ivan Mitic via suresh) - - HADOOP-8949. Remove FileUtil.CygPathCommand dead code. (Chris Nauroth via - suresh) - - HADOOP-8956. FileSystem.primitiveMkdir failures on Windows cause multiple - test suites to fail. (Chris Nauroth via suresh) - - HADOOP-8978. TestTrash fails on Windows. (Chris Nauroth via suresh) - - HADOOP-8979. TestHttpServer fails on Windows. (Chris Nauroth via suresh) - - HADOOP-8953. Shell PathData parsing failures on Windows. (Arpit Agarwal via - suresh) - - HADOOP-8975. TestFileContextResolveAfs fails on Windows. (Chris Nauroth via - suresh) - - HADOOP-8977. Multiple FsShell test failures on Windows. (Chris Nauroth via - suresh) - - HADOOP-9005. Merge hadoop cmd line scripts from branch-1-win. (David Lao, - Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh) - - HADOOP-9008. Building hadoop tarball fails on Windows. (Chris Nauroth via - suresh) - - HADOOP-9011. saveVersion.py does not include branch in version annotation. - (Chris Nauroth via suresh) - - HADOOP-9110. winutils ls off-by-one error indexing MONTHS array can cause - access violation. (Chris Nauroth via suresh) - - HADOOP-9056. Build native library on Windows. (Chuan Liu, Arpit Agarwal via - suresh) - - HADOOP-9144. Fix findbugs warnings. (Chris Nauroth via suresh) - - HADOOP-9081. Add TestWinUtils. (Chuan Liu, Ivan Mitic, Chris Nauroth, - and Bikas Saha via suresh) - - HADOOP-9146. Fix sticky bit regression on branch-trunk-win. - (Chris Nauroth via suresh) - - HADOOP-9266. Fix javac, findbugs, and release audit warnings on - branch-trunk-win. (Chris Nauroth via suresh) - - HADOOP-9270. Remove a stale java comment from FileUtil. (Chris Nauroth via - szetszwo) - - HADOOP-9271. Revert Python build scripts from branch-trunk-win. - (Chris Nauroth via suresh) - - HADOOP-9313. Remove spurious mkdir from hadoop-config.cmd. - (Ivan Mitic via suresh) - - HADOOP-9309. Test failures on Windows due to UnsatisfiedLinkError - in NativeCodeLoader#buildSupportsSnappy. (Arpit Agarwal via suresh) - - HADOOP-9347. Add instructions to BUILDING.txt describing how to - build on Windows. (Chris Nauroth via suresh) - - HADOOP-9348. Address TODO in winutils to add more command line usage - and examples. (Chris Nauroth via suresh) - - HADOOP-9354. Windows native project files missing license headers. - (Chris Nauroth via suresh) - - HADOOP-9356. Remove remaining references to cygwin/cygpath from scripts. - (Chris Nauroth via suresh) - - HADOOP-9232. JniBasedUnixGroupsMappingWithFallback fails on Windows - with UnsatisfiedLinkError. (Ivan Mitic via suresh) - - HADOOP-9368. Add timeouts to new tests in branch-trunk-win. - (Arpit Agarwal via suresh) - - HADOOP-9373. Merge CHANGES.branch-trunk-win.txt to CHANGES.txt. - (suresh) - - HADOOP-9372. Fix bad timeout annotations on tests. - (Arpit Agarwal via suresh) - - HADOOP-9376. TestProxyUserFromEnv fails on a Windows domain joined machine. - (Ivan Mitic via suresh) - - HADOOP-9365. TestHAZKUtil fails on Windows. (Ivan Mitic via suresh) - - HADOOP-9364. PathData#expandAsGlob does not return correct results for - absolute paths on Windows. (Ivan Mitic via suresh) - - HADOOP-8973. DiskChecker cannot reliably detect an inaccessible disk on - Windows with NTFS ACLs. (Chris Nauroth via suresh) - - HADOOP-9388. TestFsShellCopy fails on Windows. (Ivan Mitic via suresh) - - HADOOP-9387. Fix DF so that it won't execute a shell command on Windows - to compute the file system/mount point. (Ivan Mitic via szetszwo) - - HADOOP-9353. Activate native-win maven profile by default on Windows. - (Arpit Agarwal via szetszwo) - - HADOOP-9437. TestNativeIO#testRenameTo fails on Windows due to assumption - that POSIX errno is embedded in NativeIOException. (Chris Nauroth via - suresh) - - HADOOP-9443. Port winutils static code analysis change to trunk. - (Chuan Liu via suresh) - - HADOOP-9290. Some tests cannot load native library on windows. - (Chris Nauroth via suresh) - - HADOOP-9500. TestUserGroupInformation#testGetServerSideGroups fails on - Windows due to failure to find winutils.exe. (Chris Nauroth via suresh) - - HADOOP-9490. LocalFileSystem#reportChecksumFailure not closing the - checksum file handle before rename. (Ivan Mitic via suresh) - - HADOOP-9524. Fix ShellCommandFencer to work on Windows. - (Arpit Agarwal via suresh) - - HADOOP-9413. Add common utils for File#setReadable/Writable/Executable & - File#canRead/Write/Execute that work cross-platform. (Ivan Mitic via suresh) - - HADOOP-9532. HADOOP_CLIENT_OPTS is appended twice by Windows cmd scripts. - (Chris Nauroth via suresh) - - HADOOP-9043. Disallow in winutils creating symlinks with forwards slashes. - (Chris Nauroth and Arpit Agarwal via suresh) - - HADOOP-9483. winutils support for readlink command. - (Arpit Agarwal via suresh) - - HADOOP-9488. FileUtil#createJarWithClassPath only substitutes environment - variables from current process environment/does not support overriding - when launching new process (Chris Nauroth via bikas) - - HADOOP-9556. disable HA tests on Windows that fail due to ZooKeeper client - connection management bug. (Chris Nauroth via suresh) - - HADOOP-9553. TestAuthenticationToken fails on Windows. - (Arpit Agarwal via suresh) - - HADOOP-9397. Incremental dist tar build fails. (Chris Nauroth via jlowe) - - HADOOP-9131. Turn off TestLocalFileSystem#testListStatusWithColons on - Windows. (Chris Nauroth via suresh) - - HADOOP-9526. TestShellCommandFencer and TestShell fail on Windows. - (Arpit Agarwal via suresh) - - HADOOP-8982. TestSocketIOWithTimeout fails on Windows. - (Chris Nauroth via suresh) - - HADOOP-8958. ViewFs:Non absolute mount name failures when running - multiple tests on Windows. (Chris Nauroth via suresh) - - HADOOP-9599. hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly. - (Mostafa Elhemali via ivanmi) - - HADOOP-9637. Adding Native Fstat for Windows as needed by YARN. (Chuan Liu - via cnauroth) - - HADOOP-9264. Port change to use Java untar API on Windows from - branch-1-win to trunk. (Chris Nauroth via suresh) - - HADOOP-9678. TestRPC#testStopsAllThreads intermittently fails on Windows. - (Ivan Mitic via cnauroth) - - HADOOP-9681. FileUtil.unTarUsingJava() should close the InputStream upon - finishing. (Chuan Liu via cnauroth) - - HADOOP-9665. Fixed BlockDecompressorStream#decompress to return -1 rather - than throw EOF at end of file. (Zhijie Shen via acmurthy) - - HADOOP-8440. HarFileSystem.decodeHarURI fails for URIs whose host contains - numbers. (Ivan Mitic via cnauroth) - - HADOOP-9643. org.apache.hadoop.security.SecurityUtil calls - toUpperCase(Locale.getDefault()) as well as toLowerCase(Locale.getDefault()) - on hadoop.security.authentication value. (markrmiller@gmail.com via tucu) - - HADOOP-9701. mvn site ambiguous links in hadoop-common. (kkambatl via tucu) - -Release 2.0.6-alpha - 08/22/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.0.5-alpha - 06/06/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9407. commons-daemon 1.0.3 dependency has bad group id causing - build issues. (Sangjin Lee via suresh) - -Release 2.0.4-alpha - 2013-04-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9467. Metrics2 record filter should check name as well as tags. - (Chris Nauroth and Ganeshan Iyler via llu) - - HADOOP-9406. hadoop-client leaks dependency on JDK tools jar. (tucu) - - HADOOP-9301. hadoop client servlet/jsp/jetty/tomcat JARs creating - conflicts in Oozie & HttpFS. (tucu) - - HADOOP-9299. kerberos name resolution is kicking in even when kerberos - is not configured (daryn) - - HADOOP-9408. misleading description for net.topology.table.file.name - property in core-default.xml. (rajeshbabu via suresh) - - HADOOP-9444. Modify hadoop-policy.xml to replace unexpanded variables to a - default value of '*'. (Roman Shaposhnik via vinodkv) - - HADOOP-9471. hadoop-client wrongfully excludes jetty-util JAR, - breaking webhdfs. (tucu) - -Release 2.0.3-alpha - 2013-02-06 - - INCOMPATIBLE CHANGES - - HADOOP-8999. SASL negotiation is flawed (daryn) - - NEW FEATURES - - HADOOP-8561. Introduce HADOOP_PROXY_USER for secure impersonation in child - hadoop client processes. (Yu Gao via llu) - - HADOOP-8597. Permit FsShell's text command to read Avro files. - (Ivan Vladimirov Ivanov via cutting) - - HADOOP-9020. Add a SASL PLAIN server (daryn via bobby) - - HADOOP-9090. Support on-demand publish of metrics. (Mostafa Elhemali via - suresh) - - HADOOP-9054. Add AuthenticationHandler that uses Kerberos but allows for - an alternate form of authentication for browsers. (rkanter via tucu) - - IMPROVEMENTS - - HADOOP-8789. Tests setLevel(Level.OFF) should be Level.ERROR. - (Andy Isaacson via eli) - - HADOOP-8755. Print thread dump when tests fail due to timeout. (Andrey - Klochkov via atm) - - HADOOP-8806. libhadoop.so: dlopen should be better at locating - libsnappy.so, etc. (Colin Patrick McCabe via eli) - - HADOOP-8812. ExitUtil#terminate should print Exception#toString. (eli) - - HADOOP-8736. Add Builder for building RPC server. (Brandon Li via Suresh) - - HDFS-3957. Change MutableQuantiles to use a shared thread for rolling - over metrics. (Andrew Wang via todd) - - HADOOP-8851. Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked - tests. (Ivan A. Veselovsky via atm) - - HADOOP-8783. Improve RPC.Server's digest auth (daryn) - - HADOOP-8889. Upgrade to Surefire 2.12.3 (todd) - - HADOOP-8804. Improve Web UIs when the wildcard address is used. - (Senthil Kumar via eli) - - HADOOP-8894. GenericTestUtils.waitFor should dump thread stacks on timeout - (todd) - - HADOOP-8909. Hadoop Common Maven protoc calls must not depend on external - sh script. (Chris Nauroth via suresh) - - HADOOP-8911. CRLF characters in source and text files. - (Raja Aluri via suresh) - - HADOOP-8912. Add .gitattributes file to prevent CRLF and LF mismatches - for source and text files. (Raja Aluri via suresh) - - HADOOP-8784. Improve IPC.Client's token use (daryn) - - HADOOP-8929. Add toString, other improvements for SampleQuantiles (todd) - - HADOOP-8922. Provide alternate JSONP output for JMXJsonServlet to allow - javascript in browser (Damien Hardy via bobby) - - HADOOP-8931. Add Java version to startup message. (eli) - - HADOOP-8925. Remove the packaging. (eli) - - HADOOP-8985. Add namespace declarations in .proto files for languages - other than java. (Binglin Chan via suresh) - - HADOOP-9009. Add SecurityUtil methods to get/set authentication method - (daryn via bobby) - - HADOOP-9010. Map UGI authenticationMethod to RPC authMethod (daryn via - bobby) - - HADOOP-9013. UGI should not hardcode loginUser's authenticationType (daryn - via bobby) - - HADOOP-9014. Standardize creation of SaslRpcClients (daryn via bobby) - - HADOOP-9015. Standardize creation of SaslRpcServers (daryn via bobby) - - HADOOP-8860. Split MapReduce and YARN sections in documentation navigation. - (tomwhite via tucu) - - HADOOP-9021. Enforce configured SASL method on the server (daryn via - bobby) - - HADOO-8998. set Cache-Control no-cache header on all dynamic content. (tucu) - - HADOOP-9035. Generalize setup of LoginContext (daryn via bobby) - - HADOOP-9093. Move all the Exception in PathExceptions to o.a.h.fs package. - (suresh) - - HADOOP-9042. Add a test for umask in FileSystemContractBaseTest. - (Colin McCabe via eli) - - HADOOP-9127. Update documentation for ZooKeeper Failover Controller. - (Daisuke Kobayashi via atm) - - HADOOP-9004. Allow security unit tests to use external KDC. (Stephen Chu - via suresh) - - HADOOP-9147. Add missing fields to FIleStatus.toString. - (Jonathan Allen via suresh) - - HADOOP-8427. Convert Forrest docs to APT, incremental. (adi2 via tucu) - - HADOOP-9162. Add utility to check native library availability. - (Binglin Chang via suresh) - - HADOOP-9173. Add security token protobuf definition to common and - use it in hdfs. (suresh) - - HADOOP-9119. Add test to FileSystemContractBaseTest to verify integrity - of overwritten files. (Steve Loughran via suresh) - - HADOOP-9192. Move token related request/response messages to common. - (suresh) - - HADOOP-8712. Change default hadoop.security.group.mapping to - JniBasedUnixGroupsNetgroupMappingWithFallback (Robert Parker via todd) - - HADOOP-9106. Allow configuration of IPC connect timeout. - (Rober Parker via suresh) - - HADOOP-9216. CompressionCodecFactory#getCodecClasses should trim the - result of parsing by Configuration. (Tsuyoshi Ozawa via todd) - - HADOOP-9231. Parametrize staging URL for the uniformity of - distributionManagement. (Konstantin Boudnik via suresh) - - HADOOP-9276. Allow BoundedByteArrayOutputStream to be resettable. - (Arun Murthy via hitesh) - - HADOOP-7688. Add servlet handler check in HttpServer.start(). - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7886. Add toString to FileStatus. (SreeHari via jghoman) - - OPTIMIZATIONS - - HADOOP-8866. SampleQuantiles#query is O(N^2) instead of O(N). (Andrew Wang - via atm) - - HADOOP-8926. hadoop.util.PureJavaCrc32 cache hit-ratio is low for static - data (Gopal V via bobby) - - BUG FIXES - - HADOOP-9041. FsUrlStreamHandlerFactory could cause an infinite loop in - FileSystem initialization. (Yanbo Liang and Radim Kolar via llu) - - HADOOP-8418. Update UGI Principal classes name for running with - IBM JDK on 64 bits Windows. (Yu Gao via eyang) - - HADOOP-8795. BASH tab completion doesn't look in PATH, assumes path to - executable is specified. (Sean Mackrory via atm) - - HADOOP-8780. Update DeprecatedProperties apt file. (Ahmed Radwan via - tomwhite) - - HADOOP-8833. fs -text should make sure to call inputstream.seek(0) - before using input stream. (tomwhite and harsh) - - HADOOP-8791. Fix rm command documentation to indicte it deletes - files and not directories. (Jing Zhao via suresh) - - HADOOP-8855. SSL-based image transfer does not work when Kerberos - is disabled. (todd via eli) - - HADOOP-8616. ViewFS configuration requires a trailing slash. (Sandy Ryza - via atm) - - HADOOP-8756. Fix SEGV when libsnappy is in java.library.path but - not LD_LIBRARY_PATH. (Colin Patrick McCabe via eli) - - HADOOP-8881. FileBasedKeyStoresFactory initialization logging should - be debug not info. (tucu) - - HADOOP-8913. hadoop-metrics2.properties should give units in comment - for sampling period. (Sandy Ryza via suresh) - - HADOOP-8878. Uppercase namenode hostname causes hadoop dfs calls with - webhdfs filesystem and fsck to fail when security is on. - (Arpit Gupta via suresh) - - HADOOP-8901. GZip and Snappy support may not work without unversioned - libraries (Colin Patrick McCabe via todd) - - HADOOP-8883. Anonymous fallback in KerberosAuthenticator is broken. - (rkanter via tucu) - - HADOOP-8900. BuiltInGzipDecompressor throws IOException - stored gzip size - doesn't match decompressed size. (Andy Isaacson via suresh) - - HADOOP-8948. TestFileUtil.testGetDU fails on Windows due to incorrect - assumption of line separator. (Chris Nauroth via suresh) - - HADOOP-8951. RunJar to fail with user-comprehensible error - message if jar missing. (stevel via suresh) - - HADOOP-8713. TestRPCCompatibility fails intermittently with JDK7 - (Trevor Robinson via tgraves) - - HADOOP-9012. IPC Client sends wrong connection context (daryn via bobby) - - HADOOP-7115. Add a cache for getpwuid_r and getpwgid_r calls (tucu) - - HADOOP-6607. Add different variants of non caching HTTP headers. (tucu) - - HADOOP-9049. DelegationTokenRenewer needs to be Singleton and FileSystems - should register/deregister to/from. (Karthik Kambatla via tomwhite) - - HADOOP-9064. Augment DelegationTokenRenewer API to cancel the tokens on - calls to removeRenewAction. (kkambatl via tucu) - - HADOOP-9103. UTF8 class does not properly decode Unicode characters - outside the basic multilingual plane. (todd) - - HADOOP-9070. Kerberos SASL server cannot find kerberos key. (daryn via atm) - - HADOOP-6762. Exception while doing RPC I/O closes channel - (Sam Rash and todd via todd) - - HADOOP-9126. FormatZK and ZKFC startup can fail due to zkclient connection - establishment delay. (Rakesh R and todd via todd) - - HADOOP-9113. o.a.h.fs.TestDelegationTokenRenewer is failing intermittently. - (Karthik Kambatla via eli) - - HADOOP-9135. JniBasedUnixGroupsMappingWithFallback should log at debug - rather than info during fallback. (Colin Patrick McCabe via todd) - - HADOOP-9152. HDFS can report negative DFS Used on clusters with very small - amounts of data. (Brock Noland via atm) - - HADOOP-9153. Support createNonRecursive in ViewFileSystem. - (Sandy Ryza via tomwhite) - - HADOOP-9181. Set daemon flag for HttpServer's QueuedThreadPool. - (Liang Xie via suresh) - - HADOOP-9155. FsPermission should have different default value, 777 for - directory and 666 for file. (Binglin Chang via atm) - - HADOOP-9183. Potential deadlock in ActiveStandbyElector. (tomwhite) - - HADOOP-9203. RPCCallBenchmark should find a random available port. - (Andrew Purtell via suresh) - - HADOOP-9178. src/main/conf is missing hadoop-policy.xml. - (Sandy Ryza via eli) - - HADOOP-8816. HTTP Error 413 full HEAD if using kerberos authentication. - (moritzmoeller via tucu) - - HADOOP-9212. Potential deadlock in FileSystem.Cache/IPC/UGI. (tomwhite) - - HADOOP-8589 ViewFs tests fail when tests and home dirs are nested. - (sanjay Radia) - - HADOOP-9193. hadoop script can inadvertently expand wildcard arguments - when delegating to hdfs script. (Andy Isaacson via todd) - - HADOOP-9215. when using cmake-2.6, libhadoop.so doesn't get created - (only libhadoop.so.1.0.0) (Colin Patrick McCabe via todd) - - HADOOP-8857. hadoop.http.authentication.signature.secret.file docs - should not state that secret is randomly generated. (tucu) - - HADOOP-9190. packaging docs is broken. (Andy Isaacson via tgraves) - - HADOOP-9221. Convert remaining xdocs to APT. (Andy Isaacson via atm) - - HADOOP-8981. TestMetricsSystemImpl fails on Windows. (Xuan Gong via suresh) - - HADOOP-9124. SortedMapWritable violates contract of Map interface for - equals() and hashCode(). (Surenkumar Nihalani via tomwhite) - - HADOOP-9278. Fix the file handle leak in HarMetaData.parseMetaData() in - HarFileSystem. (Chris Nauroth via szetszwo) - - HADOOP-9252. In StringUtils, humanReadableInt(..) has a race condition and - the synchronization of limitDecimalTo2(double) can be avoided. (szetszwo) - - HADOOP-9260. Hadoop version may be not correct when starting name node or - data node. (Chris Nauroth via jlowe) - - HADOOP-9289. FsShell rm -f fails for non-matching globs. (Daryn Sharp via - suresh) - -Release 2.0.2-alpha - 2012-09-07 - - INCOMPATIBLE CHANGES - - HADOOP-8388. Remove unused BlockLocation serialization. - (Colin Patrick McCabe via eli) - - HADOOP-8689. Make trash a server side configuration option. (eli) - - HADOOP-8710. Remove ability for users to easily run the trash emptire. (eli) - - HADOOP-8794. Rename YARN_HOME to HADOOP_YARN_HOME. (vinodkv via acmurthy) - - NEW FEATURES - - HDFS-3042. Automatic failover support for NameNode HA (todd) - (see dedicated section below for breakdown of subtasks) - - HADOOP-8135. Add ByteBufferReadable interface to FSDataInputStream. (Henry - Robinson via atm) - - HADOOP-8458. Add management hook to AuthenticationHandler to enable - delegation token operations support (tucu) - - HADOOP-8465. hadoop-auth should support ephemeral authentication (tucu) - - HADOOP-8644. AuthenticatedURL should be able to use SSLFactory. (tucu) - - HADOOP-8581. add support for HTTPS to the web UIs. (tucu) - - HADOOP-7754. Expose file descriptors from Hadoop-wrapped local - FileSystems (todd and ahmed via tucu) - - HADOOP-8240. Add a new API to allow users to specify a checksum type - on FileSystem.create(..). (Kihwal Lee via szetszwo) - - IMPROVEMENTS - - HADOOP-8340. SNAPSHOT build versions should compare as less than their eventual - final release. (todd) - - HADOOP-8361. Avoid out-of-memory problems when deserializing strings. - (Colin Patrick McCabe via eli) - - HADOOP-8224. Don't hardcode hdfs.audit.logger in the scripts. - (Tomohiko Kinebuchi via eli) - - HADOOP-8398. Cleanup BlockLocation. (eli) - - HADOOP-8422. Deprecate FileSystem#getDefault* and getServerDefault - methods that don't take a Path argument. (eli) - - HADOOP-8323. Add javadoc and tests for Text.clear() behavior (harsh) - - HADOOP-8358. Config-related WARN for dfs.web.ugi can be avoided. (harsh) - - HADOOP-8450. Remove src/test/system. (eli) - - HADOOP-8244. Improve comments on ByteBufferReadable.read. (Henry Robinson - via atm) - - HADOOP-8368. Use CMake rather than autotools to build native code (ccccabe via tucu) - - HADOOP-8524. Allow users to get source of a Configuration - parameter (harsh) - - HADOOP-8449. hadoop fs -text fails with compressed sequence files - with the codec file extension (harsh) - - HADOOP-6802. Remove FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" - from CommonConfigurationKeys.java (not used, deprecated) - (Sho Shimauchi via harsh) - - HADOOP-3450. Add tests to Local Directory Allocator for - asserting their URI-returning capability (Sho Shimauchi via harsh) - - HADOOP-8463. hadoop.security.auth_to_local needs a key definition and doc. - (Madhukara Phatak via eli) - - HADOOP-8533. Remove parallel call ununsed capability in RPC. - (Brandon Li via suresh) - - HADOOP-8423. MapFile.Reader.get() crashes jvm or throws - EOFException on Snappy or LZO block-compressed data - (todd via harsh) - - HADOOP-8541. Better high-percentile latency metrics. (Andrew Wang via atm) - - HADOOP-8362. Improve exception message when Configuration.set() is - called with a null key or value. (Madhukara Phatak - and Suresh Srinivas via harsh) - - HADOOP-7818. DiskChecker#checkDir should fail if the directory is - not executable. (Madhukara Phatak via harsh) - - HADOOP-8531. SequenceFile Writer can throw out a better error if a - serializer or deserializer isn't available - (Madhukara Phatak via harsh) - - HADOOP-8609. IPC server logs a useless message when shutting down socket. - (Jon Zuanich via atm) - - HADOOP-8620. Add -Drequire.fuse and -Drequire.snappy. (Colin - Patrick McCabe via eli) - - HADOOP-8687. Upgrade log4j to 1.2.17. (eli) - - HADOOP-8278. Make sure components declare correct set of dependencies. - (tomwhite) - - HADOOP-8700. Use enum to define the checksum constants in DataChecksum. - (szetszwo) - - HADOOP-8686. Fix warnings in native code. (Colin Patrick McCabe via eli) - - HADOOP-8239. Add subclasses of MD5MD5CRC32FileChecksum to support file - checksum with CRC32C. (Kihwal Lee via szetszwo) - - HADOOP-8619. WritableComparator must implement no-arg constructor. - (Chris Douglas via Suresh) - - HADOOP-8075. Lower native-hadoop library log from info to debug. - (Hızır Sefa İrken via eli) - - HADOOP-8748. Refactor DFSClient retry utility methods to a new class - in org.apache.hadoop.io.retry. (Arun C Murthy via szetszwo) - - HADOOP-8754. Deprecate all the RPC.getServer() variants. (Brandon Li - via szetszwo) - - HADOOP-8801. ExitUtil#terminate should capture the exception stack trace. (eli) - - HADOOP-8819. Incorrectly & is used instead of && in some file system - implementations. (Brandon Li via suresh) - - HADOOP-7808. Port HADOOP-7510 - Add configurable option to use original - hostname in token instead of IP to allow server IP change. - (Daryn Sharp via suresh) - - HADOOP-8367 Improve documentation of declaringClassProtocolName in - rpc headers. (Sanjay Radia) - - HADOOP-8624. ProtobufRpcEngine should log all RPCs if TRACE logging is - enabled (todd) - - BUG FIXES - - HADOOP-8372. NetUtils.normalizeHostName() incorrectly handles hostname - starting with a numeric character. (Junping Du via suresh) - - HADOOP-8393. hadoop-config.sh missing variable exports, causes Yarn jobs - to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) - - HADOOP-8316. Audit logging should be disabled by default. (eli) - - HADOOP-8400. All commands warn "Kerberos krb5 configuration not found" - when security is not enabled. (tucu) - - HADOOP-8406. CompressionCodecFactory.CODEC_PROVIDERS iteration is - thread-unsafe (todd) - - HADOOP-8287. etc/hadoop is missing hadoop-env.sh (eli) - - HADOOP-8408. MR doesn't work with a non-default ViewFS mount table - and security enabled. (atm via eli) - - HADOOP-8329. Build fails with Java 7. (eli) - - HADOOP-8268. A few pom.xml across Hadoop project - may fail XML validation. (Radim Kolar via harsh) - - HADOOP-8444. Fix the tests FSMainOperationsBaseTest.java and - FileContextMainOperationsBaseTest.java to avoid potential - test failure (Madhukara Phatak via harsh) - - HADOOP-8452. DN logs backtrace when running under jsvc and /jmx is loaded - (Andy Isaacson via bobby) - - HADOOP-8460. Document proper setting of HADOOP_PID_DIR and - HADOOP_SECURE_DN_PID_DIR (bobby) - - HADOOP-8466. hadoop-client POM incorrectly excludes avro. (bmahe via tucu) - - HADOOP-8481. update BUILDING.txt to talk about cmake rather than autotools. - (Colin Patrick McCabe via eli) - - HADOOP-8485. Don't hardcode "Apache Hadoop 0.23" in the docs. (eli) - - HADOOP-8488. test-patch.sh gives +1 even if the native build fails. - (Colin Patrick McCabe via eli) - - HADOOP-8507. Avoid OOM while deserializing DelegationTokenIdentifer. - (Colin Patrick McCabe via eli) - - HADOOP-8433. Don't set HADOOP_LOG_DIR in hadoop-env.sh. - (Brahma Reddy Battula via eli) - - HADOOP-8509. JarFinder duplicate entry: META-INF/MANIFEST.MF exception (tucu) - - HADOOP-8512. AuthenticatedURL should reset the Token when the server returns - other than OK on authentication (tucu) - - HADOOP-8168. empty-string owners or groups causes {{MissingFormatWidthException}} - in o.a.h.fs.shell.Ls.ProcessPath() (ekoontz via tucu) - - HADOOP-8438. hadoop-validate-setup.sh refers to examples jar file which doesn't exist - (Devaraj K via umamahesh) - - HADOOP-8538. CMake builds fail on ARM. (Trevor Robinson via eli) - - HADOOP-8547. Package hadoop-pipes examples/bin directory (again). - (Colin Patrick McCabe via eli) - - HADOOP-8563. don't package hadoop-pipes examples/bin - (Colin Patrick McCabe via tgraves) - - HADOOP-8566. AvroReflectSerializer.accept(Class) throws a NPE if the class has no - package (primitive types and arrays). (tucu) - - HADOOP-8586. Fixup a bunch of SPNEGO misspellings. (eli) - - HADOOP-3886. Error in javadoc of Reporter, Mapper and Progressable - (Jingguo Yao via harsh) - - HADOOP-8587. HarFileSystem access of harMetaCache isn't threadsafe. (eli) - - HADOOP-8585. Fix initialization circularity between UserGroupInformation - and HadoopConfiguration. (Colin Patrick McCabe via atm) - - HADOOP-8552. Conflict: Same security.log.file for multiple users. - (kkambatl via tucu) - - HADOOP-8537. Fix TFile tests to pass even when native zlib support is not - compiled. (todd) - - HADOOP-8626. Typo in default setting for - hadoop.security.group.mapping.ldap.search.filter.user. (Jonathan Natkins - via atm) - - HADOOP-8480. The native build should honor -DskipTests. - (Colin Patrick McCabe via eli) - - HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM - on ARM. (Trevor Robinson via todd) - - HADOOP-8654. TextInputFormat delimiter bug (Gelesh and Jason Lowe via - bobby) - - HADOOP-8614. IOUtils#skipFully hangs forever on EOF. - (Colin Patrick McCabe via eli) - - HADOOP-8720. TestLocalFileSystem should use test root subdirectory. - (Vlad Rozov via eli) - - HADOOP-8721. ZKFC should not retry 45 times when attempting a graceful - fence during a failover. (Vinayakumar B via atm) - - HADOOP-8632. Configuration leaking class-loaders (Costin Leau via bobby) - - HADOOP-4572. Can not access user logs - Jetty is not configured by default - to serve aliases/symlinks (ahmed via tucu) - - HADOOP-8660. TestPseudoAuthenticator failing with NPE. (tucu) - - HADOOP-8699. some common testcases create core-site.xml in test-classes - making other testcases to fail. (tucu) - - HADOOP-8031. Configuration class fails to find embedded .jar resources; - should use URL.openStream() (genman via tucu) - - HADOOP-8737. cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h. - (Colin Patrick McCabe via eli) - - HADOOP-8747. Syntax error on cmake version 2.6 patch 2 in JNIFlags.cmake. (cmccabe via tucu) - - HADOOP-8722. Update BUILDING.txt with latest snappy info. - (Colin Patrick McCabe via eli) - - HADOOP-8764. CMake: HADOOP-8737 broke ARM build. (Trevor Robinson via eli) - - HADOOP-8770. NN should not RPC to self to find trash defaults. (eli) - - HADOOP-8648. libhadoop: native CRC32 validation crashes when - io.bytes.per.checksum=1. (Colin Patrick McCabe via eli) - - HADOOP-8766. FileContextMainOperationsBaseTest should randomize the root - dir. (Colin Patrick McCabe via atm) - - HADOOP-8749. HADOOP-8031 changed the way in which relative xincludes are handled in - Configuration. (ahmed via tucu) - - HADOOP-8431. Running distcp wo args throws IllegalArgumentException. - (Sandy Ryza via eli) - - HADOOP-8775. MR2 distcp permits non-positive value to -bandwidth option - which causes job never to complete. (Sandy Ryza via atm) - - HADOOP-8781. hadoop-config.sh should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH. (tucu) - - BREAKDOWN OF HDFS-3042 SUBTASKS - - HADOOP-8220. ZKFailoverController doesn't handle failure to become active - correctly (todd) - - HADOOP-8228. Auto HA: Refactor tests and add stress tests. (todd) - - HADOOP-8215. Security support for ZK Failover controller (todd) - - HADOOP-8245. Fix flakiness in TestZKFailoverController (todd) - - HADOOP-8257. TestZKFailoverControllerStress occasionally fails with Mockito - error (todd) - - HADOOP-8260. Replace ClientBaseWithFixes with our own modified copy of the - class (todd) - - HADOOP-8246. Auto-HA: automatically scope znode by nameservice ID (todd) - - HADOOP-8247. Add a config to enable auto-HA, which disables manual - FailoverController (todd) - - HADOOP-8306. ZKFC: improve error message when ZK is not running. (todd) - - HADOOP-8279. Allow manual failover to be invoked when auto-failover is - enabled. (todd) - - HADOOP-8276. Auto-HA: add config for java options to pass to zkfc daemon - (todd via eli) - - HADOOP-8405. ZKFC tests leak ZK instances. (todd) - -Release 2.0.0-alpha - 05-23-2012 - - INCOMPATIBLE CHANGES - - HADOOP-7920. Remove Avro Rpc. (suresh) - - NEW FEATURES - - HADOOP-7773. Add support for protocol buffer based RPC engine. - (suresh) - - HADOOP-7875. Add helper class to unwrap protobuf ServiceException. - (suresh) - - HADOOP-7454. Common side of High Availability Framework (HDFS-1623) - Contributed by Todd Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, - Bikas Saha, Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, - Sanjay Radia, Mingjie Lai, and Gregory Chanan - - HADOOP-8121. Active Directory Group Mapping Service. (Jonathan Natkins via - atm) - - HADOOP-7030. Add TableMapping topology implementation to read host to rack - mapping from a file. (Patrick Angeles and tomwhite via tomwhite) - - HADOOP-8206. Common portion of a ZK-based failover controller (todd) - - HADOOP-8210. Common side of HDFS-3148: The client should be able - to use multiple local interfaces for data transfer. (eli) - - HADOOP-8343. Allow configuration of authorization for JmxJsonServlet and - MetricsServlet (tucu) - - IMPROVEMENTS - - HADOOP-7524. Change RPC to allow multiple protocols including multuple - versions of the same protocol (sanjay Radia) - - HADOOP-7607. Simplify the RPC proxy cleanup process. (atm) - - HADOOP-7687. Make getProtocolSignature public (sanjay) - - HADOOP-7693. Enhance AvroRpcEngine to support the new #addProtocol - interface introduced in HADOOP-7524. (cutting) - - HADOOP-7716. RPC protocol registration on SS does not log the protocol name - (only the class which may be different) (sanjay) - - HADOOP-7776. Make the Ipc-Header in a RPC-Payload an explicit header. - (sanjay) - - HADOOP-7862. Move the support for multiple protocols to lower layer so - that Writable, PB and Avro can all use it (Sanjay) - - HADOOP-7876. Provided access to encoded key in DelegationKey for - use in protobuf based RPCs. (suresh) - - HADOOP-7899. Generate proto java files as part of the build. (tucu) - - HADOOP-7957. Classes deriving GetGroupsBase should be able to override - proxy creation. (jitendra) - - HADOOP-7965. Support for protocol version and signature in PB. (jitendra) - - HADOOP-8070. Add a standalone benchmark for RPC call performance. (todd) - - HADOOP-8084. Updates ProtoBufRpc engine to not do an unnecessary copy - for RPC request/response. (ddas) - - HADOOP-8085. Add RPC metrics to ProtobufRpcEngine. (Hari Mankude via - suresh) - - HADOOP-8098. KerberosAuthenticatorHandler should use _HOST replacement to - resolve principal name (tucu) - - HADOOP-8118. In metrics2.util.MBeans, change log level to trace for the - stack trace of InstanceAlreadyExistsException. (szetszwo) - - HADOOP-8125. make hadoop-client set of curated jars available in a - distribution tarball (rvs via tucu) - - HADOOP-7717. Move handling of concurrent client fail-overs to - RetryInvocationHandler (atm) - - HADOOP-7728. Enable task memory management to be configurable in hadoop - config setup script. (ramya) - - HADOOP-7358. Improve log levels when exceptions caught in RPC handler - (Todd Lipcon via shv) - - HADOOP-7557 Make IPC header be extensible (sanjay radia) - - HADOOP-7806. Support binding to sub-interfaces (eli) - - HADOOP-6941. Adds support for building Hadoop with IBM's JDK - (Stephen Watt, Eli and ddas) - - HADOOP-8183. Stop using "mapred.used.genericoptions.parser" (harsh) - - HADOOP-6924. Adds a directory to the list of directories to search - for the libjvm.so file. The new directory is found by running a 'find' - command and the first output is taken. This was done to handle the - build of Hadoop with IBM's JDK. (Stephen Watt, Guillermo Cabrera and ddas) - - HADOOP-8200. Remove HADOOP_[JOBTRACKER|TASKTRACKER]_OPTS. (eli) - - HADOOP-8184. ProtoBuf RPC engine uses the IPC layer reply packet. - (Sanjay Radia via szetszwo) - - HADOOP-8163. Improve ActiveStandbyElector to provide hooks for - fencing old active. (todd) - - HADOOP-8193. Refactor FailoverController/HAAdmin code to add an abstract - class for "target" services. (todd) - - HADOOP-8212. Improve ActiveStandbyElector's behavior when session expires - (todd) - - HADOOP-8216. Address log4j.properties inconsistencies btw main and - template dirs. (Patrick Hunt via eli) - - HADOOP-8149. Cap space usage of default log4j rolling policy. - (Patrick Hunt via eli) - - HADOOP-8211. Update commons-net version to 3.1. (eli) - - HADOOP-8236. haadmin should have configurable timeouts for failover - commands. (todd) - - HADOOP-8242. AbstractDelegationTokenIdentifier: add getter methods - for owner and realuser. (Colin Patrick McCabe via eli) - - HADOOP-8007. Use substitution tokens for fencing argument (todd) - - HADOOP-8077. HA: fencing method should be able to be configured on - a per-NN or per-NS basis (todd) - - HADOOP-8086. KerberosName silently sets defaultRealm to "" if the - Kerberos config is not found, it should log a WARN (tucu) - - HADOOP-8280. Move VersionUtil/TestVersionUtil and GenericTestUtils from - HDFS into Common. (Ahmed Radwan via atm) - - HADOOP-8117. Upgrade test build to Surefire 2.12 (todd) - - HADOOP-8152. Expand public APIs for security library classes. (atm via eli) - - HADOOP-7549. Use JDK ServiceLoader mechanism to find FileSystem implementations. (tucu) - - HADOOP-8185. Update namenode -format documentation and add -nonInteractive - and -force. (Arpit Gupta via atm) - - HADOOP-8214. make hadoop script recognize a full set of deprecated commands (rvs via tucu) - - HADOOP-8347. Hadoop Common logs misspell 'successful'. - (Philip Zeyliger via eli) - - HADOOP-8350. Improve NetUtils.getInputStream to return a stream which has - a tunable timeout. (todd) - - HADOOP-8356. FileSystem service loading mechanism should print the FileSystem - impl it is failing to load (tucu) - - HADOOP-8353. hadoop-daemon.sh and yarn-daemon.sh can be misleading on stop. - (Roman Shaposhnik via atm) - - HADOOP-8113. Correction to BUILDING.txt: HDFS needs ProtocolBuffer, too - (not just MapReduce). Contributed by Eugene Koontz. - - HADOOP-8285 Use ProtoBuf for RpcPayLoadHeader (sanjay radia) - - HADOOP-8366 Use ProtoBuf for RpcResponseHeader (sanjay radia) - - HADOOP-7729. Send back valid HTTP response if user hits IPC port with - HTTP GET. (todd) - - HADOOP-7987. Support setting the run-as user in unsecure mode. (jitendra) - - HADOOP-7994. Remove getProtocolVersion and getProtocolSignature from the - client side translator and server side implementation. (jitendra) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-8199. Fix issues in start-all.sh and stop-all.sh (Devaraj K via umamahesh) - - HADOOP-7635. RetryInvocationHandler should release underlying resources on - close. (atm) - - HADOOP-7695. RPC.stopProxy can throw unintended exception while logging - error. (atm) - - HADOOP-7833. Fix findbugs warnings in protobuf generated code. - (John Lee via suresh) - - HADOOP-7897. ProtobufRpcEngine client side exception mechanism is not - consistent with WritableRpcEngine. (suresh) - - HADOOP-7913. Fix bug in ProtoBufRpcEngine. (sanjay) - - HADOOP-7892. IPC logs too verbose after "RpcKind" introduction. (todd) - - HADOOP-7968. Errant println left in RPC.getHighestSupportedProtocol. (Sho - Shimauchi via harsh) - - HADOOP-7931. o.a.h.ipc.WritableRpcEngine should have a way to force - initialization. (atm) - - HADOOP-8104. Inconsistent Jackson versions (tucu) - - HADOOP-8119. Fix javac warnings in TestAuthenticationFilter in hadoop-auth. - (szetszwo) - - HADOOP-7888. TestFailoverProxy fails intermittently on trunk. (Jason Lowe - via atm) - - HADOOP-8154. DNS#getIPs shouldn't silently return the local host - IP for bogus interface names. (eli) - - HADOOP-8169. javadoc generation fails with java.lang.OutOfMemoryError: - Java heap space (tgraves via bobby) - - HADOOP-8167. Configuration deprecation logic breaks backwards compatibility (tucu) - - HADOOP-8189. LdapGroupsMapping shouldn't throw away IOException. (Jonathan Natkins via atm) - - HADOOP-8191. SshFenceByTcpPort uses netcat incorrectly (todd) - - HADOOP-8157. Fix race condition in Configuration that could cause spurious - ClassNotFoundExceptions after a GC. (todd) - - HADOOP-8197. Configuration logs WARNs on every use of a deprecated key (tucu) - - HADOOP-8159. NetworkTopology: getLeaf should check for invalid topologies. - (Colin Patrick McCabe via eli) - - HADOOP-8204. TestHealthMonitor fails occasionally (todd) - - HADOOP-8202. RPC stopProxy() does not close the proxy correctly. - (Hari Mankude via suresh) - - HADOOP-8218. RPC.closeProxy shouldn't throw error when closing a mock - (todd) - - HADOOP-8238. NetUtils#getHostNameOfIP blows up if given ip:port - string w/o port. (eli) - - HADOOP-8243. Security support broken in CLI (manual) failover controller - (todd) - - HADOOP-8251. Fix SecurityUtil.fetchServiceTicket after HADOOP-6941 (todd) - - HADOOP-8249. invalid hadoop-auth cookies should trigger authentication - if info is avail before returning HTTP 401 (tucu) - - HADOOP-8261. Har file system doesn't deal with FS URIs with a host but no - port. (atm) - - HADOOP-8263. Stringification of IPC calls not useful (todd) - - HADOOP-8264. Remove irritating double double quotes in front of hostname - (Bernd Fondermann via bobby) - - HADOOP-8270. hadoop-daemon.sh stop action should return 0 for an - already stopped service. (Roman Shaposhnik via eli) - - HADOOP-8144. pseudoSortByDistance in NetworkTopology doesn't work - properly if no local node and first node is local rack node. - (Junping Du) - - HADOOP-8282. start-all.sh refers incorrectly start-dfs.sh - existence for starting start-yarn.sh. (Devaraj K via eli) - - HADOOP-7350. Use ServiceLoader to discover compression codec classes. - (tomwhite) - - HADOOP-8284. clover integration broken, also mapreduce poms are pulling - in clover as a dependency. (phunt via tucu) - - HADOOP-8309. Pseudo & Kerberos AuthenticationHandler should use - getType() to create token (tucu) - - HADOOP-8314. HttpServer#hasAdminAccess should return false if - authorization is enabled but user is not authenticated. (tucu) - - HADOOP-8296. hadoop/yarn daemonlog usage wrong (Devaraj K via tgraves) - - HADOOP-8310. FileContext#checkPath should handle URIs with no port. (atm) - - HADOOP-8321. TestUrlStreamHandler fails. (tucu) - - HADOOP-8325. Add a ShutdownHookManager to be used by different - components instead of the JVM shutdownhook (tucu) - - HADOOP-8275. Range check DelegationKey length. - (Colin Patrick McCabe via eli) - - HADOOP-8342. HDFS command fails with exception following merge of - HADOOP-8325 (tucu) - - HADOOP-8346. Makes oid changes to make SPNEGO work. Was broken due - to fixes introduced by the IBM JDK compatibility patch. (ddas) - - HADOOP-8355. SPNEGO filter throws/logs exception when authentication fails (tucu) - - HADOOP-8349. ViewFS doesn't work when the root of a file system is mounted. (atm) - - HADOOP-8328. Duplicate FileSystem Statistics object for 'file' scheme. - (tomwhite) - - HADOOP-8359. Fix javadoc warnings in Configuration. (Anupam Seth via - szetszwo) - - HADOOP-7988. Upper case in hostname part of the principals doesn't work with - kerberos. (jitendra) - - BREAKDOWN OF HADOOP-7454 SUBTASKS - - HADOOP-7455. HA: Introduce HA Service Protocol Interface. (suresh) - - HADOOP-7774. HA: Administrative CLI to control HA daemons. (todd) - - HADOOP-7896. HA: if both NNs are in Standby mode, client needs to try failing - back and forth several times with sleeps. (atm) - - HADOOP-7922. Improve some logging for client IPC failovers and - StandbyExceptions (todd) - - HADOOP-7921. StandbyException should extend IOException (todd) - - HADOOP-7928. HA: Client failover policy is incorrectly trying to fail over all - IOExceptions (atm) - - HADOOP-7925. Add interface and update CLI to query current state to - HAServiceProtocol (eli via todd) - - HADOOP-7932. Make client connection retries on socket time outs configurable. - (Uma Maheswara Rao G via todd) - - HADOOP-7924. FailoverController for client-based configuration (eli) - - HADOOP-7961. Move HA fencing to common. (eli) - - HADOOP-7970. HAServiceProtocol methods must throw IOException. (Hari Mankude - via suresh). - - HADOOP-7992. Add ZKClient library to facilitate leader election. (Bikas Saha - via suresh). - - HADOOP-7983. HA: failover should be able to pass args to fencers. (eli) - - HADOOP-7938. HA: the FailoverController should optionally fence the active - during failover. (eli) - - HADOOP-7991. HA: the FailoverController should check the standby is ready - before failing over. (eli) - - HADOOP-8038. Add 'ipc.client.connect.max.retries.on.timeouts' entry in - core-default.xml file. (Uma Maheswara Rao G via atm) - - HADOOP-8041. Log a warning when a failover is first attempted (todd) - - HADOOP-8068. void methods can swallow exceptions when going through failover - path (todd) - - HADOOP-8116. RetriableCommand is using RetryPolicy incorrectly after - HADOOP-7896. (atm) - - HADOOP-8317. Update maven-assembly-plugin to 2.3 - fix build on FreeBSD - (Radim Kolar via bobby) - - HADOOP-8172. Configuration no longer sets all keys in a deprecated key - list. (Anupam Seth via bobby) - - HADOOP-7868. Hadoop native fails to compile when default linker - option is -Wl,--as-needed. (Trevor Robinson via eli) - - HADOOP-8655. Fix TextInputFormat for large deliminators. (Gelesh via - bobby) - - HADOOP-7900. LocalDirAllocator confChanged() accesses conf.get() twice - (Ravi Gummadi via Uma Maheswara Rao G) - - HADOOP-8146. FsShell commands cannot be interrupted - (Daryn Sharp via Uma Maheswara Rao G) - - HADOOP-8018. Hudson auto test for HDFS has started throwing javadoc - (Jon Eagles via bobby) - - HADOOP-8001 ChecksumFileSystem's rename doesn't correctly handle checksum - files. (Daryn Sharp via bobby) - - HADOOP-8006 TestFSInputChecker is failing in trunk. - (Daryn Sharp via bobby) - - HADOOP-7998. CheckFileSystem does not correctly honor setVerifyChecksum - (Daryn Sharp via bobby) - - HADOOP-7606. Upgrade Jackson to version 1.7.1 to match the version required - by Jersey (Alejandro Abdelnur via atm) - -Release 0.23.11 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-10129. Distcp may succeed when it fails (daryn) - - HADOOP-10332. HttpServer's jetty audit log always logs 200 OK (jeagles) - - HADOOP-8826. Docs still refer to 0.20.205 as stable line (Mit Desai via - jeagles) - -Release 0.23.10 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-9686. Easy access to final parameters in Configuration (Jason Lowe - via jeagles) - - HADOOP-8704. add request logging to jetty/httpserver (jeagles) - - OPTIMIZATIONS - - HADOOP-9956. RPC listener inefficiently assigns connections to readers (daryn) - - HADOOP-9955. RPC idle connection closing is extremely inefficient (daryn) - - BUG FIXES - -Release 0.23.9 - 2013-07-08 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9581. hadoop --config non-existent directory should result in error - (Ashwin Shankar via jlowe) - - HADOOP-9582. Non-existent file to "hadoop fs -conf" doesn't throw error - (Ashwin Shankar via jlowe) - -Release 0.23.8 - 2013-06-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9222. Cover package with org.apache.hadoop.io.lz4 unit tests (Vadim - Bondarev via jlowe) - - HADOOP-9233. Cover package org.apache.hadoop.io.compress.zlib with unit - tests (Vadim Bondarev via jlowe) - - HADOOP-9469. mapreduce/yarn source jars not included in dist tarball - (Robert Parker via tgraves) - - HADOOP-9504. MetricsDynamicMBeanBase has concurrency issues in - createMBeanInfo (Liang Xie via jlowe) - - HADOOP-9614. smart-test-patch.sh hangs for new version of patch (2.7.1) - (Ravi Prakash via jeagles) - -Release 0.23.7 - 2013-04-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-8849. FileUtil#fullyDelete should grant the target directories +rwx - permissions (Ivan A. Veselovsky via bobby) - - HADOOP-9067. provide test for LocalFileSystem.reportChecksumFailure - (Ivan A. Veselovsky via bobby) - - HADOOP-9336. Allow UGI of current connection to be queried. (Daryn Sharp - via kihwal) - - HADOOP-9352. Expose UGI.setLoginUser for tests (daryn) - - HADOOP-9209. Add shell command to dump file checksums (Todd Lipcon via - jeagles) - - HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) - - HADOOP-8711. IPC Server supports adding exceptions for which - the message is printed and the stack trace is not printed to avoid chatter. - (Brandon Li via Suresh) - - OPTIMIZATIONS - - HADOOP-8462. Native-code implementation of bzip2 codec. (Govind Kamat via - jlowe) - - BUG FIXES - - HADOOP-9302. HDFS docs not linked from top level (Andy Isaacson via - tgraves) - - HADOOP-9303. command manual dfsadmin missing entry for restoreFailedStorage - option (Andy Isaacson via tgraves) - - HADOOP-9339. IPC.Server incorrectly sets UGI auth type (Daryn Sharp via - kihwal) - -Release 0.23.6 - 2013-02-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - HADOOP-9217. Print thread dumps when hadoop-common tests fail. - (Andrey Klochkov via suresh) - - HADOOP-9242. Duplicate surefire plugin config in hadoop-common. - (Andrey Klochkov via suresh) - - HADOOP-9247. Parametrize Clover "generateXxx" properties to make them - re-definable via -D in mvn calls. (Ivan A. Veselovsky via suresh) - - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9072. Hadoop-Common-0.23-Build Fails to build in Jenkins - (Robert Parker via tgraves) - - HADOOP-8992. Enhance unit-test coverage of class HarFileSystem (Ivan A. - Veselovsky via bobby) - - HADOOP-9038. unit-tests for AllocatorPerContext.PathIterator (Ivan A. - Veselovsky via bobby) - - HADOOP-9105. FsShell -moveFromLocal erroneously fails (daryn via bobby) - - HADOOP-9097. Maven RAT plugin is not checking all source files (tgraves) - - HADOOP-9255. relnotes.py missing last jira (tgraves) - -Release 0.23.5 - 2012-11-28 - - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-8932. JNI-based user-group mapping modules can be too chatty on - lookup failures. (Kihwal Lee via suresh) - - HADOOP-8930. Cumulative code coverage calculation (Andrey Klochkov via - bobby) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-8906. paths with multiple globs are unreliable. (Daryn Sharp via - jlowe) - - HADOOP-8811. Compile hadoop native library in FreeBSD (Radim Kolar via - bobby) - - HADOOP-8962. RawLocalFileSystem.listStatus fails when a child filename - contains a colon (jlowe via bobby) - - HADOOP-8986. Server$Call object is never released after it is sent (bobby) - - HADOOP-9022. Hadoop distcp tool fails to copy file if -m 0 specified - (Jonathan Eagles vai bobby) - - HADOOP-9025. org.apache.hadoop.tools.TestCopyListing failing (Jonathan - Eagles via jlowe) - -Release 0.23.4 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-8822. relnotes.py was deleted post mavenization (bobby) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-8843. Old trash directories are never deleted on upgrade - from 1.x (jlowe) - -Release 0.23.3 - - INCOMPATIBLE CHANGES - - HADOOP-7967. Need generalized multi-token filesystem support (daryn) - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-8108. Move method getHostPortString() from NameNode to NetUtils. - (Brandon Li via jitendra) - - HADOOP-8288. Remove references of mapred.child.ulimit etc. since they are - not being used any more (Ravi Prakash via bobby) - - HADOOP-8535. Cut hadoop build times in half (Job Eagles via bobby) - - HADOOP-8525. Provide Improved Traceability for Configuration (bobby) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-8088. User-group mapping cache incorrectly does negative caching on - transient failures (Khiwal Lee via bobby) - - HADOOP-8179. risk of NPE in CopyCommands processArguments() (Daryn Sharp - via bobby) - - HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories - nor follow symbolic links. (Ravi Prakash via szetszwo) - - HADOOP-8180. Remove hsqldb since its not needed from pom.xml (Ravi Prakash - via tgraves) - - HADOOP-8014. ViewFileSystem does not correctly implement getDefaultBlockSize, - getDefaultReplication, getContentSummary (John George via bobby) - - HADOOP-7510. Tokens should use original hostname provided instead of ip - (Daryn Sharp via bobby) - - HADOOP-8283. Allow tests to control token service value (Daryn Sharp via - bobby) - - HADOOP-8286. Simplify getting a socket address from conf (Daryn Sharp via - bobby) - - HADOOP-8227. Allow RPC to limit ephemeral port range. (bobby) - - HADOOP-8305. distcp over viewfs is broken (John George via bobby) - - HADOOP-8334. HttpServer sometimes returns incorrect port (Daryn Sharp via - bobby) - - HADOOP-8330. Update TestSequenceFile.testCreateUsesFsArg() for HADOOP-8305. - (John George via szetszwo) - - HADOOP-8335. Improve Configuration's address handling (Daryn Sharp via - bobby) - - HADOOP-8327. distcpv2 and distcpv1 jars should not coexist (Dave Thompson - via bobby) - - HADOOP-8341. Fix or filter findbugs issues in hadoop-tools (bobby) - - HADOOP-8373. Port RPC.getServerAddress to 0.23 (Daryn Sharp via bobby) - - HADOOP-8495. Update Netty to avoid leaking file descriptors during shuffle - (Jason Lowe via tgraves) - - HADOOP-8129. ViewFileSystemTestSetup setupForViewFileSystem is erring - (Ahmed Radwan and Ravi Prakash via bobby) - - HADOOP-8573. Configuration tries to read from an inputstream resource - multiple times (Robert Evans via tgraves) - - HADOOP-8599. Non empty response from FileSystem.getFileBlockLocations when - asking for data beyond the end of file. (Andrey Klochkov via todd) - - HADOOP-8606. FileSystem.get may return the wrong filesystem (Daryn Sharp - via bobby) - - HADOOP-8551. fs -mkdir creates parent directories without the -p option - (John George via bobby) - - HADOOP-8613. AbstractDelegationTokenIdentifier#getUser() should set token - auth type. (daryn) - - HADOOP-8627. FS deleteOnExit may delete the wrong path (daryn via bobby) - - HADOOP-8634. Ensure FileSystem#close doesn't squawk for deleteOnExit paths - (daryn via bobby) - - HADOOP-8550. hadoop fs -touchz automatically created parent directories - (John George via bobby) - - HADOOP-8635. Cannot cancel paths registered deleteOnExit (daryn via bobby) - - HADOOP-8637. FilterFileSystem#setWriteChecksum is broken (daryn via bobby) - - HADOOP-8370. Native build failure: javah: class file for - org.apache.hadoop.classification.InterfaceAudience not found (Trevor - Robinson via tgraves) - - HADOOP-8633. Interrupted FsShell copies may leave tmp files (Daryn Sharp - via tgraves) - - HADOOP-8703. distcpV2: turn CRC checking off for 0 byte size (Dave - Thompson via bobby) - - HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor - Robinson via tgraves) - - HADOOP-8692. TestLocalDirAllocator fails intermittently with JDK7 - (Trevor Robinson via tgraves) - - HADOOP-8693. TestSecurityUtil fails intermittently with JDK7 (Trevor - Robinson via tgraves) - - HADOOP-8697. TestWritableName fails intermittently with JDK7 (Trevor - Robinson via tgraves) - - HADOOP-8695. TestPathData fails intermittently with JDK7 (Trevor - Robinson via tgraves) - - HADOOP-8611. Allow fall-back to the shell-based implementation when - JNI-based users-group mapping fails (Robert Parker via bobby) - - HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) - - HADOOP-8709. globStatus changed behavior from 0.20/1.x (Jason Lowe via - bobby) - - HADOOP-8725. MR is broken when security is off (daryn via bobby) - - HADOOP-8726. The Secrets in Credentials are not available to MR tasks - (daryn and Benoy Antony via bobby) - - HADOOP-8727. Gracefully deprecate dfs.umaskmode in 2.x onwards (Harsh J - via bobby) - -Release 0.23.2 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HADOOP-8048. Allow merging of Credentials (Daryn Sharp via tgraves) - - HADOOP-8032. mvn site:stage-deploy should be able to use the scp protocol - to stage documents (Ravi Prakash via tgraves) - - HADOOP-7923. Automate the updating of version numbers in the doc system. - (szetszwo) - - HADOOP-8137. Added links to CLI manuals to the site. (tgraves via - acmurthy) - - OPTIMIZATIONS - HADOOP-8071. Avoid an extra packet in client code when nagling is - disabled. (todd) - - HADOOP-6502. Improve the performance of Configuration.getClassByName when - the class is not found by caching negative results. - (sharad, todd via todd) - - BUG FIXES - - HADOOP-7660. Maven generated .classpath doesnot includes - "target/generated-test-source/java" as source directory. - (Laxman via bobby) - - HADOOP-8042 When copying a file out of HDFS, modifying it, and uploading - it back into HDFS, the put fails due to a CRC mismatch - (Daryn Sharp via bobby) - - HADOOP-8035 Hadoop Maven site is inefficient and runs phases redundantly - (abayer via tucu) - - HADOOP-8051 HttpFS documentation it is not wired to the generated site (tucu) - - HADOOP-8055. Hadoop tarball distribution lacks a core-site.xml (harsh) - - HADOOP-8052. Hadoop Metrics2 should emit Float.MAX_VALUE (instead of - Double.MAX_VALUE) to avoid making Ganglia's gmetad core. (Varun Kapoor - via mattf) - - HADOOP-8074. Small bug in hadoop error message for unknown commands. - (Colin Patrick McCabe via eli) - - HADOOP-8082 add hadoop-client and hadoop-minicluster to the - dependency-management section. (tucu) - - HADOOP-8066 The full docs build intermittently fails (abayer via tucu) - - HADOOP-8083 javadoc generation for some modules is not done under target/ (tucu) - - HADOOP-8036. TestViewFsTrash assumes the user's home directory is - 2 levels deep. (Colin Patrick McCabe via eli) - - HADOOP-8046 Revert StaticMapping semantics to the existing ones, add DNS - mapping diagnostics in progress (stevel) - - HADOOP-8057 hadoop-setup-conf.sh not working because of some extra spaces. - (Vinayakumar B via stevel) - - HADOOP-7680 TestHardLink fails on Mac OS X, when gnu stat is in path. - (Milind Bhandarkar via stevel) - - HADOOP-8050. Deadlock in metrics. (Kihwal Lee via mattf) - - HADOOP-8131. FsShell put doesn't correctly handle a non-existent dir - (Daryn Sharp via bobby) - - HADOOP-8123. Use java.home rather than env.JAVA_HOME for java in the - project. (Jonathan Eagles via acmurthy) - - HADOOP-8064. Remove unnecessary dependency on w3c.org in document processing - (Khiwal Lee via bobby) - - HADOOP-8140. dfs -getmerge should process its argments better (Daryn Sharp - via bobby) - - HADOOP-8164. Back slash as path separator is handled for Windows only. - (Daryn Sharp via suresh) - - HADOOP-8173. FsShell needs to handle quoted metachars. (Daryn Sharp via - szetszwo) - - HADOOP-8175. Add -p option to mkdir in FsShell. (Daryn Sharp via szetszwo) - - HADOOP-8176. Disambiguate the destination of FsShell copies (Daryn Sharp - via bobby) - - HADOOP-8208. Disallow self failover. (eli) - -Release 0.23.1 - 2012-02-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HADOOP-7777 Implement a base class for DNSToSwitchMapping implementations - that can offer extra topology information. (stevel) - - HADOOP-7657. Add support for LZ4 compression. (Binglin Chang via todd) - - HADOOP-7910. Add Configuration.getLongBytes to handle human readable byte size values. (Sho Shimauchi via harsh) - - - IMPROVEMENTS - - HADOOP-7801. HADOOP_PREFIX cannot be overriden. (Bruno Mahé via tomwhite) - - HADOOP-7802. Hadoop scripts unconditionally source - "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) - - HADOOP-7858. Drop some info logging to DEBUG level in IPC, - metrics, and HTTP. (todd via eli) - - HADOOP-7424. Log an error if the topology script doesn't handle multiple args. - (Uma Maheswara Rao G via eli) - - HADOOP-7804. Enable hadoop config generator to set configurations to enable - short circuit read. (Arpit Gupta via jitendra) - - HADOOP-7877. Update balancer CLI usage documentation to include the new - -policy option. (szetszwo) - - HADOOP-6840. Support non-recursive create() in FileSystem and - SequenceFile.Writer. (jitendra and eli via eli) - - HADOOP-6886. LocalFileSystem Needs createNonRecursive API. - (Nicolas Spiegelberg and eli via eli) - - HADOOP-7912. test-patch should run eclipse:eclipse to verify that it does - not break again. (Robert Joseph Evans via tomwhite) - - HADOOP-7890. Redirect hadoop script's deprecation message to stderr. - (Koji Knoguchi via mahadev) - - HADOOP-7504. Add the missing Ganglia31 opts to hadoop-metrics.properties as a comment. (harsh) - - HADOOP-7933. Add a getDelegationTokens api to FileSystem which checks - for known tokens in the passed Credentials object. (sseth) - - HADOOP-7737. normalize hadoop-mapreduce & hadoop-dist dist/tar build with - common/hdfs. (tucu) - - HADOOP-7743. Add Maven profile to create a full source tarball. (tucu) - - HADOOP-7758. Make GlobFilter class public. (tucu) - - HADOOP-7590. Mavenize streaming and MR examples. (tucu) - - HADOOP-7934. Normalize dependencies versions across all modules. (tucu) - - HADOOP-7348. Change 'addnl' in getmerge util to be a flag '-nl' instead. - (XieXianshan via harsh) - - HADOOP-7975. Add LZ4 as an entry in the default codec list, missed by HADOOP-7657 (harsh) - - HADOOP-4515. Configuration#getBoolean must not be case sensitive. (Sho Shimauchi via harsh) - - HADOOP-6490. Use StringUtils over String#replace in Path#normalizePath. - (Uma Maheswara Rao G via harsh) - - HADOOP-7574. Improve FSShell -stat, add user/group elements. - (XieXianshan via harsh) - - HADOOP-7736. Remove duplicate Path#normalizePath call. (harsh) - - HADOOP-7919. Remove the unused hadoop.logfile.* properties from the - core-default.xml file. (harsh) - - HADOOP-7939. Improve Hadoop subcomponent integration in Hadoop 0.23. (rvs via tucu) - - HADOOP-8002. SecurityUtil acquired token message should be a debug rather than info. - (Arpit Gupta via mahadev) - - HADOOP-8009. Create hadoop-client and hadoop-minicluster artifacts for downstream - projects. (tucu) - - HADOOP-7470. Move up to Jackson 1.8.8. (Enis Soztutar via szetszwo) - - HADOOP-8027. Visiting /jmx on the daemon web interfaces may print - unnecessary error in logs. (atm) - - HADOOP-7792. Add verifyToken method to AbstractDelegationTokenSecretManager. - (jitendra) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-7811. TestUserGroupInformation#testGetServerSideGroups test fails in chroot. - (Jonathan Eagles via mahadev) - - HADOOP-7813. Fix test-patch to use proper numerical comparison when checking - javadoc and findbugs warning counts. (Jonathan Eagles via tlipcon) - - HADOOP-7841. Run tests with non-secure random. (tlipcon) - - HADOOP-7851. Configuration.getClasses() never returns the default value. - (Uma Maheswara Rao G via amarrk) - - HADOOP-7787. Make source tarball use conventional name. - (Bruno Mahé via tomwhite) - - HADOOP-6614. RunJar should provide more diags when it can't create - a temp file. (Jonathan Hsieh via eli) - - HADOOP-7859. TestViewFsHdfs.testgetFileLinkStatus is failing an assert. (eli) - - HADOOP-7864. Building mvn site with Maven < 3.0.2 causes OOM errors. - (Andrew Bayer via eli) - - HADOOP-7854. UGI getCurrentUser is not synchronized. (Daryn Sharp via jitendra) - - HADOOP-7870. fix SequenceFile#createWriter with boolean - createParent arg to respect createParent. (Jon Hsieh via eli) - - HADOOP-7898. Fix javadoc warnings in AuthenticationToken.java. (suresh) - - HADOOP-7878 Regression: HADOOP-7777 switch changes break HDFS tests when the - isSingleSwitch() predicate is used. (stevel) - - HADOOP-7914. Remove the duplicated declaration of hadoop-hdfs test-jar in - hadoop-project/pom.xml. (szetszwo) - - HADOOP-7837. no NullAppender in the log4j config. (eli) - - HADOOP-7948. Shell scripts created by hadoop-dist/pom.xml to build tar do not - properly propagate failure. (cim_michajlomatijkiw via tucu) - - HADOOP-7949. Updated maxIdleTime default in the code to match - core-default.xml (eli) - - HADOOP-7853. multiple javax security configurations cause conflicts. - (daryn via tucu) - - HDFS-2614. hadoop dist tarball is missing hdfs headers. (tucu) - - HADOOP-7874. native libs should be under lib/native/ dir. (tucu) - - HADOOP-7887. KerberosAuthenticatorHandler is not setting - KerberosName name rules from configuration. (tucu) - - HADOOP-7902. skipping name rules setting (if already set) should be done - on UGI initialization only. (tucu) - - HADOOP-7810. move hadoop archive to core from tools. (tucu) - - HADOOP_7917. compilation of protobuf files fails in windows/cygwin. (tucu) - - HADOOP-7907. hadoop-tools JARs are not part of the distro. (tucu) - - HADOOP-7936. There's a Hoop README in the root dir of the tarball. (tucu) - - HADOOP-7963. Fix ViewFS to catch a null canonical service-name and pass - tests TestViewFileSystem* (Siddharth Seth via vinodkv) - - HADOOP-7964. Deadlock in NetUtils and SecurityUtil class initialization. - (Daryn Sharp via suresh) - - HADOOP-7974. TestViewFsTrash incorrectly determines the user's home - directory. (harsh via eli) - - HADOOP-7971. Adding back job/pipes/queue commands to bin/hadoop for - backward compatibility. (Prashath Sharma via acmurthy) - - HADOOP-7982. UserGroupInformation fails to login if thread's context - classloader can't load HadoopLoginModule. (todd) - - HADOOP-7986. Adding config for MapReduce History Server protocol in - hadoop-policy.xml for service level authorization. (Mahadev Konar via vinodkv) - - HADOOP-7981. Improve documentation for org.apache.hadoop.io.compress. - Decompressor.getRemaining (Jonathan Eagles via mahadev) - - HADOOP-7997. SequenceFile.createWriter(...createParent...) no - longer works on existing file. (Gregory Chanan via eli) - - HADOOP-7993. Hadoop ignores old-style config options for enabling compressed - output. (Anupam Seth via mahadev) - - HADOOP-8000. fetchdt command not available in bin/hadoop. - (Arpit Gupta via mahadev) - - HADOOP-7999. "hadoop archive" fails with ClassNotFoundException. - (Jason Lowe via mahadev) - - HADOOP-8012. hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir - and chown log/pid dirs which can fail. (Roman Shaposhnik via eli) - - HADOOP-8013. ViewFileSystem does not honor setVerifyChecksum - (Daryn Sharp via bobby) - - HADOOP-8054 NPE with FilterFileSystem (Daryn Sharp via bobby) - -Release 0.23.0 - 2011-11-01 - - INCOMPATIBLE CHANGES - - HADOOP-6904. Support method based RPC compatiblity. (hairong) - - HADOOP-6432. Add Statistics support in FileContext. (jitendra) - - HADOOP-7136. Remove failmon contrib component. (nigel) - - NEW FEATURES - - HADOOP-7324. Ganglia plugins for metrics v2. (Priyo Mustafi via llu) - - HADOOP-7342. Add an utility API in FileUtil for JDK File.list - avoid NPEs on File.list() (Bharath Mundlapudi via mattf) - - HADOOP-7322. Adding a util method in FileUtil for directory listing, - avoid NPEs on File.listFiles() (Bharath Mundlapudi via mattf) - - HADOOP-7023. Add listCorruptFileBlocks to Filesysem. (Patrick Kling - via hairong) - - HADOOP-7096. Allow setting of end-of-record delimiter for TextInputFormat - (Ahmed Radwan via todd) - - HADOOP-6994. Api to get delegation token in AbstractFileSystem. (jitendra) - - HADOOP-7171. Support UGI in FileContext API. (jitendra) - - HADOOP-7257 Client side mount tables (sanjay) - - HADOOP-6919. New metrics2 framework. (Luke Lu via acmurthy) - - HADOOP-6920. Metrics instrumentation to move new metrics2 framework. - (Luke Lu via suresh) - - HADOOP-7214. Add Common functionality necessary to provide an equivalent - of /usr/bin/groups for Hadoop. (Aaron T. Myers via todd) - - HADOOP-6832. Add an authentication plugin using a configurable static user - for the web UI. (Owen O'Malley and Todd Lipcon via cdouglas) - - HADOOP-7144. Expose JMX metrics via JSON servlet. (Robert Joseph Evans via - cdouglas) - - HADOOP-7379. Add the ability to serialize and deserialize protocol buffers - in ObjectWritable. (todd) - - HADOOP-7206. Support Snappy compression. (Issei Yoshida and - Alejandro Abdelnur via eli) - - HADOOP-7329. Add the capability of getting invividual attribute of a mbean - using JMXProxyServlet. (tanping) - - HADOOP-7380. Add client failover functionality to o.a.h.io.(ipc|retry). - (atm via eli) - - HADOOP-7460. Support pluggable trash policies. (Usman Masoon via suresh) - - HADOOP-6385. dfs should support -rmdir (was HDFS-639). (Daryn Sharp - via mattf) - - HADOOP-7119. add Kerberos HTTP SPNEGO authentication support to Hadoop - JT/NN/DN/TT web-consoles. (Alejandro Abdelnur via atm) - - IMPROVEMENTS - - HADOOP-7655. Provide a small validation script that smoke tests the installed - cluster. (Arpit Gupta via mattf) - - HADOOP-7042. Updates to test-patch.sh to include failed test names and - improve other messaging. (nigel) - - HADOOP-7001. Configuration changes can occur via the Reconfigurable - interface. (Patrick Kling via dhruba) - - HADOOP-6764. Add number of reader threads and queue length as - configuration parameters in RPC.getServer. (Dmytro Molkov via hairong) - - HADOOP-7049. TestReconfiguration should be junit v4. - (Patrick Kling via eli) - - HADOOP-7054 Change NN LoadGenerator to use FileContext APIs - (Sanjay Radia) - - HADOOP-7060. A more elegant FileSystem#listCorruptFileBlocks API. - (Patrick Kling via hairong) - - HADOOP-7058. Expose number of bytes in FSOutputSummer buffer to - implementatins. (Todd Lipcon via hairong) - - HADOOP-7061. unprecise javadoc for CompressionCodec. (Jingguo Yao via eli) - - HADOOP-7059. Remove "unused" warning in native code. (Noah Watkins via eli) - - HADOOP-6864. Provide a JNI-based implementation of - ShellBasedUnixGroupsNetgroupMapping - (implementation of GroupMappingServiceProvider) (Erik Seffl via boryas) - - HADOOP-7078. Improve javadocs for RawComparator interface. - (Harsh J Chouraria via todd) - - HADOOP-6995. Allow wildcards to be used in ProxyUsers configurations. - (todd) - - HADOOP-6376. Add a comment header to conf/slaves that specifies the file - format. (Kay Kay via todd) - - HADOOP-7151. Document need for stable hashCode() in WritableComparable. - (Dmitriy V. Ryaboy via todd) - - HADOOP-7112. Issue a warning when GenericOptionsParser libjars are not on - local filesystem. (tomwhite) - - HADOOP-7114. FsShell should dump all exceptions at DEBUG level. - (todd via tomwhite) - - HADOOP-7159. RPC server should log the client hostname when read exception - happened. (Scott Chen via todd) - - HADOOP-7167. Allow using a file to exclude certain tests from build. (todd) - - HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink(). - (Matt Foley via jghoman) - - HADOOP-7166. Add DaemonFactory to common. (Erik Steffl & jitendra) - - HADOOP-7175. Add isEnabled() to Trash. (Daryn Sharp via szetszwo) - - HADOOP-7180. Better support on CommandFormat on the API and exceptions. - (Daryn Sharp via szetszwo) - - HADOOP-7202. Improve shell Command base class. (Daryn Sharp via szetszwo) - - HADOOP-7224. Add CommandFactory to shell. (Daryn Sharp via szetszwo) - - HADOOP-7014. Generalize CLITest structure and interfaces to facilitate - upstream adoption (e.g. for web testing). (cos) - - HADOOP-7230. Move "fs -help" shell command tests from HDFS to COMMOM; see - also HDFS-1844. (Daryn Sharp via szetszwo) - - HADOOP-7233. Refactor ls to conform to new FsCommand class. (Daryn Sharp - via szetszwo) - - HADOOP-7235. Refactor the tail command to conform to new FsCommand class. - (Daryn Sharp via szetszwo) - - HADOOP-7179. Federation: Improve HDFS startup scripts. (Erik Steffl - and Tanping Wang via suresh) - - HADOOP-7227. Remove protocol version check at proxy creation in Hadoop - RPC. (jitendra) - - HADOOP-7236. Refactor the mkdir command to conform to new FsCommand class. - (Daryn Sharp via szetszwo) - - HADOOP-7250. Refactor the setrep command to conform to new FsCommand class. - (Daryn Sharp via szetszwo) - - HADOOP-7249. Refactor the chmod/chown/chgrp command to conform to new - FsCommand class. (Daryn Sharp via szetszwo) - - HADOOP-7251. Refactor the getmerge command to conform to new FsCommand - class. (Daryn Sharp via szetszwo) - - HADOOP-7265. Keep track of relative paths in PathData. (Daryn Sharp - via szetszwo) - - HADOOP-7238. Refactor the cat and text commands to conform to new FsCommand - class. (Daryn Sharp via szetszwo) - - HADOOP-7271. Standardize shell command error messages. (Daryn Sharp - via szetszwo) - - HADOOP-7272. Remove unnecessary security related info logs. (suresh) - - HADOOP-7275. Refactor the stat command to conform to new FsCommand - class. (Daryn Sharp via szetszwo) - - HADOOP-7237. Refactor the touchz command to conform to new FsCommand - class. (Daryn Sharp via szetszwo) - - HADOOP-7267. Refactor the rm/rmr/expunge commands to conform to new - FsCommand class. (Daryn Sharp via szetszwo) - - HADOOP-7285. Refactor the test command to conform to new FsCommand - class. (Daryn Sharp via todd) - - HADOOP-7289. In ivy.xml, test conf should not extend common conf. - (Eric Yang via szetszwo) - - HADOOP-7291. Update Hudson job not to run test-contrib. (Nigel Daley via eli) - - HADOOP-7286. Refactor the du/dus/df commands to conform to new FsCommand - class. (Daryn Sharp via todd) - - HADOOP-7301. FSDataInputStream should expose a getWrappedStream method. - (Jonathan Hsieh via eli) - - HADOOP-7306. Start metrics system even if config files are missing - (Luke Lu via todd) - - HADOOP-7302. webinterface.private.actions should be renamed and moved to - the MapReduce project. (Ari Rabkin via todd) - - HADOOP-7329. Improve help message for "df" to include "-h" flag. - (Xie Xianshan via todd) - - HADOOP-7320. Refactor the copy and move commands to conform to new - FsCommand class. (Daryn Sharp via todd) - - HADOOP-7312. Update value of hadoop.common.configuration.version. - (Harsh J Chouraria via todd) - - HADOOP-7337. Change PureJavaCrc32 annotations to public stable. (szetszwo) - - HADOOP-7331. Make hadoop-daemon.sh return exit code 1 if daemon processes - did not get started. (Tanping Wang via todd) - - HADOOP-7316. Add public javadocs to FSDataInputStream and - FSDataOutputStream. (eli) - - HADOOP-7323. Add capability to resolve compression codec based on codec - name. (Alejandro Abdelnur via tomwhite) - - HADOOP-1886. Undocumented parameters in FilesSystem. (Frank Conrad via eli) - - HADOOP-7375. Add resolvePath method to FileContext. (Sanjay Radia via eli) - - HADOOP-7383. HDFS needs to export protobuf library dependency in pom. - (todd via eli) - - HADOOP-7374. Don't add tools.jar to the classpath when running Hadoop. - (eli) - - HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. - (todd, nigel) - - HADOOP-6605. Add JAVA_HOME detection to hadoop-config. (eli) - - HADOOP-7384. Allow test-patch to be more flexible about patch format. (todd) - - HADOOP-6929. RPC should have a way to pass Security information other than - protocol annotations. (sharad and omalley via mahadev) - - HADOOP-7385. Remove StringUtils.stringifyException(ie) in logger functions. - (Bharath Mundlapudi via Tanping Wang). - - HADOOP-310. Additional constructor requested in BytesWritable. (Brock - Noland via atm) - - HADOOP-7429. Add another IOUtils#copyBytes method. (eli) - - HADOOP-7451. Generalize StringUtils#join. (Chris Douglas via mattf) - - HADOOP-7449. Add Data(In,Out)putByteBuffer to work with ByteBuffer similar - to Data(In,Out)putBuffer for byte[]. Merge from yahoo-merge branch, - -r 1079163. Fix missing Apache license headers. (Chris Douglas via mattf) - - HADOOP-7361. Provide an option, -overwrite/-f, in put and copyFromLocal - shell commands. (Uma Maheswara Rao G via szetszwo) - - HADOOP-7430. Improve error message when moving to trash fails due to - quota issue. (Ravi Prakash via mattf) - - HADOOP-7444. Add Checksum API to verify and calculate checksums "in bulk" - (todd) - - HADOOP-7443. Add CRC32C as another DataChecksum implementation (todd) - - HADOOP-7305. Eclipse project files are incomplete. (Niels Basjes via eli) - - HADOOP-7314. Add support for throwing UnknownHostException when a host doesn't - resolve. (Jeffrey Naisbitt via jitendra) - - HADOOP-7465. A several tiny improvements for the LOG format. - (Xie Xianshan via eli) - - HADOOP-7434. Display error when using "daemonlog -setlevel" with - illegal level. (yanjinshuang via eli) - - HADOOP-7463. Adding a configuration parameter to SecurityInfo interface. - (mahadev) - - HADOOP-7298. Add test utility for writing multi-threaded tests. (todd and - Harsh J Chouraria via todd) - - HADOOP-7485. Add -h option to ls to list file sizes in human readable - format. (XieXianshan via suresh) - - HADOOP-7378. Add -d option to ls to not expand directories. - (Daryn Sharp via suresh) - - HADOOP-7474. Refactor ClientCache out of WritableRpcEngine. (jitendra) - - HADOOP-7491. hadoop command should respect HADOOP_OPTS when given - a class name. (eli) - - HADOOP-7178. Add a parameter, useRawLocalFileSystem, to copyToLocalFile(..) - in FileSystem. (Uma Maheswara Rao G via szetszwo) - - HADOOP-6671. Use maven for hadoop common builds. (Alejandro Abdelnur - via tomwhite) - - HADOOP-7502. Make generated sources IDE friendly. - (Alejandro Abdelnur via llu) - - HADOOP-7501. Publish Hadoop Common artifacts (post HADOOP-6671) to Apache - SNAPSHOTs repo. (Alejandro Abdelnur via tomwhite) - - HADOOP-7525. Make arguments to test-patch optional. (tomwhite) - - HADOOP-7472. RPC client should deal with IP address change. - (Kihwal Lee via suresh) - - HADOOP-7499. Add method for doing a sanity check on hostnames in NetUtils. - (Jeffrey Naisbit via mahadev) - - HADOOP-6158. Move CyclicIteration to HDFS. (eli) - - HADOOP-7526. Add TestPath tests for URI conversion and reserved - characters. (eli) - - HADOOP-7531. Add servlet util methods for handling paths in requests. (eli) - - HADOOP-7493. Add ShortWritable. (Uma Maheswara Rao G via szetszwo) - - HADOOP-7555. Add a eclipse-generated files to .gitignore. (atm) - - HADOOP-7264. Bump avro version to at least 1.4.1. (Alejandro Abdelnur via - tomwhite) - - HADOOP-7498. Remove legacy TAR layout creation. (Alejandro Abdelnur via - tomwhite) - - HADOOP-7496. Break Maven TAR & bintar profiles into just LAYOUT & TAR proper. - (Alejandro Abdelnur via tomwhite) - - HADOOP-7561. Make test-patch only run tests for changed modules. (tomwhite) - - HADOOP-7547. Add generic type in WritableComparable subclasses. - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7579. Rename package names from alfredo to auth. - (Alejandro Abdelnur via szetszwo) - - HADOOP-7594. Support HTTP REST in HttpServer. (szetszwo) - - HADOOP-7552. FileUtil#fullyDelete doesn't throw IOE but lists it - in the throws clause. (eli) - - HADOOP-7580. Add a version of getLocalPathForWrite to LocalDirAllocator - which doesn't create dirs. (Chris Douglas & Siddharth Seth via acmurthy) - - HADOOP-7507. Allow ganglia metrics to include the metrics system tags - in the gmetric names. (Alejandro Abdelnur via todd) - - HADOOP-7612. Change test-patch to run tests for all nested modules. - (tomwhite) - - HADOOP-7599. Script improvements to setup a secure Hadoop cluster - (Eric Yang via ddas) - - HADOOP-7639. Enhance HttpServer to allow passing path-specs for filtering, - so that servers like Yarn WebApp can get filtered the paths served by - their own injected servlets. (Thomas Graves via vinodkv) - - HADOOP-7575. Enhanced LocalDirAllocator to support fully-qualified - paths. (Jonathan Eagles via vinodkv) - - HADOOP-7469 Add a standard handler for socket connection problems which - improves diagnostics (Uma Maheswara Rao G and stevel via stevel) - - HADOOP-7710. Added hadoop-setup-application.sh for creating - application directory (Arpit Gupta via Eric Yang) - - HADOOP-7707. Added toggle for dfs.support.append, webhdfs and hadoop proxy - user to setup config script. (Arpit Gupta via Eric Yang) - - HADOOP-7720. Added parameter for HBase user to setup config script. - (Arpit Gupta via Eric Yang) - - HADOOP-7624. Set things up for a top level hadoop-tools module. (tucu) - - HADOOP-7627. Improve MetricsAsserts to give more understandable output - on failure. (todd) - - HADOOP-7642. create hadoop-dist module where TAR stitching would happen. - (Thomas White via tucu) - - HADOOP-7709. Running a set of methods in a Single Test Class. - (Jonathan Eagles via mahadev) - - HADOOP-7705. Add a log4j back end that can push out JSON data, - one per line. (stevel) - - HADOOP-7749. Add a NetUtils createSocketAddr call which provides more - help in exception messages. (todd) - - HADOOP-7762. Common side of MR-2736. (eli) - - HADOOP-7668. Add a NetUtils method that can tell if an InetAddress - belongs to local host. (suresh) - - HADOOP-7509. Improve exception message thrown when Authentication is - required. (Ravi Prakash via suresh) - - HADOOP-7745. Fix wrong variable name in exception message introduced - in HADOOP-7509. (Ravi Prakash via suresh) - - MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra) - - HADOOP-7360. Preserve relative paths that do not contain globs in FsShell. - (Daryn Sharp and Kihwal Lee via szetszwo) - - HADOOP-7771. FsShell -copyToLocal, -get, etc. commands throw NPE if the - destination directory does not exist. (John George and Daryn Sharp - via szetszwo) - - HADOOP-7782. Aggregate project javadocs. (tomwhite) - - HADOOP-7789. Improvements to site navigation. (acmurthy) - - OPTIMIZATIONS - - HADOOP-7333. Performance improvement in PureJavaCrc32. (Eric Caspole - via todd) - - HADOOP-7445. Implement bulk checksum verification using efficient native - code. (todd) - - HADOOP-7753. Support fadvise and sync_file_range in NativeIO. Add - ReadaheadPool infrastructure for use in HDFS and MR. (todd) - - HADOOP-7446. Implement CRC32C native code using SSE4.2 instructions. - (Kihwal Lee and todd via todd) - - HADOOP-7763. Add top-level navigation to APT docs. (tomwhite) - - HADOOP-7785. Add equals, hashcode, toString to DataChecksum (todd) - - BUG FIXES - - HADOOP-7740. Fixed security audit logger configuration. (Arpit Gupta via Eric Yang) - - HADOOP-7630. hadoop-metrics2.properties should have a property *.period - set to a default value for metrics. (Eric Yang via mattf) - - HADOOP-7327. FileSystem.listStatus() throws NullPointerException instead of - IOException upon access permission failure. (mattf) - - HADOOP-7015. RawLocalFileSystem#listStatus does not deal with a directory - whose entries are changing (e.g. in a multi-thread or multi-process - environment). (Sanjay Radia via eli) - - HADOOP-7045. TestDU fails on systems with local file systems with - extended attributes. (eli) - - HADOOP-6939. Inconsistent lock ordering in - AbstractDelegationTokenSecretManager. (Todd Lipcon via tomwhite) - - HADOOP-7129. Fix typo in method name getProtocolSigature (todd) - - HADOOP-7048. Wrong description of Block-Compressed SequenceFile Format in - SequenceFile's javadoc. (Jingguo Yao via tomwhite) - - HADOOP-7153. MapWritable violates contract of Map interface for equals() - and hashCode(). (Nicholas Telford via todd) - - HADOOP-6754. DefaultCodec.createOutputStream() leaks memory. - (Aaron Kimball via tomwhite) - - HADOOP-7098. Tasktracker property not set in conf/hadoop-env.sh. - (Bernd Fondermann via tomwhite) - - HADOOP-7131. Exceptions thrown by Text methods should include the causing - exception. (Uma Maheswara Rao G via todd) - - HADOOP-6912. Guard against NPE when calling UGI.isLoginKeytabBased(). - (Kan Zhang via jitendra) - - HADOOP-7204. remove local unused fs variable from CmdHandler - and FsShellPermissions.changePermissions (boryas) - - HADOOP-7210. Chown command is not working from FSShell - (Uma Maheswara Rao G via todd) - - HADOOP-7215. RPC clients must use network interface corresponding to - the host in the client's kerberos principal key. (suresh) - - HADOOP-7019. Refactor build targets to enable faster cross project dev - cycles. (Luke Lu via cos) - - HADOOP-7216. Add FsCommand.runAll() with deprecated annotation for the - transition of Command base class improvement. (Daryn Sharp via szetszwo) - - HADOOP-7207. fs member of FSShell is not really needed (boryas) - - HADOOP-7223. FileContext createFlag combinations are not clearly defined. - (suresh) - - HADOOP-7231. Fix synopsis for -count. (Daryn Sharp via eli). - - HADOOP-7261. Disable IPV6 for junit tests. (suresh) - - HADOOP-7268. FileContext.getLocalFSFileContext() behavior needs to be fixed - w.r.t tokens. (jitendra) - - HADOOP-7290. Unit test failure in - TestUserGroupInformation.testGetServerSideGroups. (Trevor Robison via eli) - - HADOOP-7292. Fix racy test case TestSinkQueue. (Luke Lu via todd) - - HADOOP-7282. ipc.Server.getRemoteIp() may return null. (John George - via szetszwo) - - HADOOP-7208. Fix implementation of equals() and hashCode() in - StandardSocketFactory. (Uma Maheswara Rao G via todd) - - HADOOP-7336. TestFileContextResolveAfs will fail with default - test.build.data property. (jitendra) - - HADOOP-7284 Trash and shell's rm does not work for viewfs (Sanjay Radia) - - HADOOP-7341. Fix options parsing in CommandFormat (Daryn Sharp via todd) - - HADOOP-7353. Cleanup FsShell and prevent masking of RTE stack traces. - (Daryn Sharp via todd) - - HADOOP-7356. RPM packages broke bin/hadoop script in developer environment. - (Eric Yang via todd) - - HADOOP-7389. Use of TestingGroups by tests causes subsequent tests to fail. - (atm via tomwhite) - - HADOOP-7377. Fix command name handling affecting DFSAdmin. (Daryn Sharp - via mattf) - - HADOOP-7402. TestConfiguration doesn't clean up after itself. (atm via eli) - - HADOOP-7428. IPC connection is orphaned with null 'out' member. - (todd via eli) - - HADOOP-7437. IOUtils.copybytes will suppress the stream closure exceptions. - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7090. Fix resource leaks in s3.INode, BloomMapFile, WritableUtils - and CBZip2OutputStream. (Uma Maheswara Rao G via szetszwo) - - HADOOP-7440. HttpServer.getParameterValues throws NPE for missing - parameters. (Uma Maheswara Rao G and todd via todd) - - HADOOP-7442. Docs in core-default.xml still reference deprecated config - "topology.script.file.name" (atm) - - HADOOP-7419. new hadoop-config.sh doesn't manage classpath for - HADOOP_CONF_DIR correctly. (Bing Zheng and todd via todd) - - HADOOP-7448. merge from yahoo-merge branch (via mattf): - -r 1079157: Fix content type for /stacks servlet to be - plain text (Luke Lu) - -r 1079164: No need to escape plain text (Luke Lu) - - HADOOP-7471. The saveVersion.sh script sometimes fails to extract SVN URL. - (Alejandro Abdelnur via eli) - - HADOOP-2081. Configuration getInt, getLong, and getFloat replace - invalid numbers with the default value. (Harsh J via eli) - - HADOOP-7111. Several TFile tests failing when native libraries are - present. (atm) - - HADOOP-7438. Fix deprecated warnings from hadoop-daemon.sh script. - (Ravi Prakash via suresh) - - HADOOP-7468 hadoop-core JAR contains a log4j.properties file. - (Jolly Chen) - - HADOOP-7508. Compiled nativelib is in wrong directory and it is not picked - up by surefire setup. (Alejandro Abdelnur via tomwhite) - - HADOOP-7520. Fix to add distribution management info to hadoop-main - (Alejandro Abdelnur via gkesavan) - - HADOOP-7515. test-patch reports the wrong number of javadoc warnings. - (tomwhite) - - HADOOP-7523. Test org.apache.hadoop.fs.TestFilterFileSystem fails due to - java.lang.NoSuchMethodException. (John Lee via tomwhite) - - HADOOP-7528. Maven build fails in Windows. (Alejandro Abdelnur via - tomwhite) - - HADOOP-7533. Allow test-patch to be run from any subproject directory. - (tomwhite) - - HADOOP-7512. Fix example mistake in WritableComparable javadocs. - (Harsh J via eli) - - HADOOP-7357. hadoop.io.compress.TestCodec#main() should exit with - non-zero exit code if test failed. (Philip Zeyliger via eli) - - HADOOP-6622. Token should not print the password in toString. (eli) - - HADOOP-7529. Fix lock cycles in metrics system. (llu) - - HADOOP-7545. Common -tests JAR should not include properties and configs. - (todd) - - HADOOP-7536. Correct the dependency version regressions introduced in - HADOOP-6671. (Alejandro Abdelnur via tomwhite) - - HADOOP-7566. MR tests are failing webapps/hdfs not found in CLASSPATH. - (Alejandro Abdelnur via mahadev) - - HADOOP-7567. 'mvn eclipse:eclipse' fails for hadoop-alfredo (auth). - (Alejandro Abdelnur via tomwhite) - - HADOOP-7563. Setup HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME and classpath - correction. (Eric Yang via acmurthy) - - HADOOP-7560. Change src layout to be heirarchical. (Alejandro Abdelnur - via acmurthy) - - HADOOP-7576. Fix findbugs warnings and javac warnings in hadoop-auth. - (szetszwo) - - HADOOP-7593. Fix AssertionError in TestHttpServer.testMaxThreads(). - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7598. Fix smart-apply-patch.sh to handle patching from a sub - directory correctly. (Robert Evans via acmurthy) - - HADOOP-7328. When a serializer class is missing, return null, not throw - an NPE. (Harsh J Chouraria via todd) - - HADOOP-7626. Bugfix for a config generator (Eric Yang via ddas) - - HADOOP-7629. Allow immutable FsPermission objects to be used as IPC - parameters. (todd) - - HADOOP-7608. SnappyCodec check for Hadoop native lib is wrong - (Alejandro Abdelnur via todd) - - HADOOP-7637. Fix to include FairScheduler configuration file in - RPM. (Eric Yang via ddas) - - HADOOP-7633. Adds log4j.properties to the hadoop-conf dir on - deploy (Eric Yang via ddas) - - HADOOP-7631. Fixes a config problem to do with running streaming jobs - (Eric Yang via ddas) - - HADOOP-7662. Fixed logs servlet to use the pathspec '/*' instead of '/' - for correct filtering. (Thomas Graves via vinodkv) - - HADOOP-7691. Fixed conflict uid for install packages. (Eric Yang) - - HADOOP-7603. Set hdfs, mapred uid, and hadoop uid to fixed numbers. - (Eric Yang) - - HADOOP-7658. Fixed HADOOP_SECURE_DN_USER environment variable in - hadoop-evn.sh (Eric Yang) - - HADOOP-7684. Added init.d script for jobhistory server and - secondary namenode. (Eric Yang) - - HADOOP-7715. Removed unnecessary security logger configuration. (Eric Yang) - - HADOOP-7685. Improved directory ownership check function in - hadoop-setup-conf.sh. (Eric Yang) - - HADOOP-7711. Fixed recursive sourcing of HADOOP_OPTS environment - variables (Arpit Gupta via Eric Yang) - - HADOOP-7681. Fixed security and hdfs audit log4j properties - (Arpit Gupta via Eric Yang) - - HADOOP-7708. Fixed hadoop-setup-conf.sh to handle config files - consistently. (Eric Yang) - - HADOOP-7724. Fixed hadoop-setup-conf.sh to put proxy user in - core-site.xml. (Arpit Gupta via Eric Yang) - - HADOOP-7755. Detect MapReduce PreCommit Trunk builds silently failing - when running test-patch.sh. (Jonathan Eagles via tomwhite) - - HADOOP-7744. Ensure failed tests exit with proper error code. (Jonathan - Eagles via acmurthy) - - HADOOP-7764. Allow HttpServer to set both ACL list and path spec filters. - (Jonathan Eagles via acmurthy) - - HADOOP-7766. The auth to local mappings are not being respected, with webhdfs - and security enabled. (jitendra) - - HADOOP-7721. Add log before login in KerberosAuthenticationHandler. - (jitendra) - - HADOOP-7778. FindBugs warning in Token.getKind(). (tomwhite) - - HADOOP-7798. Add support gpg signatures for maven release artifacts. - (cutting via acmurthy) - - HADOOP-7797. Fix top-level pom.xml to refer to correct staging maven - repository. (omalley via acmurthy) - - HADOOP-7101. UserGroupInformation.getCurrentUser() fails when called from - non-Hadoop JAAS context. (todd) - -Release 0.22.1 - Unreleased - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-7937. Forward port SequenceFile#syncFs and friends from Hadoop 1.x. - (tomwhite) - -Release 0.22.0 - 2011-11-29 - - INCOMPATIBLE CHANGES - - HADOOP-7137. Remove hod contrib. (nigel via eli) - - NEW FEATURES - - HADOOP-6791. Refresh for proxy superuser config - (common part for HDFS-1096) (boryas) - - HADOOP-6581. Add authenticated TokenIdentifiers to UGI so that - they can be used for authorization (Kan Zhang and Jitendra Pandey - via jghoman) - - HADOOP-6584. Provide Kerberized SSL encryption for webservices. - (jghoman and Kan Zhang via jghoman) - - HADOOP-6853. Common component of HDFS-1045. (jghoman) - - HADOOP-6859 - Introduce additional statistics to FileSystem to track - file system operations (suresh) - - HADOOP-6870. Add a new API getFiles to FileSystem and FileContext that - lists all files under the input path or the subtree rooted at the - input path if recursive is true. Block locations are returned together - with each file's status. (hairong) - - HADOOP-6888. Add a new FileSystem API closeAllForUGI(..) for closing all - file systems associated with a particular UGI. (Devaraj Das and Kan Zhang - via szetszwo) - - HADOOP-6892. Common component of HDFS-1150 (Verify datanodes' identities - to clients in secure clusters) (jghoman) - - HADOOP-6889. Make RPC to have an option to timeout. (hairong) - - HADOOP-6996. Allow CodecFactory to return a codec object given a codec' - class name. (hairong) - - HADOOP-7013. Add boolean field isCorrupt to BlockLocation. - (Patrick Kling via hairong) - - HADOOP-6978. Adds support for NativeIO using JNI. - (Todd Lipcon, Devaraj Das & Owen O'Malley via ddas) - - HADOOP-7134. configure files that are generated as part of the released - tarball need to have executable bit set. (Roman Shaposhnik via cos) - - IMPROVEMENTS - - HADOOP-6644. util.Shell getGROUPS_FOR_USER_COMMAND method name - - should use common naming convention (boryas) - - HADOOP-6778. add isRunning() method to - AbstractDelegationTokenSecretManager (for HDFS-1044) (boryas) - - HADOOP-6633. normalize property names for JT/NN kerberos principal - names in configuration (boryas) - - HADOOP-6627. "Bad Connection to FS" message in FSShell should print - message from the exception (boryas) - - HADOOP-6600. mechanism for authorization check for inter-server - protocols. (boryas) - - HADOOP-6623. Add StringUtils.split for non-escaped single-character - separator. (Todd Lipcon via tomwhite) - - HADOOP-6761. The Trash Emptier has the ability to run more frequently. - (Dmytro Molkov via dhruba) - - HADOOP-6714. Resolve compressed files using CodecFactory in FsShell::text. - (Patrick Angeles via cdouglas) - - HADOOP-6661. User document for UserGroupInformation.doAs. - (Jitendra Pandey via jghoman) - - HADOOP-6674. Makes use of the SASL authentication options in the - SASL RPC. (Jitendra Pandey via ddas) - - HADOOP-6526. Need mapping from long principal names to local OS - user names. (boryas) - - HADOOP-6814. Adds an API in UserGroupInformation to get the real - authentication method of a passed UGI. (Jitendra Pandey via ddas) - - HADOOP-6756. Documentation for common configuration keys. - (Erik Steffl via shv) - - HADOOP-6835. Add support for concatenated gzip input. (Greg Roelofs via - cdouglas) - - HADOOP-6845. Renames the TokenStorage class to Credentials. - (Jitendra Pandey via ddas) - - HADOOP-6826. FileStatus needs unit tests. (Rodrigo Schmidt via Eli - Collins) - - HADOOP-6905. add buildDTServiceName method to SecurityUtil - (as part of MAPREDUCE-1718) (boryas) - - HADOOP-6632. Adds support for using different keytabs for different - servers in a Hadoop cluster. In the earier implementation, all servers - of a certain type (like TaskTracker), would have the same keytab and the - same principal. Now the principal name is a pattern that has _HOST in it. - (Kan Zhang & Jitendra Pandey via ddas) - - HADOOP-6861. Adds new non-static methods in Credentials to read and - write token storage file. (Jitendra Pandey & Owen O'Malley via ddas) - - HADOOP-6877. Common part of HDFS-1178 (NameNode servlets should communicate - with NameNode directrly). (Kan Zhang via jghoman) - - HADOOP-6475. Adding some javadoc to Server.RpcMetrics, UGI. - (Jitendra Pandey and borya via jghoman) - - HADOOP-6656. Adds a thread in the UserGroupInformation to renew TGTs - periodically. (Owen O'Malley and ddas via ddas) - - HADOOP-6890. Improve listFiles API introduced by HADOOP-6870. (hairong) - - HADOOP-6862. Adds api to add/remove user and group to AccessControlList - (amareshwari) - - HADOOP-6911. doc update for DelegationTokenFetcher (boryas) - - HADOOP-6900. Make the iterator returned by FileSystem#listLocatedStatus to - throw IOException rather than RuntimeException when there is an IO error - fetching the next file. (hairong) - - HADOOP-6905. Better logging messages when a delegation token is invalid. - (Kan Zhang via jghoman) - - HADOOP-6693. Add metrics to track kerberol login activity. (suresh) - - HADOOP-6803. Add native gzip read/write coverage to TestCodec. - (Eli Collins via tomwhite) - - HADOOP-6950. Suggest that HADOOP_CLASSPATH should be preserved in - hadoop-env.sh.template. (Philip Zeyliger via Eli Collins) - - HADOOP-6922. Make AccessControlList a writable and update documentation - for Job ACLs. (Ravi Gummadi via vinodkv) - - HADOOP-6965. Introduces checks for whether the original tgt is valid - in the reloginFromKeytab method. - - HADOOP-6856. Simplify constructors for SequenceFile, and MapFile. (omalley) - - HADOOP-6987. Use JUnit Rule to optionally fail test cases that run more - than 10 seconds (jghoman) - - HADOOP-7005. Update test-patch.sh to remove callback to Hudson. (nigel) - - HADOOP-6985. Suggest that HADOOP_OPTS be preserved in - hadoop-env.sh.template. (Ramkumar Vadali via cutting) - - HADOOP-7007. Update the hudson-test-patch ant target to work with the - latest test-patch.sh script (gkesavan) - - HADOOP-7010. Typo in FileSystem.java. (Jingguo Yao via eli) - - HADOOP-7009. MD5Hash provides a public factory method that creates an - instance of thread local MessageDigest. (hairong) - - HADOOP-7008. Enable test-patch.sh to have a configured number of - acceptable findbugs and javadoc warnings. (nigel and gkesavan) - - HADOOP-6818. Provides a JNI implementation of group resolution. (ddas) - - HADOOP-6943. The GroupMappingServiceProvider interface should be public. - (Aaron T. Myers via tomwhite) - - HADOOP-4675. Current Ganglia metrics implementation is incompatible with - Ganglia 3.1. (Brian Bockelman via tomwhite) - - HADOOP-6977. Herriot daemon clients should vend statistics (cos) - - HADOOP-7024. Create a test method for adding file systems during tests. - (Kan Zhang via jghoman) - - HADOOP-6903. Make AbstractFSileSystem methods and some FileContext methods - to be public. (Sanjay Radia) - - HADOOP-7034. Add TestPath tests to cover dot, dot dot, and slash - normalization. (eli) - - HADOOP-7032. Assert type constraints in the FileStatus constructor. (eli) - - HADOOP-6562. FileContextSymlinkBaseTest should use FileContextTestHelper. - (eli) - - HADOOP-7028. ant eclipse does not include requisite ant.jar in the - classpath. (Patrick Angeles via eli) - - HADOOP-6298. Add copyBytes to Text and BytesWritable. (omalley) - - HADOOP-6578. Configuration should trim whitespace around a lot of value - types. (Michele Catasta via eli) - - HADOOP-6811. Remove EC2 bash scripts. They are replaced by Apache Whirr - (incubating, http://incubator.apache.org/whirr). (tomwhite) - - HADOOP-7102. Remove "fs.ramfs.impl" field from core-deafult.xml (shv) - - HADOOP-7104. Remove unnecessary DNS reverse lookups from RPC layer - (Kan Zhang via todd) - - HADOOP-6056. Use java.net.preferIPv4Stack to force IPv4. - (Michele Catasta via shv) - - HADOOP-7110. Implement chmod with JNI. (todd) - - HADOOP-6812. Change documentation for correct placement of configuration - variables: mapreduce.reduce.input.buffer.percent, - mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb - (Chris Douglas via shv) - - HADOOP-6436. Remove auto-generated native build files. (rvs via eli) - - HADOOP-6970. SecurityAuth.audit should be generated under /build. (boryas) - - HADOOP-7154. Should set MALLOC_ARENA_MAX in hadoop-env.sh (todd) - - HADOOP-7187. Fix socket leak in GangliaContext. (Uma Maheswara Rao G - via szetszwo) - - HADOOP-7241. fix typo of command 'hadoop fs -help tail'. - (Wei Yongjun via eli) - - HADOOP-7244. Documentation change for updated configuration keys. - (tomwhite via eli) - - HADOOP-7189. Add ability to enable 'debug' property in JAAS configuration. - (Ted Yu via todd) - - HADOOP-7192. Update fs -stat docs to reflect the format features. (Harsh - J Chouraria via todd) - - HADOOP-7355 Add audience and stability annotations to HttpServer class - (stack) - - HADOOP-7346. Send back nicer error message to clients using outdated IPC - version. (todd) - - HADOOP-7335. Force entropy to come from non-true random for tests. - (todd via eli) - - HADOOP-7325. The hadoop command should not accept class names starting with - a hyphen. (Brock Noland via todd) - - HADOOP-7772. javadoc the topology classes (stevel) - - HADOOP-7786. Remove HDFS-specific config keys defined in FsConfig. (eli) - - HADOOP-7861. changes2html.pl generates links to HADOOP, HDFS, and MAPREDUCE - jiras. (shv) - - OPTIMIZATIONS - - HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..). - (Erik Steffl via szetszwo) - - HADOOP-6683. ZlibCompressor does not fully utilize the buffer. - (Kang Xiao via eli) - - HADOOP-6949. Reduce RPC packet size of primitive arrays using - ArrayPrimitiveWritable instead of ObjectWritable. (Matt Foley via suresh) - - BUG FIXES - - HADOOP-6638. try to relogin in a case of failed RPC connection (expired - tgt) only in case the subject is loginUser or proxyUgi.realUser. (boryas) - - HADOOP-6781. security audit log shouldn't have exception in it. (boryas) - - HADOOP-6612. Protocols RefreshUserToGroupMappingsProtocol and - RefreshAuthorizationPolicyProtocol will fail with security enabled (boryas) - - HADOOP-6764. Remove verbose logging from the Groups class. (Boris Shkolnik) - - HADOOP-6730. Bug in FileContext#copy and provide base class for - FileContext tests. (Ravi Phulari via jghoman) - - HADOOP-6669. Respect compression configuration when creating DefaultCodec - instances. (Koji Noguchi via cdouglas) - - HADOOP-6747. TestNetUtils fails on Mac OS X. (Todd Lipcon via jghoman) - - HADOOP-6787. Factor out glob pattern code from FileContext and - Filesystem. Also fix bugs identified in HADOOP-6618 and make the - glob pattern code less restrictive and more POSIX standard - compliant. (Luke Lu via eli) - - HADOOP-6649. login object in UGI should be inside the subject (jnp via - boryas) - - HADOOP-6687. user object in the subject in UGI should be reused in case - of a relogin. (jnp via boryas) - - HADOOP-6603. Provide workaround for issue with Kerberos not resolving - cross-realm principal (Kan Zhang and Jitendra Pandey via jghoman) - - HADOOP-6620. NPE if renewer is passed as null in getDelegationToken. - (Jitendra Pandey via jghoman) - - HADOOP-6613. Moves the RPC version check ahead of the AuthMethod check. - (Kan Zhang via ddas) - - HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting - with [a-f] correctly. (jghoman) - - HADOOP-6652. Removes the unnecessary cache from - ShellBasedUnixGroupsMapping. (ddas) - - HADOOP-6815. refreshSuperUserGroupsConfiguration should use server side - configuration for the refresh (boryas) - - HADOOP-6648. Adds a check for null tokens in Credentials.addToken api. - (ddas) - - HADOOP-6647. balancer fails with "is not authorized for protocol - interface NamenodeProtocol" in secure environment (boryas) - - HADOOP-6834. TFile.append compares initial key against null lastKey - (hong tang via mahadev) - - HADOOP-6670. Use the UserGroupInformation's Subject as the criteria for - equals and hashCode. (Owen O'Malley and Kan Zhang via ddas) - - HADOOP-6536. Fixes FileUtil.fullyDelete() not to delete the contents of - the sym-linked directory. (Ravi Gummadi via amareshwari) - - HADOOP-6873. using delegation token over hftp for long - running clients (boryas) - - HADOOP-6706. Improves the sasl failure handling due to expired tickets, - and other server detected failures. (Jitendra Pandey and ddas via ddas) - - HADOOP-6715. Fixes AccessControlList.toString() to return a descriptive - String representation of the ACL. (Ravi Gummadi via amareshwari) - - HADOOP-6885. Fix java doc warnings in Groups and - RefreshUserMappingsProtocol. (Eli Collins via jghoman) - - HADOOP-6482. GenericOptionsParser constructor that takes Options and - String[] ignores options. (Eli Collins via jghoman) - - HADOOP-6906. FileContext copy() utility doesn't work with recursive - copying of directories. (vinod k v via mahadev) - - HADOOP-6453. Hadoop wrapper script shouldn't ignore an existing - JAVA_LIBRARY_PATH. (Chad Metcalf via jghoman) - - HADOOP-6932. Namenode start (init) fails because of invalid kerberos - key, even when security set to "simple" (boryas) - - HADOOP-6913. Circular initialization between UserGroupInformation and - KerberosName (Kan Zhang via boryas) - - HADOOP-6907. Rpc client doesn't use the per-connection conf to figure - out server's Kerberos principal (Kan Zhang via hairong) - - HADOOP-6938. ConnectionId.getRemotePrincipal() should check if security - is enabled. (Kan Zhang via hairong) - - HADOOP-6930. AvroRpcEngine doesn't work with generated Avro code. - (sharad) - - HADOOP-6940. RawLocalFileSystem's markSupported method misnamed - markSupport. (Tom White via eli). - - HADOOP-6951. Distinct minicluster services (e.g. NN and JT) overwrite each - other's service policies. (Aaron T. Myers via tomwhite) - - HADOOP-6879. Provide SSH based (Jsch) remote execution API for system - tests (cos) - - HADOOP-6989. Correct the parameter for SetFile to set the value type - for SetFile to be NullWritable instead of the key. (cdouglas via omalley) - - HADOOP-6984. Combine the compress kind and the codec in the same option - for SequenceFiles. (cdouglas via omalley) - - HADOOP-6933. TestListFiles is flaky. (Todd Lipcon via tomwhite) - - HADOOP-6947. Kerberos relogin should set refreshKrb5Config to true. - (Todd Lipcon via tomwhite) - - HADOOP-7006. Fix 'fs -getmerge' command to not be a no-op. - (Chris Nauroth via cutting) - - HADOOP-6663. BlockDecompressorStream get EOF exception when decompressing - the file compressed from empty file. (Kang Xiao via tomwhite) - - HADOOP-6991. Fix SequenceFile::Reader to honor file lengths and call - openFile (cdouglas via omalley) - - HADOOP-7011. Fix KerberosName.main() to not throw an NPE. - (Aaron T. Myers via tomwhite) - - HADOOP-6975. Integer overflow in S3InputStream for blocks > 2GB. - (Patrick Kling via tomwhite) - - HADOOP-6758. MapFile.fix does not allow index interval definition. - (Gianmarco De Francisci Morales via tomwhite) - - HADOOP-6926. SocketInputStream incorrectly implements read(). - (Todd Lipcon via tomwhite) - - HADOOP-6899 RawLocalFileSystem#setWorkingDir() does not work for relative names - (Sanjay Radia) - - HADOOP-6496. HttpServer sends wrong content-type for CSS files - (and others). (Todd Lipcon via tomwhite) - - HADOOP-7057. IOUtils.readFully and IOUtils.skipFully have typo in - exception creation's message. (cos) - - HADOOP-7038. saveVersion script includes an additional \r while running - whoami under windows. (Wang Xu via cos) - - HADOOP-7082. Configuration.writeXML should not hold lock while outputting - (todd) - - HADOOP-7070. JAAS configuration should delegate unknown application names - to pre-existing configuration. (todd) - - HADOOP-7087. SequenceFile.createWriter ignores FileSystem parameter (todd) - - HADOOP-7091. reloginFromKeytab() should happen even if TGT can't be found. - (Kan Zhang via jghoman) - - HADOOP-7100. Fix build to not refer to contrib/ec2 removed by HADOOP-6811 - (todd) - - HADOOP-7097. JAVA_LIBRARY_PATH missing base directory. (Noah Watkins via - todd) - - HADOOP-7093. Servlets should default to text/plain (todd) - - HADOOP-7089. Fix link resolution logic in hadoop-config.sh. (eli) - - HADOOP-7046. Fix Findbugs warning in Configuration. (Po Cheung via shv) - - HADOOP-7118. Fix NPE in Configuration.writeXml (todd) - - HADOOP-7122. Fix thread leak when shell commands time out. (todd) - - HADOOP-7126. Fix file permission setting for RawLocalFileSystem on Windows. - (Po Cheung via shv) - - HADOOP-6642. Fix javac, javadoc, findbugs warnings related to security work. - (Chris Douglas, Po Cheung via shv) - - HADOOP-7140. IPC Reader threads do not stop when server stops (todd) - - HADOOP-7094. hadoop.css got lost during project split (cos) - - HADOOP-7145. Configuration.getLocalPath should trim whitespace from - the provided directories. (todd) - - HADOOP-7156. Workaround for unsafe implementations of getpwuid_r (todd) - - HADOOP-6898. FileSystem.copyToLocal creates files with 777 permissions. - (Aaron T. Myers via tomwhite) - - HADOOP-7229. Do not default to an absolute path for kinit in Kerberos - auto-renewal thread. (Aaron T. Myers via todd) - - HADOOP-7172. SecureIO should not check owner on non-secure - clusters that have no native support. (todd via eli) - - HADOOP-7184. Remove deprecated config local.cache.size from - core-default.xml (todd) - - HADOOP-7245. FsConfig should use constants in CommonConfigurationKeys. - (tomwhite via eli) - - HADOOP-7068. Ivy resolve force mode should be turned off by default. - (Luke Lu via tomwhite) - - HADOOP-7296. The FsPermission(FsPermission) constructor does not use the - sticky bit. (Siddharth Seth via tomwhite) - - HADOOP-7300. Configuration methods that return collections are inconsistent - about mutability. (todd) - - HADOOP-7305. Eclipse project classpath should include tools.jar from JDK. - (Niels Basjes via todd) - - HADOOP-7318. MD5Hash factory should reset the digester it returns. - (todd via eli) - - HADOOP-7287. Configuration deprecation mechanism doesn't work properly for - GenericOptionsParser and Tools. (Aaron T. Myers via todd) - - HADOOP-7146. RPC server leaks file descriptors (todd) - - HADOOP-7276. Hadoop native builds fail on ARM due to -m32 (Trevor Robinson - via eli) - - HADOOP-7121. Exceptions while serializing IPC call responses are not - handled well. (todd) - - HADOOP-7351 Regression: HttpServer#getWebAppsPath used to be protected - so subclasses could supply alternate webapps path but it was made private - by HADOOP-6461 (Stack) - - HADOOP-7349. HADOOP-7121 accidentally disabled some tests in TestIPC. - (todd) - - HADOOP-7390. VersionInfo not generated properly in git after unsplit. (todd - via atm) - - HADOOP-7568. SequenceFile should not print into stdout. - (Plamen Jeliazkov via shv) - - HADOOP-7663. Fix TestHDFSTrash failure. (Mayank Bansal via shv) - - HADOOP-7457. Remove out-of-date Chinese language documentation. - (Jakob Homan via eli) - - HADOOP-7783. Add more symlink tests that cover intermediate links. (eli) - -Release 0.21.1 - Unreleased - - IMPROVEMENTS - - HADOOP-6934. Test for ByteWritable comparator. - (Johannes Zillmann via Eli Collins) - - HADOOP-6786. test-patch needs to verify Herriot integrity (cos) - - HADOOP-7177. CodecPool should report which compressor it is using. - (Allen Wittenauer via eli) - - BUG FIXES - - HADOOP-6925. BZip2Codec incorrectly implements read(). - (Todd Lipcon via Eli Collins) - - HADOOP-6833. IPC leaks call parameters when exceptions thrown. - (Todd Lipcon via Eli Collins) - - HADOOP-6971. Clover build doesn't generate per-test coverage (cos) - - HADOOP-6993. Broken link on cluster setup page of docs. (eli) - - HADOOP-6944. [Herriot] Implement a functionality for getting proxy users - definitions like groups and hosts. (Vinay Thota via cos) - - HADOOP-6954. Sources JARs are not correctly published to the Maven - repository. (tomwhite) - - HADOOP-7052. misspelling of threshold in conf/log4j.properties. - (Jingguo Yao via eli) - - HADOOP-7053. wrong FSNamesystem Audit logging setting in - conf/log4j.properties. (Jingguo Yao via eli) - - HADOOP-7120. Fix a syntax error in test-patch.sh. (szetszwo) - - HADOOP-7162. Rmove a duplicated call FileSystem.listStatus(..) in FsShell. - (Alexey Diomin via szetszwo) - - HADOOP-7117. Remove fs.checkpoint.* from core-default.xml and replace - fs.checkpoint.* with dfs.namenode.checkpoint.* in documentations. - (Harsh J Chouraria via szetszwo) - - HADOOP-7193. Correct the "fs -touchz" command help message. - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7174. Null is displayed in the "fs -copyToLocal" command. - (Uma Maheswara Rao G via szetszwo) - - HADOOP-7194. Fix resource leak in IOUtils.copyBytes(..). - (Devaraj K via szetszwo) - - HADOOP-7183. WritableComparator.get should not cache comparator objects. - (tomwhite via eli) - -Release 0.21.0 - 2010-08-13 - - INCOMPATIBLE CHANGES - - HADOOP-4895. Remove deprecated methods DFSClient.getHints(..) and - DFSClient.isDirectory(..). (szetszwo) - - HADOOP-4941. Remove deprecated FileSystem methods: getBlockSize(Path f), - getLength(Path f) and getReplication(Path src). (szetszwo) - - HADOOP-4648. Remove obsolete, deprecated InMemoryFileSystem and - ChecksumDistributedFileSystem. (cdouglas via szetszwo) - - HADOOP-4940. Remove a deprecated method FileSystem.delete(Path f). (Enis - Soztutar via szetszwo) - - HADOOP-4010. Change semantics for LineRecordReader to read an additional - line per split- rather than moving back one character in the stream- to - work with splittable compression codecs. (Abdul Qadeer via cdouglas) - - HADOOP-5094. Show hostname and separate live/dead datanodes in DFSAdmin - report. (Jakob Homan via szetszwo) - - HADOOP-4942. Remove deprecated FileSystem methods getName() and - getNamed(String name, Configuration conf). (Jakob Homan via szetszwo) - - HADOOP-5486. Removes the CLASSPATH string from the command line and instead - exports it in the environment. (Amareshwari Sriramadasu via ddas) - - HADOOP-2827. Remove deprecated NetUtils::getServerAddress. (cdouglas) - - HADOOP-5681. Change examples RandomWriter and RandomTextWriter to - use new mapreduce API. (Amareshwari Sriramadasu via sharad) - - HADOOP-5680. Change org.apache.hadoop.examples.SleepJob to use new - mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5699. Change org.apache.hadoop.examples.PiEstimator to use - new mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5720. Introduces new task types - JOB_SETUP, JOB_CLEANUP - and TASK_CLEANUP. Removes the isMap methods from TaskID/TaskAttemptID - classes. (ddas) - - HADOOP-5668. Change TotalOrderPartitioner to use new API. (Amareshwari - Sriramadasu via cdouglas) - - HADOOP-5738. Split "waiting_tasks" JobTracker metric into waiting maps and - waiting reduces. (Sreekanth Ramakrishnan via cdouglas) - - HADOOP-5679. Resolve findbugs warnings in core/streaming/pipes/examples. - (Jothi Padmanabhan via sharad) - - HADOOP-4359. Support for data access authorization checking on Datanodes. - (Kan Zhang via rangadi) - - HADOOP-5690. Change org.apache.hadoop.examples.DBCountPageView to use - new mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5694. Change org.apache.hadoop.examples.dancing to use new - mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5696. Change org.apache.hadoop.examples.Sort to use new - mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5698. Change org.apache.hadoop.examples.MultiFileWordCount to - use new mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5913. Provide ability to an administrator to stop and start - job queues. (Rahul Kumar Singh and Hemanth Yamijala via yhemanth) - - MAPREDUCE-711. Removed Distributed Cache from Common, to move it - under Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-6201. Change FileSystem::listStatus contract to throw - FileNotFoundException if the directory does not exist, rather than letting - this be implementation-specific. (Jakob Homan via cdouglas) - - HADOOP-6230. Moved process tree and memory calculator related classes - from Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-6203. FsShell rm/rmr error message indicates exceeding Trash quota - and suggests using -skpTrash, when moving to trash fails. - (Boris Shkolnik via suresh) - - HADOOP-6303. Eclipse .classpath template has outdated jar files and is - missing some new ones. (cos) - - HADOOP-6396. Fix uninformative exception message when unable to parse - umask. (jghoman) - - HADOOP-6299. Reimplement the UserGroupInformation to use the OS - specific and Kerberos JAAS login. (omalley) - - HADOOP-6686. Remove redundant exception class name from the exception - message for the exceptions thrown at RPC client. (suresh) - - HADOOP-6701. Fix incorrect exit codes returned from chmod, chown and chgrp - commands from FsShell. (Ravi Phulari via suresh) - - NEW FEATURES - - HADOOP-6332. Large-scale Automated Test Framework. (sharad, Sreekanth - Ramakrishnan, at all via cos) - - HADOOP-4268. Change fsck to use ClientProtocol methods so that the - corresponding permission requirement for running the ClientProtocol - methods will be enforced. (szetszwo) - - HADOOP-3953. Implement sticky bit for directories in HDFS. (Jakob Homan - via szetszwo) - - HADOOP-4368. Implement df in FsShell to show the status of a FileSystem. - (Craig Macdonald via szetszwo) - - HADOOP-3741. Add a web ui to the SecondaryNameNode for showing its status. - (szetszwo) - - HADOOP-5018. Add pipelined writers to Chukwa. (Ari Rabkin via cdouglas) - - HADOOP-5052. Add an example computing exact digits of pi using the - Bailey-Borwein-Plouffe algorithm. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-4927. Adds a generic wrapper around outputformat to allow creation of - output on demand (Jothi Padmanabhan via ddas) - - HADOOP-5144. Add a new DFSAdmin command for changing the setting of restore - failed storage replicas in namenode. (Boris Shkolnik via szetszwo) - - HADOOP-5258. Add a new DFSAdmin command to print a tree of the rack and - datanode topology as seen by the namenode. (Jakob Homan via szetszwo) - - HADOOP-4756. A command line tool to access JMX properties on NameNode - and DataNode. (Boris Shkolnik via rangadi) - - HADOOP-4539. Introduce backup node and checkpoint node. (shv) - - HADOOP-5363. Add support for proxying connections to multiple clusters with - different versions to hdfsproxy. (Zhiyong Zhang via cdouglas) - - HADOOP-5528. Add a configurable hash partitioner operating on ranges of - BinaryComparable keys. (Klaas Bosteels via shv) - - HADOOP-5257. HDFS servers may start and stop external components through - a plugin interface. (Carlos Valiente via dhruba) - - HADOOP-5450. Add application-specific data types to streaming's typed bytes - interface. (Klaas Bosteels via omalley) - - HADOOP-5518. Add contrib/mrunit, a MapReduce unit test framework. - (Aaron Kimball via cutting) - - HADOOP-5469. Add /metrics servlet to daemons, providing metrics - over HTTP as either text or JSON. (Philip Zeyliger via cutting) - - HADOOP-5467. Introduce offline fsimage image viewer. (Jakob Homan via shv) - - HADOOP-5752. Add a new hdfs image processor, Delimited, to oiv. (Jakob - Homan via szetszwo) - - HADOOP-5266. Adds the capability to do mark/reset of the reduce values - iterator in the Context object API. (Jothi Padmanabhan via ddas) - - HADOOP-5745. Allow setting the default value of maxRunningJobs for all - pools. (dhruba via matei) - - HADOOP-5643. Adds a way to decommission TaskTrackers while the JobTracker - is running. (Amar Kamat via ddas) - - HADOOP-4829. Allow FileSystem shutdown hook to be disabled. - (Todd Lipcon via tomwhite) - - HADOOP-5815. Sqoop: A database import tool for Hadoop. - (Aaron Kimball via tomwhite) - - HADOOP-4861. Add disk usage with human-readable size (-duh). - (Todd Lipcon via tomwhite) - - HADOOP-5844. Use mysqldump when connecting to local mysql instance in Sqoop. - (Aaron Kimball via tomwhite) - - HADOOP-5976. Add a new command, classpath, to the hadoop script. (Owen - O'Malley and Gary Murry via szetszwo) - - HADOOP-6120. Add support for Avro specific and reflect data. - (sharad via cutting) - - HADOOP-6226. Moves BoundedByteArrayOutputStream from the tfile package to - the io package and makes it available to other users (MAPREDUCE-318). - (Jothi Padmanabhan via ddas) - - HADOOP-6105. Adds support for automatically handling deprecation of - configuration keys. (V.V.Chaitanya Krishna via yhemanth) - - HADOOP-6235. Adds new method to FileSystem for clients to get server - defaults. (Kan Zhang via suresh) - - HADOOP-6234. Add new option dfs.umaskmode to set umask in configuration - to use octal or symbolic instead of decimal. (Jakob Homan via suresh) - - HADOOP-5073. Add annotation mechanism for interface classification. - (Jakob Homan via suresh) - - HADOOP-4012. Provide splitting support for bzip2 compressed files. (Abdul - Qadeer via cdouglas) - - HADOOP-6246. Add backward compatibility support to use deprecated decimal - umask from old configuration. (Jakob Homan via suresh) - - HADOOP-4952. Add new improved file system interface FileContext for the - application writer (Sanjay Radia via suresh) - - HADOOP-6170. Add facility to tunnel Avro RPCs through Hadoop RPCs. - This permits one to take advantage of both Avro's RPC versioning - features and Hadoop's proven RPC scalability. (cutting) - - HADOOP-6267. Permit building contrib modules located in external - source trees. (Todd Lipcon via cutting) - - HADOOP-6240. Add new FileContext rename operation that posix compliant - that allows overwriting existing destination. (suresh) - - HADOOP-6204. Implementing aspects development and fault injeciton - framework for Hadoop (cos) - - HADOOP-6313. Implement Syncable interface in FSDataOutputStream to expose - flush APIs to application users. (Hairong Kuang via suresh) - - HADOOP-6284. Add a new parameter, HADOOP_JAVA_PLATFORM_OPTS, to - hadoop-config.sh so that it allows setting java command options for - JAVA_PLATFORM. (Koji Noguchi via szetszwo) - - HADOOP-6337. Updates FilterInitializer class to be more visible, - and the init of the class is made to take a Configuration argument. - (Jakob Homan via ddas) - - Hadoop-6223. Add new file system interface AbstractFileSystem with - implementation of some file systems that delegate to old FileSystem. - (Sanjay Radia via suresh) - - HADOOP-6433. Introduce asychronous deletion of files via a pool of - threads. This can be used to delete files in the Distributed - Cache. (Zheng Shao via dhruba) - - HADOOP-6415. Adds a common token interface for both job token and - delegation token. (Kan Zhang via ddas) - - HADOOP-6408. Add a /conf servlet to dump running configuration. - (Todd Lipcon via tomwhite) - - HADOOP-6520. Adds APIs to read/write Token and secret keys. Also - adds the automatic loading of tokens into UserGroupInformation - upon login. The tokens are read from a file specified in the - environment variable. (ddas) - - HADOOP-6419. Adds SASL based authentication to RPC. - (Kan Zhang via ddas) - - HADOOP-6510. Adds a way for superusers to impersonate other users - in a secure environment. (Jitendra Nath Pandey via ddas) - - HADOOP-6421. Adds Symbolic links to FileContext, AbstractFileSystem. - It also adds a limited implementation for the local file system - (RawLocalFs) that allows local symlinks. (Eli Collins via Sanjay Radia) - - HADOOP-6577. Add hidden configuration option "ipc.server.max.response.size" - to change the default 1 MB, the maximum size when large IPC handler - response buffer is reset. (suresh) - - HADOOP-6568. Adds authorization for the default servlets. - (Vinod Kumar Vavilapalli via ddas) - - HADOOP-6586. Log authentication and authorization failures and successes - for RPC (boryas) - - HADOOP-6580. UGI should contain authentication method. (jnp via boryas) - - HADOOP-6657. Add a capitalization method to StringUtils for MAPREDUCE-1545. - (Luke Lu via Steve Loughran) - - HADOOP-6692. Add FileContext#listStatus that returns an iterator. - (hairong) - - HADOOP-6869. Functionality to create file or folder on a remote daemon - side (Vinay Thota via cos) - - IMPROVEMENTS - - HADOOP-6798. Align Ivy version for all Hadoop subprojects. (cos) - - HADOOP-6777. Implement a functionality for suspend and resume a process. - (Vinay Thota via cos) - - HADOOP-6772. Utilities for system tests specific. (Vinay Thota via cos) - - HADOOP-6771. Herriot's artifact id for Maven deployment should be set to - hadoop-core-instrumented (cos) - - HADOOP-6752. Remote cluster control functionality needs JavaDocs - improvement (Balaji Rajagopalan via cos). - - HADOOP-4565. Added CombineFileInputFormat to use data locality information - to create splits. (dhruba via zshao) - - HADOOP-4936. Improvements to TestSafeMode. (shv) - - HADOOP-4985. Remove unnecessary "throw IOException" declarations in - FSDirectory related methods. (szetszwo) - - HADOOP-5017. Change NameNode.namesystem declaration to private. (szetszwo) - - HADOOP-4794. Add branch information from the source version control into - the version information that is compiled into Hadoop. (cdouglas via - omalley) - - HADOOP-5070. Increment copyright year to 2009, remove assertions of ASF - copyright to licensed files. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-5037. Deprecate static FSNamesystem.getFSNamesystem(). (szetszwo) - - HADOOP-5088. Include releaseaudit target as part of developer test-patch - target. (Giridharan Kesavan via nigel) - - HADOOP-2721. Uses setsid when creating new tasks so that subprocesses of - this process will be within this new session (and this process will be - the process leader for all the subprocesses). Killing the process leader, - or the main Java task in Hadoop's case, kills the entire subtree of - processes. (Ravi Gummadi via ddas) - - HADOOP-5097. Remove static variable JspHelper.fsn, a static reference to - a non-singleton FSNamesystem object. (szetszwo) - - HADOOP-3327. Improves handling of READ_TIMEOUT during map output copying. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5124. Choose datanodes randomly instead of starting from the first - datanode for providing fairness. (hairong via szetszwo) - - HADOOP-4930. Implement a Linux native executable that can be used to - launch tasks as users. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5122. Fix format of fs.default.name value in libhdfs test conf. - (Craig Macdonald via tomwhite) - - HADOOP-5038. Direct daemon trace to debug log instead of stdout. (Jerome - Boulon via cdouglas) - - HADOOP-5101. Improve packaging by adding 'all-jars' target building core, - tools, and example jars. Let findbugs depend on this rather than the 'tar' - target. (Giridharan Kesavan via cdouglas) - - HADOOP-4868. Splits the hadoop script into three parts - bin/hadoop, - bin/mapred and bin/hdfs. (Sharad Agarwal via ddas) - - HADOOP-1722. Adds support for TypedBytes and RawBytes in Streaming. - (Klaas Bosteels via ddas) - - HADOOP-4220. Changes the JobTracker restart tests so that they take much - less time. (Amar Kamat via ddas) - - HADOOP-4885. Try to restore failed name-node storage directories at - checkpoint time. (Boris Shkolnik via shv) - - HADOOP-5209. Update year to 2009 for javadoc. (szetszwo) - - HADOOP-5279. Remove unnecessary targets from test-patch.sh. - (Giridharan Kesavan via nigel) - - HADOOP-5120. Remove the use of FSNamesystem.getFSNamesystem() from - UpgradeManagerNamenode and UpgradeObjectNamenode. (szetszwo) - - HADOOP-5222. Add offset to datanode clienttrace. (Lei Xu via cdouglas) - - HADOOP-5240. Skip re-building javadoc when it is already - up-to-date. (Aaron Kimball via cutting) - - HADOOP-5042. Add a cleanup stage to log rollover in Chukwa appender. - (Jerome Boulon via cdouglas) - - HADOOP-5264. Removes redundant configuration object from the TaskTracker. - (Sharad Agarwal via ddas) - - HADOOP-5232. Enable patch testing to occur on more than one host. - (Giri Kesavan via nigel) - - HADOOP-4546. Fix DF reporting for AIX. (Bill Habermaas via cdouglas) - - HADOOP-5023. Add Tomcat support to HdfsProxy. (Zhiyong Zhang via cdouglas) - - HADOOP-5317. Provide documentation for LazyOutput Feature. - (Jothi Padmanabhan via johan) - - HADOOP-5455. Document rpc metrics context to the extent dfs, mapred, and - jvm contexts are documented. (Philip Zeyliger via cdouglas) - - HADOOP-5358. Provide scripting functionality to the synthetic load - generator. (Jakob Homan via hairong) - - HADOOP-5442. Paginate jobhistory display and added some search - capabilities. (Amar Kamat via acmurthy) - - HADOOP-4842. Streaming now allows specifiying a command for the combiner. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5196. avoiding unnecessary byte[] allocation in - SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes. - (hong tang via mahadev) - - HADOOP-4655. New method FileSystem.newInstance() that always returns - a newly allocated FileSystem object. (dhruba) - - HADOOP-4788. Set Fair scheduler to assign both a map and a reduce on each - heartbeat by default. (matei) - - HADOOP-5491. In contrib/index, better control memory usage. - (Ning Li via cutting) - - HADOOP-5423. Include option of preserving file metadata in - SequenceFile::sort. (Michael Tamm via cdouglas) - - HADOOP-5331. Add support for KFS appends. (Sriram Rao via cdouglas) - - HADOOP-4365. Make Configuration::getProps protected in support of - meaningful subclassing. (Steve Loughran via cdouglas) - - HADOOP-2413. Remove the static variable FSNamesystem.fsNamesystemObject. - (Konstantin Shvachko via szetszwo) - - HADOOP-4584. Improve datanode block reports and associated file system - scan to avoid interefering with normal datanode operations. - (Suresh Srinivas via rangadi) - - HADOOP-5502. Documentation for backup and checkpoint nodes. - (Jakob Homan via shv) - - HADOOP-5485. Mask actions in the fair scheduler's servlet UI based on - value of webinterface.private.actions. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-5581. HDFS should throw FileNotFoundException when while opening - a file that does not exist. (Brian Bockelman via rangadi) - - HADOOP-5509. PendingReplicationBlocks does not start monitor in the - constructor. (shv) - - HADOOP-5494. Modify sorted map output merger to lazily read values, - rather than buffering at least one record for each segment. (Devaraj Das - via cdouglas) - - HADOOP-5396. Provide ability to refresh queue ACLs in the JobTracker - without having to restart the daemon. - (Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-4490. Provide ability to run tasks as job owners. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5697. Change org.apache.hadoop.examples.Grep to use new - mapreduce api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5625. Add operation duration to clienttrace. (Lei Xu via cdouglas) - - HADOOP-5705. Improve TotalOrderPartitioner efficiency by updating the trie - construction. (Dick King via cdouglas) - - HADOOP-5589. Eliminate source limit of 64 for map-side joins imposed by - TupleWritable encoding. (Jingkei Ly via cdouglas) - - HADOOP-5734. Correct block placement policy description in HDFS - Design document. (Konstantin Boudnik via shv) - - HADOOP-5657. Validate data in TestReduceFetch to improve merge test - coverage. (cdouglas) - - HADOOP-5613. Change S3Exception to checked exception. - (Andrew Hitchcock via tomwhite) - - HADOOP-5717. Create public enum class for the Framework counters in - org.apache.hadoop.mapreduce. (Amareshwari Sriramadasu via sharad) - - HADOOP-5217. Split AllTestDriver for core, hdfs and mapred. (sharad) - - HADOOP-5364. Add certificate expiration warning to HsftpFileSystem and HDFS - proxy. (Zhiyong Zhang via cdouglas) - - HADOOP-5733. Add map/reduce slot capacity and blacklisted capacity to - JobTracker metrics. (Sreekanth Ramakrishnan via cdouglas) - - HADOOP-5596. Add EnumSetWritable. (He Yongqiang via szetszwo) - - HADOOP-5727. Simplify hashcode for ID types. (Shevek via cdouglas) - - HADOOP-5500. In DBOutputFormat, where field names are absent permit the - number of fields to be sufficient to construct the select query. (Enis - Soztutar via cdouglas) - - HADOOP-5081. Split TestCLI into HDFS, Mapred and Core tests. (sharad) - - HADOOP-5015. Separate block management code from FSNamesystem. (Suresh - Srinivas via szetszwo) - - HADOOP-5080. Add new test cases to TestMRCLI and TestHDFSCLI - (V.Karthikeyan via nigel) - - HADOOP-5135. Splits the tests into different directories based on the - package. Four new test targets have been defined - run-test-core, - run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr. - (Sharad Agarwal via ddas) - - HADOOP-5771. Implements unit tests for LinuxTaskController. - (Sreekanth Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-5419. Provide a facility to query the Queue ACLs for the - current user. - (Rahul Kumar Singh via yhemanth) - - HADOOP-5780. Improve per block message prited by "-metaSave" in HDFS. - (Raghu Angadi) - - HADOOP-5823. Added a new class DeprecatedUTF8 to help with removing - UTF8 related javac warnings. These warnings are removed in - FSEditLog.java as a use case. (Raghu Angadi) - - HADOOP-5824. Deprecate DataTransferProtocol.OP_READ_METADATA and remove - the corresponding unused codes. (Kan Zhang via szetszwo) - - HADOOP-5721. Factor out EditLogFileInputStream and EditLogFileOutputStream - into independent classes. (Luca Telloli & Flavio Junqueira via shv) - - HADOOP-5838. Fix a few javac warnings in HDFS. (Raghu Angadi) - - HADOOP-5854. Fix a few "Inconsistent Synchronization" warnings in HDFS. - (Raghu Angadi) - - HADOOP-5369. Small tweaks to reduce MapFile index size. (Ben Maurer - via sharad) - - HADOOP-5858. Eliminate UTF8 and fix warnings in test/hdfs-with-mr package. - (shv) - - HADOOP-5866. Move DeprecatedUTF8 from o.a.h.io to o.a.h.hdfs since it may - not be used outside hdfs. (Raghu Angadi) - - HADOOP-5857. Move normal java methods from hdfs .jsp files to .java files. - (szetszwo) - - HADOOP-5873. Remove deprecated methods randomDataNode() and - getDatanodeByIndex(..) in FSNamesystem. (szetszwo) - - HADOOP-5572. Improves the progress reporting for the sort phase for both - maps and reduces. (Ravi Gummadi via ddas) - - HADOOP-5839. Fix EC2 scripts to allow remote job submission. - (Joydeep Sen Sarma via tomwhite) - - HADOOP-5877. Fix javac warnings in TestHDFSServerPorts, TestCheckpoint, - TestNameEditsConfig, TestStartup and TestStorageRestore. - (Jakob Homan via shv) - - HADOOP-5438. Provide a single FileSystem method to create or - open-for-append to a file. (He Yongqiang via dhruba) - - HADOOP-5472. Change DistCp to support globbing of input paths. (Dhruba - Borthakur and Rodrigo Schmidt via szetszwo) - - HADOOP-5175. Don't unpack libjars on classpath. (Todd Lipcon via tomwhite) - - HADOOP-5620. Add an option to DistCp for preserving modification and access - times. (Rodrigo Schmidt via szetszwo) - - HADOOP-5664. Change map serialization so a lock is obtained only where - contention is possible, rather than for each write. (cdouglas) - - HADOOP-5896. Remove the dependency of GenericOptionsParser on - Option.withArgPattern. (Giridharan Kesavan and Sharad Agarwal via - sharad) - - HADOOP-5784. Makes the number of heartbeats that should arrive a second - at the JobTracker configurable. (Amareshwari Sriramadasu via ddas) - - HADOOP-5955. Changes TestFileOuputFormat so that is uses LOCAL_MR - instead of CLUSTER_MR. (Jothi Padmanabhan via das) - - HADOOP-5948. Changes TestJavaSerialization to use LocalJobRunner - instead of MiniMR/DFS cluster. (Jothi Padmanabhan via das) - - HADOOP-2838. Add mapred.child.env to pass environment variables to - tasktracker's child processes. (Amar Kamat via sharad) - - HADOOP-5961. DataNode process understand generic hadoop command line - options (like -Ddfs.property=value). (Raghu Angadi) - - HADOOP-5938. Change org.apache.hadoop.mapred.jobcontrol to use new - api. (Amareshwari Sriramadasu via sharad) - - HADOOP-2141. Improves the speculative execution heuristic. The heuristic - is currently based on the progress-rates of tasks and the expected time - to complete. Also, statistics about trackers are collected, and speculative - tasks are not given to the ones deduced to be slow. - (Andy Konwinski and ddas) - - HADOOP-5952. Change "-1 tests included" wording in test-patch.sh. - (Gary Murry via szetszwo) - - HADOOP-6106. Provides an option in ShellCommandExecutor to timeout - commands that do not complete within a certain amount of time. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5925. EC2 scripts should exit on error. (tomwhite) - - HADOOP-6109. Change Text to grow its internal buffer exponentially, rather - than the max of the current length and the proposed length to improve - performance reading large values. (thushara wijeratna via cdouglas) - - HADOOP-2366. Support trimmed strings in Configuration. (Michele Catasta - via szetszwo) - - HADOOP-6099. The RPC module can be configured to not send period pings. - The default behaviour of sending periodic pings remain unchanged. (dhruba) - - HADOOP-6142. Update documentation and use of harchives for relative paths - added in MAPREDUCE-739. (Mahadev Konar via cdouglas) - - HADOOP-6148. Implement a fast, pure Java CRC32 calculator which outperforms - java.util.zip.CRC32. (Todd Lipcon and Scott Carey via szetszwo) - - HADOOP-6146. Upgrade to JetS3t version 0.7.1. (tomwhite) - - HADOOP-6161. Add get/setEnum methods to Configuration. (cdouglas) - - HADOOP-6160. Fix releaseaudit target to run on specific directories. - (gkesavan) - - HADOOP-6169. Removing deprecated method calls in TFile. (hong tang via - mahadev) - - HADOOP-6176. Add a couple package private methods to AccessTokenHandler - for testing. (Kan Zhang via szetszwo) - - HADOOP-6182. Fix ReleaseAudit warnings (Giridharan Kesavan and Lee Tucker - via gkesavan) - - HADOOP-6173. Change src/native/packageNativeHadoop.sh to package all - native library files. (Hong Tang via szetszwo) - - HADOOP-6184. Provide an API to dump Configuration in a JSON format. - (V.V.Chaitanya Krishna via yhemanth) - - HADOOP-6224. Add a method to WritableUtils performing a bounded read of an - encoded String. (Jothi Padmanabhan via cdouglas) - - HADOOP-6133. Add a caching layer to Configuration::getClassByName to - alleviate a performance regression introduced in a compatibility layer. - (Todd Lipcon via cdouglas) - - HADOOP-6252. Provide a method to determine if a deprecated key is set in - config file. (Jakob Homan via suresh) - - HADOOP-5879. Read compression level and strategy from Configuration for - gzip compression. (He Yongqiang via cdouglas) - - HADOOP-6216. Support comments in host files. (Ravi Phulari and Dmytro - Molkov via szetszwo) - - HADOOP-6217. Update documentation for project split. (Corinne Chandel via - omalley) - - HADOOP-6268. Add ivy jar to .gitignore. (Todd Lipcon via cdouglas) - - HADOOP-6270. Support deleteOnExit in FileContext. (Suresh Srinivas via - szetszwo) - - HADOOP-6233. Rename configuration keys towards API standardization and - backward compatibility. (Jithendra Pandey via suresh) - - HADOOP-6260. Add additional unit tests for FileContext util methods. - (Gary Murry via suresh). - - HADOOP-6309. Change build.xml to run tests with java asserts. (Eli - Collins via szetszwo) - - HADOOP-6326. Hundson runs should check for AspectJ warnings and report - failure if any is present (cos) - - HADOOP-6329. Add build-fi directory to the ignore lists. (szetszwo) - - HADOOP-5107. Use Maven ant tasks to publish the subproject jars. - (Giridharan Kesavan via omalley) - - HADOOP-6343. Log unexpected throwable object caught in RPC. (Jitendra Nath - Pandey via szetszwo) - - HADOOP-6367. Removes Access Token implementation from common. - (Kan Zhang via ddas) - - HADOOP-6395. Upgrade some libraries to be consistent across common, hdfs, - and mapreduce. (omalley) - - HADOOP-6398. Build is broken after HADOOP-6395 patch has been applied (cos) - - HADOOP-6413. Move TestReflectionUtils to Common. (Todd Lipcon via tomwhite) - - HADOOP-6283. Improve the exception messages thrown by - FileUtil$HardLink.getLinkCount(..). (szetszwo) - - HADOOP-6279. Add Runtime::maxMemory to JVM metrics. (Todd Lipcon via - cdouglas) - - HADOOP-6305. Unify build property names to facilitate cross-projects - modifications (cos) - - HADOOP-6312. Remove unnecessary debug logging in Configuration constructor. - (Aaron Kimball via cdouglas) - - HADOOP-6366. Reduce ivy console output to ovservable level (cos) - - HADOOP-6400. Log errors getting Unix UGI. (Todd Lipcon via tomwhite) - - HADOOP-6346. Add support for specifying unpack pattern regex to - RunJar.unJar. (Todd Lipcon via tomwhite) - - HADOOP-6422. Make RPC backend plugable, protocol-by-protocol, to - ease evolution towards Avro. (cutting) - - HADOOP-5958. Use JDK 1.6 File APIs in DF.java wherever possible. - (Aaron Kimball via tomwhite) - - HADOOP-6222. Core doesn't have TestCommonCLI facility. (cos) - - HADOOP-6394. Add a helper class to simplify FileContext related tests and - improve code reusability. (Jitendra Nath Pandey via suresh) - - HADOOP-4656. Add a user to groups mapping service. (boryas, acmurthy) - - HADOOP-6435. Make RPC.waitForProxy with timeout public. (Steve Loughran - via tomwhite) - - HADOOP-6472. add tokenCache option to GenericOptionsParser for passing - file with secret keys to a map reduce job. (boryas) - - HADOOP-3205. Read multiple chunks directly from FSInputChecker subclass - into user buffers. (Todd Lipcon via tomwhite) - - HADOOP-6479. TestUTF8 assertions could fail with better text. - (Steve Loughran via tomwhite) - - HADOOP-6155. Deprecate RecordIO anticipating Avro. (Tom White via cdouglas) - - HADOOP-6492. Make some Avro serialization APIs public. - (Aaron Kimball via cutting) - - HADOOP-6497. Add an adapter for Avro's SeekableInput interface, so - that Avro can read FileSystem data. - (Aaron Kimball via cutting) - - HADOOP-6495. Identifier should be serialized after the password is - created In Token constructor (jnp via boryas) - - HADOOP-6518. Makes the UGI honor the env var KRB5CCNAME. - (Owen O'Malley via ddas) - - HADOOP-6531. Enhance FileUtil with an API to delete all contents of a - directory. (Amareshwari Sriramadasu via yhemanth) - - HADOOP-6547. Move DelegationToken into Common, so that it can be used by - MapReduce also. (devaraj via omalley) - - HADOOP-6552. Puts renewTGT=true and useTicketCache=true for the keytab - kerberos options. (ddas) - - HADOOP-6534. Trim whitespace from directory lists initializing - LocalDirAllocator. (Todd Lipcon via cdouglas) - - HADOOP-6559. Makes the RPC client automatically re-login when the SASL - connection setup fails. This is applicable only to keytab based logins. - (Devaraj Das) - - HADOOP-6551. Delegation token renewing and cancelling should provide - meaningful exceptions when there are failures instead of returning - false. (omalley) - - HADOOP-6583. Captures authentication and authorization metrics. (ddas) - - HADOOP-6543. Allows secure clients to talk to unsecure clusters. - (Kan Zhang via ddas) - - HADOOP-6579. Provide a mechanism for encoding/decoding Tokens from - a url-safe string and change the commons-code library to 1.4. (omalley) - - HADOOP-6596. Add a version field to the AbstractDelegationTokenIdentifier's - serialized value. (omalley) - - HADOOP-6573. Support for persistent delegation tokens. - (Jitendra Pandey via shv) - - HADOOP-6594. Provide a fetchdt tool via bin/hdfs. (jhoman via acmurthy) - - HADOOP-6589. Provide better error messages when RPC authentication fails. - (Kan Zhang via omalley) - - HADOOP-6599 Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics. - (Suresh Srinivas via Sanjay Radia) - - HADOOP-6537 Declare more detailed exceptions in FileContext and - AbstractFileSystem (Suresh Srinivas via Sanjay Radia) - - HADOOP-6486. fix common classes to work with Avro 1.3 reflection. - (cutting via tomwhite) - - HADOOP-6591. HarFileSystem can handle paths with the whitespace characters. - (Rodrigo Schmidt via dhruba) - - HADOOP-6407. Have a way to automatically update Eclipse .classpath file - when new libs are added to the classpath through Ivy. (tomwhite) - - HADOOP-3659. Patch to allow hadoop native to compile on Mac OS X. - (Colin Evans and Allen Wittenauer via tomwhite) - - HADOOP-6471. StringBuffer -> StringBuilder - conversion of references - as necessary. (Kay Kay via tomwhite) - - HADOOP-6646. Move HarfileSystem out of Hadoop Common. (mahadev) - - HADOOP-6566. Add methods supporting, enforcing narrower permissions on - local daemon directories. (Arun Murthy and Luke Lu via cdouglas) - - HADOOP-6705. Fix to work with 1.5 version of jiracli - (Giridharan Kesavan) - - HADOOP-6658. Exclude Private elements from generated Javadoc. (tomwhite) - - HADOOP-6635. Install/deploy source jars to Maven repo. - (Patrick Angeles via jghoman) - - HADOOP-6717. Log levels in o.a.h.security.Groups too high - (Todd Lipcon via jghoman) - - HADOOP-6667. RPC.waitForProxy should retry through NoRouteToHostException. - (Todd Lipcon via tomwhite) - - HADOOP-6677. InterfaceAudience.LimitedPrivate should take a string not an - enum. (tomwhite) - - HADOOP-678. Remove FileContext#isFile, isDirectory, and exists. - (Eli Collins via hairong) - - HADOOP-6515. Make maximum number of http threads configurable. - (Scott Chen via zshao) - - HADOOP-6563. Add more symlink tests to cover intermediate symlinks - in paths. (Eli Collins via suresh) - - HADOOP-6585. Add FileStatus#isDirectory and isFile. (Eli Collins via - tomwhite) - - HADOOP-6738. Move cluster_setup.xml from MapReduce to Common. - (Tom White via tomwhite) - - HADOOP-6794. Move configuration and script files post split. (tomwhite) - - HADOOP-6403. Deprecate EC2 bash scripts. (tomwhite) - - HADOOP-6769. Add an API in FileSystem to get FileSystem instances based - on users(ddas via boryas) - - HADOOP-6813. Add a new newInstance method in FileSystem that takes - a "user" as argument (ddas via boryas) - - HADOOP-6668. Apply audience and stability annotations to classes in - common. (tomwhite) - - HADOOP-6821. Document changes to memory monitoring. (Hemanth Yamijala - via tomwhite) - - OPTIMIZATIONS - - HADOOP-5595. NameNode does not need to run a replicator to choose a - random DataNode. (hairong) - - HADOOP-5603. Improve NameNode's block placement performance. (hairong) - - HADOOP-5638. More improvement on block placement performance. (hairong) - - HADOOP-6180. NameNode slowed down when many files with same filename - were moved to Trash. (Boris Shkolnik via hairong) - - HADOOP-6166. Further improve the performance of the pure-Java CRC32 - implementation. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-6271. Add recursive and non recursive create and mkdir to - FileContext. (Sanjay Radia via suresh) - - HADOOP-6261. Add URI based tests for FileContext. - (Ravi Pulari via suresh). - - HADOOP-6307. Add a new SequenceFile.Reader constructor in order to support - reading on un-closed file. (szetszwo) - - HADOOP-6467. Improve the performance on HarFileSystem.listStatus(..). - (mahadev via szetszwo) - - HADOOP-6569. FsShell#cat should avoid calling unecessary getFileStatus - before opening a file to read. (hairong) - - HADOOP-6689. Add directory renaming test to existing FileContext tests. - (Eli Collins via suresh) - - HADOOP-6713. The RPC server Listener thread is a scalability bottleneck. - (Dmytro Molkov via hairong) - - BUG FIXES - - HADOOP-6748. Removes hadoop.cluster.administrators, cluster administrators - acl is passed as parameter in constructor. (amareshwari) - - HADOOP-6828. Herrior uses old way of accessing logs directories (Sreekanth - Ramakrishnan via cos) - - HADOOP-6788. [Herriot] Exception exclusion functionality is not working - correctly. (Vinay Thota via cos) - - HADOOP-6773. Ivy folder contains redundant files (cos) - - HADOOP-5379. CBZip2InputStream to throw IOException on data crc error. - (Rodrigo Schmidt via zshao) - - HADOOP-5326. Fixes CBZip2OutputStream data corruption problem. - (Rodrigo Schmidt via zshao) - - HADOOP-4963. Fixes a logging to do with getting the location of - map output file. (Amareshwari Sriramadasu via ddas) - - HADOOP-2337. Trash should close FileSystem on exit and should not start - emtying thread if disabled. (shv) - - HADOOP-5072. Fix failure in TestCodec because testSequenceFileGzipCodec - won't pass without native gzip codec. (Zheng Shao via dhruba) - - HADOOP-5050. TestDFSShell.testFilePermissions should not assume umask - setting. (Jakob Homan via szetszwo) - - HADOOP-4975. Set classloader for nested mapred.join configs. (Jingkei Ly - via cdouglas) - - HADOOP-5078. Remove invalid AMI kernel in EC2 scripts. (tomwhite) - - HADOOP-5045. FileSystem.isDirectory() should not be deprecated. (Suresh - Srinivas via szetszwo) - - HADOOP-4960. Use datasource time, rather than system time, during metrics - demux. (Eric Yang via cdouglas) - - HADOOP-5032. Export conf dir set in config script. (Eric Yang via cdouglas) - - HADOOP-5176. Fix a typo in TestDFSIO. (Ravi Phulari via szetszwo) - - HADOOP-4859. Distinguish daily rolling output dir by adding a timestamp. - (Jerome Boulon via cdouglas) - - HADOOP-4959. Correct system metric collection from top on Redhat 5.1. (Eric - Yang via cdouglas) - - HADOOP-5039. Fix log rolling regex to process only the relevant - subdirectories. (Jerome Boulon via cdouglas) - - HADOOP-5095. Update Chukwa watchdog to accept config parameter. (Jerome - Boulon via cdouglas) - - HADOOP-5147. Correct reference to agent list in Chukwa bin scripts. (Ari - Rabkin via cdouglas) - - HADOOP-5148. Fix logic disabling watchdog timer in Chukwa daemon scripts. - (Ari Rabkin via cdouglas) - - HADOOP-5100. Append, rather than truncate, when creating log4j metrics in - Chukwa. (Jerome Boulon via cdouglas) - - HADOOP-5204. Fix broken trunk compilation on Hudson by letting - task-controller be an independent target in build.xml. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5212. Fix the path translation problem introduced by HADOOP-4868 - running on cygwin. (Sharad Agarwal via omalley) - - HADOOP-5226. Add license headers to html and jsp files. (szetszwo) - - HADOOP-5172. Disable misbehaving Chukwa unit test until it can be fixed. - (Jerome Boulon via nigel) - - HADOOP-4933. Fixes a ConcurrentModificationException problem that shows up - when the history viewer is accessed concurrently. - (Amar Kamat via ddas) - - HADOOP-5253. Remove duplicate call to cn-docs target. - (Giri Kesavan via nigel) - - HADOOP-5251. Fix classpath for contrib unit tests to include clover jar. - (nigel) - - HADOOP-5206. Synchronize "unprotected*" methods of FSDirectory on the root. - (Jakob Homan via shv) - - HADOOP-5292. Fix NPE in KFS::getBlockLocations. (Sriram Rao via lohit) - - HADOOP-5219. Adds a new property io.seqfile.local.dir for use by - SequenceFile, which earlier used mapred.local.dir. (Sharad Agarwal - via ddas) - - HADOOP-5300. Fix ant javadoc-dev target and the typo in the class name - NameNodeActivtyMBean. (szetszwo) - - HADOOP-5218. libhdfs unit test failed because it was unable to - start namenode/datanode. Fixed. (dhruba) - - HADOOP-5273. Add license header to TestJobInProgress.java. (Jakob Homan - via szetszwo) - - HADOOP-5229. Remove duplicate version variables in build files - (Stefan Groschupf via johan) - - HADOOP-5383. Avoid building an unused string in NameNode's - verifyReplication(). (Raghu Angadi) - - HADOOP-5347. Create a job output directory for the bbp examples. (szetszwo) - - HADOOP-5341. Make hadoop-daemon scripts backwards compatible with the - changes in HADOOP-4868. (Sharad Agarwal via yhemanth) - - HADOOP-5456. Fix javadoc links to ClientProtocol#restoreFailedStorage(..). - (Boris Shkolnik via szetszwo) - - HADOOP-5458. Remove leftover Chukwa entries from build, etc. (cdouglas) - - HADOOP-5386. Modify hdfsproxy unit test to start on a random port, - implement clover instrumentation. (Zhiyong Zhang via cdouglas) - - HADOOP-5511. Add Apache License to EditLogBackupOutputStream. (shv) - - HADOOP-5507. Fix JMXGet javadoc warnings. (Boris Shkolnik via szetszwo) - - HADOOP-5191. Accessing HDFS with any ip or hostname should work as long - as it points to the interface NameNode is listening on. (Raghu Angadi) - - HADOOP-5561. Add javadoc.maxmemory parameter to build, preventing OOM - exceptions from javadoc-dev. (Jakob Homan via cdouglas) - - HADOOP-5149. Modify HistoryViewer to ignore unfamiliar files in the log - directory. (Hong Tang via cdouglas) - - HADOOP-5477. Fix rare failure in TestCLI for hosts returning variations of - 'localhost'. (Jakob Homan via cdouglas) - - HADOOP-5194. Disables setsid for tasks run on cygwin. - (Ravi Gummadi via ddas) - - HADOOP-5322. Fix misleading/outdated comments in JobInProgress. - (Amareshwari Sriramadasu via cdouglas) - - HADOOP-5198. Fixes a problem to do with the task PID file being absent and - the JvmManager trying to look for it. (Amareshwari Sriramadasu via ddas) - - HADOOP-5464. DFSClient did not treat write timeout of 0 properly. - (Raghu Angadi) - - HADOOP-4045. Fix processing of IO errors in EditsLog. - (Boris Shkolnik via shv) - - HADOOP-5462. Fixed a double free bug in the task-controller - executable. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5652. Fix a bug where in-memory segments are incorrectly retained in - memory. (cdouglas) - - HADOOP-5533. Recovery duration shown on the jobtracker webpage is - inaccurate. (Amar Kamat via sharad) - - HADOOP-5647. Fix TestJobHistory to not depend on /tmp. (Ravi Gummadi - via sharad) - - HADOOP-5661. Fixes some findbugs warnings in o.a.h.mapred* packages and - supresses a bunch of them. (Jothi Padmanabhan via ddas) - - HADOOP-5704. Fix compilation problems in TestFairScheduler and - TestCapacityScheduler. (Chris Douglas via szetszwo) - - HADOOP-5650. Fix safemode messages in the Namenode log. (Suresh Srinivas - via szetszwo) - - HADOOP-5488. Removes the pidfile management for the Task JVM from the - framework and instead passes the PID back and forth between the - TaskTracker and the Task processes. (Ravi Gummadi via ddas) - - HADOOP-5658. Fix Eclipse templates. (Philip Zeyliger via shv) - - HADOOP-5709. Remove redundant synchronization added in HADOOP-5661. (Jothi - Padmanabhan via cdouglas) - - HADOOP-5715. Add conf/mapred-queue-acls.xml to the ignore lists. - (szetszwo) - - HADOOP-5592. Fix typo in Streaming doc in reference to GzipCodec. - (Corinne Chandel via tomwhite) - - HADOOP-5656. Counter for S3N Read Bytes does not work. (Ian Nowland - via tomwhite) - - HADOOP-5406. Fix JNI binding for ZlibCompressor::setDictionary. (Lars - Francke via cdouglas) - - HADOOP-3426. Fix/provide handling when DNS lookup fails on the loopback - address. Also cache the result of the lookup. (Steve Loughran via cdouglas) - - HADOOP-5476. Close the underlying InputStream in SequenceFile::Reader when - the constructor throws an exception. (Michael Tamm via cdouglas) - - HADOOP-5675. Do not launch a job if DistCp has no work to do. (Tsz Wo - (Nicholas), SZE via cdouglas) - - HADOOP-5737. Fixes a problem in the way the JobTracker used to talk to - other daemons like the NameNode to get the job's files. Also adds APIs - in the JobTracker to get the FileSystem objects as per the JobTracker's - configuration. (Amar Kamat via ddas) - - HADOOP-5648. Not able to generate gridmix.jar on the already compiled - version of hadoop. (gkesavan) - - HADOOP-5808. Fix import never used javac warnings in hdfs. (szetszwo) - - HADOOP-5203. TT's version build is too restrictive. (Rick Cox via sharad) - - HADOOP-5818. Revert the renaming from FSNamesystem.checkSuperuserPrivilege - to checkAccess by HADOOP-5643. (Amar Kamat via szetszwo) - - HADOOP-5820. Fix findbugs warnings for http related codes in hdfs. - (szetszwo) - - HADOOP-5822. Fix javac warnings in several dfs tests related to unncessary - casts. (Jakob Homan via szetszwo) - - HADOOP-5842. Fix a few javac warnings under packages fs and util. - (Hairong Kuang via szetszwo) - - HADOOP-5845. Build successful despite test failure on test-core target. - (sharad) - - HADOOP-5314. Prevent unnecessary saving of the file system image during - name-node startup. (Jakob Homan via shv) - - HADOOP-5855. Fix javac warnings for DisallowedDatanodeException and - UnsupportedActionException. (szetszwo) - - HADOOP-5582. Fixes a problem in Hadoop Vaidya to do with reading - counters from job history files. (Suhas Gogate via ddas) - - HADOOP-5829. Fix javac warnings found in ReplicationTargetChooser, - FSImage, Checkpointer, SecondaryNameNode and a few other hdfs classes. - (Suresh Srinivas via szetszwo) - - HADOOP-5835. Fix findbugs warnings found in Block, DataNode, NameNode and - a few other hdfs classes. (Suresh Srinivas via szetszwo) - - HADOOP-5853. Undeprecate HttpServer.addInternalServlet method. (Suresh - Srinivas via szetszwo) - - HADOOP-5801. Fixes the problem: If the hosts file is changed across restart - then it should be refreshed upon recovery so that the excluded hosts are - lost and the maps are re-executed. (Amar Kamat via ddas) - - HADOOP-5841. Resolve findbugs warnings in DistributedFileSystem, - DatanodeInfo, BlocksMap, DataNodeDescriptor. (Jakob Homan via szetszwo) - - HADOOP-5878. Fix import and Serializable javac warnings found in hdfs jsp. - (szetszwo) - - HADOOP-5782. Revert a few formatting changes introduced in HADOOP-5015. - (Suresh Srinivas via rangadi) - - HADOOP-5687. NameNode throws NPE if fs.default.name is the default value. - (Philip Zeyliger via shv) - - HADOOP-5867. Fix javac warnings found in NNBench and NNBenchWithoutMR. - (Konstantin Boudnik via szetszwo) - - HADOOP-5728. Fixed FSEditLog.printStatistics IndexOutOfBoundsException. - (Wang Xu via johan) - - HADOOP-5847. Fixed failing Streaming unit tests (gkesavan) - - HADOOP-5252. Streaming overrides -inputformat option (Klaas Bosteels - via sharad) - - HADOOP-5710. Counter MAP_INPUT_BYTES missing from new mapreduce api. - (Amareshwari Sriramadasu via sharad) - - HADOOP-5809. Fix job submission, broken by errant directory creation. - (Sreekanth Ramakrishnan and Jothi Padmanabhan via cdouglas) - - HADOOP-5635. Change distributed cache to work with other distributed file - systems. (Andrew Hitchcock via tomwhite) - - HADOOP-5856. Fix "unsafe multithreaded use of DateFormat" findbugs warning - in DataBlockScanner. (Kan Zhang via szetszwo) - - HADOOP-4864. Fixes a problem to do with -libjars with multiple jars when - client and cluster reside on different OSs. (Amareshwari Sriramadasu via - ddas) - - HADOOP-5623. Fixes a problem to do with status messages getting overwritten - in streaming jobs. (Rick Cox and Jothi Padmanabhan via ddas) - - HADOOP-5895. Fixes computation of count of merged bytes for logging. - (Ravi Gummadi via ddas) - - HADOOP-5805. problem using top level s3 buckets as input/output - directories. (Ian Nowland via tomwhite) - - HADOOP-5940. trunk eclipse-plugin build fails while trying to copy - commons-cli jar from the lib dir (Giridharan Kesavan via gkesavan) - - HADOOP-5864. Fix DMI and OBL findbugs in packages hdfs and metrics. - (hairong) - - HADOOP-5935. Fix Hudson's release audit warnings link is broken. - (Giridharan Kesavan via gkesavan) - - HADOOP-5947. Delete empty TestCombineFileInputFormat.java - - HADOOP-5899. Move a log message in FSEditLog to the right place for - avoiding unnecessary log. (Suresh Srinivas via szetszwo) - - HADOOP-5944. Add Apache license header to BlockManager.java. (Suresh - Srinivas via szetszwo) - - HADOOP-5891. SecondaryNamenode is able to converse with the NameNode - even when the default value of dfs.http.address is not overridden. - (Todd Lipcon via dhruba) - - HADOOP-5953. The isDirectory(..) and isFile(..) methods in KosmosFileSystem - should not be deprecated. (szetszwo) - - HADOOP-5954. Fix javac warnings in TestFileCreation, TestSmallBlock, - TestFileStatus, TestDFSShellGenericOptions, TestSeekBug and - TestDFSStartupVersions. (szetszwo) - - HADOOP-5956. Fix ivy dependency in hdfsproxy and capacity-scheduler. - (Giridharan Kesavan via szetszwo) - - HADOOP-5836. Bug in S3N handling of directory markers using an object with - a trailing "/" causes jobs to fail. (Ian Nowland via tomwhite) - - HADOOP-5861. s3n files are not getting split by default. (tomwhite) - - HADOOP-5762. Fix a problem that DistCp does not copy empty directory. - (Rodrigo Schmidt via szetszwo) - - HADOOP-5859. Fix "wait() or sleep() with locks held" findbugs warnings in - DFSClient. (Kan Zhang via szetszwo) - - HADOOP-5457. Fix to continue to run builds even if contrib test fails - (Giridharan Kesavan via gkesavan) - - HADOOP-5963. Remove an unnecessary exception catch in NNBench. (Boris - Shkolnik via szetszwo) - - HADOOP-5989. Fix streaming test failure. (gkesavan) - - HADOOP-5981. Fix a bug in HADOOP-2838 in parsing mapred.child.env. - (Amar Kamat via sharad) - - HADOOP-5420. Fix LinuxTaskController to kill tasks using the process - groups they are launched with. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-6031. Remove @author tags from Java source files. (Ravi Phulari - via szetszwo) - - HADOOP-5980. Fix LinuxTaskController so tasks get passed - LD_LIBRARY_PATH and other environment variables. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4041. IsolationRunner does not work as documented. - (Philip Zeyliger via tomwhite) - - HADOOP-6004. Fixes BlockLocation deserialization. (Jakob Homan via - szetszwo) - - HADOOP-6079. Serialize proxySource as DatanodeInfo in DataTransferProtocol. - (szetszwo) - - HADOOP-6096. Fix Eclipse project and classpath files following project - split. (tomwhite) - - HADOOP-6122. The great than operator in test-patch.sh should be "-gt" but - not ">". (szetszwo) - - HADOOP-6114. Fix javadoc documentation for FileStatus.getLen. - (Dmitry Rzhevskiy via dhruba) - - HADOOP-6131. A sysproperty should not be set unless the property - is set on the ant command line in build.xml (hong tang via mahadev) - - HADOOP-6137. Fix project specific test-patch requirements - (Giridharan Kesavan) - - HADOOP-6138. Eliminate the deprecated warnings introduced by H-5438. - (He Yongqiang via szetszwo) - - HADOOP-6132. RPC client create an extra connection because of incorrect - key for connection cache. (Kan Zhang via rangadi) - - HADOOP-6123. Add missing classpaths in hadoop-config.sh. (Sharad Agarwal - via szetszwo) - - HADOOP-6172. Fix jar file names in hadoop-config.sh and include - ${build.src} as a part of the source list in build.xml. (Hong Tang via - szetszwo) - - HADOOP-6124. Fix javac warning detection in test-patch.sh. (Giridharan - Kesavan via szetszwo) - - HADOOP-6177. FSInputChecker.getPos() would return position greater - than the file size. (Hong Tang via hairong) - - HADOOP-6188. TestTrash uses java.io.File api but not hadoop FileSystem api. - (Boris Shkolnik via szetszwo) - - HADOOP-6192. Fix Shell.getUlimitMemoryCommand to not rely on Map-Reduce - specific configs. (acmurthy) - - HADOOP-6103. Clones the classloader as part of Configuration clone. - (Amareshwari Sriramadasu via ddas) - - HADOOP-6152. Fix classpath variables in bin/hadoop-config.sh and some - other scripts. (Aaron Kimball via szetszwo) - - HADOOP-6215. fix GenericOptionParser to deal with -D with '=' in the - value. (Amar Kamat via sharad) - - HADOOP-6227. Fix Configuration to allow final parameters to be set to null - and prevent them from being overridden. - (Amareshwari Sriramadasu via yhemanth) - - HADOOP-6199. Move io.map.skip.index property to core-default from mapred. - (Amareshwari Sriramadasu via cdouglas) - - HADOOP-6229. Attempt to make a directory under an existing file on - LocalFileSystem should throw an Exception. (Boris Shkolnik via tomwhite) - - HADOOP-6243. Fix a NullPointerException in processing deprecated keys. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-6009. S3N listStatus incorrectly returns null instead of empty - array when called on empty root. (Ian Nowland via tomwhite) - - HADOOP-6181. Fix .eclipse.templates/.classpath for avro and jets3t jar - files. (Carlos Valiente via szetszwo) - - HADOOP-6196. Fix a bug in SequenceFile.Reader where syncing within the - header would cause the reader to read the sync marker as a record. (Jay - Booth via cdouglas) - - HADOOP-6250. Modify test-patch to delete copied XML files before running - patch build. (Rahul Kumar Singh via yhemanth) - - HADOOP-6257. Two TestFileSystem classes are confusing - hadoop-hdfs-hdfwithmr. (Philip Zeyliger via tomwhite) - - HADOOP-6151. Added a input filter to all of the http servlets that quotes - html characters in the parameters, to prevent cross site scripting - attacks. (omalley) - - HADOOP-6274. Fix TestLocalFSFileContextMainOperations test failure. - (Gary Murry via suresh). - - HADOOP-6281. Avoid null pointer exceptions when the jsps don't have - paramaters (omalley) - - HADOOP-6285. Fix the result type of the getParameterMap method in the - HttpServer.QuotingInputFilter. (omalley) - - HADOOP-6286. Fix bugs in related to URI handling in glob methods in - FileContext. (Boris Shkolnik via suresh) - - HADOOP-6292. Update native libraries guide. (Corinne Chandel via cdouglas) - - HADOOP-6327. FileContext tests should not use /tmp and should clean up - files. (Sanjay Radia via szetszwo) - - HADOOP-6318. Upgrade to Avro 1.2.0. (cutting) - - HADOOP-6334. Fix GenericOptionsParser to understand URI for -files, - -libjars and -archives options and fix Path to support URI with fragment. - (Amareshwari Sriramadasu via szetszwo) - - HADOOP-6344. Fix rm and rmr immediately delete files rather than sending - to trash, if a user is over-quota. (Jakob Homan via suresh) - - HADOOP-6347. run-test-core-fault-inject runs a test case twice if - -Dtestcase is set (cos) - - HADOOP-6375. Sync documentation for FsShell du with its implementation. - (Todd Lipcon via cdouglas) - - HADOOP-6441. Protect web ui from cross site scripting attacks (XSS) on - the host http header and using encoded utf-7. (omalley) - - HADOOP-6451. Fix build to run contrib unit tests. (Tom White via cdouglas) - - HADOOP-6374. JUnit tests should never depend on anything in conf. - (Anatoli Fomenko via cos) - - HADOOP-6290. Prevent duplicate slf4j-simple jar via Avro's classpath. - (Owen O'Malley via cdouglas) - - HADOOP-6293. Fix FsShell -text to work on filesystems other than the - default. (cdouglas) - - HADOOP-6341. Fix test-patch.sh for checkTests function. (gkesavan) - - HADOOP-6314. Fix "fs -help" for the "-count" commond. (Ravi Phulari via - szetszwo) - - HADOOP-6405. Update Eclipse configuration to match changes to Ivy - configuration (Edwin Chan via cos) - - HADOOP-6411. Remove deprecated file src/test/hadoop-site.xml. (cos) - - HADOOP-6386. NameNode's HttpServer can't instantiate InetSocketAddress: - IllegalArgumentException is thrown (cos) - - HADOOP-6254. Slow reads cause s3n to fail with SocketTimeoutException. - (Andrew Hitchcock via tomwhite) - - HADOOP-6428. HttpServer sleeps with negative values. (cos) - - HADOOP-6414. Add command line help for -expunge command. - (Ravi Phulari via tomwhite) - - HADOOP-6391. Classpath should not be part of command line arguments. - (Cristian Ivascu via tomwhite) - - HADOOP-6462. Target "compile" does not exist in contrib/cloud. (tomwhite) - - HADOOP-6402. testConf.xsl is not well-formed XML. (Steve Loughran - via tomwhite) - - HADOOP-6489. Fix 3 findbugs warnings. (Erik Steffl via suresh) - - HADOOP-6517. Fix UserGroupInformation so that tokens are saved/retrieved - to/from the embedded Subject (Owen O'Malley & Kan Zhang via ddas) - - HADOOP-6538. Sets hadoop.security.authentication to simple by default. - (ddas) - - HADOOP-6540. Contrib unit tests have invalid XML for core-site, etc. - (Aaron Kimball via tomwhite) - - HADOOP-6521. User specified umask using deprecated dfs.umask must override - server configured using new dfs.umaskmode for backward compatibility. - (suresh) - - HADOOP-6522. Fix decoding of codepoint zero in UTF8. (cutting) - - HADOOP-6505. Use tr rather than sed to effect literal substitution in the - build script. (Allen Wittenauer via cdouglas) - - HADOOP-6548. Replace mortbay imports with commons logging. (cdouglas) - - HADOOP-6560. Handle invalid har:// uri in HarFileSystem. (szetszwo) - - HADOOP-6549. TestDoAsEffectiveUser should use ip address of the host - for superuser ip check(jnp via boryas) - - HADOOP-6570. RPC#stopProxy throws NPE if getProxyEngine(proxy) returns - null. (hairong) - - HADOOP-6558. Return null in HarFileSystem.getFileChecksum(..) since no - checksum algorithm is implemented. (szetszwo) - - HADOOP-6572. Makes sure that SASL encryption and push to responder - queue for the RPC response happens atomically. (Kan Zhang via ddas) - - HADOOP-6545. Changes the Key for the FileSystem cache to be UGI (ddas) - - HADOOP-6609. Fixed deadlock in RPC by replacing shared static - DataOutputBuffer in the UTF8 class with a thread local variable. (omalley) - - HADOOP-6504. Invalid example in the documentation of - org.apache.hadoop.util.Tool. (Benoit Sigoure via tomwhite) - - HADOOP-6546. BloomMapFile can return false negatives. (Clark Jefcoat - via tomwhite) - - HADOOP-6593. TextRecordInputStream doesn't close SequenceFile.Reader. - (Chase Bradford via tomwhite) - - HADOOP-6175. Incorrect version compilation with es_ES.ISO8859-15 locale - on Solaris 10. (Urko Benito via tomwhite) - - HADOOP-6645. Bugs on listStatus for HarFileSystem (rodrigo via mahadev) - - HADOOP-6645. Re: Bugs on listStatus for HarFileSystem (rodrigo via - mahadev) - - HADOOP-6654. Fix code example in WritableComparable javadoc. (Tom White - via szetszwo) - - HADOOP-6640. FileSystem.get() does RPC retries within a static - synchronized block. (hairong) - - HADOOP-6691. TestFileSystemCaching sometimes hangs. (hairong) - - HADOOP-6507. Hadoop Common Docs - delete 3 doc files that do not belong - under Common. (Corinne Chandel via tomwhite) - - HADOOP-6439. Fixes handling of deprecated keys to follow order in which - keys are defined. (V.V.Chaitanya Krishna via yhemanth) - - HADOOP-6690. FilterFileSystem correctly handles setTimes call. - (Rodrigo Schmidt via dhruba) - - HADOOP-6703. Prevent renaming a file, directory or symbolic link to - itself. (Eli Collins via suresh) - - HADOOP-6710. Symbolic umask for file creation is not conformant with posix. - (suresh) - - HADOOP-6719. Insert all missing methods in FilterFs. - (Rodrigo Schmidt via dhruba) - - HADOOP-6724. IPC doesn't properly handle IOEs thrown by socket factory. - (Todd Lipcon via tomwhite) - - HADOOP-6722. NetUtils.connect should check that it hasn't connected a socket - to itself. (Todd Lipcon via tomwhite) - - HADOOP-6634. Fix AccessControlList to use short names to verify access - control. (Vinod Kumar Vavilapalli via sharad) - - HADOOP-6709. Re-instate deprecated FileSystem methods that were removed - after 0.20. (tomwhite) - - HADOOP-6630. hadoop-config.sh fails to get executed if hadoop wrapper - scripts are in path. (Allen Wittenauer via tomwhite) - - HADOOP-6742. Add methods HADOOP-6709 from to TestFilterFileSystem. - (Eli Collins via tomwhite) - - HADOOP-6727. Remove UnresolvedLinkException from public FileContext APIs. - (Eli Collins via tomwhite) - - HADOOP-6631. Fix FileUtil.fullyDelete() to continue deleting other files - despite failure at any level. (Contributed by Ravi Gummadi and - Vinod Kumar Vavilapalli) - - HADOOP-6723. Unchecked exceptions thrown in IPC Connection should not - orphan clients. (Todd Lipcon via tomwhite) - - HADOOP-6404. Rename the generated artifacts to common instead of core. - (tomwhite) - - HADOOP-6461. Webapps aren't located correctly post-split. - (Todd Lipcon and Steve Loughran via tomwhite) - - HADOOP-6826. Revert FileSystem create method that takes CreateFlags. - (tomwhite) - - HADOOP-6800. Harmonize JAR library versions. (tomwhite) - - HADOOP-6847. Problem staging 0.21.0 artifacts to Apache Nexus Maven - Repository (Giridharan Kesavan via cos) - - HADOOP-6819. [Herriot] Shell command for getting the new exceptions in - the logs returning exitcode 1 after executing successfully. (Vinay Thota - via cos) - - HADOOP-6839. [Herriot] Implement a functionality for getting the user list - for creating proxy users. (Vinay Thota via cos) - - HADOOP-6836. [Herriot]: Generic method for adding/modifying the attributes - for new configuration. (Vinay Thota via cos) - - HADOOP-6860. 'compile-fault-inject' should never be called directly. - (Konstantin Boudnik) - - HADOOP-6790. Instrumented (Herriot) build uses too wide mask to include - aspect files. (Konstantin Boudnik) - - HADOOP-6875. [Herriot] Cleanup of temp. configurations is needed upon - restart of a cluster (Vinay Thota via cos) - -Release 0.20.3 - Unreleased - - NEW FEATURES - - HADOOP-6637. Benchmark for establishing RPC session. (shv) - - BUG FIXES - - HADOOP-6760. WebServer shouldn't increase port number in case of negative - port setting caused by Jetty's race (cos) - - HADOOP-6881. Make WritableComparator intialize classes when - looking for their raw comparator, as classes often register raw - comparators in initializers, which are no longer automatically run - in Java 6 when a class is referenced. (cutting via omalley) - - HADOOP-7072. Remove java5 dependencies from build. (cos) - -Release 0.20.204.0 - Unreleased - - NEW FEATURES - - HADOOP-6255. Create RPM and Debian packages for common. Changes deployment - layout to be consistent across the binary tgz, rpm, and deb. Adds setup - scripts for easy one node cluster configuration and user creation. - (Eric Yang via omalley) - -Release 0.20.203.0 - 2011-5-11 - - BUG FIXES - - HADOOP-7258. The Gzip codec should not return null decompressors. (omalley) - -Release 0.20.2 - 2010-2-16 - - NEW FEATURES - - HADOOP-6218. Adds a feature where TFile can be split by Record - Sequence number. (Hong Tang and Raghu Angadi via ddas) - - BUG FIXES - - HADOOP-6231. Allow caching of filesystem instances to be disabled on a - per-instance basis. (tomwhite) - - HADOOP-5759. Fix for IllegalArgumentException when CombineFileInputFormat - is used as job InputFormat. (Amareshwari Sriramadasu via dhruba) - - HADOOP-6097. Fix Path conversion in makeQualified and reset LineReader byte - count at the start of each block in Hadoop archives. (Ben Slusky, Tom - White, and Mahadev Konar via cdouglas) - - HADOOP-6269. Fix threading issue with defaultResource in Configuration. - (Sreekanth Ramakrishnan via cdouglas) - - HADOOP-6460. Reinitializes buffers used for serializing responses in ipc - server on exceeding maximum response size to free up Java heap. (suresh) - - HADOOP-6315. Avoid incorrect use of BuiltInflater/BuiltInDeflater in - GzipCodec. (Aaron Kimball via cdouglas) - - HADOOP-6498. IPC client bug may cause rpc call hang. (Ruyue Ma and - hairong via hairong) - - IMPROVEMENTS - - HADOOP-5611. Fix C++ libraries to build on Debian Lenny. (Todd Lipcon - via tomwhite) - - HADOOP-5612. Some c++ scripts are not chmodded before ant execution. - (Todd Lipcon via tomwhite) - - HADOOP-1849. Add undocumented configuration parameter for per handler - call queue size in IPC Server. (shv) - -Release 0.20.1 - 2009-09-01 - - INCOMPATIBLE CHANGES - - HADOOP-5726. Remove pre-emption from capacity scheduler code base. - (Rahul Kumar Singh via yhemanth) - - HADOOP-5881. Simplify memory monitoring and scheduling related - configuration. (Vinod Kumar Vavilapalli via yhemanth) - - NEW FEATURES - - HADOOP-6080. Introduce -skipTrash option to rm and rmr. - (Jakob Homan via shv) - - HADOOP-3315. Add a new, binary file foramt, TFile. (Hong Tang via cdouglas) - - IMPROVEMENTS - - HADOOP-5711. Change Namenode file close log to info. (szetszwo) - - HADOOP-5736. Update the capacity scheduler documentation for features - like memory based scheduling, job initialization and removal of pre-emption. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5714. Add a metric for NameNode getFileInfo operation. (Jakob Homan - via szetszwo) - - HADOOP-4372. Improves the way history filenames are obtained and manipulated. - (Amar Kamat via ddas) - - HADOOP-5897. Add name-node metrics to capture java heap usage. - (Suresh Srinivas via shv) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-5691. Makes org.apache.hadoop.mapreduce.Reducer concrete class - instead of abstract. (Amareshwari Sriramadasu via sharad) - - HADOOP-5646. Fixes a problem in TestQueueCapacities. - (Vinod Kumar Vavilapalli via ddas) - - HADOOP-5655. TestMRServerPorts fails on java.net.BindException. (Devaraj - Das via hairong) - - HADOOP-5654. TestReplicationPolicy. fails on java.net.BindException. - (hairong) - - HADOOP-5688. Fix HftpFileSystem checksum path construction. (Tsz Wo - (Nicholas) Sze via cdouglas) - - HADOOP-4674. Fix fs help messages for -test, -text, -tail, -stat - and -touchz options. (Ravi Phulari via szetszwo) - - HADOOP-5718. Remove the check for the default queue in capacity scheduler. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5719. Remove jobs that failed initialization from the waiting queue - in the capacity scheduler. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4744. Attaching another fix to the jetty port issue. The TaskTracker - kills itself if it ever discovers that the port to which jetty is actually - bound is invalid (-1). (ddas) - - HADOOP-5349. Fixes a problem in LocalDirAllocator to check for the return - path value that is returned for the case where the file we want to write - is of an unknown size. (Vinod Kumar Vavilapalli via ddas) - - HADOOP-5636. Prevents a job from going to RUNNING state after it has been - KILLED (this used to happen when the SetupTask would come back with a - success after the job has been killed). (Amar Kamat via ddas) - - HADOOP-5641. Fix a NullPointerException in capacity scheduler's memory - based scheduling code when jobs get retired. (yhemanth) - - HADOOP-5828. Use absolute path for mapred.local.dir of JobTracker in - MiniMRCluster. (yhemanth) - - HADOOP-4981. Fix capacity scheduler to schedule speculative tasks - correctly in the presence of High RAM jobs. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5210. Solves a problem in the progress report of the reduce task. - (Ravi Gummadi via ddas) - - HADOOP-5850. Fixes a problem to do with not being able to jobs with - 0 maps/reduces. (Vinod K V via ddas) - - HADOOP-4626. Correct the API links in hdfs forrest doc so that they - point to the same version of hadoop. (szetszwo) - - HADOOP-5883. Fixed tasktracker memory monitoring to account for - momentary spurts in memory usage due to java's fork() model. - (yhemanth) - - HADOOP-5539. Fixes a problem to do with not preserving intermediate - output compression for merged data. - (Jothi Padmanabhan and Billy Pearson via ddas) - - HADOOP-5932. Fixes a problem in capacity scheduler in computing - available memory on a tasktracker. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-5908. Fixes a problem to do with ArithmeticException in the - JobTracker when there are jobs with 0 maps. (Amar Kamat via ddas) - - HADOOP-5924. Fixes a corner case problem to do with job recovery with - empty history files. Also, after a JT restart, sends KillTaskAction to - tasks that report back but the corresponding job hasn't been initialized - yet. (Amar Kamat via ddas) - - HADOOP-5882. Fixes a reducer progress update problem for new mapreduce - api. (Amareshwari Sriramadasu via sharad) - - HADOOP-5746. Fixes a corner case problem in Streaming, where if an exception - happens in MROutputThread after the last call to the map/reduce method, the - exception goes undetected. (Amar Kamat via ddas) - - HADOOP-5884. Fixes accounting in capacity scheduler so that high RAM jobs - take more slots. (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-5937. Correct a safemode message in FSNamesystem. (Ravi Phulari - via szetszwo) - - HADOOP-5869. Fix bug in assignment of setup / cleanup task that was - causing TestQueueCapacities to fail. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5921. Fixes a problem in the JobTracker where it sometimes never used - to come up due to a system file creation on JobTracker's system-dir failing. - This problem would sometimes show up only when the FS for the system-dir - (usually HDFS) is started at nearly the same time as the JobTracker. - (Amar Kamat via ddas) - - HADOOP-5920. Fixes a testcase failure for TestJobHistory. - (Amar Kamat via ddas) - - HADOOP-6139. Fix the FsShell help messages for rm and rmr. (Jakob Homan - via szetszwo) - - HADOOP-6145. Fix FsShell rm/rmr error messages when there is a FNFE. - (Jakob Homan via szetszwo) - - HADOOP-6150. Users should be able to instantiate comparator using TFile - API. (Hong Tang via rangadi) - -Release 0.20.0 - 2009-04-15 - - INCOMPATIBLE CHANGES - - HADOOP-4210. Fix findbugs warnings for equals implementations of mapred ID - classes. Removed public, static ID::read and ID::forName; made ID an - abstract class. (Suresh Srinivas via cdouglas) - - HADOOP-4253. Fix various warnings generated by findbugs. - Following deprecated methods in RawLocalFileSystem are removed: - public String getName() - public void lock(Path p, boolean shared) - public void release(Path p) - (Suresh Srinivas via johan) - - HADOOP-4618. Move http server from FSNamesystem into NameNode. - FSNamesystem.getNameNodeInfoPort() is removed. - FSNamesystem.getDFSNameNodeMachine() and FSNamesystem.getDFSNameNodePort() - replaced by FSNamesystem.getDFSNameNodeAddress(). - NameNode(bindAddress, conf) is removed. - (shv) - - HADOOP-4567. GetFileBlockLocations returns the NetworkTopology - information of the machines where the blocks reside. (dhruba) - - HADOOP-4435. The JobTracker WebUI displays the amount of heap memory - in use. (dhruba) - - HADOOP-4628. Move Hive into a standalone subproject. (omalley) - - HADOOP-4188. Removes task's dependency on concrete filesystems. - (Sharad Agarwal via ddas) - - HADOOP-1650. Upgrade to Jetty 6. (cdouglas) - - HADOOP-3986. Remove static Configuration from JobClient. (Amareshwari - Sriramadasu via cdouglas) - JobClient::setCommandLineConfig is removed - JobClient::getCommandLineConfig is removed - JobShell, TestJobShell classes are removed - - HADOOP-4422. S3 file systems should not create bucket. - (David Phillips via tomwhite) - - HADOOP-4035. Support memory based scheduling in capacity scheduler. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-3497. Fix bug in overly restrictive file globbing with a - PathFilter. (tomwhite) - - HADOOP-4445. Replace running task counts with running task - percentage in capacity scheduler UI. (Sreekanth Ramakrishnan via - yhemanth) - - HADOOP-4631. Splits the configuration into three parts - one for core, - one for mapred and the last one for HDFS. (Sharad Agarwal via cdouglas) - - HADOOP-3344. Fix libhdfs build to use autoconf and build the same - architecture (32 vs 64 bit) of the JVM running Ant. The libraries for - pipes, utils, and libhdfs are now all in c++//lib. - (Giridharan Kesavan via nigel) - - HADOOP-4874. Remove LZO codec because of licensing issues. (omalley) - - HADOOP-4970. The full path name of a file is preserved inside Trash. - (Prasad Chakka via dhruba) - - HADOOP-4103. NameNode keeps a count of missing blocks. It warns on - WebUI if there are such blocks. '-report' and '-metaSave' have extra - info to track such blocks. (Raghu Angadi) - - HADOOP-4783. Change permissions on history files on the jobtracker - to be only group readable instead of world readable. - (Amareshwari Sriramadasu via yhemanth) - - NEW FEATURES - - HADOOP-4575. Add a proxy service for relaying HsftpFileSystem requests. - Includes client authentication via user certificates and config-based - access control. (Kan Zhang via cdouglas) - - HADOOP-4661. Add DistCh, a new tool for distributed ch{mod,own,grp}. - (szetszwo) - - HADOOP-4709. Add several new features and bug fixes to Chukwa. - Added Hadoop Infrastructure Care Center (UI for visualize data collected - by Chukwa) - Added FileAdaptor for streaming small file in one chunk - Added compression to archive and demux output - Added unit tests and validation for agent, collector, and demux map - reduce job - Added database loader for loading demux output (sequence file) to jdbc - connected database - Added algorithm to distribute collector load more evenly - (Jerome Boulon, Eric Yang, Andy Konwinski, Ariel Rabkin via cdouglas) - - HADOOP-4179. Add Vaidya tool to analyze map/reduce job logs for performanc - problems. (Suhas Gogate via omalley) - - HADOOP-4029. Add NameNode storage information to the dfshealth page and - move DataNode information to a separated page. (Boris Shkolnik via - szetszwo) - - HADOOP-4348. Add service-level authorization for Hadoop. (acmurthy) - - HADOOP-4826. Introduce admin command saveNamespace. (shv) - - HADOOP-3063 BloomMapFile - fail-fast version of MapFile for sparsely - populated key space (Andrzej Bialecki via stack) - - HADOOP-1230. Add new map/reduce API and deprecate the old one. Generally, - the old code should work without problem. The new api is in - org.apache.hadoop.mapreduce and the old classes in org.apache.hadoop.mapred - are deprecated. Differences in the new API: - 1. All of the methods take Context objects that allow us to add new - methods without breaking compatability. - 2. Mapper and Reducer now have a "run" method that is called once and - contains the control loop for the task, which lets applications - replace it. - 3. Mapper and Reducer by default are Identity Mapper and Reducer. - 4. The FileOutputFormats use part-r-00000 for the output of reduce 0 and - part-m-00000 for the output of map 0. - 5. The reduce grouping comparator now uses the raw compare instead of - object compare. - 6. The number of maps in FileInputFormat is controlled by min and max - split size rather than min size and the desired number of maps. - (omalley) - - HADOOP-3305. Use Ivy to manage dependencies. (Giridharan Kesavan - and Steve Loughran via cutting) - - IMPROVEMENTS - - HADOOP-4749. Added a new counter REDUCE_INPUT_BYTES. (Yongqiang He via - zshao) - - HADOOP-4234. Fix KFS "glue" layer to allow applications to interface - with multiple KFS metaservers. (Sriram Rao via lohit) - - HADOOP-4245. Update to latest version of KFS "glue" library jar. - (Sriram Rao via lohit) - - HADOOP-4244. Change test-patch.sh to check Eclipse classpath no matter - it is run by Hudson or not. (szetszwo) - - HADOOP-3180. Add name of missing class to WritableName.getClass - IOException. (Pete Wyckoff via omalley) - - HADOOP-4178. Make the capacity scheduler's default values configurable. - (Sreekanth Ramakrishnan via omalley) - - HADOOP-4262. Generate better error message when client exception has null - message. (stevel via omalley) - - HADOOP-4226. Refactor and document LineReader to make it more readily - understandable. (Yuri Pradkin via cdouglas) - - HADOOP-4238. When listing jobs, if scheduling information isn't available - print NA instead of empty output. (Sreekanth Ramakrishnan via johan) - - HADOOP-4284. Support filters that apply to all requests, or global filters, - to HttpServer. (Kan Zhang via cdouglas) - - HADOOP-4276. Improve the hashing functions and deserialization of the - mapred ID classes. (omalley) - - HADOOP-4485. Add a compile-native ant task, as a shorthand. (enis) - - HADOOP-4454. Allow # comments in slaves file. (Rama Ramasamy via omalley) - - HADOOP-3461. Remove hdfs.StringBytesWritable. (szetszwo) - - HADOOP-4437. Use Halton sequence instead of java.util.Random in - PiEstimator. (szetszwo) - - HADOOP-4572. Change INode and its sub-classes to package private. - (szetszwo) - - HADOOP-4187. Does a runtime lookup for JobConf/JobConfigurable, and if - found, invokes the appropriate configure method. (Sharad Agarwal via ddas) - - HADOOP-4453. Improve ssl configuration and handling in HsftpFileSystem, - particularly when used with DistCp. (Kan Zhang via cdouglas) - - HADOOP-4583. Several code optimizations in HDFS. (Suresh Srinivas via - szetszwo) - - HADOOP-3923. Remove org.apache.hadoop.mapred.StatusHttpServer. (szetszwo) - - HADOOP-4622. Explicitly specify interpretor for non-native - pipes binaries. (Fredrik Hedberg via johan) - - HADOOP-4505. Add a unit test to test faulty setup task and cleanup - task killing the job. (Amareshwari Sriramadasu via johan) - - HADOOP-4608. Don't print a stack trace when the example driver gets an - unknown program to run. (Edward Yoon via omalley) - - HADOOP-4645. Package HdfsProxy contrib project without the extra level - of directories. (Kan Zhang via omalley) - - HADOOP-4126. Allow access to HDFS web UI on EC2 (tomwhite via omalley) - - HADOOP-4612. Removes RunJar's dependency on JobClient. - (Sharad Agarwal via ddas) - - HADOOP-4185. Adds setVerifyChecksum() method to FileSystem. - (Sharad Agarwal via ddas) - - HADOOP-4523. Prevent too many tasks scheduled on a node from bringing - it down by monitoring for cumulative memory usage across tasks. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-4640. Adds an input format that can split lzo compressed - text files. (johan) - - HADOOP-4666. Launch reduces only after a few maps have run in the - Fair Scheduler. (Matei Zaharia via johan) - - HADOOP-4339. Remove redundant calls from FileSystem/FsShell when - generating/processing ContentSummary. (David Phillips via cdouglas) - - HADOOP-2774. Add counters tracking records spilled to disk in MapTask and - ReduceTask. (Ravi Gummadi via cdouglas) - - HADOOP-4513. Initialize jobs asynchronously in the capacity scheduler. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4649. Improve abstraction for spill indices. (cdouglas) - - HADOOP-3770. Add gridmix2, an iteration on the gridmix benchmark. (Runping - Qi via cdouglas) - - HADOOP-4708. Add support for dfsadmin commands in TestCLI. (Boris Shkolnik - via cdouglas) - - HADOOP-4758. Add a splitter for metrics contexts to support more than one - type of collector. (cdouglas) - - HADOOP-4722. Add tests for dfsadmin quota error messages. (Boris Shkolnik - via cdouglas) - - HADOOP-4690. fuse-dfs - create source file/function + utils + config + - main source files. (pete wyckoff via mahadev) - - HADOOP-3750. Fix and enforce module dependencies. (Sharad Agarwal via - tomwhite) - - HADOOP-4747. Speed up FsShell::ls by removing redundant calls to the - filesystem. (David Phillips via cdouglas) - - HADOOP-4305. Improves the blacklisting strategy, whereby, tasktrackers - that are blacklisted are not given tasks to run from other jobs, subject - to the following conditions (all must be met): - 1) The TaskTracker has been blacklisted by at least 4 jobs (configurable) - 2) The TaskTracker has been blacklisted 50% more number of times than - the average (configurable) - 3) The cluster has less than 50% trackers blacklisted - Once in 24 hours, a TaskTracker blacklisted for all jobs is given a chance. - Restarting the TaskTracker moves it out of the blacklist. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4688. Modify the MiniMRDFSSort unit test to spill multiple times, - exercising the map-side merge code. (cdouglas) - - HADOOP-4737. Adds the KILLED notification when jobs get killed. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4728. Add a test exercising different namenode configurations. - (Boris Shkolnik via cdouglas) - - HADOOP-4807. Adds JobClient commands to get the active/blacklisted tracker - names. Also adds commands to display running/completed task attempt IDs. - (ddas) - - HADOOP-4699. Remove checksum validation from map output servlet. (cdouglas) - - HADOOP-4838. Added a registry to automate metrics and mbeans management. - (Sanjay Radia via acmurthy) - - HADOOP-3136. Fixed the default scheduler to assign multiple tasks to each - tasktracker per heartbeat, when feasible. To ensure locality isn't hurt - too badly, the scheudler will not assign more than one off-switch task per - heartbeat. The heartbeat interval is also halved since the task-tracker is - fixed to no longer send out heartbeats on each task completion. A - slow-start for scheduling reduces is introduced to ensure that reduces - aren't started till sufficient number of maps are done, else reduces of - jobs whose maps aren't scheduled might swamp the cluster. - Configuration changes to mapred-default.xml: - add mapred.reduce.slowstart.completed.maps - (acmurthy) - - HADOOP-4545. Add example and test case of secondary sort for the reduce. - (omalley) - - HADOOP-4753. Refactor gridmix2 to reduce code duplication. (cdouglas) - - HADOOP-4909. Fix Javadoc and make some of the API more consistent in their - use of the JobContext instead of Configuration. (omalley) - - HADOOP-4920. Stop storing Forrest output in Subversion. (cutting) - - HADOOP-4948. Add parameters java5.home and forrest.home to the ant commands - in test-patch.sh. (Giridharan Kesavan via szetszwo) - - HADOOP-4830. Add end-to-end test cases for testing queue capacities. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-4980. Improve code layout of capacity scheduler to make it - easier to fix some blocker bugs. (Vivek Ratan via yhemanth) - - HADOOP-4916. Make user/location of Chukwa installation configurable by an - external properties file. (Eric Yang via cdouglas) - - HADOOP-4950. Make the CompressorStream, DecompressorStream, - BlockCompressorStream, and BlockDecompressorStream public to facilitate - non-Hadoop codecs. (omalley) - - HADOOP-4843. Collect job history and configuration in Chukwa. (Eric Yang - via cdouglas) - - HADOOP-5030. Build Chukwa RPM to install into configured directory. (Eric - Yang via cdouglas) - - HADOOP-4828. Updates documents to do with configuration (HADOOP-4631). - (Sharad Agarwal via ddas) - - HADOOP-4939. Adds a test that would inject random failures for tasks in - large jobs and would also inject TaskTracker failures. (ddas) - - HADOOP-4944. A configuration file can include other configuration - files. (Rama Ramasamy via dhruba) - - HADOOP-4804. Provide Forrest documentation for the Fair Scheduler. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-5248. A testcase that checks for the existence of job directory - after the job completes. Fails if it exists. (ddas) - - HADOOP-4664. Introduces multiple job initialization threads, where the - number of threads are configurable via mapred.jobinit.threads. - (Matei Zaharia and Jothi Padmanabhan via ddas) - - HADOOP-4191. Adds a testcase for JobHistory. (Ravi Gummadi via ddas) - - HADOOP-5466. Change documenation CSS style for headers and code. (Corinne - Chandel via szetszwo) - - HADOOP-5275. Add ivy directory and files to built tar. - (Giridharan Kesavan via nigel) - - HADOOP-5468. Add sub-menus to forrest documentation and make some minor - edits. (Corinne Chandel via szetszwo) - - HADOOP-5437. Fix TestMiniMRDFSSort to properly test jvm-reuse. (omalley) - - HADOOP-5521. Removes dependency of TestJobInProgress on RESTART_COUNT - JobHistory tag. (Ravi Gummadi via ddas) - - OPTIMIZATIONS - - HADOOP-3293. Fixes FileInputFormat to do provide locations for splits - based on the rack/host that has the most number of bytes. - (Jothi Padmanabhan via ddas) - - HADOOP-4683. Fixes Reduce shuffle scheduler to invoke - getMapCompletionEvents in a separate thread. (Jothi Padmanabhan - via ddas) - - BUG FIXES - - HADOOP-4204. Fix findbugs warnings related to unused variables, naive - Number subclass instantiation, Map iteration, and badly scoped inner - classes. (Suresh Srinivas via cdouglas) - - HADOOP-4207. Update derby jar file to release 10.4.2 release. - (Prasad Chakka via dhruba) - - HADOOP-4325. SocketInputStream.read() should return -1 in case EOF. - (Raghu Angadi) - - HADOOP-4408. FsAction functions need not create new objects. (cdouglas) - - HADOOP-4440. TestJobInProgressListener tests for jobs killed in queued - state (Amar Kamat via ddas) - - HADOOP-4346. Implement blocking connect so that Hadoop is not affected - by selector problem with JDK default implementation. (Raghu Angadi) - - HADOOP-4388. If there are invalid blocks in the transfer list, Datanode - should handle them and keep transferring the remaining blocks. (Suresh - Srinivas via szetszwo) - - HADOOP-4587. Fix a typo in Mapper javadoc. (Koji Noguchi via szetszwo) - - HADOOP-4530. In fsck, HttpServletResponse sendError fails with - IllegalStateException. (hairong) - - HADOOP-4377. Fix a race condition in directory creation in - NativeS3FileSystem. (David Phillips via cdouglas) - - HADOOP-4621. Fix javadoc warnings caused by duplicate jars. (Kan Zhang via - cdouglas) - - HADOOP-4566. Deploy new hive code to support more types. - (Zheng Shao via dhruba) - - HADOOP-4571. Add chukwa conf files to svn:ignore list. (Eric Yang via - szetszwo) - - HADOOP-4589. Correct PiEstimator output messages and improve the code - readability. (szetszwo) - - HADOOP-4650. Correct a mismatch between the default value of - local.cache.size in the config and the source. (Jeff Hammerbacher via - cdouglas) - - HADOOP-4606. Fix cygpath error if the log directory does not exist. - (szetszwo via omalley) - - HADOOP-4141. Fix bug in ScriptBasedMapping causing potential infinite - loop on misconfigured hadoop-site. (Aaron Kimball via tomwhite) - - HADOOP-4691. Correct a link in the javadoc of IndexedSortable. (szetszwo) - - HADOOP-4598. '-setrep' command skips under-replicated blocks. (hairong) - - HADOOP-4429. Set defaults for user, group in UnixUserGroupInformation so - login fails more predictably when misconfigured. (Alex Loddengaard via - cdouglas) - - HADOOP-4676. Fix broken URL in blacklisted tasktrackers page. (Amareshwari - Sriramadasu via cdouglas) - - HADOOP-3422 Ganglia counter metrics are all reported with the metric - name "value", so the counter values can not be seen. (Jason Attributor - and Brian Bockelman via stack) - - HADOOP-4704. Fix javadoc typos "the the". (szetszwo) - - HADOOP-4677. Fix semantics of FileSystem::getBlockLocations to return - meaningful values. (Hong Tang via cdouglas) - - HADOOP-4669. Use correct operator when evaluating whether access time is - enabled (Dhruba Borthakur via cdouglas) - - HADOOP-4732. Pass connection and read timeouts in the correct order when - setting up fetch in reduce. (Amareshwari Sriramadasu via cdouglas) - - HADOOP-4558. Fix capacity reclamation in capacity scheduler. - (Amar Kamat via yhemanth) - - HADOOP-4770. Fix rungridmix_2 script to work with RunJar. (cdouglas) - - HADOOP-4738. When using git, the saveVersion script will use only the - commit hash for the version and not the message, which requires escaping. - (cdouglas) - - HADOOP-4576. Show pending job count instead of task count in the UI per - queue in capacity scheduler. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4623. Maintain running tasks even if speculative execution is off. - (Amar Kamat via yhemanth) - - HADOOP-4786. Fix broken compilation error in - TestTrackerBlacklistAcrossJobs. (yhemanth) - - HADOOP-4785. Fixes theJobTracker heartbeat to not make two calls to - System.currentTimeMillis(). (Amareshwari Sriramadasu via ddas) - - HADOOP-4792. Add generated Chukwa configuration files to version control - ignore lists. (cdouglas) - - HADOOP-4796. Fix Chukwa test configuration, remove unused components. (Eric - Yang via cdouglas) - - HADOOP-4708. Add binaries missed in the initial checkin for Chukwa. (Eric - Yang via cdouglas) - - HADOOP-4805. Remove black list collector from Chukwa Agent HTTP Sender. - (Eric Yang via cdouglas) - - HADOOP-4837. Move HADOOP_CONF_DIR configuration to chukwa-env.sh (Jerome - Boulon via cdouglas) - - HADOOP-4825. Use ps instead of jps for querying process status in Chukwa. - (Eric Yang via cdouglas) - - HADOOP-4844. Fixed javadoc for - org.apache.hadoop.fs.permission.AccessControlException to document that - it's deprecated in favour of - org.apache.hadoop.security.AccessControlException. (acmurthy) - - HADOOP-4706. Close the underlying output stream in - IFileOutputStream::close. (Jothi Padmanabhan via cdouglas) - - HADOOP-4855. Fixed command-specific help messages for refreshServiceAcl in - DFSAdmin and MRAdmin. (acmurthy) - - HADOOP-4820. Remove unused method FSNamesystem::deleteInSafeMode. (Suresh - Srinivas via cdouglas) - - HADOOP-4698. Lower io.sort.mb to 10 in the tests and raise the junit memory - limit to 512m from 256m. (Nigel Daley via cdouglas) - - HADOOP-4860. Split TestFileTailingAdapters into three separate tests to - avoid contention. (Eric Yang via cdouglas) - - HADOOP-3921. Fixed clover (code coverage) target to work with JDK 6. - (tomwhite via nigel) - - HADOOP-4845. Modify the reduce input byte counter to record only the - compressed size and add a human-readable label. (Yongqiang He via cdouglas) - - HADOOP-4458. Add a test creating symlinks in the working directory. - (Amareshwari Sriramadasu via cdouglas) - - HADOOP-4879. Fix org.apache.hadoop.mapred.Counters to correctly define - Object.equals rather than depend on contentEquals api. (omalley via - acmurthy) - - HADOOP-4791. Fix rpm build process for Chukwa. (Eric Yang via cdouglas) - - HADOOP-4771. Correct initialization of the file count for directories - with quotas. (Ruyue Ma via shv) - - HADOOP-4878. Fix eclipse plugin classpath file to point to ivy's resolved - lib directory and added the same to test-patch.sh. (Giridharan Kesavan via - acmurthy) - - HADOOP-4774. Fix default values of some capacity scheduler configuration - items which would otherwise not work on a fresh checkout. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4876. Fix capacity scheduler reclamation by updating count of - pending tasks correctly. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4849. Documentation for Service Level Authorization implemented in - HADOOP-4348. (acmurthy) - - HADOOP-4827. Replace Consolidator with Aggregator macros in Chukwa (Eric - Yang via cdouglas) - - HADOOP-4894. Correctly parse ps output in Chukwa jettyCollector.sh. (Ari - Rabkin via cdouglas) - - HADOOP-4892. Close fds out of Chukwa ExecPlugin. (Ari Rabkin via cdouglas) - - HADOOP-4889. Fix permissions in RPM packaging. (Eric Yang via cdouglas) - - HADOOP-4869. Fixes the TT-JT heartbeat to have an explicit flag for - restart apart from the initialContact flag that there was earlier. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4716. Fixes ReduceTask.java to clear out the mapping between - hosts and MapOutputLocation upon a JT restart (Amar Kamat via ddas) - - HADOOP-4880. Removes an unnecessary testcase from TestJobTrackerRestart. - (Amar Kamat via ddas) - - HADOOP-4924. Fixes a race condition in TaskTracker re-init. (ddas) - - HADOOP-4854. Read reclaim capacity interval from capacity scheduler - configuration. (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4896. HDFS Fsck does not load HDFS configuration. (Raghu Angadi) - - HADOOP-4956. Creates TaskStatus for failed tasks with an empty Counters - object instead of null. (ddas) - - HADOOP-4979. Fix capacity scheduler to block cluster for failed high - RAM requirements across task types. (Vivek Ratan via yhemanth) - - HADOOP-4949. Fix native compilation. (Chris Douglas via acmurthy) - - HADOOP-4787. Fixes the testcase TestTrackerBlacklistAcrossJobs which was - earlier failing randomly. (Amareshwari Sriramadasu via ddas) - - HADOOP-4914. Add description fields to Chukwa init.d scripts (Eric Yang via - cdouglas) - - HADOOP-4884. Make tool tip date format match standard HICC format. (Eric - Yang via cdouglas) - - HADOOP-4925. Make Chukwa sender properties configurable. (Ari Rabkin via - cdouglas) - - HADOOP-4947. Make Chukwa command parsing more forgiving of whitespace. (Ari - Rabkin via cdouglas) - - HADOOP-5026. Make chukwa/bin scripts executable in repository. (Andy - Konwinski via cdouglas) - - HADOOP-4977. Fix a deadlock between the reclaimCapacity and assignTasks - in capacity scheduler. (Vivek Ratan via yhemanth) - - HADOOP-4988. Fix reclaim capacity to work even when there are queues with - no capacity. (Vivek Ratan via yhemanth) - - HADOOP-5065. Remove generic parameters from argument to - setIn/OutputFormatClass so that it works with SequenceIn/OutputFormat. - (cdouglas via omalley) - - HADOOP-4818. Pass user config to instrumentation API. (Eric Yang via - cdouglas) - - HADOOP-4993. Fix Chukwa agent configuration and startup to make it both - more modular and testable. (Ari Rabkin via cdouglas) - - HADOOP-5048. Fix capacity scheduler to correctly cleanup jobs that are - killed after initialization, but before running. - (Sreekanth Ramakrishnan via yhemanth) - - HADOOP-4671. Mark loop control variables shared between threads as - volatile. (cdouglas) - - HADOOP-5079. HashFunction inadvertently destroys some randomness - (Jonathan Ellis via stack) - - HADOOP-4999. A failure to write to FsEditsLog results in - IndexOutOfBounds exception. (Boris Shkolnik via rangadi) - - HADOOP-5139. Catch IllegalArgumentException during metrics registration - in RPC. (Hairong Kuang via szetszwo) - - HADOOP-5085. Copying a file to local with Crc throws an exception. - (hairong) - - HADOOP-5211. Fix check for job completion in TestSetupAndCleanupFailure. - (enis) - - HADOOP-5254. The Configuration class should be able to work with XML - parsers that do not support xmlinclude. (Steve Loughran via dhruba) - - HADOOP-4692. Namenode in infinite loop for replicating/deleting corrupt - blocks. (hairong) - - HADOOP-5255. Fix use of Math.abs to avoid overflow. (Jonathan Ellis via - cdouglas) - - HADOOP-5269. Fixes a problem to do with tasktracker holding on to - FAILED_UNCLEAN or KILLED_UNCLEAN tasks forever. (Amareshwari Sriramadasu - via ddas) - - HADOOP-5214. Fixes a ConcurrentModificationException while the Fairshare - Scheduler accesses the tasktrackers stored by the JobTracker. - (Rahul Kumar Singh via yhemanth) - - HADOOP-5233. Addresses the three issues - Race condition in updating - status, NPE in TaskTracker task localization when the conf file is missing - (HADOOP-5234) and NPE in handling KillTaskAction of a cleanup task - (HADOOP-5235). (Amareshwari Sriramadasu via ddas) - - HADOOP-5247. Introduces a broadcast of KillJobAction to all trackers when - a job finishes. This fixes a bunch of problems to do with NPE when a - completed job is not in memory and a tasktracker comes to the jobtracker - with a status report of a task belonging to that job. (Amar Kamat via ddas) - - HADOOP-5282. Fixed job history logs for task attempts that are - failed by the JobTracker, say due to lost task trackers. (Amar - Kamat via yhemanth) - - HADOOP-5241. Fixes a bug in disk-space resource estimation. Makes - the estimation formula linear where blowUp = - Total-Output/Total-Input. (Sharad Agarwal via ddas) - - HADOOP-5142. Fix MapWritable#putAll to store key/value classes. - (Do??acan G??ney via enis) - - HADOOP-4744. Workaround for jetty6 returning -1 when getLocalPort - is invoked on the connector. The workaround patch retries a few - times before failing. (Jothi Padmanabhan via yhemanth) - - HADOOP-5280. Adds a check to prevent a task state transition from - FAILED to any of UNASSIGNED, RUNNING, COMMIT_PENDING or - SUCCEEDED. (ddas) - - HADOOP-5272. Fixes a problem to do with detecting whether an - attempt is the first attempt of a Task. This affects JobTracker - restart. (Amar Kamat via ddas) - - HADOOP-5306. Fixes a problem to do with logging/parsing the http port of a - lost tracker. Affects JobTracker restart. (Amar Kamat via ddas) - - HADOOP-5111. Fix Job::set* methods to work with generics. (cdouglas) - - HADOOP-5274. Fix gridmix2 dependency on wordcount example. (cdouglas) - - HADOOP-5145. Balancer sometimes runs out of memory after running - days or weeks. (hairong) - - HADOOP-5338. Fix jobtracker restart to clear task completion - events cached by tasktrackers forcing them to fetch all events - afresh, thus avoiding missed task completion events on the - tasktrackers. (Amar Kamat via yhemanth) - - HADOOP-4695. Change TestGlobalFilter so that it allows a web page to be - filtered more than once for a single access. (Kan Zhang via szetszwo) - - HADOOP-5298. Change TestServletFilter so that it allows a web page to be - filtered more than once for a single access. (szetszwo) - - HADOOP-5432. Disable ssl during unit tests in hdfsproxy, as it is unused - and causes failures. (cdouglas) - - HADOOP-5416. Correct the shell command "fs -test" forrest doc description. - (Ravi Phulari via szetszwo) - - HADOOP-5327. Fixed job tracker to remove files from system directory on - ACL check failures and also check ACLs on restart. - (Amar Kamat via yhemanth) - - HADOOP-5395. Change the exception message when a job is submitted to an - invalid queue. (Rahul Kumar Singh via yhemanth) - - HADOOP-5276. Fixes a problem to do with updating the start time of - a task when the tracker that ran the task is lost. (Amar Kamat via - ddas) - - HADOOP-5278. Fixes a problem to do with logging the finish time of - a task during recovery (after a JobTracker restart). (Amar Kamat - via ddas) - - HADOOP-5490. Fixes a synchronization problem in the - EagerTaskInitializationListener class. (Jothi Padmanabhan via - ddas) - - HADOOP-5493. The shuffle copier threads return the codecs back to - the pool when the shuffle completes. (Jothi Padmanabhan via ddas) - - HADOOP-5414. Fixes IO exception while executing hadoop fs -touchz - fileName by making sure that lease renewal thread exits before dfs - client exits. (hairong) - - HADOOP-5103. FileInputFormat now reuses the clusterMap network - topology object and that brings down the log messages in the - JobClient to do with NetworkTopology.add significantly. (Jothi - Padmanabhan via ddas) - - HADOOP-5483. Fixes a problem in the Directory Cleanup Thread due to which - TestMiniMRWithDFS sometimes used to fail. (ddas) - - HADOOP-5281. Prevent sharing incompatible ZlibCompressor instances between - GzipCodec and DefaultCodec. (cdouglas) - - HADOOP-5463. Balancer throws "Not a host:port pair" unless port is - specified in fs.default.name. (Stuart White via hairong) - - HADOOP-5514. Fix JobTracker metrics and add metrics for wating, failed - tasks. (cdouglas) - - HADOOP-5516. Fix NullPointerException in TaskMemoryManagerThread - that comes when monitored processes disappear when the thread is - running. (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-5382. Support combiners in the new context object API. (omalley) - - HADOOP-5471. Fixes a problem to do with updating the log.index file in the - case where a cleanup task is run. (Amareshwari Sriramadasu via ddas) - - HADOOP-5534. Fixed a deadlock in Fair scheduler's servlet. - (Rahul Kumar Singh via yhemanth) - - HADOOP-5328. Fixes a problem in the renaming of job history files during - job recovery. (Amar Kamat via ddas) - - HADOOP-5417. Don't ignore InterruptedExceptions that happen when calling - into rpc. (omalley) - - HADOOP-5320. Add a close() in TestMapReduceLocal. (Jothi Padmanabhan - via szetszwo) - - HADOOP-5520. Fix a typo in disk quota help message. (Ravi Phulari - via szetszwo) - - HADOOP-5519. Remove claims from mapred-default.xml that prime numbers - of tasks are helpful. (Owen O'Malley via szetszwo) - - HADOOP-5484. TestRecoveryManager fails wtih FileAlreadyExistsException. - (Amar Kamat via hairong) - - HADOOP-5564. Limit the JVM heap size in the java command for initializing - JAVA_PLATFORM. (Suresh Srinivas via szetszwo) - - HADOOP-5565. Add API for failing/finalized jobs to the JT metrics - instrumentation. (Jerome Boulon via cdouglas) - - HADOOP-5390. Remove duplicate jars from tarball, src from binary tarball - added by hdfsproxy. (Zhiyong Zhang via cdouglas) - - HADOOP-5066. Building binary tarball should not build docs/javadocs, copy - src, or run jdiff. (Giridharan Kesavan via cdouglas) - - HADOOP-5459. Fix undetected CRC errors where intermediate output is closed - before it has been completely consumed. (cdouglas) - - HADOOP-5571. Remove widening primitive conversion in TupleWritable mask - manipulation. (Jingkei Ly via cdouglas) - - HADOOP-5588. Remove an unnecessary call to listStatus(..) in - FileSystem.globStatusInternal(..). (Hairong Kuang via szetszwo) - - HADOOP-5473. Solves a race condition in killing a task - the state is KILLED - if there is a user request pending to kill the task and the TT reported - the state as SUCCESS. (Amareshwari Sriramadasu via ddas) - - HADOOP-5576. Fix LocalRunner to work with the new context object API in - mapreduce. (Tom White via omalley) - - HADOOP-4374. Installs a shutdown hook in the Task JVM so that log.index is - updated before the JVM exits. Also makes the update to log.index atomic. - (Ravi Gummadi via ddas) - - HADOOP-5577. Add a verbose flag to mapreduce.Job.waitForCompletion to get - the running job's information printed to the user's stdout as it runs. - (omalley) - - HADOOP-5607. Fix NPE in TestCapacityScheduler. (cdouglas) - - HADOOP-5605. All the replicas incorrectly got marked as corrupt. (hairong) - - HADOOP-5337. JobTracker, upon restart, now waits for the TaskTrackers to - join back before scheduling new tasks. This fixes race conditions associated - with greedy scheduling as was the case earlier. (Amar Kamat via ddas) - - HADOOP-5227. Fix distcp so -update and -delete can be meaningfully - combined. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-5305. Increase number of files and print debug messages in - TestCopyFiles. (szetszwo) - - HADOOP-5548. Add synchronization for JobTracker methods in RecoveryManager. - (Amareshwari Sriramadasu via sharad) - - HADOOP-3810. NameNode seems unstable on a cluster with little space left. - (hairong) - - HADOOP-5068. Fix NPE in TestCapacityScheduler. (Vinod Kumar Vavilapalli - via szetszwo) - - HADOOP-5585. Clear FileSystem statistics between tasks when jvm-reuse - is enabled. (omalley) - - HADOOP-5394. JobTracker might schedule 2 attempts of the same task - with the same attempt id across restarts. (Amar Kamat via sharad) - - HADOOP-5645. After HADOOP-4920 we need a place to checkin - releasenotes.html. (nigel) - -Release 0.19.2 - 2009-06-30 - - BUG FIXES - - HADOOP-5154. Fixes a deadlock in the fairshare scheduler. - (Matei Zaharia via yhemanth) - - HADOOP-5146. Fixes a race condition that causes LocalDirAllocator to miss - files. (Devaraj Das via yhemanth) - - HADOOP-4638. Fixes job recovery to not crash the job tracker for problems - with a single job file. (Amar Kamat via yhemanth) - - HADOOP-5384. Fix a problem that DataNodeCluster creates blocks with - generationStamp == 1. (szetszwo) - - HADOOP-5376. Fixes the code handling lost tasktrackers to set the task state - to KILLED_UNCLEAN only for relevant type of tasks. - (Amareshwari Sriramadasu via yhemanth) - - HADOOP-5285. Fixes the issues - (1) obtainTaskCleanupTask checks whether job is - inited before trying to lock the JobInProgress (2) Moves the CleanupQueue class - outside the TaskTracker and makes it a generic class that is used by the - JobTracker also for deleting the paths on the job's output fs. (3) Moves the - references to completedJobStore outside the block where the JobTracker is locked. - (ddas) - - HADOOP-5392. Fixes a problem to do with JT crashing during recovery when - the job files are garbled. (Amar Kamat via ddas) - - HADOOP-5332. Appending to files is not allowed (by default) unless - dfs.support.append is set to true. (dhruba) - - HADOOP-5333. libhdfs supports appending to files. (dhruba) - - HADOOP-3998. Fix dfsclient exception when JVM is shutdown. (dhruba) - - HADOOP-5440. Fixes a problem to do with removing a taskId from the list - of taskIds that the TaskTracker's TaskMemoryManager manages. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5446. Restore TaskTracker metrics. (cdouglas) - - HADOOP-5449. Fixes the history cleaner thread. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5479. NameNode should not send empty block replication request to - DataNode. (hairong) - - HADOOP-5259. Job with output hdfs:/user//outputpath (no - authority) fails with Wrong FS. (Doug Cutting via hairong) - - HADOOP-5522. Documents the setup/cleanup tasks in the mapred tutorial. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5549. ReplicationMonitor should schedule both replication and - deletion work in one iteration. (hairong) - - HADOOP-5554. DataNodeCluster and CreateEditsLog should create blocks with - the same generation stamp value. (hairong via szetszwo) - - HADOOP-5231. Clones the TaskStatus before passing it to the JobInProgress. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4719. Fix documentation of 'ls' format for FsShell. (Ravi Phulari - via cdouglas) - - HADOOP-5374. Fixes a NPE problem in getTasksToSave method. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4780. Cache the size of directories in DistributedCache, avoiding - long delays in recalculating it. (He Yongqiang via cdouglas) - - HADOOP-5551. Prevent directory destruction on file create. - (Brian Bockelman via shv) - - HADOOP-5671. Fix FNF exceptions when copying from old versions of - HftpFileSystem. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-5213. Fix Null pointer exception caused when bzip2compression - was used and user closed a output stream without writing any data. - (Zheng Shao via dhruba) - - HADOOP-5579. Set errno correctly in libhdfs for permission, quota, and FNF - conditions. (Brian Bockelman via cdouglas) - - HADOOP-5816. Fixes a problem in the KeyFieldBasedComparator to do with - ArrayIndexOutOfBounds exception. (He Yongqiang via ddas) - - HADOOP-5951. Add Apache license header to StorageInfo.java. (Suresh - Srinivas via szetszwo) - -Release 0.19.1 - 2009-02-23 - - IMPROVEMENTS - - HADOOP-4739. Fix spelling and grammar, improve phrasing of some sections in - mapred tutorial. (Vivek Ratan via cdouglas) - - HADOOP-3894. DFSClient logging improvements. (Steve Loughran via shv) - - HADOOP-5126. Remove empty file BlocksWithLocations.java (shv) - - HADOOP-5127. Remove public methods in FSDirectory. (Jakob Homan via shv) - - BUG FIXES - - HADOOP-4697. Fix getBlockLocations in KosmosFileSystem to handle multiple - blocks correctly. (Sriram Rao via cdouglas) - - HADOOP-4420. Add null checks for job, caused by invalid job IDs. - (Aaron Kimball via tomwhite) - - HADOOP-4632. Fix TestJobHistoryVersion to use test.build.dir instead of the - current workding directory for scratch space. (Amar Kamat via cdouglas) - - HADOOP-4508. Fix FSDataOutputStream.getPos() for append. (dhruba via - szetszwo) - - HADOOP-4727. Fix a group checking bug in fill_stat_structure(...) in - fuse-dfs. (Brian Bockelman via szetszwo) - - HADOOP-4836. Correct typos in mapred related documentation. (Jord? Polo - via szetszwo) - - HADOOP-4821. Usage description in the Quotas guide documentations are - incorrect. (Boris Shkolnik via hairong) - - HADOOP-4847. Moves the loading of OutputCommitter to the Task. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4966. Marks completed setup tasks for removal. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4982. TestFsck should run in Eclipse. (shv) - - HADOOP-5008. TestReplication#testPendingReplicationRetry leaves an opened - fd unclosed. (hairong) - - HADOOP-4906. Fix TaskTracker OOM by keeping a shallow copy of JobConf in - TaskTracker.TaskInProgress. (Sharad Agarwal via acmurthy) - - HADOOP-4918. Fix bzip2 compression to work with Sequence Files. - (Zheng Shao via dhruba). - - HADOOP-4965. TestFileAppend3 should close FileSystem. (shv) - - HADOOP-4967. Fixes a race condition in the JvmManager to do with killing - tasks. (ddas) - - HADOOP-5009. DataNode#shutdown sometimes leaves data block scanner - verification log unclosed. (hairong) - - HADOOP-5086. Use the appropriate FileSystem for trash URIs. (cdouglas) - - HADOOP-4955. Make DBOutputFormat us column names from setOutput(). - (Kevin Peterson via enis) - - HADOOP-4862. Minor : HADOOP-3678 did not remove all the cases of - spurious IOExceptions logged by DataNode. (Raghu Angadi) - - HADOOP-5034. NameNode should send both replication and deletion requests - to DataNode in one reply to a heartbeat. (hairong) - - HADOOP-4759. Removes temporary output directory for failed and killed - tasks by launching special CLEANUP tasks for the same. - (Amareshwari Sriramadasu via ddas) - - HADOOP-5161. Accepted sockets do not get placed in - DataXceiverServer#childSockets. (hairong) - - HADOOP-5193. Correct calculation of edits modification time. (shv) - - HADOOP-4494. Allow libhdfs to append to files. - (Pete Wyckoff via dhruba) - - HADOOP-5166. Fix JobTracker restart to work when ACLs are configured - for the JobTracker. (Amar Kamat via yhemanth). - - HADOOP-5067. Fixes TaskInProgress.java to keep track of count of failed and - killed tasks correctly. (Amareshwari Sriramadasu via ddas) - - HADOOP-4760. HDFS streams should not throw exceptions when closed twice. - (enis) - -Release 0.19.0 - 2008-11-18 - - INCOMPATIBLE CHANGES - - HADOOP-3595. Remove deprecated methods for mapred.combine.once - functionality, which was necessary to providing backwards - compatible combiner semantics for 0.18. (cdouglas via omalley) - - HADOOP-3667. Remove the following deprecated methods from JobConf: - addInputPath(Path) - getInputPaths() - getMapOutputCompressionType() - getOutputPath() - getSystemDir() - setInputPath(Path) - setMapOutputCompressionType(CompressionType style) - setOutputPath(Path) - (Amareshwari Sriramadasu via omalley) - - HADOOP-3652. Remove deprecated class OutputFormatBase. - (Amareshwari Sriramadasu via cdouglas) - - HADOOP-2885. Break the hadoop.dfs package into separate packages under - hadoop.hdfs that reflect whether they are client, server, protocol, - etc. DistributedFileSystem and DFSClient have moved and are now - considered package private. (Sanjay Radia via omalley) - - HADOOP-2325. Require Java 6. (cutting) - - HADOOP-372. Add support for multiple input paths with a different - InputFormat and Mapper for each path. (Chris Smith via tomwhite) - - HADOOP-1700. Support appending to file in HDFS. (dhruba) - - HADOOP-3792. Make FsShell -test consistent with unix semantics, returning - zero for true and non-zero for false. (Ben Slusky via cdouglas) - - HADOOP-3664. Remove the deprecated method InputFormat.validateInput, - which is no longer needed. (tomwhite via omalley) - - HADOOP-3549. Give more meaningful errno's in libhdfs. In particular, - EACCES is returned for permission problems. (Ben Slusky via omalley) - - HADOOP-4036. ResourceStatus was added to TaskTrackerStatus by HADOOP-3759, - so increment the InterTrackerProtocol version. (Hemanth Yamijala via - omalley) - - HADOOP-3150. Moves task promotion to tasks. Defines a new interface for - committing output files. Moves job setup to jobclient, and moves jobcleanup - to a separate task. (Amareshwari Sriramadasu via ddas) - - HADOOP-3446. Keep map outputs in memory during the reduce. Remove - fs.inmemory.size.mb and replace with properties defining in memory map - output retention during the shuffle and reduce relative to maximum heap - usage. (cdouglas) - - HADOOP-3245. Adds the feature for supporting JobTracker restart. Running - jobs can be recovered from the history file. The history file format has - been modified to support recovery. The task attempt ID now has the - JobTracker start time to disinguish attempts of the same TIP across - restarts. (Amar Ramesh Kamat via ddas) - - HADOOP-4007. REMOVE DFSFileInfo - FileStatus is sufficient. - (Sanjay Radia via hairong) - - HADOOP-3722. Fixed Hadoop Streaming and Hadoop Pipes to use the Tool - interface and GenericOptionsParser. (Enis Soztutar via acmurthy) - - HADOOP-2816. Cluster summary at name node web reports the space - utilization as: - Configured Capacity: capacity of all the data directories - Reserved space - Present Capacity: Space available for dfs,i.e. remaining+used space - DFS Used%: DFS used space/Present Capacity - (Suresh Srinivas via hairong) - - HADOOP-3938. Disk space quotas for HDFS. This is similar to namespace - quotas in 0.18. (rangadi) - - HADOOP-4293. Make Configuration Writable and remove unreleased - WritableJobConf. Configuration.write is renamed to writeXml. (omalley) - - HADOOP-4281. Change dfsadmin to report available disk space in a format - consistent with the web interface as defined in HADOOP-2816. (Suresh - Srinivas via cdouglas) - - HADOOP-4430. Further change the cluster summary at name node web that was - changed in HADOOP-2816: - Non DFS Used - This indicates the disk space taken by non DFS file from - the Configured capacity - DFS Used % - DFS Used % of Configured Capacity - DFS Remaining % - Remaing % Configured Capacity available for DFS use - DFS command line report reflects the same change. Config parameter - dfs.datanode.du.pct is no longer used and is removed from the - hadoop-default.xml. (Suresh Srinivas via hairong) - - HADOOP-4116. Balancer should provide better resource management. (hairong) - - HADOOP-4599. BlocksMap and BlockInfo made package private. (shv) - - NEW FEATURES - - HADOOP-3341. Allow streaming jobs to specify the field separator for map - and reduce input and output. The new configuration values are: - stream.map.input.field.separator - stream.map.output.field.separator - stream.reduce.input.field.separator - stream.reduce.output.field.separator - All of them default to "\t". (Zheng Shao via omalley) - - HADOOP-3479. Defines the configuration file for the resource manager in - Hadoop. You can configure various parameters related to scheduling, such - as queues and queue properties here. The properties for a queue follow a - naming convention,such as, hadoop.rm.queue.queue-name.property-name. - (Hemanth Yamijala via ddas) - - HADOOP-3149. Adds a way in which map/reducetasks can create multiple - outputs. (Alejandro Abdelnur via ddas) - - HADOOP-3714. Add a new contrib, bash-tab-completion, which enables - bash tab completion for the bin/hadoop script. See the README file - in the contrib directory for the installation. (Chris Smith via enis) - - HADOOP-3730. Adds a new JobConf constructor that disables loading - default configurations. (Alejandro Abdelnur via ddas) - - HADOOP-3772. Add a new Hadoop Instrumentation api for the JobTracker and - the TaskTracker, refactor Hadoop Metrics as an implementation of the api. - (Ari Rabkin via acmurthy) - - HADOOP-2302. Provides a comparator for numerical sorting of key fields. - (ddas) - - HADOOP-153. Provides a way to skip bad records. (Sharad Agarwal via ddas) - - HADOOP-657. Free disk space should be modelled and used by the scheduler - to make scheduling decisions. (Ari Rabkin via omalley) - - HADOOP-3719. Initial checkin of Chukwa, which is a data collection and - analysis framework. (Jerome Boulon, Andy Konwinski, Ari Rabkin, - and Eric Yang) - - HADOOP-3873. Add -filelimit and -sizelimit options to distcp to cap the - number of files/bytes copied in a particular run to support incremental - updates and mirroring. (TszWo (Nicholas), SZE via cdouglas) - - HADOOP-3585. FailMon package for hardware failure monitoring and - analysis of anomalies. (Ioannis Koltsidas via dhruba) - - HADOOP-1480. Add counters to the C++ Pipes API. (acmurthy via omalley) - - HADOOP-3854. Add support for pluggable servlet filters in the HttpServers. - (Tsz Wo (Nicholas) Sze via omalley) - - HADOOP-3759. Provides ability to run memory intensive jobs without - affecting other running tasks on the nodes. (Hemanth Yamijala via ddas) - - HADOOP-3746. Add a fair share scheduler. (Matei Zaharia via omalley) - - HADOOP-3754. Add a thrift interface to access HDFS. (dhruba via omalley) - - HADOOP-3828. Provides a way to write skipped records to DFS. - (Sharad Agarwal via ddas) - - HADOOP-3948. Separate name-node edits and fsimage directories. - (Lohit Vijayarenu via shv) - - HADOOP-3939. Add an option to DistCp to delete files at the destination - not present at the source. (Tsz Wo (Nicholas) Sze via cdouglas) - - HADOOP-3601. Add a new contrib module for Hive, which is a sql-like - query processing tool that uses map/reduce. (Ashish Thusoo via omalley) - - HADOOP-3866. Added sort and multi-job updates in the JobTracker web ui. - (Craig Weisenfluh via omalley) - - HADOOP-3698. Add access control to control who is allowed to submit or - modify jobs in the JobTracker. (Hemanth Yamijala via omalley) - - HADOOP-1869. Support access times for HDFS files. (dhruba) - - HADOOP-3941. Extend FileSystem API to return file-checksums. - (szetszwo) - - HADOOP-3581. Prevents memory intensive user tasks from taking down - nodes. (Vinod K V via ddas) - - HADOOP-3970. Provides a way to recover counters written to JobHistory. - (Amar Kamat via ddas) - - HADOOP-3702. Adds ChainMapper and ChainReducer classes allow composing - chains of Maps and Reduces in a single Map/Reduce job, something like - MAP+ / REDUCE MAP*. (Alejandro Abdelnur via ddas) - - HADOOP-3445. Add capacity scheduler that provides guaranteed capacities to - queues as a percentage of the cluster. (Vivek Ratan via omalley) - - HADOOP-3992. Add a synthetic load generation facility to the test - directory. (hairong via szetszwo) - - HADOOP-3981. Implement a distributed file checksum algorithm in HDFS - and change DistCp to use file checksum for comparing src and dst files - (szetszwo) - - HADOOP-3829. Narrown down skipped records based on user acceptable value. - (Sharad Agarwal via ddas) - - HADOOP-3930. Add common interfaces for the pluggable schedulers and the - cli & gui clients. (Sreekanth Ramakrishnan via omalley) - - HADOOP-4176. Implement getFileChecksum(Path) in HftpFileSystem. (szetszwo) - - HADOOP-249. Reuse JVMs across Map-Reduce Tasks. - Configuration changes to hadoop-default.xml: - add mapred.job.reuse.jvm.num.tasks - (Devaraj Das via acmurthy) - - HADOOP-4070. Provide a mechanism in Hive for registering UDFs from the - query language. (tomwhite) - - HADOOP-2536. Implement a JDBC based database input and output formats to - allow Map-Reduce applications to work with databases. (Fredrik Hedberg and - Enis Soztutar via acmurthy) - - HADOOP-3019. A new library to support total order partitions. - (cdouglas via omalley) - - HADOOP-3924. Added a 'KILLED' job status. (Subramaniam Krishnan via - acmurthy) - - IMPROVEMENTS - - HADOOP-4205. hive: metastore and ql to use the refactored SerDe library. - (zshao) - - HADOOP-4106. libhdfs: add time, permission and user attribute support - (part 2). (Pete Wyckoff through zshao) - - HADOOP-4104. libhdfs: add time, permission and user attribute support. - (Pete Wyckoff through zshao) - - HADOOP-3908. libhdfs: better error message if llibhdfs.so doesn't exist. - (Pete Wyckoff through zshao) - - HADOOP-3732. Delay intialization of datanode block verification till - the verification thread is started. (rangadi) - - HADOOP-1627. Various small improvements to 'dfsadmin -report' output. - (rangadi) - - HADOOP-3577. Tools to inject blocks into name node and simulated - data nodes for testing. (Sanjay Radia via hairong) - - HADOOP-2664. Add a lzop compatible codec, so that files compressed by lzop - may be processed by map/reduce. (cdouglas via omalley) - - HADOOP-3655. Add additional ant properties to control junit. (Steve - Loughran via omalley) - - HADOOP-3543. Update the copyright year to 2008. (cdouglas via omalley) - - HADOOP-3587. Add a unit test for the contrib/data_join framework. - (cdouglas) - - HADOOP-3402. Add terasort example program (omalley) - - HADOOP-3660. Add replication factor for injecting blocks in simulated - datanodes. (Sanjay Radia via cdouglas) - - HADOOP-3684. Add a cloning function to the contrib/data_join framework - permitting users to define a more efficient method for cloning values from - the reduce than serialization/deserialization. (Runping Qi via cdouglas) - - HADOOP-3478. Improves the handling of map output fetching. Now the - randomization is by the hosts (and not the map outputs themselves). - (Jothi Padmanabhan via ddas) - - HADOOP-3617. Removed redundant checks of accounting space in MapTask and - makes the spill thread persistent so as to avoid creating a new one for - each spill. (Chris Douglas via acmurthy) - - HADOOP-3412. Factor the scheduler out of the JobTracker and make - it pluggable. (Tom White and Brice Arnould via omalley) - - HADOOP-3756. Minor. Remove unused dfs.client.buffer.dir from - hadoop-default.xml. (rangadi) - - HADOOP-3747. Adds counter suport for MultipleOutputs. - (Alejandro Abdelnur via ddas) - - HADOOP-3169. LeaseChecker daemon should not be started in DFSClient - constructor. (TszWo (Nicholas), SZE via hairong) - - HADOOP-3824. Move base functionality of StatusHttpServer to a core - package. (TszWo (Nicholas), SZE via cdouglas) - - HADOOP-3646. Add a bzip2 compatible codec, so bzip compressed data - may be processed by map/reduce. (Abdul Qadeer via cdouglas) - - HADOOP-3861. MapFile.Reader and Writer should implement Closeable. - (tomwhite via omalley) - - HADOOP-3791. Introduce generics into ReflectionUtils. (Chris Smith via - cdouglas) - - HADOOP-3694. Improve unit test performance by changing - MiniDFSCluster to listen only on 127.0.0.1. (cutting) - - HADOOP-3620. Namenode should synchronously resolve a datanode's network - location when the datanode registers. (hairong) - - HADOOP-3860. NNThroughputBenchmark is extended with rename and delete - benchmarks. (shv) - - HADOOP-3892. Include unix group name in JobConf. (Matei Zaharia via johan) - - HADOOP-3875. Change the time period between heartbeats to be relative to - the end of the heartbeat rpc, rather than the start. This causes better - behavior if the JobTracker is overloaded. (acmurthy via omalley) - - HADOOP-3853. Move multiple input format (HADOOP-372) extension to - library package. (tomwhite via johan) - - HADOOP-9. Use roulette scheduling for temporary space when the size - is not known. (Ari Rabkin via omalley) - - HADOOP-3202. Use recursive delete rather than FileUtil.fullyDelete. - (Amareshwari Sriramadasu via omalley) - - HADOOP-3368. Remove common-logging.properties from conf. (Steve Loughran - via omalley) - - HADOOP-3851. Fix spelling mistake in FSNamesystemMetrics. (Steve Loughran - via omalley) - - HADOOP-3780. Remove asynchronous resolution of network topology in the - JobTracker (Amar Kamat via omalley) - - HADOOP-3852. Add ShellCommandExecutor.toString method to make nicer - error messages. (Steve Loughran via omalley) - - HADOOP-3844. Include message of local exception in RPC client failures. - (Steve Loughran via omalley) - - HADOOP-3935. Split out inner classes from DataNode.java. (johan) - - HADOOP-3905. Create generic interfaces for edit log streams. (shv) - - HADOOP-3062. Add metrics to DataNode and TaskTracker to record network - traffic for HDFS reads/writes and MR shuffling. (cdouglas) - - HADOOP-3742. Remove HDFS from public java doc and add javadoc-dev for - generative javadoc for developers. (Sanjay Radia via omalley) - - HADOOP-3944. Improve documentation for public TupleWritable class in - join package. (Chris Douglas via enis) - - HADOOP-2330. Preallocate HDFS transaction log to improve performance. - (dhruba and hairong) - - HADOOP-3965. Convert DataBlockScanner into a package private class. (shv) - - HADOOP-3488. Prevent hadoop-daemon from rsync'ing log files (Stefan - Groshupf and Craig Macdonald via omalley) - - HADOOP-3342. Change the kill task actions to require http post instead of - get to prevent accidental crawls from triggering it. (enis via omalley) - - HADOOP-3937. Limit the job name in the job history filename to 50 - characters. (Matei Zaharia via omalley) - - HADOOP-3943. Remove unnecessary synchronization in - NetworkTopology.pseudoSortByDistance. (hairong via omalley) - - HADOOP-3498. File globbing alternation should be able to span path - components. (tomwhite) - - HADOOP-3361. Implement renames for NativeS3FileSystem. - (Albert Chern via tomwhite) - - HADOOP-3605. Make EC2 scripts show an error message if AWS_ACCOUNT_ID is - unset. (Al Hoang via tomwhite) - - HADOOP-4147. Remove unused class JobWithTaskContext from class - JobInProgress. (Amareshwari Sriramadasu via johan) - - HADOOP-4151. Add a byte-comparable interface that both Text and - BytesWritable implement. (cdouglas via omalley) - - HADOOP-4174. Move fs image/edit log methods from ClientProtocol to - NamenodeProtocol. (shv via szetszwo) - - HADOOP-4181. Include a .gitignore and saveVersion.sh change to support - developing under git. (omalley) - - HADOOP-4186. Factor LineReader out of LineRecordReader. (tomwhite via - omalley) - - HADOOP-4184. Break the module dependencies between core, hdfs, and - mapred. (tomwhite via omalley) - - HADOOP-4075. test-patch.sh now spits out ant commands that it runs. - (Ramya R via nigel) - - HADOOP-4117. Improve configurability of Hadoop EC2 instances. - (tomwhite) - - HADOOP-2411. Add support for larger CPU EC2 instance types. - (Chris K Wensel via tomwhite) - - HADOOP-4083. Changed the configuration attribute queue.name to - mapred.job.queue.name. (Hemanth Yamijala via acmurthy) - - HADOOP-4194. Added the JobConf and JobID to job-related methods in - JobTrackerInstrumentation for better metrics. (Mac Yang via acmurthy) - - HADOOP-3975. Change test-patch script to report working the dir - modifications preventing the suite from being run. (Ramya R via cdouglas) - - HADOOP-4124. Added a command-line switch to allow users to set job - priorities, also allow it to be manipulated via the web-ui. (Hemanth - Yamijala via acmurthy) - - HADOOP-2165. Augmented JobHistory to include the URIs to the tasks' - userlogs. (Vinod Kumar Vavilapalli via acmurthy) - - HADOOP-4062. Remove the synchronization on the output stream when a - connection is closed and also remove an undesirable exception when - a client is stoped while there is no pending RPC request. (hairong) - - HADOOP-4227. Remove the deprecated class org.apache.hadoop.fs.ShellCommand. - (szetszwo) - - HADOOP-4006. Clean up FSConstants and move some of the constants to - better places. (Sanjay Radia via rangadi) - - HADOOP-4279. Trace the seeds of random sequences in append unit tests to - make itermitant failures reproducible. (szetszwo via cdouglas) - - HADOOP-4209. Remove the change to the format of task attempt id by - incrementing the task attempt numbers by 1000 when the job restarts. - (Amar Kamat via omalley) - - HADOOP-4301. Adds forrest doc for the skip bad records feature. - (Sharad Agarwal via ddas) - - HADOOP-4354. Separate TestDatanodeDeath.testDatanodeDeath() into 4 tests. - (szetszwo) - - HADOOP-3790. Add more unit tests for testing HDFS file append. (szetszwo) - - HADOOP-4321. Include documentation for the capacity scheduler. (Hemanth - Yamijala via omalley) - - HADOOP-4424. Change menu layout for Hadoop documentation (Boris Shkolnik - via cdouglas). - - HADOOP-4438. Update forrest documentation to include missing FsShell - commands. (Suresh Srinivas via cdouglas) - - HADOOP-4105. Add forrest documentation for libhdfs. - (Pete Wyckoff via cutting) - - HADOOP-4510. Make getTaskOutputPath public. (Chris Wensel via omalley) - - OPTIMIZATIONS - - HADOOP-3556. Removed lock contention in MD5Hash by changing the - singleton MessageDigester by an instance per Thread using - ThreadLocal. (Iv?n de Prado via omalley) - - HADOOP-3328. When client is writing data to DFS, only the last - datanode in the pipeline needs to verify the checksum. Saves around - 30% CPU on intermediate datanodes. (rangadi) - - HADOOP-3863. Use a thread-local string encoder rather than a static one - that is protected by a lock. (acmurthy via omalley) - - HADOOP-3864. Prevent the JobTracker from locking up when a job is being - initialized. (acmurthy via omalley) - - HADOOP-3816. Faster directory listing in KFS. (Sriram Rao via omalley) - - HADOOP-2130. Pipes submit job should have both blocking and non-blocking - versions. (acmurthy via omalley) - - HADOOP-3769. Make the SampleMapper and SampleReducer from - GenericMRLoadGenerator public, so they can be used in other contexts. - (Lingyun Yang via omalley) - - HADOOP-3514. Inline the CRCs in intermediate files as opposed to reading - it from a different .crc file. (Jothi Padmanabhan via ddas) - - HADOOP-3638. Caches the iFile index files in memory to reduce seeks - (Jothi Padmanabhan via ddas) - - HADOOP-4225. FSEditLog.logOpenFile() should persist accessTime - rather than modificationTime. (shv) - - HADOOP-4380. Made several new classes (Child, JVMId, - JobTrackerInstrumentation, QueueManager, ResourceEstimator, - TaskTrackerInstrumentation, and TaskTrackerMetricsInst) in - org.apache.hadoop.mapred package private instead of public. (omalley) - - BUG FIXES - - HADOOP-3563. Refactor the distributed upgrade code so that it is - easier to identify datanode and namenode related code. (dhruba) - - HADOOP-3640. Fix the read method in the NativeS3InputStream. (tomwhite via - omalley) - - HADOOP-3711. Fixes the Streaming input parsing to properly find the - separator. (Amareshwari Sriramadasu via ddas) - - HADOOP-3725. Prevent TestMiniMRMapDebugScript from swallowing exceptions. - (Steve Loughran via cdouglas) - - HADOOP-3726. Throw exceptions from TestCLI setup and teardown instead of - swallowing them. (Steve Loughran via cdouglas) - - HADOOP-3721. Refactor CompositeRecordReader and related mapred.join classes - to make them clearer. (cdouglas) - - HADOOP-3720. Re-read the config file when dfsadmin -refreshNodes is invoked - so dfs.hosts and dfs.hosts.exclude are observed. (lohit vijayarenu via - cdouglas) - - HADOOP-3485. Allow writing to files over fuse. - (Pete Wyckoff via dhruba) - - HADOOP-3723. The flags to the libhdfs.create call can be treated as - a bitmask. (Pete Wyckoff via dhruba) - - HADOOP-3643. Filter out completed tasks when asking for running tasks in - the JobTracker web/ui. (Amar Kamat via omalley) - - HADOOP-3777. Ensure that Lzo compressors/decompressors correctly handle the - case where native libraries aren't available. (Chris Douglas via acmurthy) - - HADOOP-3728. Fix SleepJob so that it doesn't depend on temporary files, - this ensures we can now run more than one instance of SleepJob - simultaneously. (Chris Douglas via acmurthy) - - HADOOP-3795. Fix saving image files on Namenode with different checkpoint - stamps. (Lohit Vijayarenu via mahadev) - - HADOOP-3624. Improving createeditslog to create tree directory structure. - (Lohit Vijayarenu via mahadev) - - HADOOP-3778. DFSInputStream.seek() did not retry in case of some errors. - (Luo Ning via rangadi) - - HADOOP-3661. The handling of moving files deleted through fuse-dfs to - Trash made similar to the behaviour from dfs shell. - (Pete Wyckoff via dhruba) - - HADOOP-3819. Unset LANG and LC_CTYPE in saveVersion.sh to make it - compatible with non-English locales. (Rong-En Fan via cdouglas) - - HADOOP-3848. Cache calls to getSystemDir in the TaskTracker instead of - calling it for each task start. (acmurthy via omalley) - - HADOOP-3131. Fix reduce progress reporting for compressed intermediate - data. (Matei Zaharia via acmurthy) - - HADOOP-3796. fuse-dfs configuration is implemented as file system - mount options. (Pete Wyckoff via dhruba) - - HADOOP-3836. Fix TestMultipleOutputs to correctly clean up. (Alejandro - Abdelnur via acmurthy) - - HADOOP-3805. Improve fuse-dfs write performance. - (Pete Wyckoff via zshao) - - HADOOP-3846. Fix unit test CreateEditsLog to generate paths correctly. - (Lohit Vjayarenu via cdouglas) - - HADOOP-3904. Fix unit tests using the old dfs package name. - (TszWo (Nicholas), SZE via johan) - - HADOOP-3319. Fix some HOD error messages to go stderr instead of - stdout. (Vinod Kumar Vavilapalli via omalley) - - HADOOP-3907. Move INodeDirectoryWithQuota to its own .java file. - (Tsz Wo (Nicholas), SZE via hairong) - - HADOOP-3919. Fix attribute name in hadoop-default for - mapred.jobtracker.instrumentation. (Ari Rabkin via omalley) - - HADOOP-3903. Change the package name for the servlets to be hdfs instead of - dfs. (Tsz Wo (Nicholas) Sze via omalley) - - HADOOP-3773. Change Pipes to set the default map output key and value - types correctly. (Koji Noguchi via omalley) - - HADOOP-3952. Fix compilation error in TestDataJoin referencing dfs package. - (omalley) - - HADOOP-3951. Fix package name for FSNamesystem logs and modify other - hard-coded Logs to use the class name. (cdouglas) - - HADOOP-3889. Improve error reporting from HftpFileSystem, handling in - DistCp. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3946. Fix TestMapRed after hadoop-3664. (tomwhite via omalley) - - HADOOP-3949. Remove duplicate jars from Chukwa. (Jerome Boulon via omalley) - - HADOOP-3933. DataNode sometimes sends up to io.byte.per.checksum bytes - more than required to client. (Ning Li via rangadi) - - HADOOP-3962. Shell command "fs -count" should support paths with different - file systems. (Tsz Wo (Nicholas), SZE via mahadev) - - HADOOP-3957. Fix javac warnings in DistCp and TestCopyFiles. (Tsz Wo - (Nicholas), SZE via cdouglas) - - HADOOP-3958. Fix TestMapRed to check the success of test-job. (omalley via - acmurthy) - - HADOOP-3985. Fix TestHDFSServerPorts to use random ports. (Hairong Kuang - via omalley) - - HADOOP-3964. Fix javadoc warnings introduced by FailMon. (dhruba) - - HADOOP-3785. Fix FileSystem cache to be case-insensitive for scheme and - authority. (Bill de hOra via cdouglas) - - HADOOP-3506. Fix a rare NPE caused by error handling in S3. (Tom White via - cdouglas) - - HADOOP-3705. Fix mapred.join parser to accept InputFormats named with - underscore and static, inner classes. (cdouglas) - - HADOOP-4023. Fix javadoc warnings introduced when the HDFS javadoc was - made private. (omalley) - - HADOOP-4030. Remove lzop from the default list of codecs. (Arun Murthy via - cdouglas) - - HADOOP-3961. Fix task disk space requirement estimates for virtual - input jobs. Delays limiting task placement until after 10% of the maps - have finished. (Ari Rabkin via omalley) - - HADOOP-2168. Fix problem with C++ record reader's progress not being - reported to framework. (acmurthy via omalley) - - HADOOP-3966. Copy findbugs generated output files to PATCH_DIR while - running test-patch. (Ramya R via lohit) - - HADOOP-4037. Fix the eclipse plugin for versions of kfs and log4j. (nigel - via omalley) - - HADOOP-3950. Cause the Mini MR cluster to wait for task trackers to - register before continuing. (enis via omalley) - - HADOOP-3910. Remove unused ClusterTestDFSNamespaceLogging and - ClusterTestDFS. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3954. Disable record skipping by default. (Sharad Agarwal via - cdouglas) - - HADOOP-4050. Fix TestFairScheduler to use absolute paths for the work - directory. (Matei Zaharia via omalley) - - HADOOP-4069. Keep temporary test files from TestKosmosFileSystem under - test.build.data instead of /tmp. (lohit via omalley) - - HADOOP-4078. Create test files for TestKosmosFileSystem in separate - directory under test.build.data. (lohit) - - HADOOP-3968. Fix getFileBlockLocations calls to use FileStatus instead - of Path reflecting the new API. (Pete Wyckoff via lohit) - - HADOOP-3963. libhdfs does not exit on its own, instead it returns error - to the caller and behaves as a true library. (Pete Wyckoff via dhruba) - - HADOOP-4100. Removes the cleanupTask scheduling from the Scheduler - implementations and moves it to the JobTracker. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4097. Make hive work well with speculative execution turned on. - (Joydeep Sen Sarma via dhruba) - - HADOOP-4113. Changes to libhdfs to not exit on its own, rather return - an error code to the caller. (Pete Wyckoff via dhruba) - - HADOOP-4054. Remove duplicate lease removal during edit log loading. - (hairong) - - HADOOP-4071. FSNameSystem.isReplicationInProgress should add an - underReplicated block to the neededReplication queue using method - "add" not "update". (hairong) - - HADOOP-4154. Fix type warnings in WritableUtils. (szetszwo via omalley) - - HADOOP-4133. Log files generated by Hive should reside in the - build directory. (Prasad Chakka via dhruba) - - HADOOP-4094. Hive now has hive-default.xml and hive-site.xml similar - to core hadoop. (Prasad Chakka via dhruba) - - HADOOP-4112. Handles cleanupTask in JobHistory - (Amareshwari Sriramadasu via ddas) - - HADOOP-3831. Very slow reading clients sometimes failed while reading. - (rangadi) - - HADOOP-4155. Use JobTracker's start time while initializing JobHistory's - JobTracker Unique String. (lohit) - - HADOOP-4099. Fix null pointer when using HFTP from an 0.18 server. - (dhruba via omalley) - - HADOOP-3570. Includes user specified libjar files in the client side - classpath path. (Sharad Agarwal via ddas) - - HADOOP-4129. Changed memory limits of TaskTracker and Tasks to be in - KiloBytes rather than bytes. (Vinod Kumar Vavilapalli via acmurthy) - - HADOOP-4139. Optimize Hive multi group-by. - (Namin Jain via dhruba) - - HADOOP-3911. Add a check to fsck options to make sure -files is not - the first option to resolve conflicts with GenericOptionsParser - (lohit) - - HADOOP-3623. Refactor LeaseManager. (szetszwo) - - HADOOP-4125. Handles Reduce cleanup tip on the web ui. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4087. Hive Metastore API for php and python clients. - (Prasad Chakka via dhruba) - - HADOOP-4197. Update DATA_TRANSFER_VERSION for HADOOP-3981. (szetszwo) - - HADOOP-4138. Refactor the Hive SerDe library to better structure - the interfaces to the serializer and de-serializer. - (Zheng Shao via dhruba) - - HADOOP-4195. Close compressor before returning to codec pool. - (acmurthy via omalley) - - HADOOP-2403. Escapes some special characters before logging to - history files. (Amareshwari Sriramadasu via ddas) - - HADOOP-4200. Fix a bug in the test-patch.sh script. - (Ramya R via nigel) - - HADOOP-4084. Add explain plan capabilities to Hive Query Language. - (Ashish Thusoo via dhruba) - - HADOOP-4121. Preserve cause for exception if the initialization of - HistoryViewer for JobHistory fails. (Amareshwari Sri Ramadasu via - acmurthy) - - HADOOP-4213. Fixes NPE in TestLimitTasksPerJobTaskScheduler. - (Sreekanth Ramakrishnan via ddas) - - HADOOP-4077. Setting access and modification time for a file - requires write permissions on the file. (dhruba) - - HADOOP-3592. Fix a couple of possible file leaks in FileUtil - (Bill de hOra via rangadi) - - HADOOP-4120. Hive interactive shell records the time taken by a - query. (Raghotham Murthy via dhruba) - - HADOOP-4090. The hive scripts pick up hadoop from HADOOP_HOME - and then the path. (Raghotham Murthy via dhruba) - - HADOOP-4242. Remove extra ";" in FSDirectory that blocks compilation - in some IDE's. (szetszwo via omalley) - - HADOOP-4249. Fix eclipse path to include the hsqldb.jar. (szetszwo via - omalley) - - HADOOP-4247. Move InputSampler into org.apache.hadoop.mapred.lib, so that - examples.jar doesn't depend on tools.jar. (omalley) - - HADOOP-4269. Fix the deprecation of LineReader by extending the new class - into the old name and deprecating it. Also update the tests to test the - new class. (cdouglas via omalley) - - HADOOP-4280. Fix conversions between seconds in C and milliseconds in - Java for access times for files. (Pete Wyckoff via rangadi) - - HADOOP-4254. -setSpaceQuota command does not convert "TB" extenstion to - terabytes properly. Implementation now uses StringUtils for parsing this. - (Raghu Angadi) - - HADOOP-4259. Findbugs should run over tools.jar also. (cdouglas via - omalley) - - HADOOP-4275. Move public method isJobValidName from JobID to a private - method in JobTracker. (omalley) - - HADOOP-4173. fix failures in TestProcfsBasedProcessTree and - TestTaskTrackerMemoryManager tests. ProcfsBasedProcessTree and - memory management in TaskTracker are disabled on Windows. - (Vinod K V via rangadi) - - HADOOP-4189. Fixes the history blocksize & intertracker protocol version - issues introduced as part of HADOOP-3245. (Amar Kamat via ddas) - - HADOOP-4190. Fixes the backward compatibility issue with Job History. - introduced by HADOOP-3245 and HADOOP-2403. (Amar Kamat via ddas) - - HADOOP-4237. Fixes the TestStreamingBadRecords.testNarrowDown testcase. - (Sharad Agarwal via ddas) - - HADOOP-4274. Capacity scheduler accidently modifies the underlying - data structures when browing the job lists. (Hemanth Yamijala via omalley) - - HADOOP-4309. Fix eclipse-plugin compilation. (cdouglas) - - HADOOP-4232. Fix race condition in JVM reuse when multiple slots become - free. (ddas via acmurthy) - - HADOOP-4302. Fix a race condition in TestReduceFetch that can yield false - negatvies. (cdouglas) - - HADOOP-3942. Update distcp documentation to include features introduced in - HADOOP-3873, HADOOP-3939. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-4319. fuse-dfs dfs_read function returns as many bytes as it is - told to read unlesss end-of-file is reached. (Pete Wyckoff via dhruba) - - HADOOP-4246. Ensure we have the correct lower bound on the number of - retries for fetching map-outputs; also fixed the case where the reducer - automatically kills on too many unique map-outputs could not be fetched - for small jobs. (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-4163. Report FSErrors from map output fetch threads instead of - merely logging them. (Sharad Agarwal via cdouglas) - - HADOOP-4261. Adds a setup task for jobs. This is required so that we - don't setup jobs that haven't been inited yet (since init could lead - to job failure). Only after the init has successfully happened do we - launch the setupJob task. (Amareshwari Sriramadasu via ddas) - - HADOOP-4256. Removes Completed and Failed Job tables from - jobqueue_details.jsp. (Sreekanth Ramakrishnan via ddas) - - HADOOP-4267. Occasional exceptions during shutting down HSQLDB is logged - but not rethrown. (enis) - - HADOOP-4018. The number of tasks for a single job cannot exceed a - pre-configured maximum value. (dhruba) - - HADOOP-4288. Fixes a NPE problem in CapacityScheduler. - (Amar Kamat via ddas) - - HADOOP-4014. Create hard links with 'fsutil hardlink' on Windows. (shv) - - HADOOP-4393. Merged org.apache.hadoop.fs.permission.AccessControlException - and org.apache.hadoop.security.AccessControlIOException into a single - class hadoop.security.AccessControlException. (omalley via acmurthy) - - HADOOP-4287. Fixes an issue to do with maintaining counts of running/pending - maps/reduces. (Sreekanth Ramakrishnan via ddas) - - HADOOP-4361. Makes sure that jobs killed from command line are killed - fast (i.e., there is a slot to run the cleanup task soon). - (Amareshwari Sriramadasu via ddas) - - HADOOP-4400. Add "hdfs://" to fs.default.name on quickstart.html. - (Jeff Hammerbacher via omalley) - - HADOOP-4378. Fix TestJobQueueInformation to use SleepJob rather than - WordCount via TestMiniMRWithDFS. (Sreekanth Ramakrishnan via acmurthy) - - HADOOP-4376. Fix formatting in hadoop-default.xml for - hadoop.http.filter.initializers. (Enis Soztutar via acmurthy) - - HADOOP-4410. Adds an extra arg to the API FileUtil.makeShellPath to - determine whether to canonicalize file paths or not. - (Amareshwari Sriramadasu via ddas) - - HADOOP-4236. Ensure un-initialized jobs are killed correctly on - user-demand. (Sharad Agarwal via acmurthy) - - HADOOP-4373. Fix calculation of Guaranteed Capacity for the - capacity-scheduler. (Hemanth Yamijala via acmurthy) - - HADOOP-4053. Schedulers must be notified when jobs complete. (Amar Kamat via omalley) - - HADOOP-4335. Fix FsShell -ls for filesystems without owners/groups. (David - Phillips via cdouglas) - - HADOOP-4426. TestCapacityScheduler broke due to the two commits HADOOP-4053 - and HADOOP-4373. This patch fixes that. (Hemanth Yamijala via ddas) - - HADOOP-4418. Updates documentation in forrest for Mapred, streaming and pipes. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3155. Ensure that there is only one thread fetching - TaskCompletionEvents on TaskTracker re-init. (Dhruba Borthakur via - acmurthy) - - HADOOP-4425. Fix EditLogInputStream to overload the bulk read method. - (cdouglas) - - HADOOP-4427. Adds the new queue/job commands to the manual. - (Sreekanth Ramakrishnan via ddas) - - HADOOP-4278. Increase debug logging for unit test TestDatanodeDeath. - Fix the case when primary is dead. (dhruba via szetszwo) - - HADOOP-4423. Keep block length when the block recovery is triggered by - append. (szetszwo) - - HADOOP-4449. Fix dfsadmin usage. (Raghu Angadi via cdouglas) - - HADOOP-4455. Added TestSerDe so that unit tests can run successfully. - (Ashish Thusoo via dhruba) - - HADOOP-4457. Fixes an input split logging problem introduced by - HADOOP-3245. (Amareshwari Sriramadasu via ddas) - - HADOOP-4464. Separate out TestFileCreationClient from TestFileCreation. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-4404. saveFSImage() removes files from a storage directory that do - not correspond to its type. (shv) - - HADOOP-4149. Fix handling of updates to the job priority, by changing the - list of jobs to be keyed by the priority, submit time, and job tracker id. - (Amar Kamat via omalley) - - HADOOP-4296. Fix job client failures by not retiring a job as soon as it - is finished. (dhruba) - - HADOOP-4439. Remove configuration variables that aren't usable yet, in - particular mapred.tasktracker.tasks.maxmemory and mapred.task.max.memory. - (Hemanth Yamijala via omalley) - - HADOOP-4230. Fix for serde2 interface, limit operator, select * operator, - UDF trim functions and sampling. (Ashish Thusoo via dhruba) - - HADOOP-4358. No need to truncate access time in INode. Also fixes NPE - in CreateEditsLog. (Raghu Angadi) - - HADOOP-4387. TestHDFSFileSystemContract fails on windows nightly builds. - (Raghu Angadi) - - HADOOP-4466. Ensure that SequenceFileOutputFormat isn't tied to Writables - and can be used with other Serialization frameworks. (Chris Wensel via - acmurthy) - - HADOOP-4525. Fix ipc.server.ipcnodelay originally missed in in HADOOP-2232. - (cdouglas via Clint Morgan) - - HADOOP-4498. Ensure that JobHistory correctly escapes the job name so that - regex patterns work. (Chris Wensel via acmurthy) - - HADOOP-4446. Modify guaranteed capacity labels in capacity scheduler's UI - to reflect the information being displayed. (Sreekanth Ramakrishnan via - yhemanth) - - HADOOP-4282. Some user facing URLs are not filtered by user filters. - (szetszwo) - - HADOOP-4595. Fixes two race conditions - one to do with updating free slot count, - and another to do with starting the MapEventsFetcher thread. (ddas) - - HADOOP-4552. Fix a deadlock in RPC server. (Raghu Angadi) - - HADOOP-4471. Sort running jobs by priority in the capacity scheduler. - (Amar Kamat via yhemanth) - - HADOOP-4500. Fix MultiFileSplit to get the FileSystem from the relevant - path rather than the JobClient. (Joydeep Sen Sarma via cdouglas) - -Release 0.18.4 - Unreleased - - BUG FIXES - - HADOOP-5114. Remove timeout for accept() in DataNode. This makes accept() - fail in JDK on Windows and causes many tests to fail. (Raghu Angadi) - - HADOOP-5192. Block receiver should not remove a block that's created or - being written by other threads. (hairong) - - HADOOP-5134. FSNamesystem#commitBlockSynchronization adds under-construction - block locations to blocksMap. (Dhruba Borthakur via hairong) - - HADOOP-5412. Simulated DataNode should not write to a block that's being - written by another thread. (hairong) - - HADOOP-5465. Fix the problem of blocks remaining under-replicated by - providing synchronized modification to the counter xmitsInProgress in - DataNode. (hairong) - - HADOOP-5557. Fixes some minor problems in TestOverReplicatedBlocks. - (szetszwo) - - HADOOP-5644. Namenode is stuck in safe mode. (suresh Srinivas via hairong) - - HADOOP-6017. Lease Manager in NameNode does not handle certain characters - in filenames. This results in fatal errors in Secondary NameNode and while - restrating NameNode. (Tsz Wo (Nicholas), SZE via rangadi) - -Release 0.18.3 - 2009-01-27 - - IMPROVEMENTS - - HADOOP-4150. Include librecordio in hadoop releases. (Giridharan Kesavan - via acmurthy) - - HADOOP-4668. Improve documentation for setCombinerClass to clarify the - restrictions on combiners. (omalley) - - BUG FIXES - - HADOOP-4499. DFSClient should invoke checksumOk only once. (Raghu Angadi) - - HADOOP-4597. Calculate mis-replicated blocks when safe-mode is turned - off manually. (shv) - - HADOOP-3121. lsr should keep listing the remaining items but not - terminate if there is any IOException. (szetszwo) - - HADOOP-4610. Always calculate mis-replicated blocks when safe-mode is - turned off. (shv) - - HADOOP-3883. Limit namenode to assign at most one generation stamp for - a particular block within a short period. (szetszwo) - - HADOOP-4556. Block went missing. (hairong) - - HADOOP-4643. NameNode should exclude excessive replicas when counting - live replicas for a block. (hairong) - - HADOOP-4703. Should not wait for proxy forever in lease recovering. - (szetszwo) - - HADOOP-4647. NamenodeFsck should close the DFSClient it has created. - (szetszwo) - - HADOOP-4616. Fuse-dfs can handle bad values from FileSystem.read call. - (Pete Wyckoff via dhruba) - - HADOOP-4061. Throttle Datanode decommission monitoring in Namenode. - (szetszwo) - - HADOOP-4659. Root cause of connection failure is being lost to code that - uses it for delaying startup. (Steve Loughran and Hairong via hairong) - - HADOOP-4614. Lazily open segments when merging map spills to avoid using - too many file descriptors. (Yuri Pradkin via cdouglas) - - HADOOP-4257. The DFS client should pick only one datanode as the candidate - to initiate lease recovery. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-4713. Fix librecordio to handle records larger than 64k. (Christian - Kunz via cdouglas) - - HADOOP-4635. Fix a memory leak in fuse dfs. (pete wyckoff via mahadev) - - HADOOP-4714. Report status between merges and make the number of records - between progress reports configurable. (Jothi Padmanabhan via cdouglas) - - HADOOP-4726. Fix documentation typos "the the". (Edward J. Yoon via - szetszwo) - - HADOOP-4679. Datanode prints tons of log messages: waiting for threadgroup - to exit, active threads is XX. (hairong) - - HADOOP-4746. Job output directory should be normalized. (hairong) - - HADOOP-4717. Removal of default port# in NameNode.getUri() causes a - map/reduce job failed to prompt temporary output. (hairong) - - HADOOP-4778. Check for zero size block meta file when updating a block. - (szetszwo) - - HADOOP-4742. Replica gets deleted by mistake. (Wang Xu via hairong) - - HADOOP-4702. Failed block replication leaves an incomplete block in - receiver's tmp data directory. (hairong) - - HADOOP-4613. Fix block browsing on Web UI. (Johan Oskarsson via shv) - - HADOOP-4806. HDFS rename should not use src path as a regular expression. - (szetszwo) - - HADOOP-4795. Prevent lease monitor getting into an infinite loop when - leases and the namespace tree does not match. (szetszwo) - - HADOOP-4620. Fixes Streaming to handle well the cases of map/reduce with empty - input/output. (Ravi Gummadi via ddas) - - HADOOP-4857. Fixes TestUlimit to have exactly 1 map in the jobs spawned. - (Ravi Gummadi via ddas) - - HADOOP-4810. Data lost at cluster startup time. (hairong) - - HADOOP-4797. Improve how RPC server reads and writes large buffers. Avoids - soft-leak of direct buffers and excess copies in NIO layer. (Raghu Angadi) - - HADOOP-4840. TestNodeCount sometimes fails with NullPointerException. - (hairong) - - HADOOP-4904. Fix deadlock while leaving safe mode. (shv) - - HADOOP-1980. 'dfsadmin -safemode enter' should prevent the namenode from - leaving safemode automatically. (shv & Raghu Angadi) - - HADOOP-4951. Lease monitor should acquire the LeaseManager lock but not the - Monitor lock. (szetszwo) - - HADOOP-4935. processMisReplicatedBlocks() should not clear - excessReplicateMap. (shv) - - HADOOP-4961. Fix ConcurrentModificationException in lease recovery - of empty files. (shv) - - HADOOP-4971. A long (unexpected) delay at datanodes could make subsequent - block reports from many datanode at the same time. (Raghu Angadi) - - HADOOP-4910. NameNode should exclude replicas when choosing excessive - replicas to delete to avoid data lose. (hairong) - - HADOOP-4983. Fixes a problem in updating Counters in the status reporting. - (Amareshwari Sriramadasu via ddas) - -Release 0.18.2 - 2008-11-03 - - BUG FIXES - - HADOOP-3614. Fix a bug that Datanode may use an old GenerationStamp to get - meta file. (szetszwo) - - HADOOP-4314. Simulated datanodes should not include blocks that are still - being written in their block report. (Raghu Angadi) - - HADOOP-4228. dfs datanode metrics, bytes_read and bytes_written, overflow - due to incorrect type used. (hairong) - - HADOOP-4395. The FSEditLog loading is incorrect for the case OP_SET_OWNER. - (szetszwo) - - HADOOP-4351. FSNamesystem.getBlockLocationsInternal throws - ArrayIndexOutOfBoundsException. (hairong) - - HADOOP-4403. Make TestLeaseRecovery and TestFileCreation more robust. - (szetszwo) - - HADOOP-4292. Do not support append() for LocalFileSystem. (hairong) - - HADOOP-4399. Make fuse-dfs multi-thread access safe. - (Pete Wyckoff via dhruba) - - HADOOP-4369. Use setMetric(...) instead of incrMetric(...) for metrics - averages. (Brian Bockelman via szetszwo) - - HADOOP-4469. Rename and add the ant task jar file to the tar file. (nigel) - - HADOOP-3914. DFSClient sends Checksum Ok only once for a block. - (Christian Kunz via hairong) - - HADOOP-4467. SerializationFactory now uses the current context ClassLoader - allowing for user supplied Serialization instances. (Chris Wensel via - acmurthy) - - HADOOP-4517. Release FSDataset lock before joining ongoing create threads. - (szetszwo) - - HADOOP-4526. fsck failing with NullPointerException. (hairong) - - HADOOP-4483 Honor the max parameter in DatanodeDescriptor.getBlockArray(..) - (Ahad Rana and Hairong Kuang via szetszwo) - - HADOOP-4340. Correctly set the exit code from JobShell.main so that the - 'hadoop jar' command returns the right code to the user. (acmurthy) - - NEW FEATURES - - HADOOP-2421. Add jdiff output to documentation, listing all API - changes from the prior release. (cutting) - -Release 0.18.1 - 2008-09-17 - - IMPROVEMENTS - - HADOOP-3934. Upgrade log4j to 1.2.15. (omalley) - - BUG FIXES - - HADOOP-3995. In case of quota failure on HDFS, rename does not restore - source filename. (rangadi) - - HADOOP-3821. Prevent SequenceFile and IFile from duplicating codecs in - CodecPool when closed more than once. (Arun Murthy via cdouglas) - - HADOOP-4040. Remove coded default of the IPC idle connection timeout - from the TaskTracker, which was causing HDFS client connections to not be - collected. (ddas via omalley) - - HADOOP-4046. Made WritableComparable's constructor protected instead of - private to re-enable class derivation. (cdouglas via omalley) - - HADOOP-3940. Fix in-memory merge condition to wait when there are no map - outputs or when the final map outputs are being fetched without contention. - (cdouglas) - -Release 0.18.0 - 2008-08-19 - - INCOMPATIBLE CHANGES - - HADOOP-2703. The default options to fsck skips checking files - that are being written to. The output of fsck is incompatible - with previous release. (lohit vijayarenu via dhruba) - - HADOOP-2865. FsShell.ls() printout format changed to print file names - in the end of the line. (Edward J. Yoon via shv) - - HADOOP-3283. The Datanode has a RPC server. It currently supports - two RPCs: the first RPC retrives the metadata about a block and the - second RPC sets the generation stamp of an existing block. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2797. Code related to upgrading to 0.14 (Block CRCs) is - removed. As result, upgrade to 0.18 or later from 0.13 or earlier - is not supported. If upgrading from 0.13 or earlier is required, - please upgrade to an intermediate version (0.14-0.17) and then - to this version. (rangadi) - - HADOOP-544. This issue introduces new classes JobID, TaskID and - TaskAttemptID, which should be used instead of their string counterparts. - Functions in JobClient, TaskReport, RunningJob, jobcontrol.Job and - TaskCompletionEvent that use string arguments are deprecated in favor - of the corresponding ones that use ID objects. Applications can use - xxxID.toString() and xxxID.forName() methods to convert/restore objects - to/from strings. (Enis Soztutar via ddas) - - HADOOP-2188. RPC client sends a ping rather than throw timeouts. - RPC server does not throw away old RPCs. If clients and the server are on - different versions, they are not able to function well. In addition, - The property ipc.client.timeout is removed from the default hadoop - configuration. It also removes metrics RpcOpsDiscardedOPsNum. (hairong) - - HADOOP-2181. This issue adds logging for input splits in Jobtracker log - and jobHistory log. Also adds web UI for viewing input splits in job UI - and history UI. (Amareshwari Sriramadasu via ddas) - - HADOOP-3226. Run combiners multiple times over map outputs as they - are merged in both the map and the reduce tasks. (cdouglas via omalley) - - HADOOP-3329. DatanodeDescriptor objects should not be stored in the - fsimage. (dhruba) - - HADOOP-2656. The Block object has a generation stamp inside it. - Existing blocks get a generation stamp of 0. This is needed to support - appends. (dhruba) - - HADOOP-3390. Removed deprecated ClientProtocol.abandonFileInProgress(). - (Tsz Wo (Nicholas), SZE via rangadi) - - HADOOP-3405. Made some map/reduce internal classes non-public: - MapTaskStatus, ReduceTaskStatus, JobSubmissionProtocol, - CompletedJobStatusStore. (enis via omaley) - - HADOOP-3265. Removed depcrecated API getFileCacheHints(). - (Lohit Vijayarenu via rangadi) - - HADOOP-3310. The namenode instructs the primary datanode to do lease - recovery. The block gets a new generation stamp. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2909. Improve IPC idle connection management. Property - ipc.client.maxidletime is removed from the default configuration, - instead it is defined as twice of the ipc.client.connection.maxidletime. - A connection with outstanding requests won't be treated as idle. - (hairong) - - HADOOP-3459. Change in the output format of dfs -ls to more closely match - /bin/ls. New format is: perm repl owner group size date name - (Mukund Madhugiri via omally) - - HADOOP-3113. An fsync invoked on a HDFS file really really - persists data! The datanode moves blocks in the tmp directory to - the real block directory on a datanode-restart. (dhruba) - - HADOOP-3452. Change fsck to return non-zero status for a corrupt - FileSystem. (lohit vijayarenu via cdouglas) - - HADOOP-3193. Include the address of the client that found the corrupted - block in the log. Also include a CorruptedBlocks metric to track the size - of the corrupted block map. (cdouglas) - - HADOOP-3512. Separate out the tools into a tools jar. (omalley) - - HADOOP-3598. Ensure that temporary task-output directories are not created - if they are not necessary e.g. for Maps with no side-effect files. - (acmurthy) - - HADOOP-3665. Modify WritableComparator so that it only creates instances - of the keytype if the type does not define a WritableComparator. Calling - the superclass compare will throw a NullPointerException. Also define - a RawComparator for NullWritable and permit it to be written as a key - to SequenceFiles. (cdouglas) - - HADOOP-3673. Avoid deadlock caused by DataNode RPC receoverBlock(). - (Tsz Wo (Nicholas), SZE via rangadi) - - NEW FEATURES - - HADOOP-3074. Provides a UrlStreamHandler for DFS and other FS, - relying on FileSystem (taton) - - HADOOP-2585. Name-node imports namespace data from a recent checkpoint - accessible via a NFS mount. (shv) - - HADOOP-3061. Writable types for doubles and bytes. (Andrzej - Bialecki via omalley) - - HADOOP-2857. Allow libhdfs to set jvm options. (Craig Macdonald - via omalley) - - HADOOP-3317. Add default port for HDFS namenode. The port in - "hdfs:" URIs now defaults to 8020, so that one may simply use URIs - of the form "hdfs://example.com/dir/file". (cutting) - - HADOOP-2019. Adds support for .tar, .tgz and .tar.gz files in - DistributedCache (Amareshwari Sriramadasu via ddas) - - HADOOP-3058. Add FSNamesystem status metrics. - (Lohit Vjayarenu via rangadi) - - HADOOP-1915. Allow users to specify counters via strings instead - of enumerations. (tomwhite via omalley) - - HADOOP-2065. Delay invalidating corrupt replicas of block until its - is removed from under replicated state. If all replicas are found to - be corrupt, retain all copies and mark the block as corrupt. - (Lohit Vjayarenu via rangadi) - - HADOOP-3221. Adds org.apache.hadoop.mapred.lib.NLineInputFormat, which - splits files into splits each of N lines. N can be specified by - configuration property "mapred.line.input.format.linespermap", which - defaults to 1. (Amareshwari Sriramadasu via ddas) - - HADOOP-3336. Direct a subset of annotated FSNamesystem calls for audit - logging. (cdouglas) - - HADOOP-3400. A new API FileSystem.deleteOnExit() that facilitates - handling of temporary files in HDFS. (dhruba) - - HADOOP-4. Add fuse-dfs to contrib, permitting one to mount an - HDFS filesystem on systems that support FUSE, e.g., Linux. - (Pete Wyckoff via cutting) - - HADOOP-3246. Add FTPFileSystem. (Ankur Goel via cutting) - - HADOOP-3250. Extend FileSystem API to allow appending to files. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3177. Implement Syncable interface for FileSystem. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-1328. Implement user counters in streaming. (tomwhite via - omalley) - - HADOOP-3187. Quotas for namespace management. (Hairong Kuang via ddas) - - HADOOP-3307. Support for Archives in Hadoop. (Mahadev Konar via ddas) - - HADOOP-3460. Add SequenceFileAsBinaryOutputFormat to permit direct - writes of serialized data. (Koji Noguchi via cdouglas) - - HADOOP-3230. Add ability to get counter values from command - line. (tomwhite via omalley) - - HADOOP-930. Add support for native S3 files. (tomwhite via cutting) - - HADOOP-3502. Quota API needs documentation in Forrest. (hairong) - - HADOOP-3413. Allow SequenceFile.Reader to use serialization - framework. (tomwhite via omalley) - - HADOOP-3541. Import of the namespace from a checkpoint documented - in hadoop user guide. (shv) - - IMPROVEMENTS - - HADOOP-3677. Simplify generation stamp upgrade by making is a - local upgrade on datandodes. Deleted distributed upgrade. - (rangadi) - - HADOOP-2928. Remove deprecated FileSystem.getContentLength(). - (Lohit Vijayarenu via rangadi) - - HADOOP-3130. Make the connect timeout smaller for getFile. - (Amar Ramesh Kamat via ddas) - - HADOOP-3160. Remove deprecated exists() from ClientProtocol and - FSNamesystem (Lohit Vjayarenu via rangadi) - - HADOOP-2910. Throttle IPC Clients during bursts of requests or - server slowdown. Clients retry connection for up to 15 minutes - when socket connection times out. (hairong) - - HADOOP-3295. Allow TextOutputFormat to use configurable spearators. - (Zheng Shao via cdouglas). - - HADOOP-3308. Improve QuickSort by excluding values eq the pivot from the - partition. (cdouglas) - - HADOOP-2461. Trim property names in configuration. - (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-2799. Deprecate o.a.h.io.Closable in favor of java.io.Closable. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3345. Enhance the hudson-test-patch target to cleanup messages, - fix minor defects, and add eclipse plugin and python unit tests. (nigel) - - HADOOP-3144. Improve robustness of LineRecordReader by defining a maximum - line length (mapred.linerecordreader.maxlength), thereby avoiding reading - too far into the following split. (Zheng Shao via cdouglas) - - HADOOP-3334. Move lease handling from FSNamesystem into a seperate class. - (Tsz Wo (Nicholas), SZE via rangadi) - - HADOOP-3332. Reduces the amount of logging in Reducer's shuffle phase. - (Devaraj Das) - - HADOOP-3355. Enhances Configuration class to accept hex numbers for getInt - and getLong. (Amareshwari Sriramadasu via ddas) - - HADOOP-3350. Add an argument to distcp to permit the user to limit the - number of maps. (cdouglas) - - HADOOP-3013. Add corrupt block reporting to fsck. - (lohit vijayarenu via cdouglas) - - HADOOP-3377. Remove TaskRunner::replaceAll and replace with equivalent - String::replace. (Brice Arnould via cdouglas) - - HADOOP-3398. Minor improvement to a utility function in that participates - in backoff calculation. (cdouglas) - - HADOOP-3381. Clear referenced when directories are deleted so that - effect of memory leaks are not multiplied. (rangadi) - - HADOOP-2867. Adds the task's CWD to its LD_LIBRARY_PATH. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3232. DU class runs the 'du' command in a seperate thread so - that it does not block user. DataNode misses heartbeats in large - nodes otherwise. (Johan Oskarsson via rangadi) - - HADOOP-3035. During block transfers between datanodes, the receiving - datanode, now can report corrupt replicas received from src node to - the namenode. (Lohit Vijayarenu via rangadi) - - HADOOP-3434. Retain the cause of the bind failure in Server::bind. - (Steve Loughran via cdouglas) - - HADOOP-3429. Increases the size of the buffers used for the communication - for Streaming jobs. (Amareshwari Sriramadasu via ddas) - - HADOOP-3486. Change default for initial block report to 0 seconds - and document it. (Sanjay Radia via omalley) - - HADOOP-3448. Improve the text in the assertion making sure the - layout versions are consistent in the data node. (Steve Loughran - via omalley) - - HADOOP-2095. Improve the Map-Reduce shuffle/merge by cutting down - buffer-copies; changed intermediate sort/merge to use the new IFile format - rather than SequenceFiles and compression of map-outputs is now - implemented by compressing the entire file rather than SequenceFile - compression. Shuffle also has been changed to use a simple byte-buffer - manager rather than the InMemoryFileSystem. - Configuration changes to hadoop-default.xml: - deprecated mapred.map.output.compression.type - (acmurthy) - - HADOOP-236. JobTacker now refuses connection from a task tracker with a - different version number. (Sharad Agarwal via ddas) - - HADOOP-3427. Improves the shuffle scheduler. It now waits for notifications - from shuffle threads when it has scheduled enough, before scheduling more. - (ddas) - - HADOOP-2393. Moves the handling of dir deletions in the tasktracker to - a separate thread. (Amareshwari Sriramadasu via ddas) - - HADOOP-3501. Deprecate InMemoryFileSystem. (cutting via omalley) - - HADOOP-3366. Stall the shuffle while in-memory merge is in progress. - (acmurthy) - - HADOOP-2916. Refactor src structure, but leave package structure alone. - (Raghu Angadi via mukund) - - HADOOP-3492. Add forrest documentation for user archives. - (Mahadev Konar via hairong) - - HADOOP-3467. Improve documentation for FileSystem::deleteOnExit. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3379. Documents stream.non.zero.exit.status.is.failure for Streaming. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3096. Improves documentation about the Task Execution Environment in - the Map-Reduce tutorial. (Amareshwari Sriramadasu via ddas) - - HADOOP-2984. Add forrest documentation for DistCp. (cdouglas) - - HADOOP-3406. Add forrest documentation for Profiling. - (Amareshwari Sriramadasu via ddas) - - HADOOP-2762. Add forrest documentation for controls of memory limits on - hadoop daemons and Map-Reduce tasks. (Amareshwari Sriramadasu via ddas) - - HADOOP-3535. Fix documentation and name of IOUtils.close to - reflect that it should only be used in cleanup contexts. (omalley) - - HADOOP-3593. Updates the mapred tutorial. (ddas) - - HADOOP-3547. Documents the way in which native libraries can be distributed - via the DistributedCache. (Amareshwari Sriramadasu via ddas) - - HADOOP-3606. Updates the Streaming doc. (Amareshwari Sriramadasu via ddas) - - HADOOP-3532. Add jdiff reports to the build scripts. (omalley) - - HADOOP-3100. Develop tests to test the DFS command line interface. (mukund) - - HADOOP-3688. Fix up HDFS docs. (Robert Chansler via hairong) - - OPTIMIZATIONS - - HADOOP-3274. The default constructor of BytesWritable creates empty - byte array. (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-3272. Remove redundant copy of Block object in BlocksMap. - (Lohit Vjayarenu via shv) - - HADOOP-3164. Reduce DataNode CPU usage by using FileChannel.tranferTo(). - On Linux DataNode takes 5 times less CPU while serving data. Results may - vary on other platforms. (rangadi) - - HADOOP-3248. Optimization of saveFSImage. (Dhruba via shv) - - HADOOP-3297. Fetch more task completion events from the job - tracker and task tracker. (ddas via omalley) - - HADOOP-3364. Faster image and log edits loading. (shv) - - HADOOP-3369. Fast block processing during name-node startup. (shv) - - HADOOP-1702. Reduce buffer copies when data is written to DFS. - DataNodes take 30% less CPU while writing data. (rangadi) - - HADOOP-3095. Speed up split generation in the FileInputSplit, - especially for non-HDFS file systems. Deprecates - InputFormat.validateInput. (tomwhite via omalley) - - HADOOP-3552. Add forrest documentation for Hadoop commands. - (Sharad Agarwal via cdouglas) - - BUG FIXES - - HADOOP-2905. 'fsck -move' triggers NPE in NameNode. - (Lohit Vjayarenu via rangadi) - - Increment ClientProtocol.versionID missed by HADOOP-2585. (shv) - - HADOOP-3254. Restructure internal namenode methods that process - heartbeats to use well-defined BlockCommand object(s) instead of - using the base java Object. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3176. Change lease record when a open-for-write-file - gets renamed. (dhruba) - - HADOOP-3269. Fix a case when namenode fails to restart - while processing a lease record. ((Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3282. Port issues in TestCheckpoint resolved. (shv) - - HADOOP-3268. file:// URLs issue in TestUrlStreamHandler under Windows. - (taton) - - HADOOP-3127. Deleting files in trash should really remove them. - (Brice Arnould via omalley) - - HADOOP-3300. Fix locking of explicit locks in NetworkTopology. - (tomwhite via omalley) - - HADOOP-3270. Constant DatanodeCommands are stored in static final - immutable variables for better code clarity. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2793. Fix broken links for worst performing shuffle tasks in - the job history page. (Amareshwari Sriramadasu via ddas) - - HADOOP-3313. Avoid unnecessary calls to System.currentTimeMillis - in RPC::Invoker. (cdouglas) - - HADOOP-3318. Recognize "Darwin" as an alias for "Mac OS X" to - support Soylatte. (Sam Pullara via omalley) - - HADOOP-3301. Fix misleading error message when S3 URI hostname - contains an underscore. (tomwhite via omalley) - - HADOOP-3338. Fix Eclipse plugin to compile after HADOOP-544 was - committed. Updated all references to use the new JobID representation. - (taton via nigel) - - HADOOP-3337. Loading FSEditLog was broken by HADOOP-3283 since it - changed Writable serialization of DatanodeInfo. This patch handles it. - (Tsz Wo (Nicholas), SZE via rangadi) - - HADOOP-3101. Prevent JobClient from throwing an exception when printing - usage. (Edward J. Yoon via cdouglas) - - HADOOP-3119. Update javadoc for Text::getBytes to better describe its - behavior. (Tim Nelson via cdouglas) - - HADOOP-2294. Fix documentation in libhdfs to refer to the correct free - function. (Craig Macdonald via cdouglas) - - HADOOP-3335. Prevent the libhdfs build from deleting the wrong - files on make clean. (cutting via omalley) - - HADOOP-2930. Make {start,stop}-balancer.sh work even if hadoop-daemon.sh - is not in the PATH. (Spiros Papadimitriou via hairong) - - HADOOP-3085. Catch Exception in metrics util classes to ensure that - misconfigured metrics don't prevent others from updating. (cdouglas) - - HADOOP-3299. CompositeInputFormat should configure the sub-input - formats. (cdouglas via omalley) - - HADOOP-3309. Lower io.sort.mb and fs.inmemory.size.mb for MiniMRDFSSort - unit test so it passes on Windows. (lohit vijayarenu via cdouglas) - - HADOOP-3348. TestUrlStreamHandler should set URLStreamFactory after - DataNodes are initialized. (Lohit Vijayarenu via rangadi) - - HADOOP-3371. Ignore InstanceAlreadyExistsException from - MBeanUtil::registerMBean. (lohit vijayarenu via cdouglas) - - HADOOP-3349. A file rename was incorrectly changing the name inside a - lease record. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3365. Removes an unnecessary copy of the key from SegmentDescriptor - to MergeQueue. (Devaraj Das) - - HADOOP-3388. Fix for TestDatanodeBlockScanner to handle blocks with - generation stamps in them. (dhruba) - - HADOOP-3203. Fixes TaskTracker::localizeJob to pass correct file sizes - for the jarfile and the jobfile. (Amareshwari Sriramadasu via ddas) - - HADOOP-3391. Fix a findbugs warning introduced by HADOOP-3248 (rangadi) - - HADOOP-3393. Fix datanode shutdown to call DataBlockScanner::shutdown and - close its log, even if the scanner thread is not running. (lohit vijayarenu - via cdouglas) - - HADOOP-3399. A debug message was logged at info level. (rangadi) - - HADOOP-3396. TestDatanodeBlockScanner occationally fails. - (Lohit Vijayarenu via rangadi) - - HADOOP-3339. Some of the failures on 3rd datanode in DFS write pipelie - are not detected properly. This could lead to hard failure of client's - write operation. (rangadi) - - HADOOP-3409. Namenode should save the root inode into fsimage. (hairong) - - HADOOP-3296. Fix task cache to work for more than two levels in the cache - hierarchy. This also adds a new counter to track cache hits at levels - greater than two. (Amar Kamat via cdouglas) - - HADOOP-3375. Lease paths were sometimes not removed from - LeaseManager.sortedLeasesByPath. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3424. Values returned by getPartition should be checked to - make sure they are in the range 0 to #reduces - 1 (cdouglas via - omalley) - - HADOOP-3408. Change FSNamesystem to send its metrics as integers to - accommodate collectors that don't support long values. (lohit vijayarenu - via cdouglas) - - HADOOP-3403. Fixes a problem in the JobTracker to do with handling of lost - tasktrackers. (Arun Murthy via ddas) - - HADOOP-1318. Completed maps are not failed if the number of reducers are - zero. (Amareshwari Sriramadasu via ddas). - - HADOOP-3351. Fixes the history viewer tool to not do huge StringBuffer - allocations. (Amareshwari Sriramadasu via ddas) - - HADOOP-3419. Fixes TestFsck to wait for updates to happen before - checking results to make the test more reliable. (Lohit Vijaya - Renu via omalley) - - HADOOP-3259. Makes failure to read system properties due to a - security manager non-fatal. (Edward Yoon via omalley) - - HADOOP-3451. Update libhdfs to use FileSystem::getFileBlockLocations - instead of removed getFileCacheHints. (lohit vijayarenu via cdouglas) - - HADOOP-3401. Update FileBench to set the new - "mapred.work.output.dir" property to work post-3041. (cdouglas via omalley) - - HADOOP-2669. DFSClient locks pendingCreates appropriately. (dhruba) - - HADOOP-3410. Fix KFS implemenation to return correct file - modification time. (Sriram Rao via cutting) - - HADOOP-3340. Fix DFS metrics for BlocksReplicated, HeartbeatsNum, and - BlockReportsAverageTime. (lohit vijayarenu via cdouglas) - - HADOOP-3435. Remove the assuption in the scripts that bash is at - /bin/bash and fix the test patch to require bash instead of sh. - (Brice Arnould via omalley) - - HADOOP-3471. Fix spurious errors from TestIndexedSort and add additional - logging to let failures be reproducible. (cdouglas) - - HADOOP-3443. Avoid copying map output across partitions when renaming a - single spill. (omalley via cdouglas) - - HADOOP-3454. Fix Text::find to search only valid byte ranges. (Chad Whipkey - via cdouglas) - - HADOOP-3417. Removes the static configuration variable, - commandLineConfig from JobClient. Moves the cli parsing from - JobShell to GenericOptionsParser. Thus removes the class - org.apache.hadoop.mapred.JobShell. (Amareshwari Sriramadasu via - ddas) - - HADOOP-2132. Only RUNNING/PREP jobs can be killed. (Jothi Padmanabhan - via ddas) - - HADOOP-3476. Code cleanup in fuse-dfs. - (Peter Wyckoff via dhruba) - - HADOOP-2427. Ensure that the cwd of completed tasks is cleaned-up - correctly on task-completion. (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-2565. Remove DFSPath cache of FileStatus. - (Tsz Wo (Nicholas), SZE via hairong) - - HADOOP-3326. Cleanup the local-fs and in-memory merge in the ReduceTask by - spawing only one thread each for the on-disk and in-memory merge. - (Sharad Agarwal via acmurthy) - - HADOOP-3493. Fix TestStreamingFailure to use FileUtil.fullyDelete to - ensure correct cleanup. (Lohit Vijayarenu via acmurthy) - - HADOOP-3455. Fix NPE in ipc.Client in case of connection failure and - improve its synchronization. (hairong) - - HADOOP-3240. Fix a testcase to not create files in the current directory. - Instead the file is created in the test directory (Mahadev Konar via ddas) - - HADOOP-3496. Fix failure in TestHarFileSystem.testArchives due to change - in HADOOP-3095. (tomwhite) - - HADOOP-3135. Get the system directory from the JobTracker instead of from - the conf. (Subramaniam Krishnan via ddas) - - HADOOP-3503. Fix a race condition when client and namenode start - simultaneous recovery of the same block. (dhruba & Tsz Wo - (Nicholas), SZE) - - HADOOP-3440. Fixes DistributedCache to not create symlinks for paths which - don't have fragments even when createSymLink is true. - (Abhijit Bagri via ddas) - - HADOOP-3463. Hadoop-daemons script should cd to $HADOOP_HOME. (omalley) - - HADOOP-3489. Fix NPE in SafeModeMonitor. (Lohit Vijayarenu via shv) - - HADOOP-3509. Fix NPE in FSNamesystem.close. (Tsz Wo (Nicholas), SZE via - shv) - - HADOOP-3491. Name-node shutdown causes InterruptedException in - ResolutionMonitor. (Lohit Vijayarenu via shv) - - HADOOP-3511. Fixes namenode image to not set the root's quota to an - invalid value when the quota was not saved in the image. (hairong) - - HADOOP-3516. Ensure the JobClient in HadoopArchives is initialized - with a configuration. (Subramaniam Krishnan via omalley) - - HADOOP-3513. Improve NNThroughputBenchmark log messages. (shv) - - HADOOP-3519. Fix NPE in DFS FileSystem rename. (hairong via tomwhite) - - HADOOP-3528. Metrics FilesCreated and files_deleted metrics - do not match. (Lohit via Mahadev) - - HADOOP-3418. When a directory is deleted, any leases that point to files - in the subdirectory are removed. ((Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3542. Diables the creation of _logs directory for the archives - directory. (Mahadev Konar via ddas) - - HADOOP-3544. Fixes a documentation issue for hadoop archives. - (Mahadev Konar via ddas) - - HADOOP-3517. Fixes a problem in the reducer due to which the last InMemory - merge may be missed. (Arun Murthy via ddas) - - HADOOP-3548. Fixes build.xml to copy all *.jar files to the dist. - (Owen O'Malley via ddas) - - HADOOP-3363. Fix unformatted storage detection in FSImage. (shv) - - HADOOP-3560. Fixes a problem to do with split creation in archives. - (Mahadev Konar via ddas) - - HADOOP-3545. Fixes a overflow problem in archives. - (Mahadev Konar via ddas) - - HADOOP-3561. Prevent the trash from deleting its parent directories. - (cdouglas) - - HADOOP-3575. Fix the clover ant target after package refactoring. - (Nigel Daley via cdouglas) - - HADOOP-3539. Fix the tool path in the bin/hadoop script under - cygwin. (Tsz Wo (Nicholas), Sze via omalley) - - HADOOP-3520. TestDFSUpgradeFromImage triggers a race condition in the - Upgrade Manager. Fixed. (dhruba) - - HADOOP-3586. Provide deprecated, backwards compatibile semantics for the - combiner to be run once and only once on each record. (cdouglas) - - HADOOP-3533. Add deprecated methods to provide API compatibility - between 0.18 and 0.17. Remove the deprecated methods in trunk. (omalley) - - HADOOP-3580. Fixes a problem to do with specifying a har as an input to - a job. (Mahadev Konar via ddas) - - HADOOP-3333. Don't assign a task to a tasktracker that it failed to - execute earlier (used to happen in the case of lost tasktrackers where - the tasktracker would reinitialize and bind to a different port). - (Jothi Padmanabhan and Arun Murthy via ddas) - - HADOOP-3534. Log IOExceptions that happen in closing the name - system when the NameNode shuts down. (Tsz Wo (Nicholas) Sze via omalley) - - HADOOP-3546. TaskTracker re-initialization gets stuck in cleaning up. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3576. Fix NullPointerException when renaming a directory - to its subdirectory. (Tse Wo (Nicholas), SZE via hairong) - - HADOOP-3320. Fix NullPointerException in NetworkTopology.getDistance(). - (hairong) - - HADOOP-3569. KFS input stream read() now correctly reads 1 byte - instead of 4. (Sriram Rao via omalley) - - HADOOP-3599. Fix JobConf::setCombineOnceOnly to modify the instance rather - than a parameter. (Owen O'Malley via cdouglas) - - HADOOP-3590. Null pointer exception in JobTracker when the task tracker is - not yet resolved. (Amar Ramesh Kamat via ddas) - - HADOOP-3603. Fix MapOutputCollector to spill when io.sort.spill.percent is - 1.0 and to detect spills when emitted records write no data. (cdouglas) - - HADOOP-3615. Set DatanodeProtocol.versionID to the correct value. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3559. Fix the libhdfs test script and config to work with the - current semantics. (lohit vijayarenu via cdouglas) - - HADOOP-3480. Need to update Eclipse template to reflect current trunk. - (Brice Arnould via tomwhite) - - HADOOP-3588. Fixed usability issues with archives. (mahadev) - - HADOOP-3635. Uncaught exception in DataBlockScanner. - (Tsz Wo (Nicholas), SZE via hairong) - - HADOOP-3639. Exception when closing DFSClient while multiple files are - open. (Benjamin Gufler via hairong) - - HADOOP-3572. SetQuotas usage interface has some minor bugs. (hairong) - - HADOOP-3649. Fix bug in removing blocks from the corrupted block map. - (Lohit Vijayarenu via shv) - - HADOOP-3604. Work around a JVM synchronization problem observed while - retrieving the address of direct buffers from compression code by obtaining - a lock during this call. (Arun C Murthy via cdouglas) - - HADOOP-3683. Fix dfs metrics to count file listings rather than files - listed. (lohit vijayarenu via cdouglas) - - HADOOP-3597. Fix SortValidator to use filesystems other than the default as - input. Validation job still runs on default fs. - (Jothi Padmanabhan via cdouglas) - - HADOOP-3693. Fix archives, distcp and native library documentation to - conform to style guidelines. (Amareshwari Sriramadasu via cdouglas) - - HADOOP-3653. Fix test-patch target to properly account for Eclipse - classpath jars. (Brice Arnould via nigel) - - HADOOP-3692. Fix documentation for Cluster setup and Quick start guides. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3691. Fix streaming and tutorial docs. (Jothi Padmanabhan via ddas) - - HADOOP-3630. Fix NullPointerException in CompositeRecordReader from empty - sources (cdouglas) - - HADOOP-3706. Fix a ClassLoader issue in the mapred.join Parser that - prevents it from loading user-specified InputFormats. - (Jingkei Ly via cdouglas) - - HADOOP-3718. Fix KFSOutputStream::write(int) to output a byte instead of - an int, per the OutputStream contract. (Sriram Rao via cdouglas) - - HADOOP-3647. Add debug logs to help track down a very occassional, - hard-to-reproduce, bug in shuffle/merge on the reducer. (acmurthy) - - HADOOP-3716. Prevent listStatus in KosmosFileSystem from returning - null for valid, empty directories. (Sriram Rao via cdouglas) - - HADOOP-3752. Fix audit logging to record rename events. (cdouglas) - - HADOOP-3737. Fix CompressedWritable to call Deflater::end to release - compressor memory. (Grant Glouser via cdouglas) - - HADOOP-3670. Fixes JobTracker to clear out split bytes when no longer - required. (Amareshwari Sriramadasu via ddas) - - HADOOP-3755. Update gridmix to work with HOD 0.4 (Runping Qi via cdouglas) - - HADOOP-3743. Fix -libjars, -files, -archives options to work even if - user code does not implement tools. (Amareshwari Sriramadasu via mahadev) - - HADOOP-3774. Fix typos in shell output. (Tsz Wo (Nicholas), SZE via - cdouglas) - - HADOOP-3762. Fixed FileSystem cache to work with the default port. (cutting - via omalley) - - HADOOP-3798. Fix tests compilation. (Mukund Madhugiri via omalley) - - HADOOP-3794. Return modification time instead of zero for KosmosFileSystem. - (Sriram Rao via cdouglas) - - HADOOP-3806. Remove debug statement to stdout from QuickSort. (cdouglas) - - HADOOP-3776. Fix NPE at NameNode when datanode reports a block after it is - deleted at NameNode. (rangadi) - - HADOOP-3537. Disallow adding a datanode to a network topology when its - network location is not resolved. (hairong) - - HADOOP-3571. Fix bug in block removal used in lease recovery. (shv) - - HADOOP-3645. MetricsTimeVaryingRate returns wrong value for - metric_avg_time. (Lohit Vijayarenu via hairong) - - HADOOP-3521. Reverted the missing cast to float for sending Counters' values - to Hadoop metrics which was removed by HADOOP-544. (acmurthy) - - HADOOP-3820. Fixes two problems in the gridmix-env - a syntax error, and a - wrong definition of USE_REAL_DATASET by default. (Arun Murthy via ddas) - - HADOOP-3724. Fixes two problems related to storing and recovering lease - in the fsimage. (dhruba) - - HADOOP-3827. Fixed compression of empty map-outputs. (acmurthy) - - HADOOP-3865. Remove reference to FSNamesystem from metrics preventing - garbage collection. (Lohit Vijayarenu via cdouglas) - - HADOOP-3884. Fix so that Eclipse plugin builds against recent - Eclipse releases. (cutting) - - HADOOP-3837. Streaming jobs report progress status. (dhruba) - - HADOOP-3897. Fix a NPE in secondary namenode. (Lohit Vijayarenu via - cdouglas) - - HADOOP-3901. Fix bin/hadoop to correctly set classpath under cygwin. - (Tsz Wo (Nicholas) Sze via omalley) - - HADOOP-3947. Fix a problem in tasktracker reinitialization. - (Amareshwari Sriramadasu via ddas) - -Release 0.17.3 - Unreleased - - IMPROVEMENTS - - HADOOP-4164. Chinese translation of the documentation. (Xuebing Yan via - omalley) - - BUG FIXES - - HADOOP-4277. Checksum verification was mistakenly disabled for - LocalFileSystem. (Raghu Angadi) - - HADOOP-4271. Checksum input stream can sometimes return invalid - data to the user. (Ning Li via rangadi) - - HADOOP-4318. DistCp should use absolute paths for cleanup. (szetszwo) - - HADOOP-4326. ChecksumFileSystem does not override create(...) correctly. - (szetszwo) - -Release 0.17.2 - 2008-08-11 - - BUG FIXES - - HADOOP-3678. Avoid spurious exceptions logged at DataNode when clients - read from DFS. (rangadi) - - HADOOP-3707. NameNode keeps a count of number of blocks scheduled - to be written to a datanode and uses it to avoid allocating more - blocks than a datanode can hold. (rangadi) - - HADOOP-3760. Fix a bug with HDFS file close() mistakenly introduced - by HADOOP-3681. (Lohit Vijayarenu via rangadi) - - HADOOP-3681. DFSClient can get into an infinite loop while closing - a file if there are some errors. (Lohit Vijayarenu via rangadi) - - HADOOP-3002. Hold off block removal while in safe mode. (shv) - - HADOOP-3685. Unbalanced replication target. (hairong) - - HADOOP-3758. Shutdown datanode on version mismatch instead of retrying - continuously, preventing excessive logging at the namenode. - (lohit vijayarenu via cdouglas) - - HADOOP-3633. Correct exception handling in DataXceiveServer, and throttle - the number of xceiver threads in a data-node. (shv) - - HADOOP-3370. Ensure that the TaskTracker.runningJobs data-structure is - correctly cleaned-up on task completion. (Zheng Shao via acmurthy) - - HADOOP-3813. Fix task-output clean-up on HDFS to use the recursive - FileSystem.delete rather than the FileUtil.fullyDelete. (Amareshwari - Sri Ramadasu via acmurthy) - - HADOOP-3859. Allow the maximum number of xceivers in the data node to - be configurable. (Johan Oskarsson via omalley) - - HADOOP-3931. Fix corner case in the map-side sort that causes some values - to be counted as too large and cause pre-mature spills to disk. Some values - will also bypass the combiner incorrectly. (cdouglas via omalley) - -Release 0.17.1 - 2008-06-23 - - INCOMPATIBLE CHANGES - - HADOOP-3565. Fix the Java serialization, which is not enabled by - default, to clear the state of the serializer between objects. - (tomwhite via omalley) - - IMPROVEMENTS - - HADOOP-3522. Improve documentation on reduce pointing out that - input keys and values will be reused. (omalley) - - HADOOP-3487. Balancer uses thread pools for managing its threads; - therefore provides better resource management. (hairong) - - BUG FIXES - - HADOOP-2159 Namenode stuck in safemode. The counter blockSafe should - not be decremented for invalid blocks. (hairong) - - HADOOP-3472 MapFile.Reader getClosest() function returns incorrect results - when before is true (Todd Lipcon via Stack) - - HADOOP-3442. Limit recursion depth on the stack for QuickSort to prevent - StackOverflowErrors. To avoid O(n*n) cases, when partitioning depth exceeds - a multiple of log(n), change to HeapSort. (cdouglas) - - HADOOP-3477. Fix build to not package contrib/*/bin twice in - distributions. (Adam Heath via cutting) - - HADOOP-3475. Fix MapTask to correctly size the accounting allocation of - io.sort.mb. (cdouglas) - - HADOOP-3550. Fix the serialization data structures in MapTask where the - value lengths are incorrectly calculated. (cdouglas) - - HADOOP-3526. Fix contrib/data_join framework by cloning values retained - in the reduce. (Spyros Blanas via cdouglas) - - HADOOP-1979. Speed up fsck by adding a buffered stream. (Lohit - Vijaya Renu via omalley) - -Release 0.17.0 - 2008-05-18 - - INCOMPATIBLE CHANGES - - HADOOP-2786. Move hbase out of hadoop core - - HADOOP-2345. New HDFS transactions to support appending - to files. Disk layout version changed from -11 to -12. (dhruba) - - HADOOP-2192. Error messages from "dfs mv" command improved. - (Mahadev Konar via dhruba) - - HADOOP-1902. "dfs du" command without any arguments operates on the - current working directory. (Mahadev Konar via dhruba) - - HADOOP-2873. Fixed bad disk format introduced by HADOOP-2345. - Disk layout version changed from -12 to -13. See changelist 630992 - (dhruba) - - HADOOP-1985. This addresses rack-awareness for Map tasks and for - HDFS in a uniform way. (ddas) - - HADOOP-1986. Add support for a general serialization mechanism for - Map Reduce. (tomwhite) - - HADOOP-771. FileSystem.delete() takes an explicit parameter that - specifies whether a recursive delete is intended. - (Mahadev Konar via dhruba) - - HADOOP-2470. Remove getContentLength(String), open(String, long, long) - and isDir(String) from ClientProtocol. ClientProtocol version changed - from 26 to 27. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-2822. Remove deprecated code for classes InputFormatBase and - PhasedFileSystem. (Amareshwari Sriramadasu via enis) - - HADOOP-2116. Changes the layout of the task execution directory. - (Amareshwari Sriramadasu via ddas) - - HADOOP-2828. The following deprecated methods in Configuration.java - have been removed - getObject(String name) - setObject(String name, Object value) - get(String name, Object defaultValue) - set(String name, Object value) - Iterator entries() - (Amareshwari Sriramadasu via ddas) - - HADOOP-2824. Removes one deprecated constructor from MiniMRCluster. - (Amareshwari Sriramadasu via ddas) - - HADOOP-2823. Removes deprecated methods getColumn(), getLine() from - org.apache.hadoop.record.compiler.generated.SimpleCharStream. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3060. Removes one unused constructor argument from MiniMRCluster. - (Amareshwari Sriramadasu via ddas) - - HADOOP-2854. Remove deprecated o.a.h.ipc.Server::getUserInfo(). - (lohit vijayarenu via cdouglas) - - HADOOP-2563. Remove deprecated FileSystem::listPaths. - (lohit vijayarenu via cdouglas) - - HADOOP-2818. Remove deprecated methods in Counters. - (Amareshwari Sriramadasu via tomwhite) - - HADOOP-2831. Remove deprecated o.a.h.dfs.INode::getAbsoluteName() - (lohit vijayarenu via cdouglas) - - HADOOP-2839. Remove deprecated FileSystem::globPaths. - (lohit vijayarenu via cdouglas) - - HADOOP-2634. Deprecate ClientProtocol::exists. - (lohit vijayarenu via cdouglas) - - HADOOP-2410. Make EC2 cluster nodes more independent of each other. - Multiple concurrent EC2 clusters are now supported, and nodes may be - added to a cluster on the fly with new nodes starting in the same EC2 - availability zone as the cluster. Ganglia monitoring and large - instance sizes have also been added. (Chris K Wensel via tomwhite) - - HADOOP-2826. Deprecated FileSplit.getFile(), LineRecordReader.readLine(). - (Amareshwari Sriramadasu via ddas) - - HADOOP-3239. getFileInfo() returns null for non-existing files instead - of throwing FileNotFoundException. (Lohit Vijayarenu via shv) - - HADOOP-3266. Removed HOD changes from CHANGES.txt, as they are now inside - src/contrib/hod (Hemanth Yamijala via ddas) - - HADOOP-3280. Separate the configuration of the virtual memory size - (mapred.child.ulimit) from the jvm heap size, so that 64 bit - streaming applications are supported even when running with 32 bit - jvms. (acmurthy via omalley) - - NEW FEATURES - - HADOOP-1398. Add HBase in-memory block cache. (tomwhite) - - HADOOP-2178. Job History on DFS. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2063. A new parameter to dfs -get command to fetch a file - even if it is corrupted. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2219. A new command "df -count" that counts the number of - files and directories. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2906. Add an OutputFormat capable of using keys, values, and - config params to map records to different output files. - (Runping Qi via cdouglas) - - HADOOP-2346. Utilities to support timeout while writing to sockets. - DFSClient and DataNode sockets have 10min write timeout. (rangadi) - - HADOOP-2951. Add a contrib module that provides a utility to - build or update Lucene indexes using Map/Reduce. (Ning Li via cutting) - - HADOOP-1622. Allow multiple jar files for map reduce. - (Mahadev Konar via dhruba) - - HADOOP-2055. Allows users to set PathFilter on the FileInputFormat. - (Alejandro Abdelnur via ddas) - - HADOOP-2551. More environment variables like HADOOP_NAMENODE_OPTS - for better control of HADOOP_OPTS for each component. (rangadi) - - HADOOP-3001. Add job counters that measure the number of bytes - read and written to HDFS, S3, KFS, and local file systems. (omalley) - - HADOOP-3048. A new Interface and a default implementation to convert - and restore serializations of objects to/from strings. (enis) - - IMPROVEMENTS - - HADOOP-2655. Copy on write for data and metadata files in the - presence of snapshots. Needed for supporting appends to HDFS - files. (dhruba) - - HADOOP-1967. When a Path specifies the same scheme as the default - FileSystem but no authority, the default FileSystem's authority is - used. Also add warnings for old-format FileSystem names, accessor - methods for fs.default.name, and check for null authority in HDFS. - (cutting) - - HADOOP-2895. Let the profiling string be configurable. - (Martin Traverso via cdouglas) - - HADOOP-910. Enables Reduces to do merges for the on-disk map output files - in parallel with their copying. (Amar Kamat via ddas) - - HADOOP-730. Use rename rather than copy for local renames. (cdouglas) - - HADOOP-2810. Updated the Hadoop Core logo. (nigel) - - HADOOP-2057. Streaming should optionally treat a non-zero exit status - of a child process as a failed task. (Rick Cox via tomwhite) - - HADOOP-2765. Enables specifying ulimits for streaming/pipes tasks (ddas) - - HADOOP-2888. Make gridmix scripts more readily configurable and amenable - to automated execution. (Mukund Madhugiri via cdouglas) - - HADOOP-2908. A document that describes the DFS Shell command. - (Mahadev Konar via dhruba) - - HADOOP-2981. Update README.txt to reflect the upcoming use of - cryptography. (omalley) - - HADOOP-2804. Add support to publish CHANGES.txt as HTML when running - the Ant 'docs' target. (nigel) - - HADOOP-2559. Change DFS block placement to allocate the first replica - locally, the second off-rack, and the third intra-rack from the - second. (lohit vijayarenu via cdouglas) - - HADOOP-2939. Make the automated patch testing process an executable - Ant target, test-patch. (nigel) - - HADOOP-2239. Add HsftpFileSystem to permit transferring files over ssl. - (cdouglas) - - HADOOP-2886. Track individual RPC metrics. - (girish vaitheeswaran via dhruba) - - HADOOP-2373. Improvement in safe-mode reporting. (shv) - - HADOOP-3091. Modify FsShell command -put to accept multiple sources. - (Lohit Vijaya Renu via cdouglas) - - HADOOP-3092. Show counter values from job -status command. - (Tom White via ddas) - - HADOOP-1228. Ant task to generate Eclipse project files. (tomwhite) - - HADOOP-3093. Adds Configuration.getStrings(name, default-value) and - the corresponding setStrings. (Amareshwari Sriramadasu via ddas) - - HADOOP-3106. Adds documentation in forrest for debugging. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3099. Add an option to distcp to preserve user, group, and - permission information. (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-2841. Unwrap AccessControlException and FileNotFoundException - from RemoteException for DFSClient. (shv) - - HADOOP-3152. Make index interval configuable when using - MapFileOutputFormat for map-reduce job. (Rong-En Fan via cutting) - - HADOOP-3143. Decrease number of slaves from 4 to 3 in TestMiniMRDFSSort, - as Hudson generates false negatives under the current load. - (Nigel Daley via cdouglas) - - HADOOP-3174. Illustrative example for MultipleFileInputFormat. (Enis - Soztutar via acmurthy) - - HADOOP-2993. Clarify the usage of JAVA_HOME in the Quick Start guide. - (acmurthy via nigel) - - HADOOP-3124. Make DataNode socket write timeout configurable. (rangadi) - - OPTIMIZATIONS - - HADOOP-2790. Fixed inefficient method hasSpeculativeTask by removing - repetitive calls to get the current time and late checking to see if - we want speculation on at all. (omalley) - - HADOOP-2758. Reduce buffer copies in DataNode when data is read from - HDFS, without negatively affecting read throughput. (rangadi) - - HADOOP-2399. Input key and value to combiner and reducer is reused. - (Owen O'Malley via ddas). - - HADOOP-2423. Code optimization in FSNamesystem.mkdirs. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2606. ReplicationMonitor selects data-nodes to replicate directly - from needed replication blocks instead of looking up for the blocks for - each live data-node. (shv) - - HADOOP-2148. Eliminate redundant data-node blockMap lookups. (shv) - - HADOOP-2027. Return the number of bytes in each block in a file - via a single rpc to the namenode to speed up job planning. - (Lohit Vijaya Renu via omalley) - - HADOOP-2902. Replace uses of "fs.default.name" with calls to the - accessor methods added in HADOOP-1967. (cutting) - - HADOOP-2119. Optimize scheduling of jobs with large numbers of - tasks by replacing static arrays with lists of runnable tasks. - (Amar Kamat via omalley) - - HADOOP-2919. Reduce the number of memory copies done during the - map output sorting. Also adds two config variables: - io.sort.spill.percent - the percentages of io.sort.mb that should - cause a spill (default 80%) - io.sort.record.percent - the percent of io.sort.mb that should - hold key/value indexes (default 5%) - (cdouglas via omalley) - - HADOOP-3140. Doesn't add a task in the commit queue if the task hadn't - generated any output. (Amar Kamat via ddas) - - HADOOP-3168. Reduce the amount of logging in streaming to an - exponentially increasing number of records (up to 10,000 - records/log). (Zheng Shao via omalley) - - BUG FIXES - - HADOOP-2195. '-mkdir' behaviour is now closer to Linux shell in case of - errors. (Mahadev Konar via rangadi) - - HADOOP-2190. bring behaviour '-ls' and '-du' closer to Linux shell - commands in case of errors. (Mahadev Konar via rangadi) - - HADOOP-2193. 'fs -rm' and 'fs -rmr' show error message when the target - file does not exist. (Mahadev Konar via rangadi) - - HADOOP-2738 Text is not subclassable because set(Text) and compareTo(Object) - access the other instance's private members directly. (jimk) - - HADOOP-2779. Remove the references to HBase in the build.xml. (omalley) - - HADOOP-2194. dfs cat on a non-existent file throws FileNotFoundException. - (Mahadev Konar via dhruba) - - HADOOP-2767. Fix for NetworkTopology erroneously skipping the last leaf - node on a rack. (Hairong Kuang and Mark Butler via dhruba) - - HADOOP-1593. FsShell works with paths in non-default FileSystem. - (Mahadev Konar via dhruba) - - HADOOP-2191. du and dus command on non-existent directory gives - appropriate error message. (Mahadev Konar via dhruba) - - HADOOP-2832. Remove tabs from code of DFSClient for better - indentation. (dhruba) - - HADOOP-2844. distcp closes file handles for sequence files. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2727. Fix links in Web UI of the hadoop daemons and some docs - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2871. Fixes a problem to do with file: URI in the JobHistory init. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2800. Deprecate SetFile.Writer constructor not the whole class. - (Johan Oskarsson via tomwhite) - - HADOOP-2891. DFSClient.close() closes all open files. (dhruba) - - HADOOP-2845. Fix dfsadmin disk utilization report on Solaris. - (Martin Traverso via tomwhite) - - HADOOP-2912. MiniDFSCluster restart should wait for namenode to exit - safemode. This was causing TestFsck to fail. (Mahadev Konar via dhruba) - - HADOOP-2820. The following classes in streaming are removed : - StreamLineRecordReader StreamOutputFormat StreamSequenceRecordReader. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2819. The following methods in JobConf are removed: - getInputKeyClass() setInputKeyClass getInputValueClass() - setInputValueClass(Class theClass) setSpeculativeExecution - getSpeculativeExecution() (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2817. Removes deprecated mapred.tasktracker.tasks.maximum and - ClusterStatus.getMaxTasks(). (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2821. Removes deprecated ShellUtil and ToolBase classes from - the util package. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2934. The namenode was encountreing a NPE while loading - leases from the fsimage. Fixed. (dhruba) - - HADOOP-2938. Some fs commands did not glob paths. - (Tsz Wo (Nicholas), SZE via rangadi) - - HADOOP-2943. Compression of intermediate map output causes failures - in the merge. (cdouglas) - - HADOOP-2870. DataNode and NameNode closes all connections while - shutting down. (Hairong Kuang via dhruba) - - HADOOP-2973. Fix TestLocalDFS for Windows platform. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2971. select multiple times if it returns early in - SocketIOWithTimeout. (rangadi) - - HADOOP-2955. Fix TestCrcCorruption test failures caused by HADOOP-2758 - (rangadi) - - HADOOP-2657. A flush call on the DFSOutputStream flushes the last - partial CRC chunk too. (dhruba) - - HADOOP-2974. IPC unit tests used "0.0.0.0" to connect to server, which - is not always supported. (rangadi) - - HADOOP-2996. Fixes uses of StringBuffer in StreamUtils class. - (Dave Brosius via ddas) - - HADOOP-2995. Fixes StreamBaseRecordReader's getProgress to return a - floating point number. (Dave Brosius via ddas) - - HADOOP-2972. Fix for a NPE in FSDataset.invalidate. - (Mahadev Konar via dhruba) - - HADOOP-2994. Code cleanup for DFSClient: remove redundant - conversions from string to string. (Dave Brosius via dhruba) - - HADOOP-3009. TestFileCreation sometimes fails because restarting - minidfscluster sometimes creates datanodes with ports that are - different from their original instance. (dhruba) - - HADOOP-2992. Distributed Upgrade framework works correctly with - more than one upgrade object. (Konstantin Shvachko via dhruba) - - HADOOP-2679. Fix a typo in libhdfs. (Jason via dhruba) - - HADOOP-2976. When a lease expires, the Namenode ensures that - blocks of the file are adequately replicated. (dhruba) - - HADOOP-2901. Fixes the creation of info servers in the JobClient - and JobTracker. Removes the creation from JobClient and removes - additional info server from the JobTracker. Also adds the command - line utility to view the history files (HADOOP-2896), and fixes - bugs in JSPs to do with analysis - HADOOP-2742, HADOOP-2792. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2890. If different datanodes report the same block but - with different sizes to the namenode, the namenode picks the - replica(s) with the largest size as the only valid replica(s). (dhruba) - - HADOOP-2825. Deprecated MapOutputLocation.getFile() is removed. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2806. Fixes a streaming document. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3008. SocketIOWithTimeout throws InterruptedIOException if the - thread is interrupted while it is waiting. (rangadi) - - HADOOP-3006. Fix wrong packet size reported by DataNode when a block - is being replicated. (rangadi) - - HADOOP-3029. Datanode prints log message "firstbadlink" only if - it detects a bad connection to another datanode in the pipeline. (dhruba) - - HADOOP-3030. Release reserved space for file in InMemoryFileSystem if - checksum reservation fails. (Devaraj Das via cdouglas) - - HADOOP-3036. Fix findbugs warnings in UpgradeUtilities. (Konstantin - Shvachko via cdouglas) - - HADOOP-3025. ChecksumFileSystem supports the delete method with - the recursive flag. (Mahadev Konar via dhruba) - - HADOOP-3012. dfs -mv file to user home directory throws exception if - the user home directory does not exist. (Mahadev Konar via dhruba) - - HADOOP-3066. Should not require superuser privilege to query if hdfs is in - safe mode (jimk) - - HADOOP-3040. If the input line starts with the separator char, the key - is set as empty. (Amareshwari Sriramadasu via ddas) - - HADOOP-3080. Removes flush calls from JobHistory. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3086. Adds the testcase missed during commit of hadoop-3040. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3046. Fix the raw comparators for Text and BytesWritables - to use the provided length rather than recompute it. (omalley) - - HADOOP-3094. Fix BytesWritable.toString to avoid extending the sign bit - (Owen O'Malley via cdouglas) - - HADOOP-3067. DFSInputStream's position read does not close the sockets. - (rangadi) - - HADOOP-3073. close() on SocketInputStream or SocketOutputStream should - close the underlying channel. (rangadi) - - HADOOP-3087. Fixes a problem to do with refreshing of loadHistory.jsp. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3065. Better logging message if the rack location of a datanode - cannot be determined. (Devaraj Das via dhruba) - - HADOOP-3064. Commas in a file path should not be treated as delimiters. - (Hairong Kuang via shv) - - HADOOP-2997. Adds test for non-writable serialier. Also fixes a problem - introduced by HADOOP-2399. (Tom White via ddas) - - HADOOP-3114. Fix TestDFSShell on Windows. (Lohit Vijaya Renu via cdouglas) - - HADOOP-3118. Fix Namenode NPE while loading fsimage after a cluster - upgrade from older disk format. (dhruba) - - HADOOP-3161. Fix FIleUtil.HardLink.getLinkCount on Mac OS. (nigel - via omalley) - - HADOOP-2927. Fix TestDU to acurately calculate the expected file size. - (shv via nigel) - - HADOOP-3123. Fix the native library build scripts to work on Solaris. - (tomwhite via omalley) - - HADOOP-3089. Streaming should accept stderr from task before - first key arrives. (Rick Cox via tomwhite) - - HADOOP-3146. A DFSOutputStream.flush method is renamed as - DFSOutputStream.fsync. (dhruba) - - HADOOP-3165. -put/-copyFromLocal did not treat input file "-" as stdin. - (Lohit Vijayarenu via rangadi) - - HADOOP-3041. Deprecate JobConf.setOutputPath and JobConf.getOutputPath. - Deprecate OutputFormatBase. Add FileOutputFormat. Existing output formats - extending OutputFormatBase, now extend FileOutputFormat. Add the following - APIs in FileOutputFormat: setOutputPath, getOutputPath, getWorkOutputPath. - (Amareshwari Sriramadasu via nigel) - - HADOOP-3083. The fsimage does not store leases. This would have to be - reworked in the next release to support appends. (dhruba) - - HADOOP-3166. Fix an ArrayIndexOutOfBoundsException in the spill thread - and make exception handling more promiscuous to catch this condition. - (cdouglas) - - HADOOP-3050. DataNode sends one and only one block report after - it registers with the namenode. (Hairong Kuang) - - HADOOP-3044. NNBench sets the right configuration for the mapper. - (Hairong Kuang) - - HADOOP-3178. Fix GridMix scripts for small and medium jobs - to handle input paths differently. (Mukund Madhugiri via nigel) - - HADOOP-1911. Fix an infinite loop in DFSClient when all replicas of a - block are bad (cdouglas) - - HADOOP-3157. Fix path handling in DistributedCache and TestMiniMRLocalFS. - (Doug Cutting via rangadi) - - HADOOP-3018. Fix the eclipse plug-in contrib wrt removed deprecated - methods (taton) - - HADOOP-3183. Fix TestJobShell to use 'ls' instead of java.io.File::exists - since cygwin symlinks are unsupported. - (Mahadev konar via cdouglas) - - HADOOP-3175. Fix FsShell.CommandFormat to handle "-" in arguments. - (Edward J. Yoon via rangadi) - - HADOOP-3220. Safemode message corrected. (shv) - - HADOOP-3208. Fix WritableDeserializer to set the Configuration on - deserialized Writables. (Enis Soztutar via cdouglas) - - HADOOP-3224. 'dfs -du /dir' does not return correct size. - (Lohit Vjayarenu via rangadi) - - HADOOP-3223. Fix typo in help message for -chmod. (rangadi) - - HADOOP-1373. checkPath() should ignore case when it compares authoriy. - (Edward J. Yoon via rangadi) - - HADOOP-3204. Fixes a problem to do with ReduceTask's LocalFSMerger not - catching Throwable. (Amar Ramesh Kamat via ddas) - - HADOOP-3229. Report progress when collecting records from the mapper and - the combiner. (Doug Cutting via cdouglas) - - HADOOP-3225. Unwrapping methods of RemoteException should initialize - detailedMassage field. (Mahadev Konar, shv, cdouglas) - - HADOOP-3247. Fix gridmix scripts to use the correct globbing syntax and - change maxentToSameCluster to run the correct number of jobs. - (Runping Qi via cdouglas) - - HADOOP-3242. Fix the RecordReader of SequenceFileAsBinaryInputFormat to - correctly read from the start of the split and not the beginning of the - file. (cdouglas via acmurthy) - - HADOOP-3256. Encodes the job name used in the filename for history files. - (Arun Murthy via ddas) - - HADOOP-3162. Ensure that comma-separated input paths are treated correctly - as multiple input paths. (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-3263. Ensure that the job-history log file always follows the - pattern of hostname_timestamp_jobid_username_jobname even if username - and/or jobname are not specfied. This helps to avoid wrong assumptions - made about the job-history log filename in jobhistory.jsp. (acmurthy) - - HADOOP-3251. Fixes getFilesystemName in JobTracker and LocalJobRunner to - use FileSystem.getUri instead of FileSystem.getName. (Arun Murthy via ddas) - - HADOOP-3237. Fixes TestDFSShell.testErrOutPut on Windows platform. - (Mahadev Konar via ddas) - - HADOOP-3279. TaskTracker checks for SUCCEEDED task status in addition to - COMMIT_PENDING status when it fails maps due to lost map. - (Devaraj Das) - - HADOOP-3286. Prevent collisions in gridmix output dirs by increasing the - granularity of the timestamp. (Runping Qi via cdouglas) - - HADOOP-3285. Fix input split locality when the splits align to - fs blocks. (omalley) - - HADOOP-3372. Fix heap management in streaming tests. (Arun Murthy via - cdouglas) - - HADOOP-3031. Fix javac warnings in test classes. (cdouglas) - - HADOOP-3382. Fix memory leak when files are not cleanly closed (rangadi) - - HADOOP-3322. Fix to push MetricsRecord for rpc metrics. (Eric Yang via - mukund) - -Release 0.16.4 - 2008-05-05 - - BUG FIXES - - HADOOP-3138. DFS mkdirs() should not throw an exception if the directory - already exists. (rangadi via mukund) - - HADOOP-3294. Fix distcp to check the destination length and retry the copy - if it doesn't match the src length. (Tsz Wo (Nicholas), SZE via mukund) - - HADOOP-3186. Fix incorrect permission checkding for mv and renameTo - in HDFS. (Tsz Wo (Nicholas), SZE via mukund) - -Release 0.16.3 - 2008-04-16 - - BUG FIXES - - HADOOP-3010. Fix ConcurrentModificationException in ipc.Server.Responder. - (rangadi) - - HADOOP-3154. Catch all Throwables from the SpillThread in MapTask, rather - than IOExceptions only. (ddas via cdouglas) - - HADOOP-3159. Avoid file system cache being overwritten whenever - configuration is modified. (Tsz Wo (Nicholas), SZE via hairong) - - HADOOP-3139. Remove the consistency check for the FileSystem cache in - closeAll() that causes spurious warnings and a deadlock. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3195. Fix TestFileSystem to be deterministic. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-3069. Primary name-node should not truncate image when transferring - it from the secondary. (shv) - - HADOOP-3182. Change permissions of the job-submission directory to 777 - from 733 to ensure sharing of HOD clusters works correctly. (Tsz Wo - (Nicholas), Sze and Amareshwari Sri Ramadasu via acmurthy) - -Release 0.16.2 - 2008-04-02 - - BUG FIXES - - HADOOP-3011. Prohibit distcp from overwriting directories on the - destination filesystem with files. (cdouglas) - - HADOOP-3033. The BlockReceiver thread in the datanode writes data to - the block file, changes file position (if needed) and flushes all by - itself. The PacketResponder thread does not flush block file. (dhruba) - - HADOOP-2978. Fixes the JobHistory log format for counters. - (Runping Qi via ddas) - - HADOOP-2985. Fixes LocalJobRunner to tolerate null job output path. - Also makes the _temporary a constant in MRConstants.java. - (Amareshwari Sriramadasu via ddas) - - HADOOP-3003. FileSystem cache key is updated after a - FileSystem object is created. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-3042. Updates the Javadoc in JobConf.getOutputPath to reflect - the actual temporary path. (Amareshwari Sriramadasu via ddas) - - HADOOP-3007. Tolerate mirror failures while DataNode is replicating - blocks as it used to before. (rangadi) - - HADOOP-2944. Fixes a "Run on Hadoop" wizard NPE when creating a - Location from the wizard. (taton) - - HADOOP-3049. Fixes a problem in MultiThreadedMapRunner to do with - catching RuntimeExceptions. (Alejandro Abdelnur via ddas) - - HADOOP-3039. Fixes a problem to do with exceptions in tasks not - killing jobs. (Amareshwari Sriramadasu via ddas) - - HADOOP-3027. Fixes a problem to do with adding a shutdown hook in - FileSystem. (Amareshwari Sriramadasu via ddas) - - HADOOP-3056. Fix distcp when the target is an empty directory by - making sure the directory is created first. (cdouglas and acmurthy - via omalley) - - HADOOP-3070. Protect the trash emptier thread from null pointer - exceptions. (Koji Noguchi via omalley) - - HADOOP-3084. Fix HftpFileSystem to work for zero-lenghth files. - (cdouglas) - - HADOOP-3107. Fix NPE when fsck invokes getListings. (dhruba) - - HADOOP-3104. Limit MultithreadedMapRunner to have a fixed length queue - between the RecordReader and the map threads. (Alejandro Abdelnur via - omalley) - - HADOOP-2833. Do not use "Dr. Who" as the default user in JobClient. - A valid user name is required. (Tsz Wo (Nicholas), SZE via rangadi) - - HADOOP-3128. Throw RemoteException in setPermissions and setOwner of - DistributedFileSystem. (shv via nigel) - -Release 0.16.1 - 2008-03-13 - - INCOMPATIBLE CHANGES - - HADOOP-2869. Deprecate SequenceFile.setCompressionType in favor of - SequenceFile.createWriter, SequenceFileOutputFormat.setCompressionType, - and JobConf.setMapOutputCompressionType. (Arun C Murthy via cdouglas) - Configuration changes to hadoop-default.xml: - deprecated io.seqfile.compression.type - - IMPROVEMENTS - - HADOOP-2371. User guide for file permissions in HDFS. - (Robert Chansler via rangadi) - - HADOOP-3098. Allow more characters in user and group names while - using -chown and -chgrp commands. (rangadi) - - BUG FIXES - - HADOOP-2789. Race condition in IPC Server Responder that could close - connections early. (Raghu Angadi) - - HADOOP-2785. minor. Fix a typo in Datanode block verification - (Raghu Angadi) - - HADOOP-2788. minor. Fix help message for chgrp shell command (Raghu Angadi). - - HADOOP-1188. fstime file is updated when a storage directory containing - namespace image becomes inaccessible. (shv) - - HADOOP-2787. An application can set a configuration variable named - dfs.umask to set the umask that is used by DFS. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2780. The default socket buffer size for DataNodes is 128K. - (dhruba) - - HADOOP-2716. Superuser privileges for the Balancer. - (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-2754. Filter out .crc files from local file system listing. - (Hairong Kuang via shv) - - HADOOP-2733. Fix compiler warnings in test code. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HADOOP-2725. Modify distcp to avoid leaving partially copied files at - the destination after encountering an error. (Tsz Wo (Nicholas), SZE - via cdouglas) - - HADOOP-2391. Cleanup job output directory before declaring a job as - SUCCESSFUL. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2808. Minor fix to FileUtil::copy to mind the overwrite - formal. (cdouglas) - - HADOOP-2683. Moving UGI out of the RPC Server. - (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-2814. Fix for NPE in datanode in unit test TestDataTransferProtocol. - (Raghu Angadi via dhruba) - - HADOOP-2811. Dump of counters in job history does not add comma between - groups. (runping via omalley) - - HADOOP-2735. Enables setting TMPDIR for tasks. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2843. Fix protections on map-side join classes to enable derivation. - (cdouglas via omalley) - - HADOOP-2840. Fix gridmix scripts to correctly invoke the java sort through - the proper jar. (Mukund Madhugiri via cdouglas) - - HADOOP-2769. TestNNThroughputBnechmark should not use a fixed port for - the namenode http port. (omalley) - - HADOOP-2852. Update gridmix benchmark to avoid an artifically long tail. - (cdouglas) - - HADOOP-2894. Fix a problem to do with tasktrackers failing to connect to - JobTracker upon reinitialization. (Owen O'Malley via ddas). - - HADOOP-2903. Fix exception generated by Metrics while using pushMetric(). - (girish vaitheeswaran via dhruba) - - HADOOP-2904. Fix to RPC metrics to log the correct host name. - (girish vaitheeswaran via dhruba) - - HADOOP-2918. Improve error logging so that dfs writes failure with - "No lease on file" can be diagnosed. (dhruba) - - HADOOP-2923. Add SequenceFileAsBinaryInputFormat, which was - missed in the commit for HADOOP-2603. (cdouglas via omalley) - - HADOOP-2931. IOException thrown by DFSOutputStream had wrong stack - trace in some cases. (Michael Bieniosek via rangadi) - - HADOOP-2883. Write failures and data corruptions on HDFS files. - The write timeout is back to what it was on 0.15 release. Also, the - datnodes flushes the block file buffered output stream before - sending a positive ack for the packet back to the client. (dhruba) - - HADOOP-2756. NPE in DFSClient while closing DFSOutputStreams - under load. (rangadi) - - HADOOP-2958. Fixed FileBench which broke due to HADOOP-2391 which performs - a check for existence of the output directory and a trivial bug in - GenericMRLoadGenerator where min/max word lenghts were identical since - they were looking at the same config variables (Chris Douglas via - acmurthy) - - HADOOP-2915. Fixed FileSystem.CACHE so that a username is included - in the cache key. (Tsz Wo (Nicholas), SZE via nigel) - - HADOOP-2813. TestDU unit test uses its own directory to run its - sequence of tests. (Mahadev Konar via dhruba) - -Release 0.16.0 - 2008-02-07 - - INCOMPATIBLE CHANGES - - HADOOP-1245. Use the mapred.tasktracker.tasks.maximum value - configured on each tasktracker when allocating tasks, instead of - the value configured on the jobtracker. InterTrackerProtocol - version changed from 5 to 6. (Michael Bieniosek via omalley) - - HADOOP-1843. Removed code from Configuration and JobConf deprecated by - HADOOP-785 and a minor fix to Configuration.toString. Specifically the - important change is that mapred-default.xml is no longer supported and - Configuration no longer supports the notion of default/final resources. - (acmurthy) - - HADOOP-1302. Remove deprecated abacus code from the contrib directory. - This also fixes a configuration bug in AggregateWordCount, so that the - job now works. (enis) - - HADOOP-2288. Enhance FileSystem API to support access control. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2184. RPC Support for user permissions and authentication. - (Raghu Angadi via dhruba) - - HADOOP-2185. RPC Server uses any available port if the specified - port is zero. Otherwise it uses the specified port. Also combines - the configuration attributes for the servers' bind address and - port from "x.x.x.x" and "y" to "x.x.x.x:y". - Deprecated configuration variables: - dfs.info.bindAddress - dfs.info.port - dfs.datanode.bindAddress - dfs.datanode.port - dfs.datanode.info.bindAdress - dfs.datanode.info.port - dfs.secondary.info.bindAddress - dfs.secondary.info.port - mapred.job.tracker.info.bindAddress - mapred.job.tracker.info.port - mapred.task.tracker.report.bindAddress - tasktracker.http.bindAddress - tasktracker.http.port - New configuration variables (post HADOOP-2404): - dfs.secondary.http.address - dfs.datanode.address - dfs.datanode.http.address - dfs.http.address - mapred.job.tracker.http.address - mapred.task.tracker.report.address - mapred.task.tracker.http.address - (Konstantin Shvachko via dhruba) - - HADOOP-2401. Only the current leaseholder can abandon a block for - a HDFS file. ClientProtocol version changed from 20 to 21. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2381. Support permission information in FileStatus. Client - Protocol version changed from 21 to 22. (Raghu Angadi via dhruba) - - HADOOP-2110. Block report processing creates fewer transient objects. - Datanode Protocol version changed from 10 to 11. - (Sanjay Radia via dhruba) - - HADOOP-2567. Add FileSystem#getHomeDirectory(), which returns the - user's home directory in a FileSystem as a fully-qualified path. - FileSystem#getWorkingDirectory() is also changed to return a - fully-qualified path, which can break applications that attempt - to, e.g., pass LocalFileSystem#getWorkingDir().toString() directly - to java.io methods that accept file names. (cutting) - - HADOOP-2514. Change trash feature to maintain a per-user trash - directory, named ".Trash" in the user's home directory. The - "fs.trash.root" parameter is no longer used. Full source paths - are also no longer reproduced within the trash. - - HADOOP-2012. Periodic data verification on Datanodes. - (Raghu Angadi via dhruba) - - HADOOP-1707. The DFSClient does not use a local disk file to cache - writes to a HDFS file. Changed Data Transfer Version from 7 to 8. - (dhruba) - - HADOOP-2652. Fix permission issues for HftpFileSystem. This is an - incompatible change since distcp may not be able to copy files - from cluster A (compiled with this patch) to cluster B (compiled - with previous versions). (Tsz Wo (Nicholas), SZE via dhruba) - - NEW FEATURES - - HADOOP-1857. Ability to run a script when a task fails to capture stack - traces. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2299. Defination of a login interface. A simple implementation for - Unix users and groups. (Hairong Kuang via dhruba) - - HADOOP-1652. A utility to balance data among datanodes in a HDFS cluster. - (Hairong Kuang via dhruba) - - HADOOP-2085. A library to support map-side joins of consistently - partitioned and sorted data sets. (Chris Douglas via omalley) - - HADOOP-2336. Shell commands to modify file permissions. (rangadi) - - HADOOP-1298. Implement file permissions for HDFS. - (Tsz Wo (Nicholas) & taton via cutting) - - HADOOP-2447. HDFS can be configured to limit the total number of - objects (inodes and blocks) in the file system. (dhruba) - - HADOOP-2487. Added an option to get statuses for all submitted/run jobs. - This information can be used to develop tools for analysing jobs. - (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-1873. Implement user permissions for Map/Reduce framework. - (Hairong Kuang via shv) - - HADOOP-2532. Add to MapFile a getClosest method that returns the key - that comes just before if the key is not present. (stack via tomwhite) - - HADOOP-1883. Add versioning to Record I/O. (Vivek Ratan via ddas) - - HADOOP-2603. Add SeqeunceFileAsBinaryInputFormat, which reads - sequence files as BytesWritable/BytesWritable regardless of the - key and value types used to write the file. (cdouglas via omalley) - - HADOOP-2367. Add ability to profile a subset of map/reduce tasks and fetch - the result to the local filesystem of the submitting application. Also - includes a general IntegerRanges extension to Configuration for setting - positive, ranged parameters. (Owen O'Malley via cdouglas) - - IMPROVEMENTS - - HADOOP-2045. Change committer list on website to a table, so that - folks can list their organization, timezone, etc. (cutting) - - HADOOP-2058. Facilitate creating new datanodes dynamically in - MiniDFSCluster. (Hairong Kuang via dhruba) - - HADOOP-1855. fsck verifies block placement policies and reports - violations. (Konstantin Shvachko via dhruba) - - HADOOP-1604. An system administrator can finalize namenode upgrades - without running the cluster. (Konstantin Shvachko via dhruba) - - HADOOP-1839. Link-ify the Pending/Running/Complete/Killed grid in - jobdetails.jsp to help quickly narrow down and see categorized TIPs' - details via jobtasks.jsp. (Amar Kamat via acmurthy) - - HADOOP-1210. Log counters in job history. (Owen O'Malley via ddas) - - HADOOP-1912. Datanode has two new commands COPY and REPLACE. These are - needed for supporting data rebalance. (Hairong Kuang via dhruba) - - HADOOP-2086. This patch adds the ability to add dependencies to a job - (run via JobControl) after construction. (Adrian Woodhead via ddas) - - HADOOP-1185. Support changing the logging level of a server without - restarting the server. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2134. Remove developer-centric requirements from overview.html and - keep it end-user focussed, specifically sections related to subversion and - building Hadoop. (Jim Kellerman via acmurthy) - - HADOOP-1989. Support simulated DataNodes. This helps creating large virtual - clusters for testing purposes. (Sanjay Radia via dhruba) - - HADOOP-1274. Support different number of mappers and reducers per - TaskTracker to allow administrators to better configure and utilize - heterogenous clusters. - Configuration changes to hadoop-default.xml: - add mapred.tasktracker.map.tasks.maximum (default value of 2) - add mapred.tasktracker.reduce.tasks.maximum (default value of 2) - remove mapred.tasktracker.tasks.maximum (deprecated for 0.16.0) - (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-2104. Adds a description to the ant targets. This makes the - output of "ant -projecthelp" sensible. (Chris Douglas via ddas) - - HADOOP-2127. Added a pipes sort example to benchmark trivial pipes - application versus trivial java application. (omalley via acmurthy) - - HADOOP-2113. A new shell command "dfs -text" to view the contents of - a gziped or SequenceFile. (Chris Douglas via dhruba) - - HADOOP-2207. Add a "package" target for contrib modules that - permits each to determine what files are copied into release - builds. (stack via cutting) - - HADOOP-1984. Makes the backoff for failed fetches exponential. - Earlier, it was a random backoff from an interval. - (Amar Kamat via ddas) - - HADOOP-1327. Include website documentation for streaming. (Rob Weltman - via omalley) - - HADOOP-2000. Rewrite NNBench to measure namenode performance accurately. - It now uses the map-reduce framework for load generation. - (Mukund Madhugiri via dhruba) - - HADOOP-2248. Speeds up the framework w.r.t Counters. Also has API - updates to the Counters part. (Owen O'Malley via ddas) - - HADOOP-2326. The initial block report at Datanode startup time has - a random backoff period. (Sanjay Radia via dhruba) - - HADOOP-2432. HDFS includes the name of the file while throwing - "File does not exist" exception. (Jim Kellerman via dhruba) - - HADOOP-2457. Added a 'forrest.home' property to the 'docs' target in - build.xml. (acmurthy) - - HADOOP-2149. A new benchmark for three name-node operation: file create, - open, and block report, to evaluate the name-node performance - for optimizations or new features. (Konstantin Shvachko via shv) - - HADOOP-2466. Change FileInputFormat.computeSplitSize to a protected - non-static method to allow sub-classes to provide alternate - implementations. (Alejandro Abdelnur via acmurthy) - - HADOOP-2425. Change TextOutputFormat to handle Text specifically for better - performance. Make NullWritable implement Comparable. Make TextOutputFormat - treat NullWritable like null. (omalley) - - HADOOP-1719. Improves the utilization of shuffle copier threads. - (Amar Kamat via ddas) - - HADOOP-2390. Added documentation for user-controls for intermediate - map-outputs & final job-outputs and native-hadoop libraries. (acmurthy) - - HADOOP-1660. Add the cwd of the map/reduce task to the java.library.path - of the child-jvm to support loading of native libraries distributed via - the DistributedCache. (acmurthy) - - HADOOP-2285. Speeds up TextInputFormat. Also includes updates to the - Text API. (Owen O'Malley via cdouglas) - - HADOOP-2233. Adds a generic load generator for modeling MR jobs. (cdouglas) - - HADOOP-2369. Adds a set of scripts for simulating a mix of user map/reduce - workloads. (Runping Qi via cdouglas) - - HADOOP-2547. Removes use of a 'magic number' in build.xml. - (Hrishikesh via nigel) - - HADOOP-2268. Fix org.apache.hadoop.mapred.jobcontrol classes to use the - List/Map interfaces rather than concrete ArrayList/HashMap classes - internally. (Adrian Woodhead via acmurthy) - - HADOOP-2406. Add a benchmark for measuring read/write performance through - the InputFormat interface, particularly with compression. (cdouglas) - - HADOOP-2131. Allow finer-grained control over speculative-execution. Now - users can set it for maps and reduces independently. - Configuration changes to hadoop-default.xml: - deprecated mapred.speculative.execution - add mapred.map.tasks.speculative.execution - add mapred.reduce.tasks.speculative.execution - (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-1965. Interleave sort/spill in teh map-task along with calls to the - Mapper.map method. This is done by splitting the 'io.sort.mb' buffer into - two and using one half for collecting map-outputs and the other half for - sort/spill. (Amar Kamat via acmurthy) - - HADOOP-2464. Unit tests for chmod, chown, and chgrp using DFS. - (Raghu Angadi) - - HADOOP-1876. Persist statuses of completed jobs in HDFS so that the - JobClient can query and get information about decommissioned jobs and also - across JobTracker restarts. - Configuration changes to hadoop-default.xml: - add mapred.job.tracker.persist.jobstatus.active (default value of false) - add mapred.job.tracker.persist.jobstatus.hours (default value of 0) - add mapred.job.tracker.persist.jobstatus.dir (default value of - /jobtracker/jobsInfo) - (Alejandro Abdelnur via acmurthy) - - HADOOP-2077. Added version and build information to STARTUP_MSG for all - hadoop daemons to aid error-reporting, debugging etc. (acmurthy) - - HADOOP-2398. Additional instrumentation for NameNode and RPC server. - Add support for accessing instrumentation statistics via JMX. - (Sanjay radia via dhruba) - - HADOOP-2449. A return of the non-MR version of NNBench. - (Sanjay Radia via shv) - - HADOOP-1989. Remove 'datanodecluster' command from bin/hadoop. - (Sanjay Radia via shv) - - HADOOP-1742. Improve JavaDoc documentation for ClientProtocol, DFSClient, - and FSNamesystem. (Konstantin Shvachko) - - HADOOP-2298. Add Ant target for a binary-only distribution. - (Hrishikesh via nigel) - - HADOOP-2509. Add Ant target for Rat report (Apache license header - reports). (Hrishikesh via nigel) - - HADOOP-2469. WritableUtils.clone should take a Configuration - instead of a JobConf. (stack via omalley) - - HADOOP-2659. Introduce superuser permissions for admin operations. - (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-2596. Added a SequenceFile.createWriter api which allows the user - to specify the blocksize, replication factor and the buffersize to be - used for the underlying HDFS file. (Alejandro Abdelnur via acmurthy) - - HADOOP-2431. Test HDFS File Permissions. (Hairong Kuang via shv) - - HADOOP-2232. Add an option to disable Nagle's algorithm in the IPC stack. - (Clint Morgan via cdouglas) - - HADOOP-2342. Created a micro-benchmark for measuring - local-file versus hdfs reads. (Owen O'Malley via nigel) - - HADOOP-2529. First version of HDFS User Guide. (Raghu Angadi) - - HADOOP-2690. Add jar-test target to build.xml, separating compilation - and packaging of the test classes. (Enis Soztutar via cdouglas) - - OPTIMIZATIONS - - HADOOP-1898. Release the lock protecting the last time of the last stack - dump while the dump is happening. (Amareshwari Sri Ramadasu via omalley) - - HADOOP-1900. Makes the heartbeat and task event queries interval - dependent on the cluster size. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2208. Counter update frequency (from TaskTracker to JobTracker) is - capped at 1 minute. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2284. Reduce the number of progress updates during the sorting in - the map task. (Amar Kamat via ddas) - - BUG FIXES - - HADOOP-2583. Fixes a bug in the Eclipse plug-in UI to edit locations. - Plug-in version is now synchronized with Hadoop version. - - HADOOP-2100. Remove faulty check for existence of $HADOOP_PID_DIR and let - 'mkdir -p' check & create it. (Michael Bieniosek via acmurthy) - - HADOOP-1642. Ensure jobids generated by LocalJobRunner are unique to - avoid collissions and hence job-failures. (Doug Cutting via acmurthy) - - HADOOP-2096. Close open file-descriptors held by streams while localizing - job.xml in the JobTracker and while displaying it on the webui in - jobconf.jsp. (Amar Kamat via acmurthy) - - HADOOP-2098. Log start & completion of empty jobs to JobHistory, which - also ensures that we close the file-descriptor of the job's history log - opened during job-submission. (Amar Kamat via acmurthy) - - HADOOP-2112. Adding back changes to build.xml lost while reverting - HADOOP-1622 i.e. http://svn.apache.org/viewvc?view=rev&revision=588771. - (acmurthy) - - HADOOP-2089. Fixes the command line argument handling to handle multiple - -cacheArchive in Hadoop streaming. (Lohit Vijayarenu via ddas) - - HADOOP-2071. Fix StreamXmlRecordReader to use a BufferedInputStream - wrapped over the DFSInputStream since mark/reset aren't supported by - DFSInputStream anymore. (Lohit Vijayarenu via acmurthy) - - HADOOP-1348. Allow XML comments inside configuration files. - (Rajagopal Natarajan and Enis Soztutar via enis) - - HADOOP-1952. Improve handling of invalid, user-specified classes while - configuring streaming jobs such as combiner, input/output formats etc. - Now invalid options are caught, logged and jobs are failed early. (Lohit - Vijayarenu via acmurthy) - - HADOOP-2151. FileSystem.globPaths validates the list of Paths that - it returns. (Lohit Vijayarenu via dhruba) - - HADOOP-2121. Cleanup DFSOutputStream when the stream encountered errors - when Datanodes became full. (Raghu Angadi via dhruba) - - HADOOP-1130. The FileSystem.closeAll() method closes all existing - DFSClients. (Chris Douglas via dhruba) - - HADOOP-2204. DFSTestUtil.waitReplication was not waiting for all replicas - to get created, thus causing unit test failure. - (Raghu Angadi via dhruba) - - HADOOP-2078. An zero size file may have no blocks associated with it. - (Konstantin Shvachko via dhruba) - - HADOOP-2212. ChecksumFileSystem.getSumBufferSize might throw - java.lang.ArithmeticException. The fix is to initialize bytesPerChecksum - to 0. (Michael Bieniosek via ddas) - - HADOOP-2216. Fix jobtasks.jsp to ensure that it first collects the - taskids which satisfy the filtering criteria and then use that list to - print out only the required task-reports, previously it was oblivious to - the filtering and hence used the wrong index into the array of task-reports. - (Amar Kamat via acmurthy) - - HADOOP-2272. Fix findbugs target to reflect changes made to the location - of the streaming jar file by HADOOP-2207. (Adrian Woodhead via nigel) - - HADOOP-2244. Fixes the MapWritable.readFields to clear the instance - field variable every time readFields is called. (Michael Stack via ddas). - - HADOOP-2245. Fixes LocalJobRunner to include a jobId in the mapId. Also, - adds a testcase for JobControl. (Adrian Woodhead via ddas). - - HADOOP-2275. Fix erroneous detection of corrupted file when namenode - fails to allocate any datanodes for newly allocated block. - (Dhruba Borthakur via dhruba) - - HADOOP-2256. Fix a buf in the namenode that could cause it to encounter - an infinite loop while deleting excess replicas that were created by - block rebalancing. (Hairong Kuang via dhruba) - - HADOOP-2209. SecondaryNamenode process exits if it encounters exceptions - that it cannot handle. (Dhruba Borthakur via dhruba) - - HADOOP-2314. Prevent TestBlockReplacement from occasionally getting - into an infinite loop. (Hairong Kuang via dhruba) - - HADOOP-2300. This fixes a bug where mapred.tasktracker.tasks.maximum - would be ignored even if it was set in hadoop-site.xml. - (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2349. Improve code layout in file system transaction logging code. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2368. Fix unit tests on Windows. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2363. This fix allows running multiple instances of the unit test - in parallel. The bug was introduced in HADOOP-2185 that changed - port-rolling behaviour. (Konstantin Shvachko via dhruba) - - HADOOP-2271. Fix chmod task to be non-parallel. (Adrian Woodhead via - omalley) - - HADOOP-2313. Fail the build if building libhdfs fails. (nigel via omalley) - - HADOOP-2359. Remove warning for interruptted exception when closing down - minidfs. (dhruba via omalley) - - HADOOP-1841. Prevent slow clients from consuming threads in the NameNode. - (dhruba) - - HADOOP-2323. JobTracker.close() should not print stack traces for - normal exit. (jimk via cutting) - - HADOOP-2376. Prevents sort example from overriding the number of maps. - (Owen O'Malley via ddas) - - HADOOP-2434. FSDatasetInterface read interface causes HDFS reads to occur - in 1 byte chunks, causing performance degradation. - (Raghu Angadi via dhruba) - - HADOOP-2459. Fix package target so that src/docs/build files are not - included in the release. (nigel) - - HADOOP-2215. Fix documentation in cluster_setup.html & - mapred_tutorial.html reflect that mapred.tasktracker.tasks.maximum has - been superceeded by mapred.tasktracker.{map|reduce}.tasks.maximum. - (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-2459. Fix package target so that src/docs/build files are not - included in the release. (nigel) - - HADOOP-2352. Remove AC_CHECK_LIB for libz and liblzo to ensure that - libhadoop.so doesn't have a dependency on them. (acmurthy) - - HADOOP-2453. Fix the configuration for wordcount-simple example in Hadoop - Pipes which currently produces an XML parsing error. (Amareshwari Sri - Ramadasu via acmurthy) - - HADOOP-2476. Unit test failure while reading permission bits of local - file system (on Windows) fixed. (Raghu Angadi via dhruba) - - HADOOP-2247. Fine-tune the strategies for killing mappers and reducers - due to failures while fetching map-outputs. Now the map-completion times - and number of currently running reduces are taken into account by the - JobTracker before killing the mappers, while the progress made by the - reducer and the number of fetch-failures vis-a-vis total number of - fetch-attempts are taken into account before teh reducer kills itself. - (Amar Kamat via acmurthy) - - HADOOP-2452. Fix eclipse plug-in build.xml to refers to the right - location where hadoop-*-core.jar is generated. (taton) - - HADOOP-2492. Additional debugging in the rpc server to better - diagnose ConcurrentModificationException. (dhruba) - - HADOOP-2344. Enhance the utility for executing shell commands to read the - stdout/stderr streams while waiting for the command to finish (to free up - the buffers). Also, this patch throws away stderr of the DF utility. - @deprecated - org.apache.hadoop.fs.ShellCommand for org.apache.hadoop.util.Shell - org.apache.hadoop.util.ShellUtil for - org.apache.hadoop.util.Shell.ShellCommandExecutor - (Amar Kamat via acmurthy) - - HADOOP-2511. Fix a javadoc warning in org.apache.hadoop.util.Shell - introduced by HADOOP-2344. (acmurthy) - - HADOOP-2442. Fix TestLocalFileSystemPermission.testLocalFSsetOwner - to work on more platforms. (Raghu Angadi via nigel) - - HADOOP-2488. Fix a regression in random read performance. - (Michael Stack via rangadi) - - HADOOP-2523. Fix TestDFSShell.testFilePermissions on Windows. - (Raghu Angadi via nigel) - - HADOOP-2535. Removed support for deprecated mapred.child.heap.size and - fixed some indentation issues in TaskRunner. (acmurthy) - Configuration changes to hadoop-default.xml: - remove mapred.child.heap.size - - HADOOP-2512. Fix error stream handling in Shell. Use exit code to - detect shell command errors in RawLocalFileSystem. (Raghu Angadi) - - HADOOP-2446. Fixes TestHDFSServerPorts and TestMRServerPorts so they - do not rely on statically configured ports and cleanup better. (nigel) - - HADOOP-2537. Make build process compatible with Ant 1.7.0. - (Hrishikesh via nigel) - - HADOOP-1281. Ensure running tasks of completed map TIPs (e.g. speculative - tasks) are killed as soon as the TIP completed. (acmurthy) - - HADOOP-2571. Suppress a suprious warning in test code. (cdouglas) - - HADOOP-2481. NNBench report its progress periodically. - (Hairong Kuang via dhruba) - - HADOOP-2601. Start name-node on a free port for TestNNThroughputBenchmark. - (Konstantin Shvachko) - - HADOOP-2494. Set +x on contrib/*/bin/* in packaged tar bundle. - (stack via tomwhite) - - HADOOP-2605. Remove bogus leading slash in task-tracker report bindAddress. - (Konstantin Shvachko) - - HADOOP-2620. Trivial. 'bin/hadoop fs -help' did not list chmod, chown, and - chgrp. (Raghu Angadi) - - HADOOP-2614. The DFS WebUI accesses are configured to be from the user - specified by dfs.web.ugi. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2543. Implement a "no-permission-checking" mode for smooth - upgrade from a pre-0.16 install of HDFS. - (Hairong Kuang via dhruba) - - HADOOP-290. A DataNode log message now prints the target of a replication - request correctly. (dhruba) - - HADOOP-2538. Redirect to a warning, if plaintext parameter is true but - the filter parameter is not given in TaskLogServlet. - (Michael Bieniosek via enis) - - HADOOP-2582. Prevent 'bin/hadoop fs -copyToLocal' from creating - zero-length files when the src does not exist. - (Lohit Vijayarenu via cdouglas) - - HADOOP-2189. Incrementing user counters should count as progress. (ddas) - - HADOOP-2649. The NameNode periodically computes replication work for - the datanodes. The periodicity of this computation is now configurable. - (dhruba) - - HADOOP-2549. Correct disk size computation so that data-nodes could switch - to other local drives if current is full. (Hairong Kuang via shv) - - HADOOP-2633. Fsck should call name-node methods directly rather than - through rpc. (Tsz Wo (Nicholas), SZE via shv) - - HADOOP-2687. Modify a few log message generated by dfs client to be - logged only at INFO level. (stack via dhruba) - - HADOOP-2402. Fix BlockCompressorStream to ensure it buffers data before - sending it down to the compressor so that each write call doesn't - compress. (Chris Douglas via acmurthy) - - HADOOP-2645. The Metrics initialization code does not throw - exceptions when servers are restarted by MiniDFSCluster. - (Sanjay Radia via dhruba) - - HADOOP-2691. Fix a race condition that was causing the DFSClient - to erroneously remove a good datanode from a pipeline that actually - had another datanode that was bad. (dhruba) - - HADOOP-1195. All code in FSNamesystem checks the return value - of getDataNode for null before using it. (dhruba) - - HADOOP-2640. Fix a bug in MultiFileSplitInputFormat that was always - returning 1 split in some circumstances. (Enis Soztutar via nigel) - - HADOOP-2626. Fix paths with special characters to work correctly - with the local filesystem. (Thomas Friol via cutting) - - HADOOP-2646. Fix SortValidator to work with fully-qualified - working directories. (Arun C Murthy via nigel) - - HADOOP-2092. Added a ping mechanism to the pipes' task to periodically - check if the parent Java task is running, and exit if the parent isn't - alive and responding. (Amareshwari Sri Ramadasu via acmurthy) - - HADOOP-2714. TestDecommission failed on windows because the replication - request was timing out. (dhruba) - - HADOOP-2576. Namenode performance degradation over time triggered by - large heartbeat interval. (Raghu Angadi) - - HADOOP-2713. TestDatanodeDeath failed on windows because the replication - request was timing out. (dhruba) - - HADOOP-2639. Fixes a problem to do with incorrect maintenance of values - for runningMapTasks/runningReduceTasks. (Amar Kamat and Arun Murthy - via ddas) - - HADOOP-2723. Fixed the check for checking whether to do user task - profiling. (Amareshwari Sri Ramadasu via omalley) - - HADOOP-2734. Link forrest docs to new http://hadoop.apache.org - (Doug Cutting via nigel) - - HADOOP-2641. Added Apache license headers to 95 files. (nigel) - - HADOOP-2732. Fix bug in path globbing. (Hairong Kuang via nigel) - - HADOOP-2404. Fix backwards compatability with hadoop-0.15 configuration - files that was broken by HADOOP-2185. (omalley) - - HADOOP-2755. Fix fsck performance degradation because of permissions - issue. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-2768. Fix performance regression caused by HADOOP-1707. - (dhruba borthakur via nigel) - - HADOOP-3108. Fix NPE in setPermission and setOwner. (shv) - -Release 0.15.3 - 2008-01-18 - - BUG FIXES - - HADOOP-2562. globPaths supports {ab,cd}. (Hairong Kuang via dhruba) - - HADOOP-2540. fsck reports missing blocks incorrectly. (dhruba) - - HADOOP-2570. "work" directory created unconditionally, and symlinks - created from the task cwds. - - HADOOP-2574. Fixed mapred_tutorial.xml to correct minor errors with the - WordCount examples. (acmurthy) - -Release 0.15.2 - 2008-01-02 - - BUG FIXES - - HADOOP-2246. Moved the changelog for HADOOP-1851 from the NEW FEATURES - section to the INCOMPATIBLE CHANGES section. (acmurthy) - - HADOOP-2238. Fix TaskGraphServlet so that it sets the content type of - the response appropriately. (Paul Saab via enis) - - HADOOP-2129. Fix so that distcp works correctly when source is - HDFS but not the default filesystem. HDFS paths returned by the - listStatus() method are now fully-qualified. (cutting) - - HADOOP-2378. Fixes a problem where the last task completion event would - get created after the job completes. (Alejandro Abdelnur via ddas) - - HADOOP-2228. Checks whether a job with a certain jobId is already running - and then tries to create the JobInProgress object. - (Johan Oskarsson via ddas) - - HADOOP-2422. dfs -cat multiple files fail with 'Unable to write to - output stream'. (Raghu Angadi via dhruba) - - HADOOP-2460. When the namenode encounters ioerrors on writing a - transaction log, it stops writing new transactions to that one. - (Raghu Angadi via dhruba) - - HADOOP-2227. Use the LocalDirAllocator uniformly for handling all of the - temporary storage required for a given task. It also implies that - mapred.local.dir.minspacestart is handled by checking if there is enough - free-space on any one of the available disks. (Amareshwari Sri Ramadasu - via acmurthy) - - HADOOP-2437. Fix the LocalDirAllocator to choose the seed for the - round-robin disk selections randomly. This helps in spreading data across - multiple partitions much better. (acmurhty) - - HADOOP-2486. When the list of files from the InMemoryFileSystem is obtained - for merging, this patch will ensure that only those files whose checksums - have also got created (renamed) are returned. (ddas) - - HADOOP-2456. Hardcode English locale to prevent NumberFormatException - from occurring when starting the NameNode with certain locales. - (Matthias Friedrich via nigel) - - IMPROVEMENTS - - HADOOP-2160. Remove project-level, non-user documentation from - releases, since it's now maintained in a separate tree. (cutting) - - HADOOP-1327. Add user documentation for streaming. (cutting) - - HADOOP-2382. Add hadoop-default.html to subversion. (cutting) - - HADOOP-2158. hdfsListDirectory calls FileSystem.listStatus instead - of FileSystem.listPaths. This reduces the number of RPC calls on the - namenode, thereby improving scalability. (Christian Kunz via dhruba) - -Release 0.15.1 - 2007-11-27 - - INCOMPATIBLE CHANGES - - HADOOP-713. Reduce CPU usage on namenode while listing directories. - FileSystem.listPaths does not return the size of the entire subtree. - Introduced a new API ClientProtocol.getContentLength that returns the - size of the subtree. (Dhruba Borthakur via dhruba) - - IMPROVEMENTS - - HADOOP-1917. Addition of guides/tutorial for better overall - documentation for Hadoop. Specifically: - * quickstart.html is targetted towards first-time users and helps them - setup a single-node cluster and play with Hadoop. - * cluster_setup.html helps admins to configure and setup non-trivial - hadoop clusters. - * mapred_tutorial.html is a comprehensive Map-Reduce tutorial. - (acmurthy) - - BUG FIXES - - HADOOP-2174. Removed the unnecessary Reporter.setStatus call from - FSCopyFilesMapper.close which led to a NPE since the reporter isn't valid - in the close method. (Chris Douglas via acmurthy) - - HADOOP-2172. Restore performance of random access to local files - by caching positions of local input streams, avoiding a system - call. (cutting) - - HADOOP-2205. Regenerate the Hadoop website since some of the changes made - by HADOOP-1917 weren't correctly copied over to the trunk/docs directory. - Also fixed a couple of minor typos and broken links. (acmurthy) - -Release 0.15.0 - 2007-11-2 - - INCOMPATIBLE CHANGES - - HADOOP-1708. Make files appear in namespace as soon as they are - created. (Dhruba Borthakur via dhruba) - - HADOOP-999. A HDFS Client immediately informs the NameNode of a new - file creation. ClientProtocol version changed from 14 to 15. - (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-932. File locking interfaces and implementations (that were - earlier deprecated) are removed. Client Protocol version changed - from 15 to 16. (Raghu Angadi via dhruba) - - HADOOP-1621. FileStatus is now a concrete class and FileSystem.listPaths - is deprecated and replaced with listStatus. (Chris Douglas via omalley) - - HADOOP-1656. The blockSize of a file is stored persistently in the file - inode. (Dhruba Borthakur via dhruba) - - HADOOP-1838. The blocksize of files created with an earlier release is - set to the default block size. (Dhruba Borthakur via dhruba) - - HADOOP-785. Add support for 'final' Configuration parameters, - removing support for 'mapred-default.xml', and changing - 'hadoop-site.xml' to not override other files. Now folks should - generally use 'hadoop-site.xml' for all configurations. Values - with a 'final' tag may not be overridden by subsequently loaded - configuration files, e.g., by jobs. (Arun C. Murthy via cutting) - - HADOOP-1846. DatanodeReport in ClientProtocol can report live - datanodes, dead datanodes or all datanodes. Client Protocol version - changed from 17 to 18. (Hairong Kuang via dhruba) - - HADOOP-1851. Permit specification of map output compression type - and codec, independent of the final output's compression - parameters. (Arun C Murthy via cutting) - - HADOOP-1819. Jobtracker cleanups, including binding ports before - clearing state directories, so that inadvertently starting a - second jobtracker doesn't trash one that's already running. Removed - method JobTracker.getTracker() because the static variable, which - stored the value caused initialization problems. - (omalley via cutting) - - NEW FEATURES - - HADOOP-89. A client can access file data even before the creator - has closed the file. Introduce a new command "tail" from dfs shell. - (Dhruba Borthakur via dhruba) - - HADOOP-1636. Allow configuration of the number of jobs kept in - memory by the JobTracker. (Michael Bieniosek via omalley) - - HADOOP-1667. Reorganize CHANGES.txt into sections to make it - easier to read. Also remove numbering, to make merging easier. - (cutting) - - HADOOP-1610. Add metrics for failed tasks. - (Devaraj Das via tomwhite) - - HADOOP-1767. Add "bin/hadoop job -list" sub-command. (taton via cutting) - - HADOOP-1351. Add "bin/hadoop job [-fail-task|-kill-task]" sub-commands - to terminate a particular task-attempt. (Enis Soztutar via acmurthy) - - HADOOP-1880. SleepJob : An example job that sleeps at each map and - reduce task. (enis) - - HADOOP-1809. Add a link in web site to #hadoop IRC channel. (enis) - - HADOOP-1894. Add percentage graphs and mapred task completion graphs - to Web User Interface. Users not using Firefox may install a plugin to - their browsers to see svg graphics. (enis) - - HADOOP-1914. Introduce a new NamenodeProtocol to allow secondary - namenodes and rebalancing processes to communicate with a primary - namenode. (Hairong Kuang via dhruba) - - HADOOP-1963. Add a FileSystem implementation for the Kosmos - Filesystem (KFS). (Sriram Rao via cutting) - - HADOOP-1822. Allow the specialization and configuration of socket - factories. Provide a StandardSocketFactory, and a SocksSocketFactory to - allow the use of SOCKS proxies. (taton). - - HADOOP-1968. FileSystem supports wildcard input syntax "{ }". - (Hairong Kuang via dhruba) - - HADOOP-2566. Add globStatus method to the FileSystem interface - and deprecate globPath and listPath. (Hairong Kuang via hairong) - - OPTIMIZATIONS - - HADOOP-1910. Reduce the number of RPCs that DistributedFileSystem.create() - makes to the namenode. (Raghu Angadi via dhruba) - - HADOOP-1565. Reduce memory usage of NameNode by replacing - TreeMap in HDFS Namespace with ArrayList. - (Dhruba Borthakur via dhruba) - - HADOOP-1743. Change DFS INode from a nested class to standalone - class, with specialized subclasses for directories and files, to - save memory on the namenode. (Konstantin Shvachko via cutting) - - HADOOP-1759. Change file name in INode from String to byte[], - saving memory on the namenode. (Konstantin Shvachko via cutting) - - HADOOP-1766. Save memory in namenode by having BlockInfo extend - Block, and replace many uses of Block with BlockInfo. - (Konstantin Shvachko via cutting) - - HADOOP-1687. Save memory in namenode by optimizing BlockMap - representation. (Konstantin Shvachko via cutting) - - HADOOP-1774. Remove use of INode.parent in Block CRC upgrade. - (Raghu Angadi via dhruba) - - HADOOP-1788. Increase the buffer size on the Pipes command socket. - (Amareshwari Sri Ramadasu and Christian Kunz via omalley) - - BUG FIXES - - HADOOP-1946. The Datanode code does not need to invoke du on - every heartbeat. (Hairong Kuang via dhruba) - - HADOOP-1935. Fix a NullPointerException in internalReleaseCreate. - (Dhruba Borthakur) - - HADOOP-1933. The nodes listed in include and exclude files - are always listed in the datanode report. - (Raghu Angadi via dhruba) - - HADOOP-1953. The job tracker should wait beteween calls to try and delete - the system directory (Owen O'Malley via devaraj) - - HADOOP-1932. TestFileCreation fails with message saying filestatus.dat - is of incorrect size. (Dhruba Borthakur via dhruba) - - HADOOP-1573. Support for 0 reducers in PIPES. - (Owen O'Malley via devaraj) - - HADOOP-1500. Fix typographical errors in the DFS WebUI. - (Nigel Daley via dhruba) - - HADOOP-1076. Periodic checkpoint can continue even if an earlier - checkpoint encountered an error. (Dhruba Borthakur via dhruba) - - HADOOP-1887. The Namenode encounters an ArrayIndexOutOfBoundsException - while listing a directory that had a file that was - being actively written to. (Dhruba Borthakur via dhruba) - - HADOOP-1904. The Namenode encounters an exception because the - list of blocks per datanode-descriptor was corrupted. - (Konstantin Shvachko via dhruba) - - HADOOP-1762. The Namenode fsimage does not contain a list of - Datanodes. (Raghu Angadi via dhruba) - - HADOOP-1890. Removed debugging prints introduced by HADOOP-1774. - (Raghu Angadi via dhruba) - - HADOOP-1763. Too many lost task trackers on large clusters due to - insufficient number of RPC handler threads on the JobTracker. - (Devaraj Das) - - HADOOP-1463. HDFS report correct usage statistics for disk space - used by HDFS. (Hairong Kuang via dhruba) - - HADOOP-1692. In DFS ant task, don't cache the Configuration. - (Chris Douglas via cutting) - - HADOOP-1726. Remove lib/jetty-ext/ant.jar. (omalley) - - HADOOP-1772. Fix hadoop-daemon.sh script to get correct hostname - under Cygwin. (Tsz Wo (Nicholas), SZE via cutting) - - HADOOP-1749. Change TestDFSUpgrade to sort files, fixing sporadic - test failures. (Enis Soztutar via cutting) - - HADOOP-1748. Fix tasktracker to be able to launch tasks when log - directory is relative. (omalley via cutting) - - HADOOP-1775. Fix a NullPointerException and an - IllegalArgumentException in MapWritable. - (Jim Kellerman via cutting) - - HADOOP-1795. Fix so that jobs can generate output file names with - special characters. (Fr??d??ric Bertin via cutting) - - HADOOP-1810. Fix incorrect value type in MRBench (SmallJobs) - (Devaraj Das via tomwhite) - - HADOOP-1806. Fix ant task to compile again, also fix default - builds to compile ant tasks. (Chris Douglas via cutting) - - HADOOP-1758. Fix escape processing in librecordio to not be - quadratic. (Vivek Ratan via cutting) - - HADOOP-1817. Fix MultiFileSplit to read and write the split - length, so that it is not always zero in map tasks. - (Thomas Friol via cutting) - - HADOOP-1853. Fix contrib/streaming to accept multiple -cacheFile - options. (Prachi Gupta via cutting) - - HADOOP-1818. Fix MultiFileInputFormat so that it does not return - empty splits when numPaths < numSplits. (Thomas Friol via enis) - - HADOOP-1840. Fix race condition which leads to task's diagnostic - messages getting lost. (acmurthy) - - HADOOP-1885. Fix race condition in MiniDFSCluster shutdown. - (Chris Douglas via nigel) - - HADOOP-1889. Fix path in EC2 scripts for building your own AMI. - (tomwhite) - - HADOOP-1892. Fix a NullPointerException in the JobTracker when - trying to fetch a task's diagnostic messages from the JobClient. - (Amar Kamat via acmurthy) - - HADOOP-1897. Completely remove about.html page from the web site. - (enis) - - HADOOP-1907. Fix null pointer exception when getting task diagnostics - in JobClient. (Christian Kunz via omalley) - - HADOOP-1882. Remove spurious asterisks from decimal number displays. - (Raghu Angadi via cutting) - - HADOOP-1783. Make S3 FileSystem return Paths fully-qualified with - scheme and host. (tomwhite) - - HADOOP-1925. Make pipes' autoconf script look for libsocket and libnsl, so - that it can compile under Solaris. (omalley) - - HADOOP-1940. TestDFSUpgradeFromImage must shut down its MiniDFSCluster. - (Chris Douglas via nigel) - - HADOOP-1930. Fix the blame for failed fetchs on the right host. (Arun C. - Murthy via omalley) - - HADOOP-1934. Fix the platform name on Mac to use underscores rather than - spaces. (omalley) - - HADOOP-1959. Use "/" instead of File.separator in the StatusHttpServer. - (jimk via omalley) - - HADOOP-1626. Improve dfsadmin help messages. - (Lohit Vijayarenu via dhruba) - - HADOOP-1695. The SecondaryNamenode waits for the Primary NameNode to - start up. (Dhruba Borthakur) - - HADOOP-1983. Have Pipes flush the command socket when progress is sent - to prevent timeouts during long computations. (omalley) - - HADOOP-1875. Non-existant directories or read-only directories are - filtered from dfs.client.buffer.dir. (Hairong Kuang via dhruba) - - HADOOP-1992. Fix the performance degradation in the sort validator. - (acmurthy via omalley) - - HADOOP-1874. Move task-outputs' promotion/discard to a separate thread - distinct from the main heartbeat-processing thread. The main upside being - that we do not lock-up the JobTracker during HDFS operations, which - otherwise may lead to lost tasktrackers if the NameNode is unresponsive. - (Devaraj Das via acmurthy) - - HADOOP-2026. Namenode prints out one log line for "Number of transactions" - at most once every minute. (Dhruba Borthakur) - - HADOOP-2022. Ensure that status information for successful tasks is correctly - recorded at the JobTracker, so that, for example, one may view correct - information via taskdetails.jsp. This bug was introduced by HADOOP-1874. - (Amar Kamat via acmurthy) - - HADOOP-2031. Correctly maintain the taskid which takes the TIP to - completion, failing which the case of lost tasktrackers isn't handled - properly i.e. the map TIP is incorrectly left marked as 'complete' and it - is never rescheduled elsewhere, leading to hung reduces. - (Devaraj Das via acmurthy) - - HADOOP-2018. The source datanode of a data transfer waits for - a response from the target datanode before closing the data stream. - (Hairong Kuang via dhruba) - - HADOOP-2023. Disable TestLocalDirAllocator on Windows. - (Hairong Kuang via nigel) - - HADOOP-2016. Ignore status-updates from FAILED/KILLED tasks at the - TaskTracker. This fixes a race-condition which caused the tasks to wrongly - remain in the RUNNING state even after being killed by the JobTracker and - thus handicap the cleanup of the task's output sub-directory. (acmurthy) - - HADOOP-1771. Fix a NullPointerException in streaming caused by an - IOException in MROutputThread. (lohit vijayarenu via nigel) - - HADOOP-2028. Fix distcp so that the log dir does not need to be - specified and the destination does not need to exist. - (Chris Douglas via nigel) - - HADOOP-2044. The namenode protects all lease manipulations using a - sortedLease lock. (Dhruba Borthakur) - - HADOOP-2051. The TaskCommit thread should not die for exceptions other - than the InterruptedException. This behavior is there for the other long - running threads in the JobTracker. (Arun C Murthy via ddas) - - HADOOP-1973. The FileSystem object would be accessed on the JobTracker - through a RPC in the InterTrackerProtocol. The check for the object being - null was missing and hence NPE would be thrown sometimes. This issue fixes - that problem. (Amareshwari Sri Ramadasu via ddas) - - HADOOP-2033. The SequenceFile.Writer.sync method was a no-op, which caused - very uneven splits for applications like distcp that count on them. - (omalley) - - HADOOP-2070. Added a flush method to pipes' DownwardProtocol and call - that before waiting for the application to finish to ensure all buffered - data is flushed. (Owen O'Malley via acmurthy) - - HADOOP-2080. Fixed calculation of the checksum file size when the values - are large. (omalley) - - HADOOP-2048. Change error handling in distcp so that each map copies - as much as possible before reporting the error. Also report progress on - every copy. (Chris Douglas via omalley) - - HADOOP-2073. Change size of VERSION file after writing contents to it. - (Konstantin Shvachko via dhruba) - - HADOOP-2102. Fix the deprecated ToolBase to pass its Configuration object - to the superceding ToolRunner to ensure it picks up the appropriate - configuration resources. (Dennis Kubes and Enis Soztutar via acmurthy) - - HADOOP-2103. Fix minor javadoc bugs introduce by HADOOP-2046. (Nigel - Daley via acmurthy) - - IMPROVEMENTS - - HADOOP-1908. Restructure data node code so that block sending and - receiving are seperated from data transfer header handling. - (Hairong Kuang via dhruba) - - HADOOP-1921. Save the configuration of completed/failed jobs and make them - available via the web-ui. (Amar Kamat via devaraj) - - HADOOP-1266. Remove dependency of package org.apache.hadoop.net on - org.apache.hadoop.dfs. (Hairong Kuang via dhruba) - - HADOOP-1779. Replace INodeDirectory.getINode() by a getExistingPathINodes() - to allow the retrieval of all existing INodes along a given path in a - single lookup. This facilitates removal of the 'parent' field in the - inode. (Christophe Taton via dhruba) - - HADOOP-1756. Add toString() to some Writable-s. (ab) - - HADOOP-1727. New classes: MapWritable and SortedMapWritable. - (Jim Kellerman via ab) - - HADOOP-1651. Improve progress reporting. - (Devaraj Das via tomwhite) - - HADOOP-1595. dfsshell can wait for a file to achieve its intended - replication target. (Tsz Wo (Nicholas), SZE via dhruba) - - HADOOP-1693. Remove un-needed log fields in DFS replication classes, - since the log may be accessed statically. (Konstantin Shvachko via cutting) - - HADOOP-1231. Add generics to Mapper and Reducer interfaces. - (tomwhite via cutting) - - HADOOP-1436. Improved command-line APIs, so that all tools need - not subclass ToolBase, and generic parameter parser is public. - (Enis Soztutar via cutting) - - HADOOP-1703. DFS-internal code cleanups, removing several uses of - the obsolete UTF8. (Christophe Taton via cutting) - - HADOOP-1731. Add Hadoop's version to contrib jar file names. - (cutting) - - HADOOP-1689. Make shell scripts more portable. All shell scripts - now explicitly depend on bash, but do not require that bash be - installed in a particular location, as long as it is on $PATH. - (cutting) - - HADOOP-1744. Remove many uses of the deprecated UTF8 class from - the HDFS namenode. (Christophe Taton via cutting) - - HADOOP-1654. Add IOUtils class, containing generic io-related - utility methods. (Enis Soztutar via cutting) - - HADOOP-1158. Change JobTracker to record map-output transmission - errors and use them to trigger speculative re-execution of tasks. - (Arun C Murthy via cutting) - - HADOOP-1601. Change GenericWritable to use ReflectionUtils for - instance creation, avoiding classloader issues, and to implement - Configurable. (Enis Soztutar via cutting) - - HADOOP-1750. Log standard output and standard error when forking - task processes. (omalley via cutting) - - HADOOP-1803. Generalize build.xml to make files in all - src/contrib/*/bin directories executable. (stack via cutting) - - HADOOP-1739. Let OS always choose the tasktracker's umbilical - port. Also switch default address for umbilical connections to - loopback. (cutting) - - HADOOP-1812. Let OS choose ports for IPC and RPC unit tests. (cutting) - - HADOOP-1825. Create $HADOOP_PID_DIR when it does not exist. - (Michael Bieniosek via cutting) - - HADOOP-1425. Replace uses of ToolBase with the Tool interface. - (Enis Soztutar via cutting) - - HADOOP-1569. Reimplement DistCP to use the standard FileSystem/URI - code in Hadoop so that you can copy from and to all of the supported file - systems.(Chris Douglas via omalley) - - HADOOP-1018. Improve documentation w.r.t handling of lost hearbeats between - TaskTrackers and JobTracker. (acmurthy) - - HADOOP-1718. Add ant targets for measuring code coverage with clover. - (simonwillnauer via nigel) - - HADOOP-1592. Log error messages to the client console when tasks - fail. (Amar Kamat via cutting) - - HADOOP-1879. Remove some unneeded casts. (Nilay Vaish via cutting) - - HADOOP-1878. Add space between priority links on job details - page. (Thomas Friol via cutting) - - HADOOP-120. In ArrayWritable, prevent creation with null value - class, and improve documentation. (Cameron Pope via cutting) - - HADOOP-1926. Add a random text writer example/benchmark so that we can - benchmark compression codecs on random data. (acmurthy via omalley) - - HADOOP-1906. Warn the user if they have an obsolete madred-default.xml - file in their configuration directory. (acmurthy via omalley) - - HADOOP-1971. Warn when job does not specify a jar. (enis via cutting) - - HADOOP-1942. Increase the concurrency of transaction logging to - edits log. Reduce the number of syncs by double-buffering the changes - to the transaction log. (Dhruba Borthakur) - - HADOOP-2046. Improve mapred javadoc. (Arun C. Murthy via cutting) - - HADOOP-2105. Improve overview.html to clarify supported platforms, - software pre-requisites for hadoop, how to install them on various - platforms and a better general description of hadoop and it's utility. - (Jim Kellerman via acmurthy) - - -Release 0.14.4 - 2007-11-26 - - BUG FIXES - - HADOOP-2140. Add missing Apache Licensing text at the front of several - C and C++ files. - - HADOOP-2169. Fix the DT_SONAME field of libhdfs.so to set it to the - correct value of 'libhdfs.so', currently it is set to the absolute path of - libhdfs.so. (acmurthy) - - HADOOP-2001. Make the job priority updates and job kills synchronized on - the JobTracker. Deadlock was seen in the JobTracker because of the lack of - this synchronization. (Arun C Murthy via ddas) - - -Release 0.14.3 - 2007-10-19 - - BUG FIXES - - HADOOP-2053. Fixed a dangling reference to a memory buffer in the map - output sorter. (acmurthy via omalley) - - HADOOP-2036. Fix a NullPointerException in JvmMetrics class. (nigel) - - HADOOP-2043. Release 0.14.2 was compiled with Java 1.6 rather than - Java 1.5. (cutting) - - -Release 0.14.2 - 2007-10-09 - - BUG FIXES - - HADOOP-1948. Removed spurious error message during block crc upgrade. - (Raghu Angadi via dhruba) - - HADOOP-1862. reduces are getting stuck trying to find map outputs. - (Arun C. Murthy via ddas) - - HADOOP-1977. Fixed handling of ToolBase cli options in JobClient. - (enis via omalley) - - HADOOP-1972. Fix LzoCompressor to ensure the user has actually asked - to finish compression. (arun via omalley) - - HADOOP-1970. Fix deadlock in progress reporting in the task. (Vivek - Ratan via omalley) - - HADOOP-1978. Name-node removes edits.new after a successful startup. - (Konstantin Shvachko via dhruba) - - HADOOP-1955. The Namenode tries to not pick the same source Datanode for - a replication request if the earlier replication request for the same - block and that source Datanode had failed. - (Raghu Angadi via dhruba) - - HADOOP-1961. The -get option to dfs-shell works when a single filename - is specified. (Raghu Angadi via dhruba) - - HADOOP-1997. TestCheckpoint closes the edits file after writing to it, - otherwise the rename of this file on Windows fails. - (Konstantin Shvachko via dhruba) - -Release 0.14.1 - 2007-09-04 - - BUG FIXES - - HADOOP-1740. Fix null pointer exception in sorting map outputs. (Devaraj - Das via omalley) - - HADOOP-1790. Fix tasktracker to work correctly on multi-homed - boxes. (Torsten Curdt via cutting) - - HADOOP-1798. Fix jobtracker to correctly account for failed - tasks. (omalley via cutting) - - -Release 0.14.0 - 2007-08-17 - - INCOMPATIBLE CHANGES - - 1. HADOOP-1134. - CONFIG/API - dfs.block.size must now be a multiple of - io.byte.per.checksum, otherwise new files can not be written. - LAYOUT - DFS layout version changed from -6 to -7, which will require an - upgrade from previous versions. - PROTOCOL - Datanode RPC protocol version changed from 7 to 8. - - 2. HADOOP-1283 - API - deprecated file locking API. - - 3. HADOOP-894 - PROTOCOL - changed ClientProtocol to fetch parts of block locations. - - 4. HADOOP-1336 - CONFIG - Enable speculative execution by default. - - 5. HADOOP-1197 - API - deprecated method for Configuration.getObject, because - Configurations should only contain strings. - - 6. HADOOP-1343 - API - deprecate Configuration.set(String,Object) so that only strings are - put in Configrations. - - 7. HADOOP-1207 - CLI - Fix FsShell 'rm' command to continue when a non-existent file is - encountered. - - 8. HADOOP-1473 - CLI/API - Job, TIP, and Task id formats have changed and are now unique - across job tracker restarts. - - 9. HADOOP-1400 - API - JobClient constructor now takes a JobConf object instead of a - Configuration object. - - NEW FEATURES and BUG FIXES - - 1. HADOOP-1197. In Configuration, deprecate getObject() and add - getRaw(), which skips variable expansion. (omalley via cutting) - - 2. HADOOP-1343. In Configuration, deprecate set(String,Object) and - implement Iterable. (omalley via cutting) - - 3. HADOOP-1344. Add RunningJob#getJobName(). (Michael Bieniosek via cutting) - - 4. HADOOP-1342. In aggregators, permit one to limit the number of - unique values per key. (Runping Qi via cutting) - - 5. HADOOP-1340. Set the replication factor of the MD5 file in the filecache - to be the same as the replication factor of the original file. - (Dhruba Borthakur via tomwhite.) - - 6. HADOOP-1355. Fix null pointer dereference in - TaskLogAppender.append(LoggingEvent). (Arun C Murthy via tomwhite.) - - 7. HADOOP-1357. Fix CopyFiles to correctly avoid removing "/". - (Arun C Murthy via cutting) - - 8. HADOOP-234. Add pipes facility, which permits writing MapReduce - programs in C++. - - 9. HADOOP-1359. Fix a potential NullPointerException in HDFS. - (Hairong Kuang via cutting) - - 10. HADOOP-1364. Fix inconsistent synchronization in SequenceFile. - (omalley via cutting) - - 11. HADOOP-1379. Add findbugs target to build.xml. - (Nigel Daley via cutting) - - 12. HADOOP-1364. Fix various inconsistent synchronization issues. - (Devaraj Das via cutting) - - 13. HADOOP-1393. Remove a potential unexpected negative number from - uses of random number generator. (omalley via cutting) - - 14. HADOOP-1387. A number of "performance" code-cleanups suggested - by findbugs. (Arun C Murthy via cutting) - - 15. HADOOP-1401. Add contrib/hbase javadoc to tree. (stack via cutting) - - 16. HADOOP-894. Change HDFS so that the client only retrieves a limited - number of block locations per request from the namenode. - (Konstantin Shvachko via cutting) - - 17. HADOOP-1406. Plug a leak in MapReduce's use of metrics. - (David Bowen via cutting) - - 18. HADOOP-1394. Implement "performance" code-cleanups in HDFS - suggested by findbugs. (Raghu Angadi via cutting) - - 19. HADOOP-1413. Add example program that uses Knuth's dancing links - algorithm to solve pentomino problems. (omalley via cutting) - - 20. HADOOP-1226. Change HDFS so that paths it returns are always - fully qualified. (Dhruba Borthakur via cutting) - - 21. HADOOP-800. Improvements to HDFS web-based file browser. - (Enis Soztutar via cutting) - - 22. HADOOP-1408. Fix a compiler warning by adding a class to replace - a generic. (omalley via cutting) - - 23. HADOOP-1376. Modify RandomWriter example so that it can generate - data for the Terasort benchmark. (Devaraj Das via cutting) - - 24. HADOOP-1429. Stop logging exceptions during normal IPC server - shutdown. (stack via cutting) - - 25. HADOOP-1461. Fix the synchronization of the task tracker to - avoid lockups in job cleanup. (Arun C Murthy via omalley) - - 26. HADOOP-1446. Update the TaskTracker metrics while the task is - running. (Devaraj via omalley) - - 27. HADOOP-1414. Fix a number of issues identified by FindBugs as - "Bad Practice". (Dhruba Borthakur via cutting) - - 28. HADOOP-1392. Fix "correctness" bugs identified by FindBugs in - fs and dfs packages. (Raghu Angadi via cutting) - - 29. HADOOP-1412. Fix "dodgy" bugs identified by FindBugs in fs and - io packages. (Hairong Kuang via cutting) - - 30. HADOOP-1261. Remove redundant events from HDFS namenode's edit - log when a datanode restarts. (Raghu Angadi via cutting) - - 31. HADOOP-1336. Re-enable speculative execution by - default. (omalley via cutting) - - 32. HADOOP-1311. Fix a bug in BytesWritable#set() where start offset - was ignored. (Dhruba Borthakur via cutting) - - 33. HADOOP-1450. Move checksumming closer to user code, so that - checksums are created before data is stored in large buffers and - verified after data is read from large buffers, to better catch - memory errors. (cutting) - - 34. HADOOP-1447. Add support in contrib/data_join for text inputs. - (Senthil Subramanian via cutting) - - 35. HADOOP-1456. Fix TestDecommission assertion failure by setting - the namenode to ignore the load on datanodes while allocating - replicas. (Dhruba Borthakur via tomwhite) - - 36. HADOOP-1396. Fix FileNotFoundException on DFS block. - (Dhruba Borthakur via tomwhite) - - 37. HADOOP-1467. Remove redundant counters from WordCount example. - (Owen O'Malley via tomwhite) - - 38. HADOOP-1139. Log HDFS block transitions at INFO level, to better - enable diagnosis of problems. (Dhruba Borthakur via cutting) - - 39. HADOOP-1269. Finer grained locking in HDFS namenode. - (Dhruba Borthakur via cutting) - - 40. HADOOP-1438. Improve HDFS documentation, correcting typos and - making images appear in PDF. Also update copyright date for all - docs. (Luke Nezda via cutting) - - 41. HADOOP-1457. Add counters for monitoring task assignments. - (Arun C Murthy via tomwhite) - - 42. HADOOP-1472. Fix so that timed-out tasks are counted as failures - rather than as killed. (Arun C Murthy via cutting) - - 43. HADOOP-1234. Fix a race condition in file cache that caused - tasktracker to not be able to find cached files. - (Arun C Murthy via cutting) - - 44. HADOOP-1482. Fix secondary namenode to roll info port. - (Dhruba Borthakur via cutting) - - 45. HADOOP-1300. Improve removal of excess block replicas to be - rack-aware. Attempts are now made to keep replicas on more - racks. (Hairong Kuang via cutting) - - 46. HADOOP-1417. Disable a few FindBugs checks that generate a lot - of spurious warnings. (Nigel Daley via cutting) - - 47. HADOOP-1320. Rewrite RandomWriter example to bypass reduce. - (Arun C Murthy via cutting) - - 48. HADOOP-1449. Add some examples to contrib/data_join. - (Senthil Subramanian via cutting) - - 49. HADOOP-1459. Fix so that, in HDFS, getFileCacheHints() returns - hostnames instead of IP addresses. (Dhruba Borthakur via cutting) - - 50. HADOOP-1493. Permit specification of "java.library.path" system - property in "mapred.child.java.opts" configuration property. - (Enis Soztutar via cutting) - - 51. HADOOP-1372. Use LocalDirAllocator for HDFS temporary block - files, so that disk space, writability, etc. is considered. - (Dhruba Borthakur via cutting) - - 52. HADOOP-1193. Pool allocation of compression codecs. This - eliminates a memory leak that could cause OutOfMemoryException, - and also substantially improves performance. - (Arun C Murthy via cutting) - - 53. HADOOP-1492. Fix a NullPointerException handling version - mismatch during datanode registration. - (Konstantin Shvachko via cutting) - - 54. HADOOP-1442. Fix handling of zero-length input splits. - (Senthil Subramanian via cutting) - - 55. HADOOP-1444. Fix HDFS block id generation to check pending - blocks for duplicates. (Dhruba Borthakur via cutting) - - 56. HADOOP-1207. Fix FsShell's 'rm' command to not stop when one of - the named files does not exist. (Tsz Wo Sze via cutting) - - 57. HADOOP-1475. Clear tasktracker's file cache before it - re-initializes, to avoid confusion. (omalley via cutting) - - 58. HADOOP-1505. Remove spurious stacktrace in ZlibFactory - introduced in HADOOP-1093. (Michael Stack via tomwhite) - - 59. HADOOP-1484. Permit one to kill jobs from the web ui. Note that - this is disabled by default. One must set - "webinterface.private.actions" to enable this. - (Enis Soztutar via cutting) - - 60. HADOOP-1003. Remove flushing of namenode edit log from primary - namenode lock, increasing namenode throughput. - (Dhruba Borthakur via cutting) - - 61. HADOOP-1023. Add links to searchable mail archives. - (tomwhite via cutting) - - 62. HADOOP-1504. Fix terminate-hadoop-cluster script in contrib/ec2 - to only terminate Hadoop instances, and not other instances - started by the same user. (tomwhite via cutting) - - 63. HADOOP-1462. Improve task progress reporting. Progress reports - are no longer blocking since i/o is performed in a separate - thread. Reporting during sorting and more is also more - consistent. (Vivek Ratan via cutting) - - 64. [ intentionally blank ] - - 65. HADOOP-1453. Remove some unneeded calls to FileSystem#exists() - when opening files, reducing the namenode load somewhat. - (Raghu Angadi via cutting) - - 66. HADOOP-1489. Fix text input truncation bug due to mark/reset. - Add a unittest. (Bwolen Yang via cutting) - - 67. HADOOP-1455. Permit specification of arbitrary job options on - pipes command line. (Devaraj Das via cutting) - - 68. HADOOP-1501. Better randomize sending of block reports to - namenode, so reduce load spikes. (Dhruba Borthakur via cutting) - - 69. HADOOP-1147. Remove @author tags from Java source files. - - 70. HADOOP-1283. Convert most uses of UTF8 in the namenode to be - String. (Konstantin Shvachko via cutting) - - 71. HADOOP-1511. Speedup hbase unit tests. (stack via cutting) - - 72. HADOOP-1517. Remove some synchronization in namenode to permit - finer grained locking previously added. (Konstantin Shvachko via cutting) - - 73. HADOOP-1512. Fix failing TestTextInputFormat on Windows. - (Senthil Subramanian via nigel) - - 74. HADOOP-1518. Add a session id to job metrics, for use by HOD. - (David Bowen via cutting) - - 75. HADOOP-1292. Change 'bin/hadoop fs -get' to first copy files to - a temporary name, then rename them to their final name, so that - failures don't leave partial files. (Tsz Wo Sze via cutting) - - 76. HADOOP-1377. Add support for modification time to FileSystem and - implement in HDFS and local implementations. Also, alter access - to file properties to be through a new FileStatus interface. - (Dhruba Borthakur via cutting) - - 77. HADOOP-1515. Add MultiFileInputFormat, which can pack multiple, - typically small, input files into each split. (Enis Soztutar via cutting) - - 78. HADOOP-1514. Make reducers report progress while waiting for map - outputs, so they're not killed. (Vivek Ratan via cutting) - - 79. HADOOP-1508. Add an Ant task for FsShell operations. Also add - new FsShell commands "touchz", "test" and "stat". - (Chris Douglas via cutting) - - 80. HADOOP-1028. Add log messages for server startup and shutdown. - (Tsz Wo Sze via cutting) - - 81. HADOOP-1485. Add metrics for monitoring shuffle. - (Devaraj Das via cutting) - - 82. HADOOP-1536. Remove file locks from libhdfs tests. - (Dhruba Borthakur via nigel) - - 83. HADOOP-1520. Add appropriate synchronization to FSEditsLog. - (Dhruba Borthakur via nigel) - - 84. HADOOP-1513. Fix a race condition in directory creation. - (Devaraj via omalley) - - 85. HADOOP-1546. Remove spurious column from HDFS web UI. - (Dhruba Borthakur via cutting) - - 86. HADOOP-1556. Make LocalJobRunner delete working files at end of - job run. (Devaraj Das via tomwhite) - - 87. HADOOP-1571. Add contrib lib directories to root build.xml - javadoc classpath. (Michael Stack via tomwhite) - - 88. HADOOP-1554. Log killed tasks to the job history and display them on the - web/ui. (Devaraj Das via omalley) - - 89. HADOOP-1533. Add persistent error logging for distcp. The logs are stored - into a specified hdfs directory. (Senthil Subramanian via omalley) - - 90. HADOOP-1286. Add support to HDFS for distributed upgrades, which - permits coordinated upgrade of datanode data. - (Konstantin Shvachko via cutting) - - 91. HADOOP-1580. Improve contrib/streaming so that subprocess exit - status is displayed for errors. (John Heidemann via cutting) - - 92. HADOOP-1448. In HDFS, randomize lists of non-local block - locations returned to client, so that load is better balanced. - (Hairong Kuang via cutting) - - 93. HADOOP-1578. Fix datanode to send its storage id to namenode - during registration. (Konstantin Shvachko via cutting) - - 94. HADOOP-1584. Fix a bug in GenericWritable which limited it to - 128 types instead of 256. (Espen Amble Kolstad via cutting) - - 95. HADOOP-1473. Make job ids unique across jobtracker restarts. - (omalley via cutting) - - 96. HADOOP-1582. Fix hdfslib to return 0 instead of -1 at - end-of-file, per C conventions. (Christian Kunz via cutting) - - 97. HADOOP-911. Fix a multithreading bug in libhdfs. - (Christian Kunz) - - 98. HADOOP-1486. Fix so that fatal exceptions in namenode cause it - to exit. (Dhruba Borthakur via cutting) - - 99. HADOOP-1470. Factor checksum generation and validation out of - ChecksumFileSystem so that it can be reused by FileSystem's with - built-in checksumming. (Hairong Kuang via cutting) - -100. HADOOP-1590. Use relative urls in jobtracker jsp pages, so that - webapp can be used in non-root contexts. (Thomas Friol via cutting) - -101. HADOOP-1596. Fix the parsing of taskids by streaming and improve the - error reporting. (omalley) - -102. HADOOP-1535. Fix the user-controlled grouping to the reduce function. - (Vivek Ratan via omalley) - -103. HADOOP-1585. Modify GenericWritable to declare the classes as subtypes - of Writable (Espen Amble Kolstad via omalley) - -104. HADOOP-1576. Fix errors in count of completed tasks when - speculative execution is enabled. (Arun C Murthy via cutting) - -105. HADOOP-1598. Fix license headers: adding missing; updating old. - (Enis Soztutar via cutting) - -106. HADOOP-1547. Provide examples for aggregate library. - (Runping Qi via tomwhite) - -107. HADOOP-1570. Permit jobs to enable and disable the use of - hadoop's native library. (Arun C Murthy via cutting) - -108. HADOOP-1433. Add job priority. (Johan Oskarsson via tomwhite) - -109. HADOOP-1597. Add status reports and post-upgrade options to HDFS - distributed upgrade. (Konstantin Shvachko via cutting) - -110. HADOOP-1524. Permit user task logs to appear as they're - created. (Michael Bieniosek via cutting) - -111. HADOOP-1599. Fix distcp bug on Windows. (Senthil Subramanian via cutting) - -112. HADOOP-1562. Add JVM metrics, including GC and logging stats. - (David Bowen via cutting) - -113. HADOOP-1613. Fix "DFS Health" page to display correct time of - last contact. (Dhruba Borthakur via cutting) - -114. HADOOP-1134. Add optimized checksum support to HDFS. Checksums - are now stored with each block, rather than as parallel files. - This reduces the namenode's memory requirements and increases - data integrity. (Raghu Angadi via cutting) - -115. HADOOP-1400. Make JobClient retry requests, so that clients can - survive jobtracker problems. (omalley via cutting) - -116. HADOOP-1564. Add unit tests for HDFS block-level checksums. - (Dhruba Borthakur via cutting) - -117. HADOOP-1620. Reduce the number of abstract FileSystem methods, - simplifying implementations. (cutting) - -118. HADOOP-1625. Fix a "could not move files" exception in datanode. - (Raghu Angadi via cutting) - -119. HADOOP-1624. Fix an infinite loop in datanode. (Raghu Angadi via cutting) - -120. HADOOP-1084. Switch mapred file cache to use file modification - time instead of checksum to detect file changes, as checksums are - no longer easily accessed. (Arun C Murthy via cutting) - -130. HADOOP-1623. Fix an infinite loop when copying directories. - (Dhruba Borthakur via cutting) - -131. HADOOP-1603. Fix a bug in namenode initialization where - default replication is sometimes reset to one on restart. - (Raghu Angadi via cutting) - -132. HADOOP-1635. Remove hardcoded keypair name and fix launch-hadoop-cluster - to support later versions of ec2-api-tools. (Stu Hood via tomwhite) - -133. HADOOP-1638. Fix contrib EC2 scripts to support NAT addressing. - (Stu Hood via tomwhite) - -134. HADOOP-1632. Fix an IllegalArgumentException in fsck. - (Hairong Kuang via cutting) - -135. HADOOP-1619. Fix FSInputChecker to not attempt to read past EOF. - (Hairong Kuang via cutting) - -136. HADOOP-1640. Fix TestDecommission on Windows. - (Dhruba Borthakur via cutting) - -137. HADOOP-1587. Fix TestSymLink to get required system properties. - (Devaraj Das via omalley) - -138. HADOOP-1628. Add block CRC protocol unit tests. (Raghu Angadi via omalley) - -139. HADOOP-1653. FSDirectory code-cleanups. FSDirectory.INode - becomes a static class. (Christophe Taton via dhruba) - -140. HADOOP-1066. Restructure documentation to make more user - friendly. (Connie Kleinjans and Jeff Hammerbacher via cutting) - -141. HADOOP-1551. libhdfs supports setting replication factor and - retrieving modification time of files. (Sameer Paranjpye via dhruba) - -141. HADOOP-1647. FileSystem.getFileStatus returns valid values for "/". - (Dhruba Borthakur via dhruba) - -142. HADOOP-1657. Fix NNBench to ensure that the block size is a - multiple of bytes.per.checksum. (Raghu Angadi via dhruba) - -143. HADOOP-1553. Replace user task output and log capture code to use shell - redirection instead of copier threads in the TaskTracker. Capping the - size of the output is now done via tail in memory and thus should not be - large. The output of the tasklog servlet is not forced into UTF8 and is - not buffered entirely in memory. (omalley) - Configuration changes to hadoop-default.xml: - remove mapred.userlog.num.splits - remove mapred.userlog.purge.splits - change default mapred.userlog.limit.kb to 0 (no limit) - change default mapred.userlog.retain.hours to 24 - Configuration changes to log4j.properties: - remove log4j.appender.TLA.noKeepSplits - remove log4j.appender.TLA.purgeLogSplits - remove log4j.appender.TLA.logsRetainHours - URL changes: - http:///tasklog.jsp -> http://tasklog with - parameters limited to start and end, which may be positive (from - start) or negative (from end). - Environment: - require bash (v2 or later) and tail - -144. HADOOP-1659. Fix a job id/job name mixup. (Arun C. Murthy via omalley) - -145. HADOOP-1665. With HDFS Trash enabled and the same file was created - and deleted more than once, the suceeding deletions creates Trash item - names suffixed with a integer. (Dhruba Borthakur via dhruba) - -146. HADOOP-1666. FsShell object can be used for multiple fs commands. - (Dhruba Borthakur via dhruba) - -147. HADOOP-1654. Remove performance regression introduced by Block CRC. - (Raghu Angadi via dhruba) - -148. HADOOP-1680. Improvements to Block CRC upgrade messages. - (Raghu Angadi via dhruba) - -149. HADOOP-71. Allow Text and SequenceFile Map/Reduce inputs from non-default - filesystems. (omalley) - -150. HADOOP-1568. Expose HDFS as xml/http filesystem to provide cross-version - compatability. (Chris Douglas via omalley) - -151. HADOOP-1668. Added an INCOMPATIBILITY section to CHANGES.txt. (nigel) - -152. HADOOP-1629. Added a upgrade test for HADOOP-1134. - (Raghu Angadi via nigel) - -153. HADOOP-1698. Fix performance problems on map output sorting for jobs - with large numbers of reduces. (Devaraj Das via omalley) - -154. HADOOP-1716. Fix a Pipes wordcount example to remove the 'file:' - schema from its output path. (omalley via cutting) - -155. HADOOP-1714. Fix TestDFSUpgradeFromImage to work on Windows. - (Raghu Angadi via nigel) - -156. HADOOP-1663. Return a non-zero exit code if streaming fails. (Lohit Renu - via omalley) - -157. HADOOP-1712. Fix an unhandled exception on datanode during block - CRC upgrade. (Raghu Angadi via cutting) - -158. HADOOP-1717. Fix TestDFSUpgradeFromImage to work on Solaris. - (nigel via cutting) - -159. HADOOP-1437. Add Eclipse plugin in contrib. - (Eugene Hung and Christophe Taton via cutting) - - -Release 0.13.0 - 2007-06-08 - - 1. HADOOP-1047. Fix TestReplication to succeed more reliably. - (Hairong Kuang via cutting) - - 2. HADOOP-1063. Fix a race condition in MiniDFSCluster test code. - (Hairong Kuang via cutting) - - 3. HADOOP-1101. In web ui, split shuffle statistics from reduce - statistics, and add some task averages. (Devaraj Das via cutting) - - 4. HADOOP-1071. Improve handling of protocol version mismatch in - JobTracker. (Tahir Hashmi via cutting) - - 5. HADOOP-1116. Increase heap size used for contrib unit tests. - (Philippe Gassmann via cutting) - - 6. HADOOP-1120. Add contrib/data_join, tools to simplify joining - data from multiple sources using MapReduce. (Runping Qi via cutting) - - 7. HADOOP-1064. Reduce log level of some DFSClient messages. - (Dhruba Borthakur via cutting) - - 8. HADOOP-1137. Fix StatusHttpServer to work correctly when - resources are in a jar file. (Benjamin Reed via cutting) - - 9. HADOOP-1094. Optimize generated Writable implementations for - records to not allocate a new BinaryOutputArchive or - BinaryInputArchive per call. (Milind Bhandarkar via cutting) - -10. HADOOP-1068. Improve error message for clusters with 0 datanodes. - (Dhruba Borthakur via tomwhite) - -11. HADOOP-1122. Fix divide-by-zero exception in FSNamesystem - chooseTarget method. (Dhruba Borthakur via tomwhite) - -12. HADOOP-1131. Add a closeAll() static method to FileSystem. - (Philippe Gassmann via tomwhite) - -13. HADOOP-1085. Improve port selection in HDFS and MapReduce test - code. Ports are now selected by the OS during testing rather than - by probing for free ports, improving test reliability. - (Arun C Murthy via cutting) - -14. HADOOP-1153. Fix HDFS daemons to correctly stop their threads. - (Konstantin Shvachko via cutting) - -15. HADOOP-1146. Add a counter for reduce input keys and rename the - "reduce input records" counter to be "reduce input groups". - (David Bowen via cutting) - -16. HADOOP-1165. In records, replace idential generated toString - methods with a method on the base class. (Milind Bhandarkar via cutting) - -17. HADOOP-1164. Fix TestReplicationPolicy to specify port zero, so - that a free port is automatically selected. (omalley via cutting) - -18. HADOOP-1166. Add a NullOutputFormat and use it in the - RandomWriter example. (omalley via cutting) - -19. HADOOP-1169. Fix a cut/paste error in CopyFiles utility so that - S3-based source files are correctly copied. (Michael Stack via cutting) - -20. HADOOP-1167. Remove extra synchronization in InMemoryFileSystem. - (omalley via cutting) - -21. HADOOP-1110. Fix an off-by-one error counting map inputs. - (David Bowen via cutting) - -22. HADOOP-1178. Fix a NullPointerException during namenode startup. - (Dhruba Borthakur via cutting) - -23. HADOOP-1011. Fix a ConcurrentModificationException when viewing - job history. (Tahir Hashmi via cutting) - -24. HADOOP-672. Improve help for fs shell commands. - (Dhruba Borthakur via cutting) - -25. HADOOP-1170. Improve datanode performance by removing device - checks from common operations. (Igor Bolotin via cutting) - -26. HADOOP-1090. Fix SortValidator's detection of whether the input - file belongs to the sort-input or sort-output directory. - (Arun C Murthy via tomwhite) - -27. HADOOP-1081. Fix bin/hadoop on Darwin. (Michael Bieniosek via cutting) - -28. HADOOP-1045. Add contrib/hbase, a BigTable-like online database. - (Jim Kellerman via cutting) - -29. HADOOP-1156. Fix a NullPointerException in MiniDFSCluster. - (Hairong Kuang via cutting) - -30. HADOOP-702. Add tools to help automate HDFS upgrades. - (Konstantin Shvachko via cutting) - -31. HADOOP-1163. Fix ganglia metrics to aggregate metrics from different - hosts properly. (Michael Bieniosek via tomwhite) - -32. HADOOP-1194. Make compression style record level for map output - compression. (Arun C Murthy via tomwhite) - -33. HADOOP-1187. Improve DFS Scalability: avoid scanning entire list of - datanodes in getAdditionalBlocks. (Dhruba Borthakur via tomwhite) - -34. HADOOP-1133. Add tool to analyze and debug namenode on a production - cluster. (Dhruba Borthakur via tomwhite) - -35. HADOOP-1151. Remove spurious printing to stderr in streaming - PipeMapRed. (Koji Noguchi via tomwhite) - -36. HADOOP-988. Change namenode to use a single map of blocks to metadata. - (Raghu Angadi via tomwhite) - -37. HADOOP-1203. Change UpgradeUtilities used by DFS tests to use - MiniDFSCluster to start and stop NameNode/DataNodes. - (Nigel Daley via tomwhite) - -38. HADOOP-1217. Add test.timeout property to build.xml, so that - long-running unit tests may be automatically terminated. - (Nigel Daley via cutting) - -39. HADOOP-1149. Improve DFS Scalability: make - processOverReplicatedBlock() a no-op if blocks are not - over-replicated. (Raghu Angadi via tomwhite) - -40. HADOOP-1149. Improve DFS Scalability: optimize getDistance(), - contains(), and isOnSameRack() in NetworkTopology. - (Hairong Kuang via tomwhite) - -41. HADOOP-1218. Make synchronization on TaskTracker's RunningJob - object consistent. (Devaraj Das via tomwhite) - -42. HADOOP-1219. Ignore progress report once a task has reported as - 'done'. (Devaraj Das via tomwhite) - -43. HADOOP-1114. Permit user to specify additional CLASSPATH elements - with a HADOOP_CLASSPATH environment variable. (cutting) - -44. HADOOP-1198. Remove ipc.client.timeout parameter override from - unit test configuration. Using the default is more robust and - has almost the same run time. (Arun C Murthy via tomwhite) - -45. HADOOP-1211. Remove deprecated constructor and unused static - members in DataNode class. (Konstantin Shvachko via tomwhite) - -46. HADOOP-1136. Fix ArrayIndexOutOfBoundsException in - FSNamesystem$UnderReplicatedBlocks add() method. - (Hairong Kuang via tomwhite) - -47. HADOOP-978. Add the client name and the address of the node that - previously started to create the file to the description of - AlreadyBeingCreatedException. (Konstantin Shvachko via tomwhite) - -48. HADOOP-1001. Check the type of keys and values generated by the - mapper against the types specified in JobConf. - (Tahir Hashmi via tomwhite) - -49. HADOOP-971. Improve DFS Scalability: Improve name node performance - by adding a hostname to datanodes map. (Hairong Kuang via tomwhite) - -50. HADOOP-1189. Fix 'No space left on device' exceptions on datanodes. - (Raghu Angadi via tomwhite) - -51. HADOOP-819. Change LineRecordWriter to not insert a tab between - key and value when either is null, and to print nothing when both - are null. (Runping Qi via cutting) - -52. HADOOP-1204. Rename InputFormatBase to be FileInputFormat, and - deprecate InputFormatBase. Also make LineRecordReader easier to - extend. (Runping Qi via cutting) - -53. HADOOP-1213. Improve logging of errors by IPC server, to - consistently include the service name and the call. (cutting) - -54. HADOOP-1238. Fix metrics reporting by TaskTracker to correctly - track maps_running and reduces_running. - (Michael Bieniosek via cutting) - -55. HADOOP-1093. Fix a race condition in HDFS where blocks were - sometimes erased before they were reported written. - (Dhruba Borthakur via cutting) - -56. HADOOP-1239. Add a package name to some testjar test classes. - (Jim Kellerman via cutting) - -57. HADOOP-1241. Fix NullPointerException in processReport when - namenode is restarted. (Dhruba Borthakur via tomwhite) - -58. HADOOP-1244. Fix stop-dfs.sh to no longer incorrectly specify - slaves file for stopping datanode. - (Michael Bieniosek via tomwhite) - -59. HADOOP-1253. Fix ConcurrentModificationException and - NullPointerException in JobControl. - (Johan Oskarson via tomwhite) - -60. HADOOP-1256. Fix NameNode so that multiple DataNodeDescriptors - can no longer be created on startup. (Hairong Kuang via cutting) - -61. HADOOP-1214. Replace streaming classes with new counterparts - from Hadoop core. (Runping Qi via tomwhite) - -62. HADOOP-1250. Move a chmod utility from streaming to FileUtil. - (omalley via cutting) - -63. HADOOP-1258. Fix TestCheckpoint test case to wait for - MiniDFSCluster to be active. (Nigel Daley via tomwhite) - -64. HADOOP-1148. Re-indent all Java source code to consistently use - two spaces per indent level. (cutting) - -65. HADOOP-1251. Add a method to Reporter to get the map InputSplit. - (omalley via cutting) - -66. HADOOP-1224. Fix "Browse the filesystem" link to no longer point - to dead datanodes. (Enis Soztutar via tomwhite) - -67. HADOOP-1154. Fail a streaming task if the threads reading from or - writing to the streaming process fail. (Koji Noguchi via tomwhite) - -68. HADOOP-968. Move shuffle and sort to run in reduce's child JVM, - rather than in TaskTracker. (Devaraj Das via cutting) - -69. HADOOP-1111. Add support for client notification of job - completion. If the job configuration has a job.end.notification.url - property it will make a HTTP GET request to the specified URL. - The number of retries and the interval between retries is also - configurable. (Alejandro Abdelnur via tomwhite) - -70. HADOOP-1275. Fix misspelled job notification property in - hadoop-default.xml. (Alejandro Abdelnur via tomwhite) - -71. HADOOP-1152. Fix race condition in MapOutputCopier.copyOutput file - rename causing possible reduce task hang. - (Tahir Hashmi via tomwhite) - -72. HADOOP-1050. Distinguish between failed and killed tasks so as to - not count a lost tasktracker against the job. - (Arun C Murthy via tomwhite) - -73. HADOOP-1271. Fix StreamBaseRecordReader to be able to log record - data that's not UTF-8. (Arun C Murthy via tomwhite) - -74. HADOOP-1190. Fix unchecked warnings in main Hadoop code. - (tomwhite) - -75. HADOOP-1127. Fix AlreadyBeingCreatedException in namenode for - jobs run with speculative execution. - (Arun C Murthy via tomwhite) - -76. HADOOP-1282. Omnibus HBase patch. Improved tests & configuration. - (Jim Kellerman via cutting) - -77. HADOOP-1262. Make dfs client try to read from a different replica - of the checksum file when a checksum error is detected. - (Hairong Kuang via tomwhite) - -78. HADOOP-1279. Fix JobTracker to maintain list of recently - completed jobs by order of completion, not submission. - (Arun C Murthy via cutting) - -79. HADOOP-1284. In contrib/streaming, permit flexible specification - of field delimiter and fields for partitioning and sorting. - (Runping Qi via cutting) - -80. HADOOP-1176. Fix a bug where reduce would hang when a map had - more than 2GB of output for it. (Arun C Murthy via cutting) - -81. HADOOP-1293. Fix contrib/streaming to print more than the first - twenty lines of standard error. (Koji Noguchi via cutting) - -82. HADOOP-1297. Fix datanode so that requests to remove blocks that - do not exist no longer causes block reports to be re-sent every - second. (Dhruba Borthakur via cutting) - -83. HADOOP-1216. Change MapReduce so that, when numReduceTasks is - zero, map outputs are written directly as final output, skipping - shuffle, sort and reduce. Use this to implement reduce=NONE - option in contrib/streaming. (Runping Qi via cutting) - -84. HADOOP-1294. Fix unchecked warnings in main Hadoop code under - Java 6. (tomwhite) - -85. HADOOP-1299. Fix so that RPC will restart after RPC.stopClient() - has been called. (Michael Stack via cutting) - -86. HADOOP-1278. Improve blacklisting of TaskTrackers by JobTracker, - to reduce false positives. (Arun C Murthy via cutting) - -87. HADOOP-1290. Move contrib/abacus into mapred/lib/aggregate. - (Runping Qi via cutting) - -88. HADOOP-1272. Extract inner classes from FSNamesystem into separate - classes. (Dhruba Borthakur via tomwhite) - -89. HADOOP-1247. Add support to contrib/streaming for aggregate - package, formerly called Abacus. (Runping Qi via cutting) - -90. HADOOP-1061. Fix bug in listing files in the S3 filesystem. - NOTE: this change is not backwards compatible! You should use the - MigrationTool supplied to migrate existing S3 filesystem data to - the new format. Please backup your data first before upgrading - (using 'hadoop distcp' for example). (tomwhite) - -91. HADOOP-1304. Make configurable the maximum number of task - attempts before a job fails. (Devaraj Das via cutting) - -92. HADOOP-1308. Use generics to restrict types when classes are - passed as parameters to JobConf methods. (Michael Bieniosek via cutting) - -93. HADOOP-1312. Fix a ConcurrentModificationException in NameNode - that killed the heartbeat monitoring thread. - (Dhruba Borthakur via cutting) - -94. HADOOP-1315. Clean up contrib/streaming, switching it to use core - classes more and removing unused code. (Runping Qi via cutting) - -95. HADOOP-485. Allow a different comparator for grouping keys in - calls to reduce. (Tahir Hashmi via cutting) - -96. HADOOP-1322. Fix TaskTracker blacklisting to work correctly in - one- and two-node clusters. (Arun C Murthy via cutting) - -97. HADOOP-1144. Permit one to specify a maximum percentage of tasks - that can fail before a job is aborted. The default is zero. - (Arun C Murthy via cutting) - -98. HADOOP-1184. Fix HDFS decomissioning to complete when the only - copy of a block is on a decommissioned node. (Dhruba Borthakur via cutting) - -99. HADOOP-1263. Change DFSClient to retry certain namenode calls - with a random, exponentially increasing backoff time, to avoid - overloading the namenode on, e.g., job start. (Hairong Kuang via cutting) - -100. HADOOP-1325. First complete, functioning version of HBase. - (Jim Kellerman via cutting) - -101. HADOOP-1276. Make tasktracker expiry interval configurable. - (Arun C Murthy via cutting) - -102. HADOOP-1326. Change JobClient#RunJob() to return the job. - (omalley via cutting) - -103. HADOOP-1270. Randomize the fetch of map outputs, speeding the - shuffle. (Arun C Murthy via cutting) - -104. HADOOP-1200. Restore disk checking lost in HADOOP-1170. - (Hairong Kuang via cutting) - -105. HADOOP-1252. Changed MapReduce's allocation of local files to - use round-robin among available devices, rather than a hashcode. - More care is also taken to not allocate files on full or offline - drives. (Devaraj Das via cutting) - -106. HADOOP-1324. Change so that an FSError kills only the task that - generates it rather than the entire task tracker. - (Arun C Murthy via cutting) - -107. HADOOP-1310. Fix unchecked warnings in aggregate code. (tomwhite) - -108. HADOOP-1255. Fix a bug where the namenode falls into an infinite - loop trying to remove a dead node. (Hairong Kuang via cutting) - -109. HADOOP-1160. Fix DistributedFileSystem.close() to close the - underlying FileSystem, correctly aborting files being written. - (Hairong Kuang via cutting) - -110. HADOOP-1341. Fix intermittent failures in HBase unit tests - caused by deadlock. (Jim Kellerman via cutting) - -111. HADOOP-1350. Fix shuffle performance problem caused by forcing - chunked encoding of map outputs. (Devaraj Das via cutting) - -112. HADOOP-1345. Fix HDFS to correctly retry another replica when a - checksum error is encountered. (Hairong Kuang via cutting) - -113. HADOOP-1205. Improve synchronization around HDFS block map. - (Hairong Kuang via cutting) - -114. HADOOP-1353. Fix a potential NullPointerException in namenode. - (Dhruba Borthakur via cutting) - -115. HADOOP-1354. Fix a potential NullPointerException in FsShell. - (Hairong Kuang via cutting) - -116. HADOOP-1358. Fix a potential bug when DFSClient calls skipBytes. - (Hairong Kuang via cutting) - -117. HADOOP-1356. Fix a bug in ValueHistogram. (Runping Qi via cutting) - -118. HADOOP-1363. Fix locking bug in JobClient#waitForCompletion(). - (omalley via cutting) - -119. HADOOP-1368. Fix inconsistent synchronization in JobInProgress. - (omalley via cutting) - -120. HADOOP-1369. Fix inconsistent synchronization in TaskTracker. - (omalley via cutting) - -121. HADOOP-1361. Fix various calls to skipBytes() to check return - value. (Hairong Kuang via cutting) - -122. HADOOP-1388. Fix a potential NullPointerException in web ui. - (Devaraj Das via cutting) - -123. HADOOP-1385. Fix MD5Hash#hashCode() to generally hash to more - than 256 values. (omalley via cutting) - -124. HADOOP-1386. Fix Path to not permit the empty string as a - path, as this has lead to accidental file deletion. Instead - force applications to use "." to name the default directory. - (Hairong Kuang via cutting) - -125. HADOOP-1407. Fix integer division bug in JobInProgress which - meant failed tasks didn't cause the job to fail. - (Arun C Murthy via tomwhite) - -126. HADOOP-1427. Fix a typo that caused GzipCodec to incorrectly use - a very small input buffer. (Espen Amble Kolstad via cutting) - -127. HADOOP-1435. Fix globbing code to no longer use the empty string - to indicate the default directory, per HADOOP-1386. - (Hairong Kuang via cutting) - -128. HADOOP-1411. Make task retry framework handle - AlreadyBeingCreatedException when wrapped as a RemoteException. - (Hairong Kuang via tomwhite) - -129. HADOOP-1242. Improve handling of DFS upgrades. - (Konstantin Shvachko via cutting) - -130. HADOOP-1332. Fix so that TaskTracker exits reliably during unit - tests on Windows. (omalley via cutting) - -131. HADOOP-1431. Fix so that sort progress reporting during map runs - only while sorting, so that stuck maps are correctly terminated. - (Devaraj Das and Arun C Murthy via cutting) - -132. HADOOP-1452. Change TaskTracker.MapOutputServlet.doGet.totalRead - to a long, permitting map outputs to exceed 2^31 bytes. - (omalley via cutting) - -133. HADOOP-1443. Fix a bug opening zero-length files in HDFS. - (Konstantin Shvachko via cutting) - - -Release 0.12.3 - 2007-04-06 - - 1. HADOOP-1162. Fix bug in record CSV and XML serialization of - binary values. (Milind Bhandarkar via cutting) - - 2. HADOOP-1123. Fix NullPointerException in LocalFileSystem when - trying to recover from a checksum error. - (Hairong Kuang & Nigel Daley via tomwhite) - - 3. HADOOP-1177. Fix bug where IOException in MapOutputLocation.getFile - was not being logged. (Devaraj Das via tomwhite) - - 4. HADOOP-1175. Fix bugs in JSP for displaying a task's log messages. - (Arun C Murthy via cutting) - - 5. HADOOP-1191. Fix map tasks to wait until sort progress thread has - stopped before reporting the task done. (Devaraj Das via cutting) - - 6. HADOOP-1192. Fix an integer overflow bug in FSShell's 'dus' - command and a performance problem in HDFS's implementation of it. - (Hairong Kuang via cutting) - - 7. HADOOP-1105. Fix reducers to make "progress" while iterating - through values. (Devaraj Das & Owen O'Malley via tomwhite) - - 8. HADOOP-1179. Make Task Tracker close index file as soon as the read - is done when serving get-map-output requests. - (Devaraj Das via tomwhite) - - -Release 0.12.2 - 2007-23-17 - - 1. HADOOP-1135. Fix bug in block report processing which may cause - the namenode to delete blocks. (Dhruba Borthakur via tomwhite) - - 2. HADOOP-1145. Make XML serializer and deserializer classes public - in record package. (Milind Bhandarkar via cutting) - - 3. HADOOP-1140. Fix a deadlock in metrics. (David Bowen via cutting) - - 4. HADOOP-1150. Fix streaming -reducer and -mapper to give them - defaults. (Owen O'Malley via tomwhite) - - -Release 0.12.1 - 2007-03-17 - - 1. HADOOP-1035. Fix a StackOverflowError in FSDataSet. - (Raghu Angadi via cutting) - - 2. HADOOP-1053. Fix VInt representation of negative values. Also - remove references in generated record code to methods outside of - the record package and improve some record documentation. - (Milind Bhandarkar via cutting) - - 3. HADOOP-1067. Compile fails if Checkstyle jar is present in lib - directory. Also remove dependency on a particular Checkstyle - version number. (tomwhite) - - 4. HADOOP-1060. Fix an IndexOutOfBoundsException in the JobTracker - that could cause jobs to hang. (Arun C Murthy via cutting) - - 5. HADOOP-1077. Fix a race condition fetching map outputs that could - hang reduces. (Devaraj Das via cutting) - - 6. HADOOP-1083. Fix so that when a cluster restarts with a missing - datanode, its blocks are replicated. (Hairong Kuang via cutting) - - 7. HADOOP-1082. Fix a NullPointerException in ChecksumFileSystem. - (Hairong Kuang via cutting) - - 8. HADOOP-1088. Fix record serialization of negative values. - (Milind Bhandarkar via cutting) - - 9. HADOOP-1080. Fix bug in bin/hadoop on Windows when native - libraries are present. (ab via cutting) - -10. HADOOP-1091. Fix a NullPointerException in MetricsRecord. - (David Bowen via tomwhite) - -11. HADOOP-1092. Fix a NullPointerException in HeartbeatMonitor - thread. (Hairong Kuang via tomwhite) - -12. HADOOP-1112. Fix a race condition in Hadoop metrics. - (David Bowen via tomwhite) - -13. HADOOP-1108. Checksummed file system should retry reading if a - different replica is found when handling ChecksumException. - (Hairong Kuang via tomwhite) - -14. HADOOP-1070. Fix a problem with number of racks and datanodes - temporarily doubling. (Konstantin Shvachko via tomwhite) - -15. HADOOP-1099. Fix NullPointerException in JobInProgress. - (Gautam Kowshik via tomwhite) - -16. HADOOP-1115. Fix bug where FsShell copyToLocal doesn't - copy directories. (Hairong Kuang via tomwhite) - -17. HADOOP-1109. Fix NullPointerException in StreamInputFormat. - (Koji Noguchi via tomwhite) - -18. HADOOP-1117. Fix DFS scalability: when the namenode is - restarted it consumes 80% CPU. (Dhruba Borthakur via - tomwhite) - -19. HADOOP-1089. Make the C++ version of write and read v-int - agree with the Java versions. (Milind Bhandarkar via - tomwhite) - -20. HADOOP-1096. Rename InputArchive and OutputArchive and - make them public. (Milind Bhandarkar via tomwhite) - -21. HADOOP-1128. Fix missing progress information in map tasks. - (Espen Amble Kolstad, Andrzej Bialecki, and Owen O'Malley - via tomwhite) - -22. HADOOP-1129. Fix DFSClient to not hide IOExceptions in - flush method. (Hairong Kuang via tomwhite) - -23. HADOOP-1126. Optimize CPU usage for under replicated blocks - when cluster restarts. (Hairong Kuang via tomwhite) - - -Release 0.12.0 - 2007-03-02 - - 1. HADOOP-975. Separate stdout and stderr from tasks. - (Arun C Murthy via cutting) - - 2. HADOOP-982. Add some setters and a toString() method to - BytesWritable. (omalley via cutting) - - 3. HADOOP-858. Move contrib/smallJobsBenchmark to src/test, removing - obsolete bits. (Nigel Daley via cutting) - - 4. HADOOP-992. Fix MiniMR unit tests to use MiniDFS when specified, - rather than the local FS. (omalley via cutting) - - 5. HADOOP-954. Change use of metrics to use callback mechanism. - Also rename utility class Metrics to MetricsUtil. - (David Bowen & Nigel Daley via cutting) - - 6. HADOOP-893. Improve HDFS client's handling of dead datanodes. - The set is no longer reset with each block, but rather is now - maintained for the life of an open file. (Raghu Angadi via cutting) - - 7. HADOOP-882. Upgrade to jets3t version 0.5, used by the S3 - FileSystem. This version supports retries. (Michael Stack via cutting) - - 8. HADOOP-977. Send task's stdout and stderr to JobClient's stdout - and stderr respectively, with each line tagged by the task's name. - (Arun C Murthy via cutting) - - 9. HADOOP-761. Change unit tests to not use /tmp. (Nigel Daley via cutting) - -10. HADOOP-1007. Make names of metrics used in Hadoop unique. - (Nigel Daley via cutting) - -11. HADOOP-491. Change mapred.task.timeout to be per-job, and make a - value of zero mean no timeout. Also change contrib/streaming to - disable task timeouts. (Arun C Murthy via cutting) - -12. HADOOP-1010. Add Reporter.NULL, a Reporter implementation that - does nothing. (Runping Qi via cutting) - -13. HADOOP-923. In HDFS NameNode, move replication computation to a - separate thread, to improve heartbeat processing time. - (Dhruba Borthakur via cutting) - -14. HADOOP-476. Rewrite contrib/streaming command-line processing, - improving parameter validation. (Sanjay Dahiya via cutting) - -15. HADOOP-973. Improve error messages in Namenode. This should help - to track down a problem that was appearing as a - NullPointerException. (Dhruba Borthakur via cutting) - -16. HADOOP-649. Fix so that jobs with no tasks are not lost. - (Thomas Friol via cutting) - -17. HADOOP-803. Reduce memory use by HDFS namenode, phase I. - (Raghu Angadi via cutting) - -18. HADOOP-1021. Fix MRCaching-based unit tests on Windows. - (Nigel Daley via cutting) - -19. HADOOP-889. Remove duplicate code from HDFS unit tests. - (Milind Bhandarkar via cutting) - -20. HADOOP-943. Improve HDFS's fsck command to display the filename - for under-replicated blocks. (Dhruba Borthakur via cutting) - -21. HADOOP-333. Add validator for sort benchmark output. - (Arun C Murthy via cutting) - -22. HADOOP-947. Improve performance of datanode decomissioning. - (Dhruba Borthakur via cutting) - -23. HADOOP-442. Permit one to specify hosts allowed to connect to - namenode and jobtracker with include and exclude files. (Wendy - Chien via cutting) - -24. HADOOP-1017. Cache constructors, for improved performance. - (Ron Bodkin via cutting) - -25. HADOOP-867. Move split creation out of JobTracker to client. - Splits are now saved in a separate file, read by task processes - directly, so that user code is no longer required in the - JobTracker. (omalley via cutting) - -26. HADOOP-1006. Remove obsolete '-local' option from test code. - (Gautam Kowshik via cutting) - -27. HADOOP-952. Create a public (shared) Hadoop EC2 AMI. - The EC2 scripts now support launch of public AMIs. - (tomwhite) - -28. HADOOP-1025. Remove some obsolete code in ipc.Server. (cutting) - -29. HADOOP-997. Implement S3 retry mechanism for failed block - transfers. This includes a generic retry mechanism for use - elsewhere in Hadoop. (tomwhite) - -30. HADOOP-990. Improve HDFS support for full datanode volumes. - (Raghu Angadi via cutting) - -31. HADOOP-564. Replace uses of "dfs://" URIs with the more standard - "hdfs://". (Wendy Chien via cutting) - -32. HADOOP-1030. In unit tests, unify setting of ipc.client.timeout. - Also increase the value used from one to two seconds, in hopes of - making tests complete more reliably. (cutting) - -33. HADOOP-654. Stop assigning tasks to a tasktracker if it has - failed more than a specified number in the job. - (Arun C Murthy via cutting) - -34. HADOOP-985. Change HDFS to identify nodes by IP address rather - than by DNS hostname. (Raghu Angadi via cutting) - -35. HADOOP-248. Optimize location of map outputs to not use random - probes. (Devaraj Das via cutting) - -36. HADOOP-1029. Fix streaming's input format to correctly seek to - the start of splits. (Arun C Murthy via cutting) - -37. HADOOP-492. Add per-job and per-task counters. These are - incremented via the Reporter interface and available through the - web ui and the JobClient API. The mapreduce framework maintains a - few basic counters, and applications may add their own. Counters - are also passed to the metrics system. - (David Bowen via cutting) - -38. HADOOP-1034. Fix datanode to better log exceptions. - (Philippe Gassmann via cutting) - -39. HADOOP-878. In contrib/streaming, fix reducer=NONE to work with - multiple maps. (Arun C Murthy via cutting) - -40. HADOOP-1039. In HDFS's TestCheckpoint, avoid restarting - MiniDFSCluster so often, speeding this test. (Dhruba Borthakur via cutting) - -41. HADOOP-1040. Update RandomWriter example to use counters and - user-defined input and output formats. (omalley via cutting) - -42. HADOOP-1027. Fix problems with in-memory merging during shuffle - and re-enable this optimization. (Devaraj Das via cutting) - -43. HADOOP-1036. Fix exception handling in TaskTracker to keep tasks - from being lost. (Arun C Murthy via cutting) - -44. HADOOP-1042. Improve the handling of failed map output fetches. - (Devaraj Das via cutting) - -45. HADOOP-928. Make checksums optional per FileSystem. - (Hairong Kuang via cutting) - -46. HADOOP-1044. Fix HDFS's TestDecommission to not spuriously fail. - (Wendy Chien via cutting) - -47. HADOOP-972. Optimize HDFS's rack-aware block placement algorithm. - (Hairong Kuang via cutting) - -48. HADOOP-1043. Optimize shuffle, increasing parallelism. - (Devaraj Das via cutting) - -49. HADOOP-940. Improve HDFS's replication scheduling. - (Dhruba Borthakur via cutting) - -50. HADOOP-1020. Fix a bug in Path resolution, and a with unit tests - on Windows. (cutting) - -51. HADOOP-941. Enhance record facility. - (Milind Bhandarkar via cutting) - -52. HADOOP-1000. Fix so that log messages in task subprocesses are - not written to a task's standard error. (Arun C Murthy via cutting) - -53. HADOOP-1037. Fix bin/slaves.sh, which currently only works with - /bin/bash, to specify /bin/bash rather than /bin/sh. (cutting) - -54. HADOOP-1046. Clean up tmp from partially received stale block files. (ab) - -55. HADOOP-1041. Optimize mapred counter implementation. Also group - counters by their declaring Enum. (David Bowen via cutting) - -56. HADOOP-1032. Permit one to specify jars that will be cached - across multiple jobs. (Gautam Kowshik via cutting) - -57. HADOOP-1051. Add optional checkstyle task to build.xml. To use - this developers must download the (LGPL'd) checkstyle jar - themselves. (tomwhite via cutting) - -58. HADOOP-1049. Fix a race condition in IPC client. - (Devaraj Das via cutting) - -60. HADOOP-1056. Check HDFS include/exclude node lists with both IP - address and hostname. (Wendy Chien via cutting) - -61. HADOOP-994. In HDFS, limit the number of blocks invalidated at - once. Large lists were causing datenodes to timeout. - (Dhruba Borthakur via cutting) - -62. HADOOP-432. Add a trash feature, disabled by default. When - enabled, the FSShell 'rm' command will move things to a trash - directory in the filesystem. In HDFS, a thread periodically - checkpoints the trash and removes old checkpoints. (cutting) - - -Release 0.11.2 - 2007-02-16 - - 1. HADOOP-1009. Fix an infinite loop in the HDFS namenode. - (Dhruba Borthakur via cutting) - - 2. HADOOP-1014. Disable in-memory merging during shuffle, as this is - causing data corruption. (Devaraj Das via cutting) - - -Release 0.11.1 - 2007-02-09 - - 1. HADOOP-976. Make SequenceFile.Metadata public. (Runping Qi via cutting) - - 2. HADOOP-917. Fix a NullPointerException in SequenceFile's merger - with large map outputs. (omalley via cutting) - - 3. HADOOP-984. Fix a bug in shuffle error handling introduced by - HADOOP-331. If a map output is unavailable, the job tracker is - once more informed. (Arun C Murthy via cutting) - - 4. HADOOP-987. Fix a problem in HDFS where blocks were not removed - from neededReplications after a replication target was selected. - (Hairong Kuang via cutting) - -Release 0.11.0 - 2007-02-02 - - 1. HADOOP-781. Remove methods deprecated in 0.10 that are no longer - widely used. (cutting) - - 2. HADOOP-842. Change HDFS protocol so that the open() method is - passed the client hostname, to permit the namenode to order block - locations on the basis of network topology. - (Hairong Kuang via cutting) - - 3. HADOOP-852. Add an ant task to compile record definitions, and - use it to compile record unit tests. (Milind Bhandarkar via cutting) - - 4. HADOOP-757. Fix "Bad File Descriptor" exception in HDFS client - when an output file is closed twice. (Raghu Angadi via cutting) - - 5. [ intentionally blank ] - - 6. HADOOP-890. Replace dashes in metric names with underscores, - for better compatibility with some monitoring systems. - (Nigel Daley via cutting) - - 7. HADOOP-801. Add to jobtracker a log of task completion events. - (Sanjay Dahiya via cutting) - - 8. HADOOP-855. In HDFS, try to repair files with checksum errors. - An exception is still thrown, but corrupt blocks are now removed - when they have replicas. (Wendy Chien via cutting) - - 9. HADOOP-886. Reduce number of timer threads created by metrics API - by pooling contexts. (Nigel Daley via cutting) - -10. HADOOP-897. Add a "javac.args" property to build.xml that permits - one to pass arbitrary options to javac. (Milind Bhandarkar via cutting) - -11. HADOOP-899. Update libhdfs for changes in HADOOP-871. - (Sameer Paranjpye via cutting) - -12. HADOOP-905. Remove some dead code from JobClient. (cutting) - -13. HADOOP-902. Fix a NullPointerException in HDFS client when - closing output streams. (Raghu Angadi via cutting) - -14. HADOOP-735. Switch generated record code to use BytesWritable to - represent fields of type 'buffer'. (Milind Bhandarkar via cutting) - -15. HADOOP-830. Improve mapreduce merge performance by buffering and - merging multiple map outputs as they arrive at reduce nodes before - they're written to disk. (Devaraj Das via cutting) - -16. HADOOP-908. Add a new contrib package, Abacus, that simplifies - counting and aggregation, built on MapReduce. (Runping Qi via cutting) - -17. HADOOP-901. Add support for recursive renaming to the S3 filesystem. - (Tom White via cutting) - -18. HADOOP-912. Fix a bug in TaskTracker.isIdle() that was - sporadically causing unit test failures. (Arun C Murthy via cutting) - -19. HADOOP-909. Fix the 'du' command to correctly compute the size of - FileSystem directory trees. (Hairong Kuang via cutting) - -20. HADOOP-731. When a checksum error is encountered on a file stored - in HDFS, try another replica of the data, if any. - (Wendy Chien via cutting) - -21. HADOOP-732. Add support to SequenceFile for arbitrary metadata, - as a set of attribute value pairs. (Runping Qi via cutting) - -22. HADOOP-929. Fix PhasedFileSystem to pass configuration to - underlying FileSystem. (Sanjay Dahiya via cutting) - -23. HADOOP-935. Fix contrib/abacus to not delete pre-existing output - files, but rather to fail in this case. (Runping Qi via cutting) - -24. HADOOP-936. More metric renamings, as in HADOOP-890. - (Nigel Daley via cutting) - -25. HADOOP-856. Fix HDFS's fsck command to not report that - non-existent filesystems are healthy. (Milind Bhandarkar via cutting) - -26. HADOOP-602. Remove the dependency on Lucene's PriorityQueue - utility, by copying it into Hadoop. This facilitates using Hadoop - with different versions of Lucene without worrying about CLASSPATH - order. (Milind Bhandarkar via cutting) - -27. [ intentionally blank ] - -28. HADOOP-227. Add support for backup namenodes, which periodically - get snapshots of the namenode state. (Dhruba Borthakur via cutting) - -29. HADOOP-884. Add scripts in contrib/ec2 to facilitate running - Hadoop on an Amazon's EC2 cluster. (Tom White via cutting) - -30. HADOOP-937. Change the namenode to request re-registration of - datanodes in more circumstances. (Hairong Kuang via cutting) - -31. HADOOP-922. Optimize small forward seeks in HDFS. If data is has - likely already in flight, skip ahead rather than re-opening the - block. (Dhruba Borthakur via cutting) - -32. HADOOP-961. Add a 'job -events' sub-command that prints job - events, including task completions and failures. (omalley via cutting) - -33. HADOOP-959. Fix namenode snapshot code added in HADOOP-227 to - work on Windows. (Dhruba Borthakur via cutting) - -34. HADOOP-934. Fix TaskTracker to catch metrics exceptions that were - causing heartbeats to fail. (Arun Murthy via cutting) - -35. HADOOP-881. Fix JobTracker web interface to display the correct - number of task failures. (Sanjay Dahiya via cutting) - -36. HADOOP-788. Change contrib/streaming to subclass TextInputFormat, - permitting it to take advantage of native compression facilities. - (Sanjay Dahiya via cutting) - -37. HADOOP-962. In contrib/ec2: make scripts executable in tar file; - add a README; make the environment file use a template. - (Tom White via cutting) - -38. HADOOP-549. Fix a NullPointerException in TaskReport's - serialization. (omalley via cutting) - -39. HADOOP-963. Fix remote exceptions to have the stack trace of the - caller thread, not the IPC listener thread. (omalley via cutting) - -40. HADOOP-967. Change RPC clients to start sending a version header. - (omalley via cutting) - -41. HADOOP-964. Fix a bug introduced by HADOOP-830 where jobs failed - whose comparators and/or i/o types were in the job's jar. - (Dennis Kubes via cutting) - -42. HADOOP-969. Fix a deadlock in JobTracker. (omalley via cutting) - -43. HADOOP-862. Add support for the S3 FileSystem to the CopyFiles - tool. (Michael Stack via cutting) - -44. HADOOP-965. Fix IsolationRunner so that job's jar can be found. - (Dennis Kubes via cutting) - -45. HADOOP-309. Fix two NullPointerExceptions in StatusHttpServer. - (navychen via cutting) - -46. HADOOP-692. Add rack awareness to HDFS's placement of blocks. - (Hairong Kuang via cutting) - - -Release 0.10.1 - 2007-01-10 - - 1. HADOOP-857. Fix S3 FileSystem implementation to permit its use - for MapReduce input and output. (Tom White via cutting) - - 2. HADOOP-863. Reduce logging verbosity introduced by HADOOP-813. - (Devaraj Das via cutting) - - 3. HADOOP-815. Fix memory leaks in JobTracker. (Arun C Murthy via cutting) - - 4. HADOOP-600. Fix a race condition in JobTracker. - (Arun C Murthy via cutting) - - 5. HADOOP-864. Fix 'bin/hadoop -jar' to operate correctly when - hadoop.tmp.dir does not yet exist. (omalley via cutting) - - 6. HADOOP-866. Fix 'dfs -get' command to remove existing crc files, - if any. (Milind Bhandarkar via cutting) - - 7. HADOOP-871. Fix a bug in bin/hadoop setting JAVA_LIBRARY_PATH. - (Arun C Murthy via cutting) - - 8. HADOOP-868. Decrease the number of open files during map, - respecting io.sort.fa ctor. (Devaraj Das via cutting) - - 9. HADOOP-865. Fix S3 FileSystem so that partially created files can - be deleted. (Tom White via cutting) - -10. HADOOP-873. Pass java.library.path correctly to child processes. - (omalley via cutting) - -11. HADOOP-851. Add support for the LZO codec. This is much faster - than the default, zlib-based compression, but it is only available - when the native library is built. (Arun C Murthy via cutting) - -12. HADOOP-880. Fix S3 FileSystem to remove directories. - (Tom White via cutting) - -13. HADOOP-879. Fix InputFormatBase to handle output generated by - MapFileOutputFormat. (cutting) - -14. HADOOP-659. In HDFS, prioritize replication of blocks based on - current replication level. Blocks which are severely - under-replicated should be further replicated before blocks which - are less under-replicated. (Hairong Kuang via cutting) - -15. HADOOP-726. Deprecate FileSystem locking methods. They are not - currently usable. Locking should eventually provided as an - independent service. (Raghu Angadi via cutting) - -16. HADOOP-758. Fix exception handling during reduce so that root - exceptions are not masked by exceptions in cleanups. - (Raghu Angadi via cutting) - - -Release 0.10.0 - 2007-01-05 - - 1. HADOOP-763. Change DFS namenode benchmark to not use MapReduce. - (Nigel Daley via cutting) - - 2. HADOOP-777. Use fully-qualified hostnames for tasktrackers and - datanodes. (Mahadev Konar via cutting) - - 3. HADOOP-621. Change 'dfs -cat' to exit sooner when output has been - closed. (Dhruba Borthakur via cutting) - - 4. HADOOP-752. Rationalize some synchronization in DFS namenode. - (Dhruba Borthakur via cutting) - - 5. HADOOP-629. Fix RPC services to better check the protocol name and - version. (omalley via cutting) - - 6. HADOOP-774. Limit the number of invalid blocks returned with - heartbeats by the namenode to datanodes. Transmitting and - processing very large invalid block lists can tie up both the - namenode and datanode for too long. (Dhruba Borthakur via cutting) - - 7. HADOOP-738. Change 'dfs -get' command to not create CRC files by - default, adding a -crc option to force their creation. - (Milind Bhandarkar via cutting) - - 8. HADOOP-676. Improved exceptions and error messages for common job - input specification errors. (Sanjay Dahiya via cutting) - - 9. [Included in 0.9.2 release] - -10. HADOOP-756. Add new dfsadmin option to wait for filesystem to be - operational. (Dhruba Borthakur via cutting) - -11. HADOOP-770. Fix jobtracker web interface to display, on restart, - jobs that were running when it was last stopped. - (Sanjay Dahiya via cutting) - -12. HADOOP-331. Write all map outputs to a single file with an index, - rather than to a separate file per reduce task. This should both - speed the shuffle and make things more scalable. - (Devaraj Das via cutting) - -13. HADOOP-818. Fix contrib unit tests to not depend on core unit - tests. (omalley via cutting) - -14. HADOOP-786. Log common exception at debug level. - (Sanjay Dahiya via cutting) - -15. HADOOP-796. Provide more convenient access to failed task - information in the web interface. (Sanjay Dahiya via cutting) - -16. HADOOP-764. Reduce memory allocations in namenode some. - (Dhruba Borthakur via cutting) - -17. HADOOP-802. Update description of mapred.speculative.execution to - mention reduces. (Nigel Daley via cutting) - -18. HADOOP-806. Include link to datanodes on front page of namenode - web interface. (Raghu Angadi via cutting) - -19. HADOOP-618. Make JobSubmissionProtocol public. - (Arun C Murthy via cutting) - -20. HADOOP-782. Fully remove killed tasks. (Arun C Murthy via cutting) - -21. HADOOP-792. Fix 'dfs -mv' to return correct status. - (Dhruba Borthakur via cutting) - -22. HADOOP-673. Give each task its own working directory again. - (Mahadev Konar via cutting) - -23. HADOOP-571. Extend the syntax of Path to be a URI; to be - optionally qualified with a scheme and authority. The scheme - determines the FileSystem implementation, while the authority - determines the FileSystem instance. New FileSystem - implementations may be provided by defining an fs..impl - property, naming the FileSystem implementation class. This - permits easy integration of new FileSystem implementations. - (cutting) - -24. HADOOP-720. Add an HDFS white paper to website. - (Dhruba Borthakur via cutting) - -25. HADOOP-794. Fix a divide-by-zero exception when a job specifies - zero map tasks. (omalley via cutting) - -26. HADOOP-454. Add a 'dfs -dus' command that provides summary disk - usage. (Hairong Kuang via cutting) - -27. HADOOP-574. Add an Amazon S3 implementation of FileSystem. To - use this, one need only specify paths of the form - s3://id:secret@bucket/. Alternately, the AWS access key id and - secret can be specified in your config, with the properties - fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey. - (Tom White via cutting) - -28. HADOOP-824. Rename DFSShell to be FsShell, since it applies - generically to all FileSystem implementations. (cutting) - -29. HADOOP-813. Fix map output sorting to report progress, so that - sorts which take longer than the task timeout do not fail. - (Devaraj Das via cutting) - -30. HADOOP-825. Fix HDFS daemons when configured with new URI syntax. - (omalley via cutting) - -31. HADOOP-596. Fix a bug in phase reporting during reduce. - (Sanjay Dahiya via cutting) - -32. HADOOP-811. Add a utility, MultithreadedMapRunner. - (Alejandro Abdelnur via cutting) - -33. HADOOP-829. Within HDFS, clearly separate three different - representations for datanodes: one for RPCs, one for - namenode-internal use, and one for namespace persistence. - (Dhruba Borthakur via cutting) - -34. HADOOP-823. Fix problem starting datanode when not all configured - data directories exist. (Bryan Pendleton via cutting) - -35. HADOOP-451. Add a Split interface. CAUTION: This incompatibly - changes the InputFormat and RecordReader interfaces. Not only is - FileSplit replaced with Split, but a FileSystem parameter is no - longer passed in several methods, input validation has changed, - etc. (omalley via cutting) - -36. HADOOP-814. Optimize locking in namenode. (Dhruba Borthakur via cutting) - -37. HADOOP-738. Change 'fs -put' and 'fs -get' commands to accept - standard input and output, respectively. Standard i/o is - specified by a file named '-'. (Wendy Chien via cutting) - -38. HADOOP-835. Fix a NullPointerException reading record-compressed - SequenceFiles. (Hairong Kuang via cutting) - -39. HADOOP-836. Fix a MapReduce bug on Windows, where the wrong - FileSystem was used. Also add a static FileSystem.getLocal() - method and better Path checking in HDFS, to help avoid such issues - in the future. (omalley via cutting) - -40. HADOOP-837. Improve RunJar utility to unpack jar file - hadoop.tmp.dir, rather than the system temporary directory. - (Hairong Kuang via cutting) - -41. HADOOP-841. Fix native library to build 32-bit version even when - on a 64-bit host, if a 32-bit JVM is used. (Arun C Murthy via cutting) - -42. HADOOP-838. Fix tasktracker to pass java.library.path to - sub-processes, so that libhadoop.a is found. - (Arun C Murthy via cutting) - -43. HADOOP-844. Send metrics messages on a fixed-delay schedule - instead of a fixed-rate schedule. (David Bowen via cutting) - -44. HADOOP-849. Fix OutOfMemory exceptions in TaskTracker due to a - file handle leak in SequenceFile. (Devaraj Das via cutting) - -45. HADOOP-745. Fix a synchronization bug in the HDFS namenode. - (Dhruba Borthakur via cutting) - -46. HADOOP-850. Add Writable implementations for variable-length - integers. (ab via cutting) - -47. HADOOP-525. Add raw comparators to record types. This greatly - improves record sort performance. (Milind Bhandarkar via cutting) - -48. HADOOP-628. Fix a problem with 'fs -cat' command, where some - characters were replaced with question marks. (Wendy Chien via cutting) - -49. HADOOP-804. Reduce verbosity of MapReduce logging. - (Sanjay Dahiya via cutting) - -50. HADOOP-853. Rename 'site' to 'docs', in preparation for inclusion - in releases. (cutting) - -51. HADOOP-371. Include contrib jars and site documentation in - distributions. Also add contrib and example documentation to - distributed javadoc, in separate sections. (Nigel Daley via cutting) - -52. HADOOP-846. Report progress during entire map, as sorting of - intermediate outputs may happen at any time, potentially causing - task timeouts. (Devaraj Das via cutting) - -53. HADOOP-840. In task tracker, queue task cleanups and perform them - in a separate thread. (omalley & Mahadev Konar via cutting) - -54. HADOOP-681. Add to HDFS the ability to decommission nodes. This - causes their blocks to be re-replicated on other nodes, so that - they may be removed from a cluster. (Dhruba Borthakur via cutting) - -55. HADOOP-470. In HDFS web ui, list the datanodes containing each - copy of a block. (Hairong Kuang via cutting) - -56. HADOOP-700. Change bin/hadoop to only include core jar file on - classpath, not example, test, etc. Also rename core jar to - hadoop-${version}-core.jar so that it can be more easily - identified. (Nigel Daley via cutting) - -57. HADOOP-619. Extend InputFormatBase to accept individual files and - glob patterns as MapReduce inputs, not just directories. Also - change contrib/streaming to use this. (Sanjay Dahia via cutting) - - -Release 0.9.2 - 2006-12-15 - - 1. HADOOP-639. Restructure InterTrackerProtocol to make task - accounting more reliable. (Arun C Murthy via cutting) - - 2. HADOOP-827. Turn off speculative execution by default, since it's - currently broken. (omalley via cutting) - - 3. HADOOP-791. Fix a deadlock in the task tracker. - (Mahadev Konar via cutting) - - -Release 0.9.1 - 2006-12-06 - - 1. HADOOP-780. Use ReflectionUtils to instantiate key and value - objects. (ab) - - 2. HADOOP-779. Fix contrib/streaming to work correctly with gzipped - input files. (Hairong Kuang via cutting) - - -Release 0.9.0 - 2006-12-01 - - 1. HADOOP-655. Remove most deprecated code. A few deprecated things - remain, notably UTF8 and some methods that are still required. - Also cleaned up constructors for SequenceFile, MapFile, SetFile, - and ArrayFile a bit. (cutting) - - 2. HADOOP-565. Upgrade to Jetty version 6. (Sanjay Dahiya via cutting) - - 3. HADOOP-682. Fix DFS format command to work correctly when - configured with a non-existent directory. (Sanjay Dahiya via cutting) - - 4. HADOOP-645. Fix a bug in contrib/streaming when -reducer is NONE. - (Dhruba Borthakur via cutting) - - 5. HADOOP-687. Fix a classpath bug in bin/hadoop that blocked the - servers from starting. (Sameer Paranjpye via omalley) - - 6. HADOOP-683. Remove a script dependency on bash, so it works with - dash, the new default for /bin/sh on Ubuntu. (James Todd via cutting) - - 7. HADOOP-382. Extend unit tests to run multiple datanodes. - (Milind Bhandarkar via cutting) - - 8. HADOOP-604. Fix some synchronization issues and a - NullPointerException in DFS datanode. (Raghu Angadi via cutting) - - 9. HADOOP-459. Fix memory leaks and a host of other issues with - libhdfs. (Sameer Paranjpye via cutting) - -10. HADOOP-694. Fix a NullPointerException in jobtracker. - (Mahadev Konar via cutting) - -11. HADOOP-637. Fix a memory leak in the IPC server. Direct buffers - are not collected like normal buffers, and provided little - advantage. (Raghu Angadi via cutting) - -12. HADOOP-696. Fix TestTextInputFormat unit test to not rely on the - order of directory listings. (Sameer Paranjpye via cutting) - -13. HADOOP-611. Add support for iterator-based merging to - SequenceFile. (Devaraj Das via cutting) - -14. HADOOP-688. Move DFS administrative commands to a separate - command named 'dfsadmin'. (Dhruba Borthakur via cutting) - -15. HADOOP-708. Fix test-libhdfs to return the correct status, so - that failures will break the build. (Nigel Daley via cutting) - -16. HADOOP-646. Fix namenode to handle edits files larger than 2GB. - (Milind Bhandarkar via cutting) - -17. HADOOP-705. Fix a bug in the JobTracker when failed jobs were - not completely cleaned up. (Mahadev Konar via cutting) - -18. HADOOP-613. Perform final merge while reducing. This removes one - sort pass over the data and should consequently significantly - decrease overall processing time. (Devaraj Das via cutting) - -19. HADOOP-661. Make each job's configuration visible through the web - ui. (Arun C Murthy via cutting) - -20. HADOOP-489. In MapReduce, separate user logs from system logs. - Each task's log output is now available through the web ui. (Arun - C Murthy via cutting) - -21. HADOOP-712. Fix record io's xml serialization to correctly handle - control-characters. (Milind Bhandarkar via cutting) - -22. HADOOP-668. Improvements to the web-based DFS browser. - (Hairong Kuang via cutting) - -23. HADOOP-715. Fix build.xml so that test logs are written in build - directory, rather than in CWD. (Arun C Murthy via cutting) - -24. HADOOP-538. Add support for building an optional native library, - libhadoop.so, that improves the performance of zlib-based - compression. To build this, specify -Dcompile.native to Ant. - (Arun C Murthy via cutting) - -25. HADOOP-610. Fix an problem when the DFS block size is configured - to be smaller than the buffer size, typically only when debugging. - (Milind Bhandarkar via cutting) - -26. HADOOP-695. Fix a NullPointerException in contrib/streaming. - (Hairong Kuang via cutting) - -27. HADOOP-652. In DFS, when a file is deleted, the block count is - now decremented. (Vladimir Krokhmalyov via cutting) - -28. HADOOP-725. In DFS, optimize block placement algorithm, - previously a performance bottleneck. (Milind Bhandarkar via cutting) - -29. HADOOP-723. In MapReduce, fix a race condition during the - shuffle, which resulted in FileNotFoundExceptions. (omalley via cutting) - -30. HADOOP-447. In DFS, fix getBlockSize(Path) to work with relative - paths. (Raghu Angadi via cutting) - -31. HADOOP-733. Make exit codes in DFShell consistent and add a unit - test. (Dhruba Borthakur via cutting) - -32. HADOOP-709. Fix contrib/streaming to work with commands that - contain control characters. (Dhruba Borthakur via cutting) - -33. HADOOP-677. In IPC, permit a version header to be transmitted - when connections are established. This will permit us to change - the format of IPC requests back-compatibly in subsequent releases. - (omalley via cutting) - -34. HADOOP-699. Fix DFS web interface so that filesystem browsing - works correctly, using the right port number. Also add support - for sorting datanode list by various columns. - (Raghu Angadi via cutting) - -35. HADOOP-76. Implement speculative reduce. Now when a job is - configured for speculative execution, both maps and reduces will - execute speculatively. Reduce outputs are written to temporary - location and moved to the final location when reduce is complete. - (Sanjay Dahiya via cutting) - -36. HADOOP-736. Roll back to Jetty 5.1.4, due to performance problems - with Jetty 6.0.1. - -37. HADOOP-739. Fix TestIPC to use different port number, making it - more reliable. (Nigel Daley via cutting) - -38. HADOOP-749. Fix a NullPointerException in jobfailures.jsp. - (omalley via cutting) - -39. HADOOP-747. Fix record serialization to work correctly when - records are embedded in Maps. (Milind Bhandarkar via cutting) - -40. HADOOP-698. Fix HDFS client not to retry the same datanode on - read failures. (Milind Bhandarkar via cutting) - -41. HADOOP-689. Add GenericWritable, to facilitate polymorphism in - MapReduce, SequenceFile, etc. (Feng Jiang via cutting) - -42. HADOOP-430. Stop datanode's HTTP server when registration with - namenode fails. (Wendy Chien via cutting) - -43. HADOOP-750. Fix a potential race condition during mapreduce - shuffle. (omalley via cutting) - -44. HADOOP-728. Fix contrib/streaming-related issues, including - '-reducer NONE'. (Sanjay Dahiya via cutting) - - -Release 0.8.0 - 2006-11-03 - - 1. HADOOP-477. Extend contrib/streaming to scan the PATH environment - variables when resolving executable program names. - (Dhruba Borthakur via cutting) - - 2. HADOOP-583. In DFSClient, reduce the log level of re-connect - attempts from 'info' to 'debug', so they are not normally shown. - (Konstantin Shvachko via cutting) - - 3. HADOOP-498. Re-implement DFS integrity checker to run server-side, - for much improved performance. (Milind Bhandarkar via cutting) - - 4. HADOOP-586. Use the jar name for otherwise un-named jobs. - (Sanjay Dahiya via cutting) - - 5. HADOOP-514. Make DFS heartbeat interval configurable. - (Milind Bhandarkar via cutting) - - 6. HADOOP-588. Fix logging and accounting of failed tasks. - (Sanjay Dahiya via cutting) - - 7. HADOOP-462. Improve command line parsing in DFSShell, so that - incorrect numbers of arguments result in informative errors rather - than ArrayOutOfBoundsException. (Dhruba Borthakur via cutting) - - 8. HADOOP-561. Fix DFS so that one replica of each block is written - locally, if possible. This was the intent, but there as a bug. - (Dhruba Borthakur via cutting) - - 9. HADOOP-610. Fix TaskTracker to survive more exceptions, keeping - tasks from becoming lost. (omalley via cutting) - -10. HADOOP-625. Add a servlet to all http daemons that displays a - stack dump, useful for debugging. (omalley via cutting) - -11. HADOOP-554. Fix DFSShell to return -1 for errors. - (Dhruba Borthakur via cutting) - -12. HADOOP-626. Correct the documentation in the NNBench example - code, and also remove a mistaken call there. - (Nigel Daley via cutting) - -13. HADOOP-634. Add missing license to many files. - (Nigel Daley via cutting) - -14. HADOOP-627. Fix some synchronization problems in MiniMRCluster - that sometimes caused unit tests to fail. (Nigel Daley via cutting) - -15. HADOOP-563. Improve the NameNode's lease policy so that leases - are held for one hour without renewal (instead of one minute). - However another attempt to create the same file will still succeed - if the lease has not been renewed within a minute. This prevents - communication or scheduling problems from causing a write to fail - for up to an hour, barring some other process trying to create the - same file. (Dhruba Borthakur via cutting) - -16. HADOOP-635. In DFSShell, permit specification of multiple files - as the source for file copy and move commands. - (Dhruba Borthakur via cutting) - -17. HADOOP-641. Change NameNode to request a fresh block report from - a re-discovered DataNode, so that no-longer-needed replications - are stopped promptly. (Konstantin Shvachko via cutting) - -18. HADOOP-642. Change IPC client to specify an explicit connect - timeout. (Konstantin Shvachko via cutting) - -19. HADOOP-638. Fix an unsynchronized access to TaskTracker's - internal state. (Nigel Daley via cutting) - -20. HADOOP-624. Fix servlet path to stop a Jetty warning on startup. - (omalley via cutting) - -21. HADOOP-578. Failed tasks are no longer placed at the end of the - task queue. This was originally done to work around other - problems that have now been fixed. Re-executing failed tasks - sooner causes buggy jobs to fail faster. (Sanjay Dahiya via cutting) - -22. HADOOP-658. Update source file headers per Apache policy. (cutting) - -23. HADOOP-636. Add MapFile & ArrayFile constructors which accept a - Progressable, and pass it down to SequenceFile. This permits - reduce tasks which use MapFile to still report progress while - writing blocks to the filesystem. (cutting) - -24. HADOOP-576. Enable contrib/streaming to use the file cache. Also - extend the cache to permit symbolic links to cached items, rather - than local file copies. (Mahadev Konar via cutting) - -25. HADOOP-482. Fix unit tests to work when a cluster is running on - the same machine, removing port conflicts. (Wendy Chien via cutting) - -26. HADOOP-90. Permit dfs.name.dir to list multiple directories, - where namenode data is to be replicated. (Milind Bhandarkar via cutting) - -27. HADOOP-651. Fix DFSCk to correctly pass parameters to the servlet - on the namenode. (Milind Bhandarkar via cutting) - -28. HADOOP-553. Change main() routines of DataNode and NameNode to - log exceptions rather than letting the JVM print them to standard - error. Also, change the hadoop-daemon.sh script to rotate - standard i/o log files. (Raghu Angadi via cutting) - -29. HADOOP-399. Fix javadoc warnings. (Nigel Daley via cutting) - -30. HADOOP-599. Fix web ui and command line to correctly report DFS - filesystem size statistics. Also improve web layout. - (Raghu Angadi via cutting) - -31. HADOOP-660. Permit specification of junit test output format. - (Nigel Daley via cutting) - -32. HADOOP-663. Fix a few unit test issues. (Mahadev Konar via cutting) - -33. HADOOP-664. Cause entire build to fail if libhdfs tests fail. - (Nigel Daley via cutting) - -34. HADOOP-633. Keep jobtracker from dying when job initialization - throws exceptions. Also improve exception handling in a few other - places and add more informative thread names. - (omalley via cutting) - -35. HADOOP-669. Fix a problem introduced by HADOOP-90 that can cause - DFS to lose files. (Milind Bhandarkar via cutting) - -36. HADOOP-373. Consistently check the value returned by - FileSystem.mkdirs(). (Wendy Chien via cutting) - -37. HADOOP-670. Code cleanups in some DFS internals: use generic - types, replace Vector with ArrayList, etc. - (Konstantin Shvachko via cutting) - -38. HADOOP-647. Permit map outputs to use a different compression - type than the job output. (omalley via cutting) - -39. HADOOP-671. Fix file cache to check for pre-existence before - creating . (Mahadev Konar via cutting) - -40. HADOOP-665. Extend many DFSShell commands to accept multiple - arguments. Now commands like "ls", "rm", etc. will operate on - multiple files. (Dhruba Borthakur via cutting) - - -Release 0.7.2 - 2006-10-18 - - 1. HADOOP-607. Fix a bug where classes included in job jars were not - found by tasks. (Mahadev Konar via cutting) - - 2. HADOOP-609. Add a unit test that checks that classes in job jars - can be found by tasks. Also modify unit tests to specify multiple - local directories. (Mahadev Konar via cutting) - - -Release 0.7.1 - 2006-10-11 - - 1. HADOOP-593. Fix a NullPointerException in the JobTracker. - (omalley via cutting) - - 2. HADOOP-592. Fix a NullPointerException in the IPC Server. Also - consistently log when stale calls are discarded. (omalley via cutting) - - 3. HADOOP-594. Increase the DFS safe-mode threshold from .95 to - .999, so that nearly all blocks must be reported before filesystem - modifications are permitted. (Konstantin Shvachko via cutting) - - 4. HADOOP-598. Fix tasks to retry when reporting completion, so that - a single RPC timeout won't fail a task. (omalley via cutting) - - 5. HADOOP-597. Fix TaskTracker to not discard map outputs for errors - in transmitting them to reduce nodes. (omalley via cutting) - - -Release 0.7.0 - 2006-10-06 - - 1. HADOOP-243. Fix rounding in the display of task and job progress - so that things are not shown to be 100% complete until they are in - fact finished. (omalley via cutting) - - 2. HADOOP-438. Limit the length of absolute paths in DFS, since the - file format used to store pathnames has some limitations. - (Wendy Chien via cutting) - - 3. HADOOP-530. Improve error messages in SequenceFile when keys or - values are of the wrong type. (Hairong Kuang via cutting) - - 4. HADOOP-288. Add a file caching system and use it in MapReduce to - cache job jar files on slave nodes. (Mahadev Konar via cutting) - - 5. HADOOP-533. Fix unit test to not modify conf directory. - (Hairong Kuang via cutting) - - 6. HADOOP-527. Permit specification of the local address that various - Hadoop daemons should bind to. (Philippe Gassmann via cutting) - - 7. HADOOP-542. Updates to contrib/streaming: reformatted source code, - on-the-fly merge sort, a fix for HADOOP-540, etc. - (Michel Tourn via cutting) - - 8. HADOOP-545. Remove an unused config file parameter. - (Philippe Gassmann via cutting) - - 9. HADOOP-548. Add an Ant property "test.output" to build.xml that - causes test output to be logged to the console. (omalley via cutting) - -10. HADOOP-261. Record an error message when map output is lost. - (omalley via cutting) - -11. HADOOP-293. Report the full list of task error messages in the - web ui, not just the most recent. (omalley via cutting) - -12. HADOOP-551. Restore JobClient's console printouts to only include - a maximum of one update per one percent of progress. - (omalley via cutting) - -13. HADOOP-306. Add a "safe" mode to DFS. The name node enters this - when less than a specified percentage of file data is complete. - Currently safe mode is only used on startup, but eventually it - will also be entered when datanodes disconnect and file data - becomes incomplete. While in safe mode no filesystem - modifications are permitted and block replication is inhibited. - (Konstantin Shvachko via cutting) - -14. HADOOP-431. Change 'dfs -rm' to not operate recursively and add a - new command, 'dfs -rmr' which operates recursively. - (Sameer Paranjpye via cutting) - -15. HADOOP-263. Include timestamps for job transitions. The web - interface now displays the start and end times of tasks and the - start times of sorting and reducing for reduce tasks. Also, - extend ObjectWritable to handle enums, so that they can be passed - as RPC parameters. (Sanjay Dahiya via cutting) - -16. HADOOP-556. Contrib/streaming: send keep-alive reports to task - tracker every 10 seconds rather than every 100 records, to avoid - task timeouts. (Michel Tourn via cutting) - -17. HADOOP-547. Fix reduce tasks to ping tasktracker while copying - data, rather than only between copies, avoiding task timeouts. - (Sanjay Dahiya via cutting) - -18. HADOOP-537. Fix src/c++/libhdfs build process to create files in - build/, no longer modifying the source tree. - (Arun C Murthy via cutting) - -19. HADOOP-487. Throw a more informative exception for unknown RPC - hosts. (Sameer Paranjpye via cutting) - -20. HADOOP-559. Add file name globbing (pattern matching) support to - the FileSystem API, and use it in DFSShell ('bin/hadoop dfs') - commands. (Hairong Kuang via cutting) - -21. HADOOP-508. Fix a bug in FSDataInputStream. Incorrect data was - returned after seeking to a random location. - (Milind Bhandarkar via cutting) - -22. HADOOP-560. Add a "killed" task state. This can be used to - distinguish kills from other failures. Task state has also been - converted to use an enum type instead of an int, uncovering a bug - elsewhere. The web interface is also updated to display killed - tasks. (omalley via cutting) - -23. HADOOP-423. Normalize Paths containing directories named "." and - "..", using the standard, unix interpretation. Also add checks in - DFS, prohibiting the use of "." or ".." as directory or file - names. (Wendy Chien via cutting) - -24. HADOOP-513. Replace map output handling with a servlet, rather - than a JSP page. This fixes an issue where - IllegalStateException's were logged, sets content-length - correctly, and better handles some errors. (omalley via cutting) - -25. HADOOP-552. Improved error checking when copying map output files - to reduce nodes. (omalley via cutting) - -26. HADOOP-566. Fix scripts to work correctly when accessed through - relative symbolic links. (Lee Faris via cutting) - -27. HADOOP-519. Add positioned read methods to FSInputStream. These - permit one to read from a stream without moving its position, and - can hence be performed by multiple threads at once on a single - stream. Implement an optimized version for DFS and local FS. - (Milind Bhandarkar via cutting) - -28. HADOOP-522. Permit block compression with MapFile and SetFile. - Since these formats are always sorted, block compression can - provide a big advantage. (cutting) - -29. HADOOP-567. Record version and revision information in builds. A - package manifest is added to the generated jar file containing - version information, and a VersionInfo utility is added that - includes further information, including the build date and user, - and the subversion revision and repository. A 'bin/hadoop - version' comand is added to show this information, and it is also - added to various web interfaces. (omalley via cutting) - -30. HADOOP-568. Fix so that errors while initializing tasks on a - tasktracker correctly report the task as failed to the jobtracker, - so that it will be rescheduled. (omalley via cutting) - -31. HADOOP-550. Disable automatic UTF-8 validation in Text. This - permits, e.g., TextInputFormat to again operate on non-UTF-8 data. - (Hairong and Mahadev via cutting) - -32. HADOOP-343. Fix mapred copying so that a failed tasktracker - doesn't cause other copies to slow. (Sameer Paranjpye via cutting) - -33. HADOOP-239. Add a persistent job history mechanism, so that basic - job statistics are not lost after 24 hours and/or when the - jobtracker is restarted. (Sanjay Dahiya via cutting) - -34. HADOOP-506. Ignore heartbeats from stale task trackers. - (Sanjay Dahiya via cutting) - -35. HADOOP-255. Discard stale, queued IPC calls. Do not process - calls whose clients will likely time out before they receive a - response. When the queue is full, new calls are now received and - queued, and the oldest calls are discarded, so that, when servers - get bogged down, they no longer develop a backlog on the socket. - This should improve some DFS namenode failure modes. - (omalley via cutting) - -36. HADOOP-581. Fix datanode to not reset itself on communications - errors with the namenode. If a request to the namenode fails, the - datanode should retry, not restart. This reduces the load on the - namenode, since restarts cause a resend of the block report. - (omalley via cutting) - - -Release 0.6.2 - 2006-09-18 - -1. HADOOP-532. Fix a bug reading value-compressed sequence files, - where an exception was thrown reporting that the full value had not - been read. (omalley via cutting) - -2. HADOOP-534. Change the default value class in JobConf to be Text - instead of the now-deprecated UTF8. This fixes the Grep example - program, which was updated to use Text, but relies on this - default. (Hairong Kuang via cutting) - - -Release 0.6.1 - 2006-09-13 - - 1. HADOOP-520. Fix a bug in libhdfs, where write failures were not - correctly returning error codes. (Arun C Murthy via cutting) - - 2. HADOOP-523. Fix a NullPointerException when TextInputFormat is - explicitly specified. Also add a test case for this. - (omalley via cutting) - - 3. HADOOP-521. Fix another NullPointerException finding the - ClassLoader when using libhdfs. (omalley via cutting) - - 4. HADOOP-526. Fix a NullPointerException when attempting to start - two datanodes in the same directory. (Milind Bhandarkar via cutting) - - 5. HADOOP-529. Fix a NullPointerException when opening - value-compressed sequence files generated by pre-0.6.0 Hadoop. - (omalley via cutting) - - -Release 0.6.0 - 2006-09-08 - - 1. HADOOP-427. Replace some uses of DatanodeDescriptor in the DFS - web UI code with DatanodeInfo, the preferred public class. - (Devaraj Das via cutting) - - 2. HADOOP-426. Fix streaming contrib module to work correctly on - Solaris. This was causing nightly builds to fail. - (Michel Tourn via cutting) - - 3. HADOOP-400. Improvements to task assignment. Tasks are no longer - re-run on nodes where they have failed (unless no other node is - available). Also, tasks are better load-balanced among nodes. - (omalley via cutting) - - 4. HADOOP-324. Fix datanode to not exit when a disk is full, but - rather simply to fail writes. (Wendy Chien via cutting) - - 5. HADOOP-434. Change smallJobsBenchmark to use standard Hadoop - scripts. (Sanjay Dahiya via cutting) - - 6. HADOOP-453. Fix a bug in Text.setCapacity(). (siren via cutting) - - - 7. HADOOP-450. Change so that input types are determined by the - RecordReader rather than specified directly in the JobConf. This - facilitates jobs with a variety of input types. - - WARNING: This contains incompatible API changes! The RecordReader - interface has two new methods that all user-defined InputFormats - must now define. Also, the values returned by TextInputFormat are - no longer of class UTF8, but now of class Text. - - 8. HADOOP-436. Fix an error-handling bug in the web ui. - (Devaraj Das via cutting) - - 9. HADOOP-455. Fix a bug in Text, where DEL was not permitted. - (Hairong Kuang via cutting) - -10. HADOOP-456. Change the DFS namenode to keep a persistent record - of the set of known datanodes. This will be used to implement a - "safe mode" where filesystem changes are prohibited when a - critical percentage of the datanodes are unavailable. - (Konstantin Shvachko via cutting) - -11. HADOOP-322. Add a job control utility. This permits one to - specify job interdependencies. Each job is submitted only after - the jobs it depends on have successfully completed. - (Runping Qi via cutting) - -12. HADOOP-176. Fix a bug in IntWritable.Comparator. - (Dick King via cutting) - -13. HADOOP-421. Replace uses of String in recordio package with Text - class, for improved handling of UTF-8 data. - (Milind Bhandarkar via cutting) - -14. HADOOP-464. Improved error message when job jar not found. - (Michel Tourn via cutting) - -15. HADOOP-469. Fix /bin/bash specifics that have crept into our - /bin/sh scripts since HADOOP-352. - (Jean-Baptiste Quenot via cutting) - -16. HADOOP-468. Add HADOOP_NICENESS environment variable to set - scheduling priority for daemons. (Vetle Roeim via cutting) - -17. HADOOP-473. Fix TextInputFormat to correctly handle more EOL - formats. Things now work correctly with CR, LF or CRLF. - (Dennis Kubes & James White via cutting) - -18. HADOOP-461. Make Java 1.5 an explicit requirement. (cutting) - -19. HADOOP-54. Add block compression to SequenceFile. One may now - specify that blocks of keys and values are compressed together, - improving compression for small keys and values. - SequenceFile.Writer's constructor is now deprecated and replaced - with a factory method. (Arun C Murthy via cutting) - -20. HADOOP-281. Prohibit DFS files that are also directories. - (Wendy Chien via cutting) - -21. HADOOP-486. Add the job username to JobStatus instances returned - by JobClient. (Mahadev Konar via cutting) - -22. HADOOP-437. contrib/streaming: Add support for gzipped inputs. - (Michel Tourn via cutting) - -23. HADOOP-463. Add variable expansion to config files. - Configuration property values may now contain variable - expressions. A variable is referenced with the syntax - '${variable}'. Variables values are found first in the - configuration, and then in Java system properties. The default - configuration is modified so that temporary directories are now - under ${hadoop.tmp.dir}, which is, by default, - /tmp/hadoop-${user.name}. (Michel Tourn via cutting) - -24. HADOOP-419. Fix a NullPointerException finding the ClassLoader - when using libhdfs. (omalley via cutting) - -25. HADOOP-460. Fix contrib/smallJobsBenchmark to use Text instead of - UTF8. (Sanjay Dahiya via cutting) - -26. HADOOP-196. Fix Configuration(Configuration) constructor to work - correctly. (Sami Siren via cutting) - -27. HADOOP-501. Fix Configuration.toString() to handle URL resources. - (Thomas Friol via cutting) - -28. HADOOP-499. Reduce the use of Strings in contrib/streaming, - replacing them with Text for better performance. - (Hairong Kuang via cutting) - -29. HADOOP-64. Manage multiple volumes with a single DataNode. - Previously DataNode would create a separate daemon per configured - volume, each with its own connection to the NameNode. Now all - volumes are handled by a single DataNode daemon, reducing the load - on the NameNode. (Milind Bhandarkar via cutting) - -30. HADOOP-424. Fix MapReduce so that jobs which generate zero splits - do not fail. (Fr??d??ric Bertin via cutting) - -31. HADOOP-408. Adjust some timeouts and remove some others so that - unit tests run faster. (cutting) - -32. HADOOP-507. Fix an IllegalAccessException in DFS. - (omalley via cutting) - -33. HADOOP-320. Fix so that checksum files are correctly copied when - the destination of a file copy is a directory. - (Hairong Kuang via cutting) - -34. HADOOP-286. In DFSClient, avoid pinging the NameNode with - renewLease() calls when no files are being written. - (Konstantin Shvachko via cutting) - -35. HADOOP-312. Close idle IPC connections. All IPC connections were - cached forever. Now, after a connection has been idle for more - than a configurable amount of time (one second by default), the - connection is closed, conserving resources on both client and - server. (Devaraj Das via cutting) - -36. HADOOP-497. Permit the specification of the network interface and - nameserver to be used when determining the local hostname - advertised by datanodes and tasktrackers. - (Lorenzo Thione via cutting) - -37. HADOOP-441. Add a compression codec API and extend SequenceFile - to use it. This will permit the use of alternate compression - codecs in SequenceFile. (Arun C Murthy via cutting) - -38. HADOOP-483. Improvements to libhdfs build and documentation. - (Arun C Murthy via cutting) - -39. HADOOP-458. Fix a memory corruption bug in libhdfs. - (Arun C Murthy via cutting) - -40. HADOOP-517. Fix a contrib/streaming bug in end-of-line detection. - (Hairong Kuang via cutting) - -41. HADOOP-474. Add CompressionCodecFactory, and use it in - TextInputFormat and TextOutputFormat. Compressed input files are - automatically decompressed when they have the correct extension. - Output files will, when output compression is specified, be - generated with an approprate extension. Also add a gzip codec and - fix problems with UTF8 text inputs. (omalley via cutting) - - -Release 0.5.0 - 2006-08-04 - - 1. HADOOP-352. Fix shell scripts to use /bin/sh instead of - /bin/bash, for better portability. - (Jean-Baptiste Quenot via cutting) - - 2. HADOOP-313. Permit task state to be saved so that single tasks - may be manually re-executed when debugging. (omalley via cutting) - - 3. HADOOP-339. Add method to JobClient API listing jobs that are - not yet complete, i.e., that are queued or running. - (Mahadev Konar via cutting) - - 4. HADOOP-355. Updates to the streaming contrib module, including - API fixes, making reduce optional, and adding an input type for - StreamSequenceRecordReader. (Michel Tourn via cutting) - - 5. HADOOP-358. Fix a NPE bug in Path.equals(). - (Fr??d??ric Bertin via cutting) - - 6. HADOOP-327. Fix ToolBase to not call System.exit() when - exceptions are thrown. (Hairong Kuang via cutting) - - 7. HADOOP-359. Permit map output to be compressed. - (omalley via cutting) - - 8. HADOOP-341. Permit input URI to CopyFiles to use the HTTP - protocol. This lets one, e.g., more easily copy log files into - DFS. (Arun C Murthy via cutting) - - 9. HADOOP-361. Remove unix dependencies from streaming contrib - module tests, making them pure java. (Michel Tourn via cutting) - -10. HADOOP-354. Make public methods to stop DFS daemons. - (Barry Kaplan via cutting) - -11. HADOOP-252. Add versioning to RPC protocols. - (Milind Bhandarkar via cutting) - -12. HADOOP-356. Add contrib to "compile" and "test" build targets, so - that this code is better maintained. (Michel Tourn via cutting) - -13. HADOOP-307. Add smallJobsBenchmark contrib module. This runs - lots of small jobs, in order to determine per-task overheads. - (Sanjay Dahiya via cutting) - -14. HADOOP-342. Add a tool for log analysis: Logalyzer. - (Arun C Murthy via cutting) - -15. HADOOP-347. Add web-based browsing of DFS content. The namenode - redirects browsing requests to datanodes. Content requests are - redirected to datanodes where the data is local when possible. - (Devaraj Das via cutting) - -16. HADOOP-351. Make Hadoop IPC kernel independent of Jetty. - (Devaraj Das via cutting) - -17. HADOOP-237. Add metric reporting to DFS and MapReduce. With only - minor configuration changes, one can now monitor many Hadoop - system statistics using Ganglia or other monitoring systems. - (Milind Bhandarkar via cutting) - -18. HADOOP-376. Fix datanode's HTTP server to scan for a free port. - (omalley via cutting) - -19. HADOOP-260. Add --config option to shell scripts, specifying an - alternate configuration directory. (Milind Bhandarkar via cutting) - -20. HADOOP-381. Permit developers to save the temporary files for - tasks whose names match a regular expression, to facilliate - debugging. (omalley via cutting) - -21. HADOOP-344. Fix some Windows-related problems with DF. - (Konstantin Shvachko via cutting) - -22. HADOOP-380. Fix reduce tasks to poll less frequently for map - outputs. (Mahadev Konar via cutting) - -23. HADOOP-321. Refactor DatanodeInfo, in preparation for - HADOOP-306. (Konstantin Shvachko & omalley via cutting) - -24. HADOOP-385. Fix some bugs in record io code generation. - (Milind Bhandarkar via cutting) - -25. HADOOP-302. Add new Text class to replace UTF8, removing - limitations of that class. Also refactor utility methods for - writing zero-compressed integers (VInts and VLongs). - (Hairong Kuang via cutting) - -26. HADOOP-335. Refactor DFS namespace/transaction logging in - namenode. (Konstantin Shvachko via cutting) - -27. HADOOP-375. Fix handling of the datanode HTTP daemon's port so - that multiple datanode's can be run on a single host. - (Devaraj Das via cutting) - -28. HADOOP-386. When removing excess DFS block replicas, remove those - on nodes with the least free space first. - (Johan Oskarson via cutting) - -29. HADOOP-389. Fix intermittent failures of mapreduce unit tests. - Also fix some build dependencies. - (Mahadev & Konstantin via cutting) - -30. HADOOP-362. Fix a problem where jobs hang when status messages - are recieved out-of-order. (omalley via cutting) - -31. HADOOP-394. Change order of DFS shutdown in unit tests to - minimize errors logged. (Konstantin Shvachko via cutting) - -32. HADOOP-396. Make DatanodeID implement Writable. - (Konstantin Shvachko via cutting) - -33. HADOOP-377. Permit one to add URL resources to a Configuration. - (Jean-Baptiste Quenot via cutting) - -34. HADOOP-345. Permit iteration over Configuration key/value pairs. - (Michel Tourn via cutting) - -35. HADOOP-409. Streaming contrib module: make configuration - properties available to commands as environment variables. - (Michel Tourn via cutting) - -36. HADOOP-369. Add -getmerge option to dfs command that appends all - files in a directory into a single local file. - (Johan Oskarson via cutting) - -37. HADOOP-410. Replace some TreeMaps with HashMaps in DFS, for - a 17% performance improvement. (Milind Bhandarkar via cutting) - -38. HADOOP-411. Add unit tests for command line parser. - (Hairong Kuang via cutting) - -39. HADOOP-412. Add MapReduce input formats that support filtering - of SequenceFile data, including sampling and regex matching. - Also, move JobConf.newInstance() to a new utility class. - (Hairong Kuang via cutting) - -40. HADOOP-226. Fix fsck command to properly consider replication - counts, now that these can vary per file. (Bryan Pendleton via cutting) - -41. HADOOP-425. Add a Python MapReduce example, using Jython. - (omalley via cutting) - - -Release 0.4.0 - 2006-06-28 - - 1. HADOOP-298. Improved progress reports for CopyFiles utility, the - distributed file copier. (omalley via cutting) - - 2. HADOOP-299. Fix the task tracker, permitting multiple jobs to - more easily execute at the same time. (omalley via cutting) - - 3. HADOOP-250. Add an HTTP user interface to the namenode, running - on port 50070. (Devaraj Das via cutting) - - 4. HADOOP-123. Add MapReduce unit tests that run a jobtracker and - tasktracker, greatly increasing code coverage. - (Milind Bhandarkar via cutting) - - 5. HADOOP-271. Add links from jobtracker's web ui to tasktracker's - web ui. Also attempt to log a thread dump of child processes - before they're killed. (omalley via cutting) - - 6. HADOOP-210. Change RPC server to use a selector instead of a - thread per connection. This should make it easier to scale to - larger clusters. Note that this incompatibly changes the RPC - protocol: clients and servers must both be upgraded to the new - version to ensure correct operation. (Devaraj Das via cutting) - - 7. HADOOP-311. Change DFS client to retry failed reads, so that a - single read failure will not alone cause failure of a task. - (omalley via cutting) - - 8. HADOOP-314. Remove the "append" phase when reducing. Map output - files are now directly passed to the sorter, without first - appending them into a single file. Now, the first third of reduce - progress is "copy" (transferring map output to reduce nodes), the - middle third is "sort" (sorting map output) and the last third is - "reduce" (generating output). Long-term, the "sort" phase will - also be removed. (omalley via cutting) - - 9. HADOOP-316. Fix a potential deadlock in the jobtracker. - (omalley via cutting) - -10. HADOOP-319. Fix FileSystem.close() to remove the FileSystem - instance from the cache. (Hairong Kuang via cutting) - -11. HADOOP-135. Fix potential deadlock in JobTracker by acquiring - locks in a consistent order. (omalley via cutting) - -12. HADOOP-278. Check for existence of input directories before - starting MapReduce jobs, making it easier to debug this common - error. (omalley via cutting) - -13. HADOOP-304. Improve error message for - UnregisterdDatanodeException to include expected node name. - (Konstantin Shvachko via cutting) - -14. HADOOP-305. Fix TaskTracker to ask for new tasks as soon as a - task is finished, rather than waiting for the next heartbeat. - This improves performance when tasks are short. - (Mahadev Konar via cutting) - -15. HADOOP-59. Add support for generic command line options. One may - now specify the filesystem (-fs), the MapReduce jobtracker (-jt), - a config file (-conf) or any configuration property (-D). The - "dfs", "fsck", "job", and "distcp" commands currently support - this, with more to be added. (Hairong Kuang via cutting) - -16. HADOOP-296. Permit specification of the amount of reserved space - on a DFS datanode. One may specify both the percentage free and - the number of bytes. (Johan Oskarson via cutting) - -17. HADOOP-325. Fix a problem initializing RPC parameter classes, and - remove the workaround used to initialize classes. - (omalley via cutting) - -18. HADOOP-328. Add an option to the "distcp" command to ignore read - errors while copying. (omalley via cutting) - -19. HADOOP-27. Don't allocate tasks to trackers whose local free - space is too low. (Johan Oskarson via cutting) - -20. HADOOP-318. Keep slow DFS output from causing task timeouts. - This incompatibly changes some public interfaces, adding a - parameter to OutputFormat.getRecordWriter() and the new method - Reporter.progress(), but it makes lots of tasks succeed that were - previously failing. (Milind Bhandarkar via cutting) - - -Release 0.3.2 - 2006-06-09 - - 1. HADOOP-275. Update the streaming contrib module to use log4j for - its logging. (Michel Tourn via cutting) - - 2. HADOOP-279. Provide defaults for log4j logging parameters, so - that things still work reasonably when Hadoop-specific system - properties are not provided. (omalley via cutting) - - 3. HADOOP-280. Fix a typo in AllTestDriver which caused the wrong - test to be run when "DistributedFSCheck" was specified. - (Konstantin Shvachko via cutting) - - 4. HADOOP-240. DFS's mkdirs() implementation no longer logs a warning - when the directory already exists. (Hairong Kuang via cutting) - - 5. HADOOP-285. Fix DFS datanodes to be able to re-join the cluster - after the connection to the namenode is lost. (omalley via cutting) - - 6. HADOOP-277. Fix a race condition when creating directories. - (Sameer Paranjpye via cutting) - - 7. HADOOP-289. Improved exception handling in DFS datanode. - (Konstantin Shvachko via cutting) - - 8. HADOOP-292. Fix client-side logging to go to standard error - rather than standard output, so that it can be distinguished from - application output. (omalley via cutting) - - 9. HADOOP-294. Fixed bug where conditions for retrying after errors - in the DFS client were reversed. (omalley via cutting) - - -Release 0.3.1 - 2006-06-05 - - 1. HADOOP-272. Fix a bug in bin/hadoop setting log - parameters. (omalley & cutting) - - 2. HADOOP-274. Change applications to log to standard output rather - than to a rolling log file like daemons. (omalley via cutting) - - 3. HADOOP-262. Fix reduce tasks to report progress while they're - waiting for map outputs, so that they do not time out. - (Mahadev Konar via cutting) - - 4. HADOOP-245 and HADOOP-246. Improvements to record io package. - (Mahadev Konar via cutting) - - 5. HADOOP-276. Add logging config files to jar file so that they're - always found. (omalley via cutting) - - -Release 0.3.0 - 2006-06-02 - - 1. HADOOP-208. Enhance MapReduce web interface, adding new pages - for failed tasks, and tasktrackers. (omalley via cutting) - - 2. HADOOP-204. Tweaks to metrics package. (David Bowen via cutting) - - 3. HADOOP-209. Add a MapReduce-based file copier. This will - copy files within or between file systems in parallel. - (Milind Bhandarkar via cutting) - - 4. HADOOP-146. Fix DFS to check when randomly generating a new block - id that no existing blocks already have that id. - (Milind Bhandarkar via cutting) - - 5. HADOOP-180. Make a daemon thread that does the actual task clean ups, so - that the main offerService thread in the taskTracker doesn't get stuck - and miss his heartbeat window. This was killing many task trackers as - big jobs finished (300+ tasks / node). (omalley via cutting) - - 6. HADOOP-200. Avoid transmitting entire list of map task names to - reduce tasks. Instead just transmit the number of map tasks and - henceforth refer to them by number when collecting map output. - (omalley via cutting) - - 7. HADOOP-219. Fix a NullPointerException when handling a checksum - exception under SequenceFile.Sorter.sort(). (cutting & stack) - - 8. HADOOP-212. Permit alteration of the file block size in DFS. The - default block size for new files may now be specified in the - configuration with the dfs.block.size property. The block size - may also be specified when files are opened. - (omalley via cutting) - - 9. HADOOP-218. Avoid accessing configuration while looping through - tasks in JobTracker. (Mahadev Konar via cutting) - -10. HADOOP-161. Add hashCode() method to DFS's Block. - (Milind Bhandarkar via cutting) - -11. HADOOP-115. Map output types may now be specified. These are also - used as reduce input types, thus permitting reduce input types to - differ from reduce output types. (Runping Qi via cutting) - -12. HADOOP-216. Add task progress to task status page. - (Bryan Pendelton via cutting) - -13. HADOOP-233. Add web server to task tracker that shows running - tasks and logs. Also add log access to job tracker web interface. - (omalley via cutting) - -14. HADOOP-205. Incorporate pending tasks into tasktracker load - calculations. (Mahadev Konar via cutting) - -15. HADOOP-247. Fix sort progress to better handle exceptions. - (Mahadev Konar via cutting) - -16. HADOOP-195. Improve performance of the transfer of map outputs to - reduce nodes by performing multiple transfers in parallel, each on - a separate socket. (Sameer Paranjpye via cutting) - -17. HADOOP-251. Fix task processes to be tolerant of failed progress - reports to their parent process. (omalley via cutting) - -18. HADOOP-325. Improve the FileNotFound exceptions thrown by - LocalFileSystem to include the name of the file. - (Benjamin Reed via cutting) - -19. HADOOP-254. Use HTTP to transfer map output data to reduce - nodes. This, together with HADOOP-195, greatly improves the - performance of these transfers. (omalley via cutting) - -20. HADOOP-163. Cause datanodes that\ are unable to either read or - write data to exit, so that the namenode will no longer target - them for new blocks and will replicate their data on other nodes. - (Hairong Kuang via cutting) - -21. HADOOP-222. Add a -setrep option to the dfs commands that alters - file replication levels. (Johan Oskarson via cutting) - -22. HADOOP-75. In DFS, only check for a complete file when the file - is closed, rather than as each block is written. - (Milind Bhandarkar via cutting) - -23. HADOOP-124. Change DFS so that datanodes are identified by a - persistent ID rather than by host and port. This solves a number - of filesystem integrity problems, when, e.g., datanodes are - restarted. (Konstantin Shvachko via cutting) - -24. HADOOP-256. Add a C API for DFS. (Arun C Murthy via cutting) - -25. HADOOP-211. Switch to use the Jakarta Commons logging internally, - configured to use log4j by default. (Arun C Murthy and cutting) - -26. HADOOP-265. Tasktracker now fails to start if it does not have a - writable local directory for temporary files. In this case, it - logs a message to the JobTracker and exits. (Hairong Kuang via cutting) - -27. HADOOP-270. Fix potential deadlock in datanode shutdown. - (Hairong Kuang via cutting) - -Release 0.2.1 - 2006-05-12 - - 1. HADOOP-199. Fix reduce progress (broken by HADOOP-182). - (omalley via cutting) - - 2. HADOOP-201. Fix 'bin/hadoop dfs -report'. (cutting) - - 3. HADOOP-207. Fix JDK 1.4 incompatibility introduced by HADOOP-96. - System.getenv() does not work in JDK 1.4. (Hairong Kuang via cutting) - - -Release 0.2.0 - 2006-05-05 - - 1. Fix HADOOP-126. 'bin/hadoop dfs -cp' now correctly copies .crc - files. (Konstantin Shvachko via cutting) - - 2. Fix HADOOP-51. Change DFS to support per-file replication counts. - (Konstantin Shvachko via cutting) - - 3. Fix HADOOP-131. Add scripts to start/stop dfs and mapred daemons. - Use these in start/stop-all scripts. (Chris Mattmann via cutting) - - 4. Stop using ssh options by default that are not yet in widely used - versions of ssh. Folks can still enable their use by uncommenting - a line in conf/hadoop-env.sh. (cutting) - - 5. Fix HADOOP-92. Show information about all attempts to run each - task in the web ui. (Mahadev konar via cutting) - - 6. Fix HADOOP-128. Improved DFS error handling. (Owen O'Malley via cutting) - - 7. Fix HADOOP-129. Replace uses of java.io.File with new class named - Path. This fixes bugs where java.io.File methods were called - directly when FileSystem methods were desired, and reduces the - likelihood of such bugs in the future. It also makes the handling - of pathnames more consistent between local and dfs FileSystems and - between Windows and Unix. java.io.File-based methods are still - available for back-compatibility, but are deprecated and will be - removed once 0.2 is released. (cutting) - - 8. Change dfs.data.dir and mapred.local.dir to be comma-separated - lists of directories, no longer be space-separated. This fixes - several bugs on Windows. (cutting) - - 9. Fix HADOOP-144. Use mapred task id for dfs client id, to - facilitate debugging. (omalley via cutting) - -10. Fix HADOOP-143. Do not line-wrap stack-traces in web ui. - (omalley via cutting) - -11. Fix HADOOP-118. In DFS, improve clean up of abandoned file - creations. (omalley via cutting) - -12. Fix HADOOP-138. Stop multiple tasks in a single heartbeat, rather - than one per heartbeat. (Stefan via cutting) - -13. Fix HADOOP-139. Remove a potential deadlock in - LocalFileSystem.lock(). (Igor Bolotin via cutting) - -14. Fix HADOOP-134. Don't hang jobs when the tasktracker is - misconfigured to use an un-writable local directory. (omalley via cutting) - -15. Fix HADOOP-115. Correct an error message. (Stack via cutting) - -16. Fix HADOOP-133. Retry pings from child to parent, in case of - (local) communcation problems. Also log exit status, so that one - can distinguish patricide from other deaths. (omalley via cutting) - -17. Fix HADOOP-142. Avoid re-running a task on a host where it has - previously failed. (omalley via cutting) - -18. Fix HADOOP-148. Maintain a task failure count for each - tasktracker and display it in the web ui. (omalley via cutting) - -19. Fix HADOOP-151. Close a potential socket leak, where new IPC - connection pools were created per configuration instance that RPCs - use. Now a global RPC connection pool is used again, as - originally intended. (cutting) - -20. Fix HADOOP-69. Don't throw a NullPointerException when getting - hints for non-existing file split. (Bryan Pendelton via cutting) - -21. Fix HADOOP-157. When a task that writes dfs files (e.g., a reduce - task) failed and was retried, it would fail again and again, - eventually failing the job. The problem was that dfs did not yet - know that the failed task had abandoned the files, and would not - yet let another task create files with the same names. Dfs now - retries when creating a file long enough for locks on abandoned - files to expire. (omalley via cutting) - -22. Fix HADOOP-150. Improved task names that include job - names. (omalley via cutting) - -23. Fix HADOOP-162. Fix ConcurrentModificationException when - releasing file locks. (omalley via cutting) - -24. Fix HADOOP-132. Initial check-in of new Metrics API, including - implementations for writing metric data to a file and for sending - it to Ganglia. (David Bowen via cutting) - -25. Fix HADOOP-160. Remove some uneeded synchronization around - time-consuming operations in the TaskTracker. (omalley via cutting) - -26. Fix HADOOP-166. RPCs failed when passed subclasses of a declared - parameter type. This is fixed by changing ObjectWritable to store - both the declared type and the instance type for Writables. Note - that this incompatibly changes the format of ObjectWritable and - will render unreadable any ObjectWritables stored in files. - Nutch only uses ObjectWritable in intermediate files, so this - should not be a problem for Nutch. (Stefan & cutting) - -27. Fix HADOOP-168. MapReduce RPC protocol methods should all declare - IOException, so that timeouts are handled appropriately. - (omalley via cutting) - -28. Fix HADOOP-169. Don't fail a reduce task if a call to the - jobtracker to locate map outputs fails. (omalley via cutting) - -29. Fix HADOOP-170. Permit FileSystem clients to examine and modify - the replication count of individual files. Also fix a few - replication-related bugs. (Konstantin Shvachko via cutting) - -30. Permit specification of a higher replication levels for job - submission files (job.xml and job.jar). This helps with large - clusters, since these files are read by every node. (cutting) - -31. HADOOP-173. Optimize allocation of tasks with local data. (cutting) - -32. HADOOP-167. Reduce number of Configurations and JobConf's - created. (omalley via cutting) - -33. NUTCH-256. Change FileSystem#createNewFile() to create a .crc - file. The lack of a .crc file was causing warnings. (cutting) - -34. HADOOP-174. Change JobClient to not abort job until it has failed - to contact the job tracker for five attempts, not just one as - before. (omalley via cutting) - -35. HADOOP-177. Change MapReduce web interface to page through tasks. - Previously, when jobs had more than a few thousand tasks they - could crash web browsers. (Mahadev Konar via cutting) - -36. HADOOP-178. In DFS, piggyback blockwork requests from datanodes - on heartbeat responses from namenode. This reduces the volume of - RPC traffic. Also move startup delay in blockwork from datanode - to namenode. This fixes a problem where restarting the namenode - triggered a lot of uneeded replication. (Hairong Kuang via cutting) - -37. HADOOP-183. If the DFS namenode is restarted with different - minimum and/or maximum replication counts, existing files' - replication counts are now automatically adjusted to be within the - newly configured bounds. (Hairong Kuang via cutting) - -38. HADOOP-186. Better error handling in TaskTracker's top-level - loop. Also improve calculation of time to send next heartbeat. - (omalley via cutting) - -39. HADOOP-187. Add two MapReduce examples/benchmarks. One creates - files containing random data. The second sorts the output of the - first. (omalley via cutting) - -40. HADOOP-185. Fix so that, when a task tracker times out making the - RPC asking for a new task to run, the job tracker does not think - that it is actually running the task returned. (omalley via cutting) - -41. HADOOP-190. If a child process hangs after it has reported - completion, its output should not be lost. (Stack via cutting) - -42. HADOOP-184. Re-structure some test code to better support testing - on a cluster. (Mahadev Konar via cutting) - -43. HADOOP-191 Add streaming package, Hadoop's first contrib module. - This permits folks to easily submit MapReduce jobs whose map and - reduce functions are implemented by shell commands. Use - 'bin/hadoop jar build/hadoop-streaming.jar' to get details. - (Michel Tourn via cutting) - -44. HADOOP-189. Fix MapReduce in standalone configuration to - correctly handle job jar files that contain a lib directory with - nested jar files. (cutting) - -45. HADOOP-65. Initial version of record I/O framework that enables - the specification of record types and generates marshalling code - in both Java and C++. Generated Java code implements - WritableComparable, but is not yet otherwise used by - Hadoop. (Milind Bhandarkar via cutting) - -46. HADOOP-193. Add a MapReduce-based FileSystem benchmark. - (Konstantin Shvachko via cutting) - -47. HADOOP-194. Add a MapReduce-based FileSystem checker. This reads - every block in every file in the filesystem. (Konstantin Shvachko - via cutting) - -48. HADOOP-182. Fix so that lost task trackers to not change the - status of reduce tasks or completed jobs. Also fixes the progress - meter so that failed tasks are subtracted. (omalley via cutting) - -49. HADOOP-96. Logging improvements. Log files are now separate from - standard output and standard error files. Logs are now rolled. - Logging of all DFS state changes can be enabled, to facilitate - debugging. (Hairong Kuang via cutting) - - -Release 0.1.1 - 2006-04-08 - - 1. Added CHANGES.txt, logging all significant changes to Hadoop. (cutting) - - 2. Fix MapReduceBase.close() to throw IOException, as declared in the - Closeable interface. This permits subclasses which override this - method to throw that exception. (cutting) - - 3. Fix HADOOP-117. Pathnames were mistakenly transposed in - JobConf.getLocalFile() causing many mapred temporary files to not - be removed. (Raghavendra Prabhu via cutting) - - 4. Fix HADOOP-116. Clean up job submission files when jobs complete. - (cutting) - - 5. Fix HADOOP-125. Fix handling of absolute paths on Windows (cutting) - -Release 0.1.0 - 2006-04-01 - - 1. The first release of Hadoop. - diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt deleted file mode 100644 index 0ddecafd0d0..00000000000 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ /dev/null @@ -1,14611 +0,0 @@ -Hadoop HDFS Change Log - -Trunk (Unreleased) - - INCOMPATIBLE CHANGES - - HDFS-3034. Remove the deprecated DFSOutputStream.sync() method. (szetszwo) - - HDFS-5079. Cleaning up NNHAStatusHeartbeat.State from - DatanodeProtocolProtos. (Tao Luo via shv) - - HDFS-5570. Deprecate hftp / hsftp and replace them with webhdfs / swebhdfs. - (wheat9) - - HDFS-2538. option to disable fsck dots (Mohammad Kamrul Islam via aw) - - HDFS-7302. Remove "downgrade" from "namenode -rollingUpgrade" startup - option since it may incorrectly finalize an ongoing rolling upgrade. - (Kai Sasaki via szetszwo) - - HDFS-7985. WebHDFS should be always enabled. (Li Lu via wheat9) - - HDFS-7281. Missing block is marked as corrupted block (Ming Ma via - Yongjun Zhang) - - HDFS-8349. Remove .xml and documentation references to dfs.webhdfs.enabled. - (Ray Chiang via aajisaka) - - HDFS-8241. Remove unused NameNode startup option -finalize. - (Brahma Reddy Battula via aajisaka) - - HDFS-8135. Remove the deprecated FSConstants class. (Li Lu via wheat9) - - HDFS-8591. Remove support for deprecated configuration key - dfs.namenode.decommission.nodes.per.interval. (wang) - - HDFS-8895. Remove deprecated BlockStorageLocation APIs. (wang) - - HDFS-8981. Adding revision to data node jmx getVersion() method. (Siqi Li - via mingma) - - HDFS-9085. Show renewer information in DelegationTokenIdentifier#toString. - (Zhihai Xu via cnauroth) - - HDFS-9278. Fix preferredBlockSize typo in OIV XML output. (Nicole Pazmany via wang) - - HDFS-5165. Remove the TotalFiles metrics. (Akira Ajisaka via wheat9) - - NEW FEATURES - - HDFS-3125. Add JournalService to enable Journal Daemon. (suresh) - - HDFS-6440. Support more than 2 NameNodes. (Jesse Yates via atm) - - HDFS-9057. allow/disallow snapshots via webhdfs - (Bramma Reddy Battula via vinayakumarb) - - HDFS-9525. hadoop utilities need to support provided delegation - tokens (HeeSoo Kim via aw) - - HDFS-9804. Allow long-running Balancer to login with keytab. - (Xiao Chen via zhz) - - IMPROVEMENTS - - HDFS-9653. Added blocks pending deletion report to dfsadmin. - (Weiwei Yang via eyang) - - HDFS-4665. Move TestNetworkTopologyWithNodeGroup to common. - (Junping Du via llu) - - HDFS-1620. Rename HdfsConstants -> HdfsServerConstants, FSConstants -> - HdfsConstants. (Harsh J Chouraria via atm) - - HDFS-2197. Refactor RPC call implementations out of NameNode class (todd) - - HDFS-2572. Remove unnecessary double-check in DN#getHostName. (harsh) - - HDFS-2857. Cleanup BlockInfo class. (suresh) - - HDFS-2786. Fix host-based token incompatibilities in DFSUtil. (Kihwal Lee - via jitendra) - - HDFS-2878. Fix TestBlockRecovery and move it back into main test directory. - (todd) - - HDFS-2655. BlockReaderLocal#skip performs unnecessary IO. - (Brandon Li via jitendra) - - HDFS-3009. Remove duplicate code in DFSClient#isLocalAddress by using - NetUtils. (Hari Mankude via suresh) - - HDFS-3002. TestNameNodeMetrics need not wait for metrics update. - (suresh) - - HDFS-3016. Security in unit tests. (Jaimin Jetly via jitendra) - - HDFS-3030. Remove getProtocolVersion and getProtocolSignature from translators. - (jitendra) - - HDFS-3111. Missing license headers in trunk. (umamahesh) - - HDFS-3091. Update the usage limitations of ReplaceDatanodeOnFailure policy in - the config description for the smaller clusters. (szetszwo via umamahesh) - - HDFS-309. FSEditLog should log progress during replay. (Sho Shimauchi - via todd) - - HDFS-3131. Improve TestStorageRestore. (Brandon Li via atm) - - HDFS-3178. Add states and state handler for journal synchronization in - JournalService. (szetszwo) - - HDFS-3273. Refactor BackupImage and FSEditLog, and rename - JournalListener.rollLogs(..) to startLogSegment(..). (szetszwo) - - HDFS-3292. Remove the deprecated DiskStatus, getDiskStatus(), getRawUsed() - and getRawCapacity() from DistributedFileSystem. (Arpit Gupta via szetszwo) - - HADOOP-8285. HDFS changes for Use ProtoBuf for RpcPayLoadHeader. (sanjay - radia) - - HDFS-2743. Streamline usage of bookkeeper journal manager. - (Ivan Kelly via umamahesh) - - HDFS-3293. Add toString(), equals(..) and hashCode() to JournalInfo. - (Hari Mankude via szetszwo) - - HDFS-3197. Incorrect class comments in a few tests. (Andy Isaacson via eli) - - HDFS-3476. Correct the default used in TestDFSClientRetries.busyTest() - after HDFS-3462 (harsh) - - HDFS-3478. Test quotas with Long.Max_Value. (Sujay Rau via eli) - - HDFS-3630 Modify TestPersistBlocks to use both flush and hflush (sanjay) - - HDFS-3768. Exception in TestJettyHelper is incorrect. - (Eli Reisman via jghoman) - - HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. (harsh) - - HDFS-2127. Add a test that ensure AccessControlExceptions contain - a full path. (Stephen Chu via eli) - - HDFS-3995. Use DFSTestUtil.createFile() for file creation and - writing in test cases. (Jing Zhao via suresh) - - HDFS-3735. NameNode WebUI should allow sorting live datanode list by fields - Block Pool Used, Block Pool Used(%) and Failed Volumes. - (Brahma Reddy Battula via suresh) - - HDFS-4052. BlockManager#invalidateWork should print log outside the lock. - (Jing Zhao via suresh) - - HDFS-3358. Specify explicitly that the NN UI status total is talking - of persistent objects on heap. (harsh) - - HDFS-4904. Remove JournalService. (Arpit Agarwal via cnauroth) - - HDFS-5041. Add the time of last heartbeat to dead server Web UI (Shinichi - Yamashita via brandonli) - - HDFS-5721. sharedEditsImage in Namenode#initializeSharedEdits() should be - closed before method returns. (Ted Yu via junping_du) - - HDFS-6228. comments typo fix for FsDatasetImpl.java (zhaoyunjiong via umamahesh) - - HDFS-6246. Remove 'dfs.support.append' flag from trunk code. (umamahesh) - - HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable - directory. (Jing Zhao via wheat9) - - HDFS-7591. hdfs classpath command should support same options as hadoop - classpath (Varun Saxena via Arpit Agarwal) - - HDFS-316. Balancer should run for a configurable # of iterations (Xiaoyu - Yao via aw) - - HDFS-7322. deprecate sbin/hadoop-daemon.sh (aw) - - HDFS-7668. Convert site documentation from apt to markdown (Masatake - Iwasaki via aw) - - HDFS-7460. Rewrite httpfs to use new shell framework (John Smith via aw) - - HDFS-6353. Check and make checkpoint before stopping the NameNode. (jing9) - - HDFS-5033. Bad error message for fs -put/copyFromLocal if user - doesn't have permissions to read the source (Darrell Taylor via aw) - - HDFS-9182. Cleanup the findbugs and other issues after HDFS EC merged to trunk. - (umamahesh) - - HDFS-8632. Add InterfaceAudience annotation to the erasure coding classes. - (Rakesh R via wang) - - HDFS-9070. Allow fsck display pending replica location information for - being-written blocks. (GAO Rui via jing9) - - HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data - streamers are failed for the current block group. (Rakesh R via umamahesh) - - HDFS-9323. Randomize the DFSStripedOutputStreamWithFailure tests. (szetszwo) - - HDFS-8777. Erasure Coding: add tests for taking snapshots on EC files. - (Rakesh R via zhz) - - HADOOP-11684. S3a to use thread pool that blocks clients. - (Thomas Demoor and Aaron Fabbri via lei) - - HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. - (Surendra Singh Lilhore via xyao) - - HDFS-8968. Erasure coding: a comprehensive I/O throughput benchmark tool. - (Rui Li via zhz) - - HDFS-8831. Trash Support for deletion in HDFS encryption zone. (xyao) - - HDFS-9582. TestLeaseRecoveryStriped file missing Apache License header - and not well formatted. (umamahesh) - - HDFS-9494. Parallel optimization of DFSStripedOutputStream#flushAllInternals. - (Gao Rui via jing9) - - HDFS-9260. Improve the performance and GC friendliness of NameNode startup - and full block reports (Staffan Friberg via cmccabe) - - HDFS-9795. OIV Delimited should show which files are ACL-enabled (lei) - - HDFS-9829. Erasure Coding: Improve few exception handling logic of - ErasureCodingWorker. (Rakesh R via jing9) - - HDFS-9838. Refactor the excessReplicateMap to a class. (szetszwo) - - HDFS-9733. Refactor DFSClient#getFileChecksum and DataXceiver#blockChecksum - (Kai Zheng via umamahesh) - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-9635 Fix potential Stack Overflow in DomainSocket.c (V. Karthik Kumar - via cmccabe) - - HDFS-2299. TestOfflineEditsViewer is failing on trunk. (Uma Maheswara Rao G - via atm) - HDFS-2310. TestBackupNode fails since HADOOP-7524 went in. - (Ivan Kelly via todd) - - HDFS-2313. Rat excludes has a typo for excluding editsStored files. (atm) - - HDFS-2314. MRV1 test compilation broken after HDFS-2197 (todd) - - HDFS-46. Change default namespace quota of root directory from - Integer.MAX_VALUE to Long.MAX_VALUE. (Uma Maheswara Rao G via szetszwo) - - HDFS-2373. Commands using WebHDFS and hftp print unnecessary debug - info on the console with security enabled. (Arpit Gupta via suresh) - - HDFS-2776. Missing interface annotation on JournalSet. - (Brandon Li via jitendra) - - HDFS-3037. TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is - racy. (atm) - - HDFS-3116. Typo in fetchdt error message. (AOE Takashi via atm) - - HDFS-3126. Journal stream from Namenode to BackupNode needs to have - timeout. (Hari Mankude via suresh) - - HDFS-3121. Add HDFS tests for HADOOP-8014 change. (John George via - suresh) - - HDFS-3119. Overreplicated block is not deleted even after the replication - factor is reduced after sync follwed by closing that file. (Ashish Singhi - via umamahesh) - - HDFS-3265. PowerPc Build error. (Kumar Ravi via mattf) - - HDFS-2312. FSNamesystem javadoc incorrectly says its for DNs. (harsh) - - HDFS-3163. TestHDFSCLI.testAll fails if the user name is not all lowercase. - (Brandon Li via atm) - - HDFS-3462. TestDFSClientRetries.busyTest() should restore default - xceiver count in the config. (Madhukara Phatak via harsh) - - HDFS-3550. Fix raid javadoc warnings. (Jason Lowe via daryn) - - HDFS-3549. Fix dist tar build fails in hadoop-hdfs-raid project. (Jason Lowe via daryn) - - HDFS-3614. Revert unused MiniDFSCluster constructor from HDFS-3049. - (acmurthy via eli) - - HDFS-3792. Fix two findbugs introduced by HDFS-3695 (todd) - - HDFS-3827. TestHASafeMode#assertSafemode method should be made static. - (Jing Zhao via suresh) - - HDFS-3834. Remove unused static fields NAME, DESCRIPTION and Usage from - Command. (Jing Zhao via suresh) - - HDFS-2434. TestNameNodeMetrics.testCorruptBlock fails intermittently. - (Jing Zhao via suresh) - - HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to - ReplicaAlreadyExistsException. (Jing Zhao via suresh) - - HDFS-4115. TestHDFSCLI.testAll fails one test due to number format. - (Trevor Robinson via suresh) - - HDFS-4105. The SPNEGO user for secondary namenode should use the web - keytab. (Arpit Gupta via jitendra) - - HDFS-4003. test-patch should build the common native libs before - running hdfs tests. (Colin Patrick McCabe via eli) - - HDFS-4260 Fix HDFS tests to set test dir to a valid HDFS path as opposed - to the local build path (Chri Nauroth via Sanjay) - - HDFS-4310. fix test org.apache.hadoop.hdfs.server.datanode. - TestStartSecureDataNode (Ivan A. Veselovsky via atm) - - HDFS-4275. MiniDFSCluster-based tests fail on Windows due to failure - to delete test namenode directory. (Chris Nauroth via suresh) - - HDFS-4338. TestNameNodeMetrics#testCorruptBlock is flaky. (Andrew Wang via - atm) - - HDFS-4391. TestDataTransferKeepalive fails when tests are executed in a - certain order. (Andrew Wang via atm) - - HDFS-4757. Update FSDirectory#inodeMap when replacing an INodeDirectory - while setting quota. (Jing Zhao via szetszwo) - - HDFS-4761. When resetting FSDirectory, the inodeMap should also be reset. - (Jing Zhao via szetszwo) - - HDFS-4687. TestDelegationTokenForProxyUser#testWebHdfsDoAs is flaky with - JDK7. (Andrew Wang via atm) - - HDFS-3934. duplicative dfs_hosts entries handled wrong. (Colin Patrick - McCabe) - - HDFS-5719. FSImage#doRollback() should close prevState before return - (Ted Yu via brandonli) - - HDFS-5794. Fix the inconsistency of layout version number of - ADD_DATANODE_AND_STORAGE_UUIDS between trunk and branch-2. (jing9) - - HDFS-6893. crypto subcommand is not sorted properly in hdfs's hadoop_usage - (David Luo via aw) - - HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal) - - HDFS-7204. balancer doesn't run as a daemon (aw) - - HDFS-7319. Remove dead link to HFTP documentation from index.xml. - (Masatake Iwasaki via wheat9) - - HDFS-7407. Minor typo in privileged pid/out/log names (aw) - - HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw) - - HADOOP-11484. hadoop-mapreduce-client-nativetask fails to build on ARM - AARCH64 due to x86 asm statements (Edward Nevill via Colin P. McCabe) - - HDFS-7667. Various typos and improvements to HDFS Federation doc - (Charles Lamb via aw) - - HDFS-3750. API docs don't include HDFS (Jolly Chen via aw) - - HDFS-7320. The appearance of hadoop-hdfs-httpfs site docs is inconsistent - (Masatake Iwasaki via aw) - - HDFS-7670. HDFS Quota guide has typos, incomplete command lines - (Brahma Reddy Battula via aw) - - HDFS-7669. HDFS Design Doc references commands that no longer exist. - (Brahma Reddy Battula via aw) - - HDFS-7755. httpfs shell code has hardcoded path to bash (Dmitry - Sivachenko via aw) - - HDFS-7711. Fix various typos in ClusterSetup.md. - (Brahma Reddy Battula via wheat9) - - HDFS-7791. dfs count -v should be added to quota documentation (Akira - AJISAKA via aw) - - HDFS-7803. Wrong command mentioned in HDFSHighAvailabilityWithQJM - documentation (Arshad Mohammad via aw) - - HDFS-4681. TestBlocksWithNotEnoughRacks#testCorruptBlockRereplicatedAcrossRacks - fails using IBM java (Ayappan via aw) - - HDFS-8110. Remove unsupported 'hdfs namenode -rollingUpgrade downgrade' - from document. (J.Andreina via aajisaka) - - HDFS-7673. synthetic load generator docs give incorrect/incomplete commands - (Brahma Reddy Battula via aw) - - HDFS-8332. DFS client API calls should check filesystem closed - (Rakesh R via umamahesh) - - HDFS-8412. Fix the test failures in HTTPFS. (umamahesh) - - HDFS-8627. NPE thrown if unable to fetch token from Namenode - (J.Andreina via vinayakumarb) - - HDFS-8657. Update docs for mSNN. (Jesse Yates via atm) - - HDFS-8800. hdfs --daemon stop namenode corrupts logs (John Smith via aw) - - HDFS-9204. DatanodeDescriptor#PendingReplicationWithoutTargets is wrongly - calculated. (Mingliang Liu via jing9) - - HDFS-9217. Fix broken findbugsExcludeFile.xml for hadoop-hdfs-client module. - (Mingliang Liu via aajisaka) - - HDFS-9423. Fix intermittent failure of TestEditLogTailer. - (Masatake Iwasaki via waltersu4549) - - HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed - due to unexpected number of invalidate blocks. (Wei-Chiu Chuang via zhz) - - HDFS-9376. TestSeveralNameNodes fails occasionally. - (Masatake Iwasaki via cnauroth) - - HDFS-9615. Fix variable name typo in DFSConfigKeys. (Ray Chiang via - Arpit Agarwal) - - HDFS-9646. ErasureCodingWorker may fail when recovering data blocks with - length less than the first internal block. (jing9) - - HDFS-9789. Correctly update DataNode's scheduled block size when writing - small EC file. (jing9) - - HDFS-9794. Streamer threads may leak if failure happens when closing the - striped outputstream. (jing9) - - HDFS-9818. Correctly handle EC reconstruction work caused by not enough - racks. (jing9) - - HDFS-9837. BlockManager#countNodes should be able to detect duplicated - internal blocks. (jing9) - - HDFS-9734. Refactoring of checksum failure report related codes. - (Kai Zheng via zhz) - - HDFS-9866. BlockManager#chooseExcessReplicasStriped may weaken rack fault - tolerance. (jing9) - - HDFS-9867. Missing block exception should carry locatedBlocks information. - (Mingliang Liu via jing9) - - HDFS-9876. shouldProcessOverReplicated should not count number of pending - replicas. (jing9) - - BREAKDOWN OF HDFS-7285 SUBTASKS AND RELATED JIRAS - - HDFS-7347. Configurable erasure coding policy for individual files and - directories ( Zhe Zhang via vinayakumarb ) - - HDFS-7339. Representing striped block groups in NameNode with hierarchical - naming protocol ( zhz ) - - HDFS-7652. Process block reports for erasure coded blocks (Zhe Zhang) - - HDFS-7716. Erasure Coding: extend BlockInfo to handle EC info (Jing Zhao) - - HDFS-7749. Erasure Coding: Add striped block support in INodeFile (Jing Zhao) - - HDFS-7837. Erasure Coding: allocate and persist striped blocks in NameNode - (Jing Zhao via Zhe Zhang) - - HDFS-7872. Erasure Coding: INodeFile.dumpTreeRecursively() supports to print - striped blocks (Takuya Fukudome via jing9) - - HDFS-7853. Erasure coding: extend LocatedBlocks to support reading from - striped files (Jing Zhao) - - HDFS-7826. Erasure Coding: Update INodeFile quota computation for striped - blocks ( Kai Sasaki via jing9 ) - - HDFS-7912. Erasure Coding: track BlockInfo instead of Block in - UnderReplicatedBlocks and PendingReplicationBlocks (Jing Zhao) - - HDFS-7369. Erasure coding: distribute recovery work for striped blocks to - DataNode (Zhe Zhang) - - HDFS-7864. Erasure Coding: Update safemode calculation for striped blocks - (GAO Rui via jing9) - - HDFS-7827. Erasure Coding: support striped blocks in non-protobuf fsimage - ( Hui Zheng via jing9 ) - - HDFS-7616. Add a test for BlockGroup support in FSImage. - (Takuya Fukudome via szetszwo) - - HDFS-7907. Erasure Coding: track invalid, corrupt, and under-recovery striped - blocks in NameNode (Jing Zhao) - - HDFS-8005. Erasure Coding: simplify striped block recovery work computation - and add tests (Jing Zhao) - - HDFS-7617. Add unit tests for editlog transactions for EC - (Hui Zheng via Zhe Zhang) - - HDFS-7839. Erasure coding: implement facilities in NameNode to create and - manage EC zones (Zhe Zhang) - - HDFS-7969. Erasure coding: NameNode support for lease recovery of striped - block groups. (Zhe Zhang) - - HDFS-7782. Erasure coding: pread from files in striped layout. - (Zhe Zhang and Jing Zhao via Zhe Zhang) - - HDFS-8023. Erasure Coding: retrieve eraure coding schema for a file from - NameNode (vinayakumarb) - - HDFS-8074. Define a system-wide default EC schema. (Kai Zheng) - - HDFS-8077. Erasure coding: fix bugs in EC zone and symlinks. - (Jing Zhao and Zhe Zhang via Jing Zhao) - - HDFS-8104. Make hard-coded values consistent with the system default schema first before remove them. (Kai Zheng) - - HDFS-7889. Subclass DFSOutputStream to support writing striping layout files. (Li Bo via Kai Zheng) - - HDFS-8090. Erasure Coding: Add RPC to client-namenode to list all - ECSchemas loaded in Namenode. (vinayakumarb) - - HDFS-8122. Erasure Coding: Support specifying ECSchema during creation of ECZone. - (Vinayakumar B via Zhe Zhang) - - HDFS-8114. Erasure coding: Add auditlog FSNamesystem#createErasureCodingZone if this - operation fails. (Rakesh R via Zhe Zhang) - - HDFS-8123. Erasure Coding: Better to move EC related proto messages to a - separate erasurecoding proto file (Rakesh R via vinayakumarb) - - HDFS-7349. Support DFS command for the EC encoding (vinayakumarb) - - HDFS-8120. Erasure coding: created util class to analyze striped block groups. - (Contributed by Zhe Zhang and Li Bo via Jing Zhao) - - HDFS-7994. Detect if resevered EC Block ID is already used during namenode - startup. (Hui Zheng via szetszwo) - - HDFS-8167. BlockManager.addBlockCollectionWithCheck should check if the block is a striped block. (Hui Zheng via zhz). - - HDFS-8166. DFSStripedOutputStream should not create empty blocks. (Jing Zhao) - - HDFS-7937. Erasure Coding: INodeFile quota computation unit tests. - (Kai Sasaki via Jing Zhao) - - HDFS-8145. Fix the editlog corruption exposed by failed TestAddStripedBlocks. - (Jing Zhao) - - HDFS-8146. Protobuf changes for BlockECRecoveryCommand and its fields for - making it ready for transfer to DN (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8181. createErasureCodingZone sets retryCache state as false always - (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8190. StripedBlockUtil.getInternalBlockLength may have overflow error. - (szetszwo) - - HDFS-8216. TestDFSStripedOutputStream should use BlockReaderTestUtil to - create BlockReader. (szetszwo via Zhe Zhang) - - HDFS-8212. DistributedFileSystem.createErasureCodingZone should pass schema - in FileSystemLinkResolver. (szetszwo via Zhe Zhang) - - HDFS-8024. Erasure Coding: ECworker frame, basics, bootstraping and configuration. - (umamahesh) - - HDFS-8156. Add/implement necessary APIs even we just have the system default - schema. (Kai Zheng via Zhe Zhang) - - HDFS-8136. Client gets and uses EC schema when reads and writes a stripping - file. (Kai Sasaki via Kai Zheng) - - HDFS-8233. Fix DFSStripedOutputStream#getCurrentBlockGroupBytes when the last - stripe is at the block group boundary. (jing9) - - HDFS-8223. Should calculate checksum for parity blocks in DFSStripedOutputStream. - (Yi Liu via jing9) - - HDFS-8228. Erasure Coding: SequentialBlockGroupIdGenerator#nextValue may cause - block id conflicts (Jing Zhao via Zhe Zhang) - - HDFS-8033. Erasure coding: stateful (non-positional) read from files in - striped layout (Zhe Zhang) - - HDFS-8230. Erasure Coding: Ignore DatanodeProtocol#DNA_ERASURE_CODING_RECOVERY - commands from standbynode if any (vinayakumarb) - - HDFS-8189. ClientProtocol#createErasureCodingZone API was wrongly annotated - as Idempotent (vinayakumarb) - - HDFS-8235. Erasure Coding: Create DFSStripedInputStream in DFSClient#open. - (Kai Sasaki via jing9) - - HDFS-8272. Erasure Coding: simplify the retry logic in DFSStripedInputStream - (stateful read). (Jing Zhao via Zhe Zhang) - - HDFS-8282. Erasure coding: move striped reading logic to StripedBlockUtil. - (Zhe Zhang) - - HDFS-8183. Erasure Coding: Improve DFSStripedOutputStream closing of - datastreamer threads. (Rakesh R via Zhe Zhang) - - HDFS-8308. Erasure Coding: NameNode may get blocked in waitForLoadingFSImage() - when loading editlog. (jing9) - - HDFS-7949. WebImageViewer need support file size calculation with striped - blocks. (Rakesh R via Zhe Zhang) - - HDFS-8316. Erasure coding: refactor EC constants to be consistent with HDFS-8249. - (Zhe Zhang via jing9) - - HDFS-8281. Erasure Coding: implement parallel stateful reading for striped layout. - (jing9) - - HDFS-8137. Send the EC schema to DataNode via EC encoding/recovering command(umamahesh) - - HDFS-8242. Erasure Coding: XML based end-to-end test for ECCli commands - (Rakesh R via vinayakumarb) - - HDFS-8324. Add trace info to DFSClient#getErasureCodingZoneInfo(..) (vinayakumarb via - umamahesh) - - HDFS-7672. Handle write failure for stripping blocks and refactor the - existing code in DFSStripedOutputStream and StripedDataStreamer. (szetszwo) - - HDFS-7348. Erasure Coding: DataNode reconstruct striped blocks. - (Yi Liu via Zhe Zhang) - - HDFS-8334. Erasure coding: rename DFSStripedInputStream related test - classes. (Zhe Zhang) - - HDFS-8129. Erasure Coding: Maintain consistent naming for Erasure Coding related classes - EC/ErasureCoding - (umamahesh) - - HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream. - (Yi Liu via jing9) - - HDFS-8289. Erasure Coding: add ECSchema to HdfsFileStatus. (Yong Zhang via - jing9) - - HDFS-8355. Erasure Coding: Refactor BlockInfo and BlockInfoUnderConstruction. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-7678. Erasure coding: DFSInputStream with decode functionality (pread). - (Zhe Zhang) - - HDFS-8372. Erasure coding: compute storage type quotas for striped files, - to be consistent with HDFS-8327. (Zhe Zhang via jing9) - - HDFS-8368. Erasure Coding: DFS opening a non-existent file need to be - handled properly (Rakesh R via zhz) - - HDFS-8363. Erasure Coding: DFSStripedInputStream#seekToNewSource. (yliu) - - HDFS-8195. Erasure coding: Fix file quota change when we complete/commit - the striped blocks. (Takuya Fukudome via zhz) - - HDFS-8364. Erasure coding: fix some minor bugs in EC CLI - (Walter Su via vinayakumarb) - - HDFS-8391. NN should consider current EC tasks handling count from DN while - assigning new tasks. (umamahesh) - - HDFS-8367. BlockInfoStriped uses EC schema. (Kai Sasaki via Kai Zheng) - - HDFS-8352. Erasure Coding: test webhdfs read write stripe file. (waltersu4549) - - HDFS-8417. Erasure Coding: Pread failed to read data starting from not-first stripe. - (Walter Su via jing9) - - HDFS-8418. Fix the isNeededReplication calculation for Striped block in NN. - (Yi Liu via jing9) - - HDFS-8320. Erasure coding: consolidate striping-related terminologies. (zhz) - - HDFS-8366. Erasure Coding: Make the timeout parameter of polling blocking queue - configurable in DFSStripedOutputStream. (Li Bo) - - HDFS-8378. Erasure Coding: Few improvements for the erasure coding worker. - (Rakesh R via waltersu4549) - - HDFS-8375. Add cellSize as an XAttr to ECZone. ( Vinayakumar B via zhz). - - HDFS-8428. Erasure Coding: Fix the NullPointerException when deleting file. - (Yi Liu via zhz). - - HDFS-8323. Bump GenerationStamp for write faliure in DFSStripedOutputStream. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-8427. Remove dataBlockNum and parityBlockNum from BlockInfoStriped. - (Kai Sasaki via jing9) - - HDFS-8186. Erasure coding: Make block placement policy for EC file configurable. - (Walter Su via zhz) - - HDFS-8294. Erasure Coding: Fix Findbug warnings present in erasure coding. - (Rakesh R via zhz) - - HDFS-8441. Erasure Coding: make condition check earlier for setReplication. - (waltersu4549) - - HDFS-7768. Change fsck to support EC files. (Takanobu Asanuma via szetszwo) - - HDFS-8382. Remove chunkSize and initialize from erasure coder. (Kai Zheng) - - HDFS-8408. Revisit and refactor ErasureCodingInfo (vinayakumarb) - - HDFS-8479. Erasure coding: fix striping related logic in FSDirWriteFileOp to - sync with HDFS-8421. (Zhe Zhang via jing9) - - HDFS-8481. Erasure coding: remove workarounds in client side stripped blocks - recovering. (zhz) - - HDFS-8336. Expose some administrative erasure coding operations to HdfsAdmin - (Uma Maheswara Rao G via vinayakumarb) - - HDFS-8444. Erasure Coding: fix cannot rename a zone dir - (Walter Su via vinayakumarb) - - HDFS-8517. Fix a decoding issue in stripped block recovering in client side. - (Kai Zheng via jing9) - - HDFS-8453. Erasure coding: properly handle start offset for internal blocks - in a block group. (Zhe Zhang via jing9) - - HDFS-7621. Erasure Coding: update the Balancer/Mover data migration logic. - (Walter Su via zhz) - - HDFS-8328. Follow-on to update decode for DataNode striped blocks - reconstruction. (yliu) - - HDFS-8319. Erasure Coding: support decoding for stateful read. - (Jing Zhao via zhz) - - HDFS-8460. Erasure Coding: stateful read result doesn't match data - occasionally because of flawed test. (Walter Su via zhz) - - HDFS-8556. Erasure Coding: Fix usage of 'createZone' (vinayakumarb) - - HDFS-8571. Fix TestErasureCodingCli test (Vinayakumar B via waltersu4549) - - HDFS-8450. Erasure Coding: Consolidate erasure coding zone related - implementation into a single class (Rakesh R via vinayakumarb) - - HDFS-8585. Erasure Coding: Remove dataBlockNum and parityBlockNum from - StripedBlockProto. (Yi Liu via jing9) - - HDFS-8559. Erasure Coding: fix non-protobuf fsimage for striped blocks. - (Jing Zhao via yliu) - - HDFS-8580. Erasure coding: Persist cellSize in BlockInfoStriped and - StripedBlocksFeature. (Walter Su via jing9) - - HDFS-8466. Refactor BlockInfoContiguous and fix NPE in - TestBlockInfo#testCopyConstructor() (vinayakumarb) - - HDFS-8254. Avoid assigning a leading streamer in StripedDataStreamer to - tolerate datanode failure. (Tsz Wo Nicholas Sze via jing9) - - HDFS-8543. Erasure Coding: processOverReplicatedBlock() handles striped block. - (Walter Su via jing9) - - HDFS-8602. Erasure Coding: Client can't read(decode) the EC files which have - corrupt blocks. (jing9 and Kai Sasaki) - - HDFS-8567. Erasure Coding: SafeMode handles file smaller than a full stripe. - (Walter Su via jing9) - - HDFS-8253. DFSStripedOutputStream.closeThreads releases cellBuffers - multiple times. (Kai Sasaki via szetszwo) - - HDFS-8468. 2 RPC calls for every file read in DFSClient#open(..) resulting in - double Audit log entries (vinayakumarb) - - HDFS-8684. Erasure Coding: fix some block number calculation for striped - block. (yliu) - - HDFS-8461. Erasure coding: fix priority level of UnderReplicatedBlocks for - striped block. (Walter Su via jing9) - - HDFS-8719. Erasure Coding: client generates too many small packets when - writing parity data. (Li Bo via waltersu4549) - - HDFS-8563. Erasure Coding: fsck handles file smaller than a full stripe. - (Walter Su via jing9) - - HDFS-8484. Erasure coding: Two contiguous blocks occupy IDs belong to same - striped group. (Walter Su via jing9) - - HDFS-8744. Erasure Coding: the number of chunks in packet is not updated - when writing parity data. (Li Bo) - - HDFS-8669. Erasure Coding: handle missing internal block locations in - DFSStripedInputStream. (jing9) - - HDFS-8702. Erasure coding: update BlockManager.blockHasEnoughRacks(..) logic - for striped block. (Kai Sasaki via jing9) - - HDFS-8734. Erasure Coding: fix one cell need two packets. (Walter Su via - jing9) - - HDFS-8619. Erasure Coding: revisit replica counting for striped blocks. - (Jing Zhao via yliu) - - HDFS-8058. Erasure coding: use BlockInfo[] for both striped and contiguous - blocks in INodeFile. (Zhe Zhang and Yi Liu via zhz) - - HDFS-8787. Erasure coding: rename BlockInfoContiguousUC and BlockInfoStripedUC - to be consistent with trunk. (zhz) - - HDFS-8433. Erasure coding: set blockToken in LocatedStripedBlock.(waltersu4549) - - HDFS-8760. Erasure Coding: reuse BlockReader when reading the same block in pread. - (jing9) - - HDFS-8781. Erasure Coding: Correctly handle BlockManager#InvalidateBlocks for - striped block. (Yi Liu via jing9) - - HDFS-8813. Erasure Coding: Client no need to decode missing parity blocks. - (Walter Su via jing9) - - HDFS-8798. Erasure Coding: fix DFSStripedInputStream/DFSStripedOutputStream - re-fetch token when expired. (Walter Su via jing9) - - HDFS-8769. Erasure coding: unit test for SequentialBlockGroupIdGenerator. - (Rakesh R via waltersu4549) - - HDFS-8202. Improve end to end stirpping file test to add erasure recovering - test. (Xinwei Qin via zhz) - - HDFS-8804. Erasure Coding: use DirectBufferPool in DFSStripedInputStream for - buffer allocation. (jing9) - - HDFS-8399. Erasure Coding: unit test the behaviour of BlockManager recovery - work for the deleted blocks. (Rakesh R via zhz) - - HDFS-8857. Erasure Coding: Fix ArrayIndexOutOfBoundsException in - TestWriteStripedFileWithFailure. (Li Bo) - - HDFS-8827. Erasure Coding: Fix NPE when NameNode processes over-replicated - striped blocks. (Walter Su and Takuya Fukudome via jing9) - - HDFS-8854. Erasure coding: add ECPolicy to replace schema+cellSize in - hadoop-hdfs. (Walter Su via zhz) - - HDFS-8220. Erasure Coding: StripedDataStreamer fails to handle the - blocklocations which doesn't satisfy BlockGroupSize. (Rakesh R via zhz) - - HDFS-8838. Erasure Coding: Tolerate datanode failures in DFSStripedOutputStream - when the data length is small. (szetszwo via waltersu4549) - - HDFS-8909. Erasure coding: update BlockInfoContiguousUC and BlockInfoStripedUC - to use BlockUnderConstructionFeature. (Jing Zhao via waltersu4549) - - HDFS-8937. Erasure coding: do not throw exception when setting replication on - EC file. (Gao Rui via jing9) - - HDFS-8978. Erasure coding: fix 2 failed tests of DFSStripedOutputStream. - (Walter Su via jing9) - - HDFS-8833. Erasure coding: store EC schema and cell size in INodeFile and - eliminate notion of EC zones. (zhz) - - HDFS-8853. Erasure Coding: Provide ECSchema validation when setting EC - policy. (andreina via zhz) - - HDFS-8975. Erasure coding : Fix random failure in TestSafeModeWithStripedFile - (J.Andreina via vinayakumarb) - - HDFS-7351. Document the HDFS Erasure Coding feature. - (umamahesh and Zhe Zhang via wang) - - HDFS-8899. Erasure Coding: use threadpool for EC recovery tasks on DataNode. - (Rakesh R via zhz) - - HDFS-9088. Cleanup erasure coding documentation. (wang via zhz) - - HDFS-9097. Erasure coding: update EC command "-s" flag to "-p" when - specifying policy. (zhz) - - HDFS-9086. Rename dfs.datanode.stripedread.threshold.millis to - dfs.datanode.stripedread.timeout.millis. (wang via zhz) - - HDFS-8550. Erasure Coding: Fix FindBugs Multithreaded correctness Warning. - (Rakesh R via zhz) - - HDFS-9113. ErasureCodingWorker#processErasureCodingTasks should not fail to process - remaining tasks due to one invalid ECTask (umamahesh) - - HDFS-9091. Erasure Coding: Provide DistributedFilesystem API to - getAllErasureCodingPolicies. (Rakesh R via zhz) - - HDFS-8920. Erasure Coding: when recovering lost blocks, logs can be too - verbose and hurt performance. (Rui Li via Kai Zheng) - - HDFS-8882. Erasure Coding: Use datablocks, parityblocks and cell size from - ErasureCodingPolicy (Vinayakumar B via zhz) - - HDFS-9040. Erasure coding: coordinate data streamers in - DFSStripedOutputStream. (jing9 and Walter Su) - - HDFS-9185. Fix null tracer in ErasureCodingWorker. (Rakesh R via jing9) - - HDFS-9180. Update excluded DataNodes in DFSStripedOutputStream based on failures - in data streamers. (jing9) - - HDFS-8557. Allow to configure RS and XOR raw coders (Kai Zheng) - - HDFS-9206. Inconsistent default value of dfs.datanode.stripedread.buffer.size. - (Walter Su via jing9) - - HDFS-9209. Erasure coding: Add apache license header in - TestFileStatusWithECPolicy.java. (Surendra Singh Lilhore via jing9) - - HDFS-8438. Erasure Coding: Allow concat striped files if they have the same - ErasureCodingPolicy. (Walter Su via jing9) - - HDFS-9275. Wait previous ErasureCodingWork to finish before schedule - another one. (Walter Su via yliu) - - HDFS-9348. Erasure Coding: DFS GetErasureCodingPolicy API on a non-existent - file should be handled properly. (Rakesh R via umamahesh) - - HDFS-9373. Erasure coding: friendly log information for write operations - with some failed streamers. (Li Bo via zhz) - - HDFS-9173. Erasure Coding: Lease recovery for striped file. (Walter Su and - Jing Zhao via zhz) - - HDFS-9451. Clean up depreated umasks and related unit tests. - (Wei-Chiu Chuang via wheat9) - - HDFS-9575. Use byte array for internal block indices in a striped block. - (jing9 via szetszwo) - - HDFS-9659. EditLogTailerThread to Active Namenode RPC should timeout - (surendra singh lilhore via vinayakumarb) - - HDFS-9731. Erasure Coding: Rename BlockECRecoveryCommand to - BlockECReconstructionCommand. (Rakesh R via zhz) - - HDFS-9403. Erasure coding: some EC tests are missing timeout. - (Gao Rui via zhz) - - HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping - related tests. (Rui Li via zhz) - - HDFS-9775. Erasure Coding : Rename BlockRecoveryWork to - BlockReconstructionWork. (Rakesh R via zhz) - - HDFS-9755. Erasure Coding: allow to use multiple EC policies in striping - related tests [Part 2]. (Rui Li via zhz) - - HDFS-9787. SNNs stop uploading FSImage to ANN once isPrimaryCheckPointer - changed to false. (Guocui Mi via vinayakumarb) - - HDFS-9830. Remove references to hftp in ViewFs documentation in trunk. - (Wei-Chiu Chuang via aajisaka) - - HDFS-9816. Erasure Coding: allow to use multiple EC policies in striping - related tests [Part 3]. (Rui Li via zhz) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - HDFS-9047. Retire libwebhdfs. (wheat9) - - NEW FEATURES - HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from - an XML file (cmccabe) - - IMPROVEMENTS - - HDFS-9267. TestDiskError should get stored replicas through - FsDatasetTestUtils. (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-9491. Tests should get the number of pending async delets via - FsDatasetTestUtils. (Tony Wu via lei) - - HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for - reusability. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-8947. NameNode, DataNode and NFS gateway to support JvmPauseMonitor as - a service. (Sunil G via Stevel) - - HDFS-9129. Move the safemode block count into BlockManager. (Mingliang Liu - via jing9) - - HDFS-9281. Change TestDeleteBlockPool to not explicitly use File to check - block pool existence. (lei) - - HDFS-9371. Code cleanup for DatanodeManager. (jing9) - - HDFS-9300. TestDirectoryScanner.testThrottle() is still a little flakey. - (Daniel Templeton via wang) - - HDFS-8477. describe dfs.ha.zkfc.port in hdfs-default.xml. - (Kanaka Kumar Avvaru via wang) - - HDFS-9350. Avoid creating temprorary strings in Block.toString() and - getBlockName() (Staffan Friberg via cmccabe) - - HDFS-9624. DataNode start slowly due to the initial DU command operations. - (Lin Yiqun via wang) - - HDFS-9576: HTrace: collect position/length information on read operations - (zhz via cmccabe) - - HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. (jing9) - - HDFS-9674. The HTrace span for OpWriteBlock should record the maxWriteToDisk - time. (cmccabe via zhz) - - HDFS-9094. Add command line option to ask NameNode reload - configuration. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9541. Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes - greater than 2 GB. (cmccabe via zhz) - - HDFS-9677. Rename generationStampV1/generationStampV2 to - legacyGenerationStamp/generationStamp. (Mingliang Liu via jing9) - - HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had - an error (Rakesh R via cmccabe) - - HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) - - HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. - (Daniel Templeton via kasha) - - HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. - (jing9) - - HDFS-7964. Add support for async edit logging. (Daryn Sharp) - - HDFS-9534. Add CLI command to clear storage policy from a path. - (Xiaobing Zhou via Arpit Agarwal) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9618. Fix mismatch between log level and guard in - BlockManager#computeRecoveryWorkForBlocks (iwasakims) - - HDFS-9621. getListing wrongly associates Erasure Coding policy to pre-existing - replicated files under an EC directory. (jing9) - - HDFS-9691. TestBlockManagerSafeMode#testCheckSafeMode fails intermittently. - (Mingliang Liu via aajisaka) - - HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. - (Wei Zhou via wang) - - HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure clusters. - (Daniel Templeton via kasha) - -Release 2.8.0 - UNRELEASED - - NEW FEATURES - - HDFS-7891. A block placement policy with best rack failure tolerance. - (Walter Su via szetszwo) - - HDFS-8131. Implement a space balanced block placement policy (Liu Shaohui - via kihwal) - - HDFS-8155. Support OAuth2 in WebHDFS. (jghoman) - - HDFS-9184. Logging HDFS operation's caller context into audit logs. - (Mingliang Liu via jitendra) - - HDFS-9244. Support nested encryption zones. (zhz) - - HDFS-9711. Integrate CSRF prevention filter in WebHDFS. (cnauroth) - - IMPROVEMENTS - - HDFS-9257. improve error message for "Absolute path required" in INode.java - to contain the rejected path (Marcell Szabo via harsh) - - HDFS-2390. dfsadmin -setBalancerBandwidth does not validate -ve value - (Gautam Gopalakrishnan via harsh) - - HDFS-8821. Explain message "Operation category X is not supported - in state standby" (Gautam Gopalakrishnan via harsh) - - HDFS-3918. EditLogTailer shouldn't log WARN when other node - is in standby mode (todd via harsh) - - HDFS-4396. Add START_MSG/SHUTDOWN_MSG for ZKFC - (Liang Xie via harsh) - - HDFS-7875. Improve log message when wrong value configured for - dfs.datanode.failed.volumes.tolerated. - (nijel via harsh) - - HDFS-2360. Ugly stacktrace when quota exceeds. (harsh) - - HDFS-7835. make initial sleeptime in locateFollowingBlock configurable for - DFSClient. (Zhihai Xu via Yongjun Zhang) - - HDFS-7829. Code clean up for LocatedBlock. (Takanobu Asanuma via jing9) - - HDFS-7854. Separate class DataStreamer out of DFSOutputStream. (Li Bo via - jing9) - - HDFS-7713. Implement mkdirs in the HDFS Web UI. (Ravi Prakash via wheat9) - - HDFS-7928. Scanning blocks from disk during rolling upgrade startup takes - a lot of time if disks are busy (Rushabh S Shah via kihwal) - - HDFS-7990. IBR delete ack should not be delayed. (daryn via kihwal) - - HDFS-8004. Use KeyProviderCryptoExtension#warmUpEncryptedKeys when creating - an encryption zone. (awang via asuresh) - - HDFS-6263. Remove DRFA.MaxBackupIndex config from log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-6408. Remove redundant definitions in log4j.properties. - (Abhiraj Butala via aajisaka) - - HDFS-7890. Improve information on Top users for metrics in - RollingWindowsManager and lower log level (J.Andreina via vinayakumarb) - - HDFS-7645. Rolling upgrade is restoring blocks from trash multiple times. - (Vinayakumar B and Keisuke Ogiwara via Arpit Agarwal) - - HDFS-7944. Minor cleanup of BlockPoolManager#getAllNamenodeThreads. - (Arpit Agarwal) - - HDFS-7671. hdfs user guide should point to the common rack awareness doc. - (Kai Sasaki via aajisaka) - - HDFS-8009. Signal congestion on the DataNode. (wheat9) - - HDFS-7978. Add LOG.isDebugEnabled() guard for some LOG.debug(..). - (Walter Su via wang) - - HDFS-7888. Change DFSOutputStream and DataStreamer for convenience of - subclassing. (Li Bo via szetszwo) - - HDFS-8049. Add @InterfaceAudience.Private annotation to hdfs client - implementation. (Takuya Fukudome via szetszwo) - - HDFS-8079. Move dfs.client.retry.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Retry. (szetszwo) - - HDFS-8073. Split BlockPlacementPolicyDefault.chooseTarget(..) so it - can be easily overrided. (Walter Su via vinayakumarb) - - HDFS-8080. Separate JSON related routines used by WebHdfsFileSystem to a - package local class. (wheat9) - - HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. - (szetszwo) - - HDFS-8076. Code cleanup for DFSInputStream: use offset instead of - LocatedBlock when possible. (Zhe Zhang via wang) - - HDFS-8025. Addendum fix for HDFS-3087 Decomissioning on NN restart can - complete without blocks being replicated. (Ming Ma via wang) - - HDFS-8089. Move o.a.h.hdfs.web.resources.* to the client jars. (wheat9) - - HDFS-7979. Initialize block report IDs with a random number. (wang) - - HDFS-8101. DFSClient use of non-constant DFSConfigKeys pulls in WebHDFS - classes at runtime. (Sean Busbey via atm) - - HDFS-8102. Separate webhdfs retry configuration keys from DFSConfigKeys. - (wheat9) - - HDFS-8100. Refactor DFSClient.Conf to a standalone class and separates - short-circuit related conf to ShortCircuitConf. (szetszwo) - - HDFS-8103. Move BlockTokenSecretManager.AccessMode into - BlockTokenIdentifier. (wheat9) - - HDFS-8084. Move dfs.client.failover.* confs from DFSConfigKeys to - HdfsClientConfigKeys.Failover and fix typos in the dfs.http.client.* - configuration keys. (szetszwo) - - HDFS-7933. fsck should also report decommissioning replicas. - (Xiaoyu Yao via cnauroth) - - HDFS-8083. Move dfs.client.write.* conf from DFSConfigKeys to - HdfsClientConfigKeys.Write. (szetszwo) - - HDFS-8117. More accurate verification in SimulatedFSDataset: replace - DEFAULT_DATABYTE with patterned data. (Zhe Zhang via wang) - - HDFS-8144. Split TestLazyPersistFiles into multiple tests. (Arpit Agarwal) - - HDFS-8082. Move dfs.client.read.*, dfs.client.short.circuit.*, - dfs.client.mmap.* and dfs.client.hedged.read.* conf from DFSConfigKeys - to HdfsClientConfigKeys. (szetszwo) - - HDFS-8165. Move GRANDFATHER_GENERATION_STAMP and GRANDFATER_INODE_ID to - hdfs-client. (wheat9) - - HDFS-8152. Refactoring of lazy persist storage cases. (Arpit Agarwal) - - HDFS-8169. Move LocatedBlocks and related classes to hdfs-client. (wheat9) - - HDFS-8133. Improve readability of deleted block check (Daryn Sharp via - Colin P. McCabe) - - HDFS-8185. Separate client related routines in HAUtil into a new class. - (wheat9) - - HDFS-8218. Move classes that used by ClientProtocol into hdfs-client. - (wheat9) - - HDFS-4448. Allow HA NN to start in secure mode with wildcard address - configured (atm via asuresh) - - HDFS-8215. Refactor NamenodeFsck#check method. (Takanobu Asanuma - via szetszwo) - - HDFS-8052. Move WebHdfsFileSystem into hadoop-hdfs-client. (wheat9) - - HDFS-8176. Record from/to snapshots in audit log for snapshot diff report. - (J. Andreina via jing9) - - HDFS-8280. Code Cleanup in DFSInputStream. (Jing Zhao via wheat9) - - HDFS-8283. DataStreamer cleanup and some minor improvement. (szetszwo via - jing9) - - HDFS-5574. Remove buffer copy in BlockReader.skip. - (Binglin Chang via aajisaka) - - HDFS-8200. Refactor FSDirStatAndListingOp. (wheat9) - - HDFS-8292. Move conditional in fmt_time from dfs-dust.js to status.html. - (Charles Lamb via wang) - - HDFS-8086. Move LeaseRenewer to the hdfs.client.impl package. (Takanobu - Asanuma via szetszwo) - - HDFS-8249. Separate HdfsConstants into the client and the server side - class. (wheat9) - - HDFS-7397. Add more detail to the documentation for the conf key - "dfs.client.read.shortcircuit.streams.cache.size" (Brahma Reddy Battula via - Colin P. McCabe) - - HDFS-8237. Move all protocol classes used by ClientProtocol to hdfs-client. - (wheat9) - - HDFS-7758. Retire FsDatasetSpi#getVolumes() and use - FsDatasetSpi#getVolumeRefs() instead (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote - NameNode (Charles Lamb via Colin P. McCabe) - - HDFS-8314. Move HdfsServerConstants#IO_FILE_BUFFER_SIZE and - SMALL_BUFFER_SIZE to the users. (Li Lu via wheat9) - - HDFS-8207. Improper log message when blockreport interval compared with - initial delay. (Brahma Reddy Battula and Ashish Singhi via ozawa) - - HDFS-7559. Create unit test to automatically compare HDFS related classes - and hdfs-default.xml. (Ray Chiang via asuresh) - - HDFS-5640. Add snapshot methods to FileContext. (Rakesh R via cnauroth) - - HDFS-8284. Update documentation about how to use HTrace with HDFS (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-8113. Add check for null BlockCollection pointers in - BlockInfoContiguous structures (Chengbing Liu via Colin P. McCabe) - - HDFS-6757. Simplify lease manager with INodeID. (wheat9) - - HDFS-8327. Simplify quota calculations for snapshots and truncate. (wheat9) - - HDFS-8357. Consolidate parameters of INode.CleanSubtree() into a parameter - objects. (Li Lu via wheat9) - - HDFS-8255. Rename getBlockReplication to getPreferredBlockReplication. - (Contributed by Zhe Zhang) - - HDFS-6184. Capture NN's thread dump when it fails over. - (Ming Ma via aajisaka) - - HDFS-8350. Remove old webhdfs.xml and other outdated documentation stuff. - (Brahma Reddy Battula via aajisaka) - - HDFS-6888. Allow selectively audit logging ops (Chen He via vinayakumarb) - - HDFS-8397. Refactor the error handling code in DataStreamer. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-8394. Move getAdditionalBlock() and related functionalities into a - separate class. (wheat9) - - HDFS-8157. Writes to RAM DISK reserve locked memory for block files. - (Arpit Agarwal) - - HDFS-8345. Storage policy APIs must be exposed via the FileSystem - interface. (Arpit Agarwal) - - HDFS-4185. Add a metric for number of active leases (Rakesh R via raviprak) - - HDFS-4383. Document the lease limits. (Arshad Mohammad via aajisaka) - - HDFS-8421. Move startFile() and related functions into FSDirWriteFileOp. - (wheat9) - - HDFS-8377. Support HTTP/2 in datanode. (Duo Zhang via wheat9) - - HDFS-8482. Rename BlockInfoContiguous to BlockInfo. (Zhe Zhang via wang) - - HDFS-8443. Document dfs.namenode.service.handler.count in hdfs-site.xml. - (J.Andreina via aajisaka) - - HDFS-8489. Subclass BlockInfo to represent contiguous blocks. - (Zhe Zhang via jing9) - - HDFS-8386. Improve synchronization of 'streamer' reference in - DFSOutputStream. (Rakesh R via wang) - - HDFS-8513. Rename BlockPlacementPolicyRackFaultTolarent to - BlockPlacementPolicyRackFaultTolerant. (wang) - - HDFS-8532. Make the visibility of DFSOutputStream#streamer member variable - to private. (Rakesh R via wang) - - HDFS-8535. Clarify that dfs usage in dfsadmin -report output includes all - block replicas. (Eddy Xu via wang) - - HDFS-8432. Introduce a minimum compatible layout version to allow downgrade - in more rolling upgrade use cases. (cnauroth) - - HDFS-8116. Cleanup uncessary if LOG.isDebugEnabled() from - RollingWindowManager. (Brahma Reddy Battula via xyao) - - HDFS-8553. Document hdfs class path options. - (Brahma Reddy Battula via cnauroth) - - HDFS-8552. Fix hdfs CLI usage message for namenode and zkfc. - (Brahma Reddy Battula via xyao) - - HDFS-8568. TestClusterId#testFormatWithEmptyClusterIdOption is failing. - (Rakesh R. via xyao) - - HDFS-8549. Abort the balancer if an upgrade is in progress. (wang) - - HDFS-8573. Move creation of restartMeta file logic from BlockReceiver to - ReplicaInPipeline. (Eddy Xu via wang) - - HDFS-7923. The DataNodes should rate-limit their full block reports by - asking the NN on heartbeat messages (cmccabe) - - HDFS-8540. Mover should exit with NO_MOVE_BLOCK if no block can be moved. - (surendra singh lilhore via szetszwo) - - HDFS-8606. Cleanup DFSOutputStream by removing unwanted changes - from HDFS-8386. (Rakesh R via szetszwo) - - HDFS-9608. Merge HDFS-7912 to trunk and branch-2 (track BlockInfo instead - of Block in UnderReplicatedBlocks and PendingReplicationBlocks). - (Zhe Zhang via wang) - - HDFS-8238. Move ClientProtocol to the hdfs-client. - (Takanobu Asanuma via wheat9) - - HDFS-8446. Separate safemode related operations in GetBlockLocations(). - (wheat9) - - HDFS-8589. Fix unused imports in BPServiceActor and BlockReportLeaseManager - (cmccabe) - - HDFS-6249. Output AclEntry in PBImageXmlWriter. - (surendra singh lilhore via aajisaka) - - HDFS-8605. Merge Refactor of DFSOutputStream from HDFS-7285 branch. - (vinayakumarb via wang) - - HDFS-8582. Support getting a list of reconfigurable config properties and - do not generate spurious reconfig warnings (Lei (Eddy) Xu via Colin P. - McCabe) - - HDFS-8192. Eviction should key off used locked memory instead of - ram disk free space. (Arpit Agarwal) - - HDFS-6564. Use slf4j instead of common-logging in hdfs-client. - (Rakesh R via wheat9) - - HDFS-8639. Add Option for NameNode HTTP port in MiniDFSClusterManager. - (Kai Sasaki via jing9) - - HDFS-8462. Implement GETXATTRS and LISTXATTRS operations for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-8640. Make reserved RBW space visible through JMX. (kanaka kumar - avvaru via Arpit Agarwal) - - HDFS-8665. Fix replication check in DFSTestUtils#waitForReplication. (wang) - - HDFS-8546. Use try with resources in DataStorage and Storage. (wang) - - HDFS-8651. Make hadoop-hdfs-project Native code -Wall-clean (Alan Burlison - via Colin P. McCabe) - - HDFS-8653. Code cleanup for DatanodeManager, DatanodeDescriptor and - DatanodeStorageInfo. (Zhe Zhang via wang) - - HDFS-8493. Consolidate truncate() related implementation in a single class. - (Rakesh R via wheat9) - - HDFS-8635. Migrate HDFS native build to new CMake framework (Alan Burlison - via Colin P. McCabe) - - HDFS-8666. Speedup the TestMover tests. (Walter Su via jing9) - - HDFS-8703. Merge refactor of DFSInputStream from ErasureCoding branch - (vinayakumarb) - - HDFS-8709. Clarify automatic sync in FSEditLog#logEdit. (wang) - - HDFS-8711. setSpaceQuota command should print the available storage type - when input storage type is wrong. (Brahma Reddy Battula via xyao) - - HDFS-8620. Clean up the checkstyle warinings about ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8712. Remove 'public' and 'abstracta modifiers in FsVolumeSpi and - FsDatasetSpi (Lei (Eddy) Xu via vinayakumarb) - - HDFS-8726. Move protobuf files that define the client-sever protocols to - hdfs-client. (wheat9) - - HDFS-8751. Remove setBlocks API from INodeFile and misc code cleanup. (Zhe - Zhang via jing9) - - HDFS-8541. Mover should exit with NO_MOVE_PROGRESS if there is no move - progress. (Surendra Singh Lilhore via szetszwo) - - HDFS-8742. Inotify: Support event for OP_TRUNCATE. - (Surendra Singh Lilhore via aajisaka) - - HDFS-8794. Improve CorruptReplicasMap#corruptReplicasMap. (yliu) - - HDFS-7483. Display information per tier on the Namenode UI. - (Benoy Antony and wheat9 via wheat9) - - HDFS-8721. Add a metric for number of encryption zones. - (Rakesh R via cnauroth) - - HDFS-8495. Consolidate append() related implementation into a single class. - (Rakesh R via wheat9) - - HDFS-8795. Improve InvalidateBlocks#node2blocks. (yliu) - - HDFS-8797. WebHdfsFileSystem creates too many connections for pread. (jing9) - - HDFS-8730. Clean up the import statements in ClientProtocol. - (Takanobu Asanuma via wheat9) - - HDFS-8735. Inotify: All events classes should implement toString() API. - (Surendra Singh Lilhore via aajisaka) - - HDFS-7858. Improve HA Namenode Failover detection on the client. (asuresh) - - HDFS-8180. AbstractFileSystem Implementation for WebHdfs. (snayak via jghoman) - - HDFS-8811. Move BlockStoragePolicy name's constants from - HdfsServerConstants.java to HdfsConstants.java (vinayakumarb) - - HDFS-8822. Add SSD storagepolicy tests in TestBlockStoragePolicy# - testDefaultPolicies (vinayakumarb) - - HDFS-8816. Improve visualization for the Datanode tab in the NN UI. (wheat9) - - HDFS-7192. DN should ignore lazyPersist hint if the writer is - not local. (Arpit Agarwal) - - HDFS-6860. BlockStateChange logs are too noisy. (Chang Li and xyao via xyao) - - HDFS-8815. DFS getStoragePolicy implementation using single RPC call - (Surendra Singh Lilhore via vinayakumarb) - - HDFS-8856. Make LeaseManager#countPath O(1). (Arpit Agarwal) - - HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. - (Walter Su via wang) - - HDFS-8818. Changes the global moveExecutor to per datanode executors and - changes MAX_SIZE_TO_MOVE to be configurable. (szetszwo) - - HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. - (Brahma Reddy Battula via jing9) - - HDFS-8887. Expose storage type and storage ID in BlockLocation. (wang) - - HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer. - (Jagadesh Kiran N via aajisaka) - - HDFS-7649. Multihoming docs should emphasize using hostnames in - configurations. (Brahma Reddy Battula via Arpit Agarwal) - - HDFS-8824. Do not use small blocks for balancing the cluster. (szetszwo) - - HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. - (Anu Engineer via xyao) - - HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. (wang) - - HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. - (wheat9) - - HDFS-8801. Convert BlockInfoUnderConstruction as a feature. - (Jing Zhao via wheat9) - - HDFS-8880. NameNode metrics logging. (Arpit Agarwal) - - HDFS-8278. When computing max-size-to-move in Balancer, count only the - storage with remaining >= default block size. (szetszwo) - - HDFS-8435. Support CreateFlag in WebHDFS. (Jakob Homan via cdouglas) - - HDFS-8826. In Balancer, add an option to specify the source node list - so that balancer only selects blocks to move from those nodes. (szetszwo) - - HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. - (Anu Engineer via Arpit Agarwal) - - HDFS-8803. Move DfsClientConf to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8917. Cleanup BlockInfoUnderConstruction from comments and tests. - (Zhe Zhang via jing9) - - HDFS-8884. Fail-fast check in BlockPlacementPolicyDefault#chooseTarget. - (yliu) - - HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. - (Yufei Gu via Yongjun Zhang) - - HDFS-8823. Move replication factor into individual blocks. (wheat9) - - HDFS-8934. Move ShortCircuitShm to hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8928. Improvements for BlockUnderConstructionFeature: - ReplicaUnderConstruction as a separate class and replicas as an array. - (jing9) - - HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) - - HDFS-8951. Move the shortcircuit package to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC - root in ShutdownHookManager. (Walter Su via jing9) - - HDFS-8248. Store INodeId instead of the INodeFile object in - BlockInfoContiguous. (wheat9) - - HDFS-8962. Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf. - (Mingliang Liu via wheat9) - - HDFS-8865. Improve quota initialization performance. (kihwal) - - HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone - classes from BlockManager. (Mingliang Liu via wheat9) - - HDFS-8925. Move BlockReaderLocal to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-8983. NameNode support for protected directories. (Arpit Agarwal) - - HDFS-8980. Remove unnecessary block replacement in INodeFile. (jing9) - - HDFS-8990. Move RemoteBlockReader to hdfs-client module. - (Mingliang via wheat9) - - HDFS-8946. Improve choosing datanode storage for block placement. (yliu) - - HDFS-8965. Harden edit log reading code against out of memory errors (cmccabe) - - HDFS-2070. Add more unit tests for FsShell getmerge (Daniel Templeton via - Colin P. McCabe) - - HDFS-328. Improve fs -setrep error message for invalid replication factors. - (Daniel Templeton via wang) - - HDFS-8890. Allow admin to specify which blockpools the balancer should run - on. (Chris Trezzo via mingma) - - HDFS-9002. Move o.a.h.hdfs.net/*Peer classes to hdfs-client. - (Mingliang Liu via wheat9) - - HDFS-9021. Use a yellow elephant rather than a blue one in diagram. (wang) - - HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to - hadoop-hdfs-client module. (Mingliang Liu via wheat9) - - HDFS-8984. Move replication queues related methods in FSNamesystem to - BlockManager. (wheat9) - - HDFS-9019. Adding informative message to sticky bit permission denied - exception. (xyao) - - HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8716. Introduce a new config specifically for safe mode block count - (Chang Li via kihwal) - - HDFS-7116. Add a command to get the balancer bandwidth - (Rakesh R via vinayakumarb) - - HDFS-8974. Convert docs in xdoc format to markdown. - (Masatake Iwasaki via aajisaka) - - HDFS-6763. Initialize file system-wide quota once on transitioning to active - (kihwal) - - HDFS-9027. Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method. - (Mingliang Liu via Arpit Agarwal) - - HDFS-8996. Consolidate validateLog and scanLog in FJM#EditLogFile (Zhe - Zhang via Colin P. McCabe) - - HDFS-9010. Replace NameNode.DEFAULT_PORT with HdfsClientConfigKeys. - DFS_NAMENODE_RPC_PORT_DEFAULT config key. (Mingliang Liu via wheat9) - - HDFS-9065. Include commas on # of files, blocks, total filesystem objects - in NN Web UI. (Daniel Templeton via wheat9) - - HDFS-9008. Balancer#Parameters class could use a builder pattern. - (Chris Trezzo via mingma) - - HDFS-8953. DataNode Metrics logging (Kanaka Kumar Avvaru via vinayakumarb) - - HDFS-9082. Change the log level in WebHdfsFileSystem.initialize() from INFO - to DEBUG. (Santhosh Nayak via cnauroth) - - HDFS-7986. Allow files / directories to be deleted from the NameNode UI. - (Ravi Prakash via wheat9) - - HDFS-7995. Implement chmod in the HDFS Web UI. - (Ravi Prakash and Haohui Mai via wheat9) - - HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-5802. NameNode does not check for inode type before traversing down a - path. (Xiao Chen via Yongjun Zhang) - - HDFS-9101. Remove deprecated NameNode.getUri() static helper method. - (Mingliang Liu via wheat9) - - HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to - PBHelperClient. (Mingliang Liu via wheat9) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - HDFS-9039. Separate client and server side methods of o.a.h.hdfs. - NameNodeProxies. (Mingliang Liu via wheat9) - - HDFS-8733. Keep server related definition in hdfs.proto on server side. - (Mingliang Liu via wheat9) - - HDFS-9130. Use GenericTestUtils#setLogLevel to the logging level. - (Mingliang Liu via wheat9) - - HDFS-9131 Move config keys used by hdfs-client to HdfsClientConfigKeys. - (Mingliang Liu via wheat9) - - HDFS-7529. Consolidate encryption zone related implementation into a single - class. (Rakesh R via wheat9) - - HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from - HdfsServerConstants to HdfsConstants. (Mingliang Liu via wheat9) - - HDFS-5795. RemoteBlockReader2#checkSuccess() shoud print error status. - (Xiao Chen via Yongjun Zhang) - - HDFS-9112. Improve error message for Haadmin when multiple name service IDs - are configured. (Anu Engineer via jing9) - - HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via - Lei (Eddy) Xu) - - HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read - when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) - - HDFS-8873. Allow the directoryScanner to be rate-limited (Daniel Templeton - via Colin P. McCabe) - - HDFS-8053. Move DFSIn/OutputStream and related classes to - hadoop-hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9087. Add some jitter to DataNode.checkDiskErrorThread (Elliott Clark - via Colin P. McCabe) - - HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. (Mingliang Liu - via wheat9) - - HDFS-9080. Update htrace version to 4.0.1 (cmccabe) - - HDFS-9148. Incorrect assert message in TestWriteToReplica#testWriteToTemporary - (Tony Wu via lei) - - HDFS-8859. Improve DataNode ReplicaMap memory footprint to save about 45%. - (yliu) - - HDFS-9165. Move entries in META-INF/services/o.a.h.fs.FileSystem to - hdfs-client. (Mingliang Liu via wheat9) - - HDFS-8696. Make the lower and higher watermark in the DN Netty server - configurable. (Xiaobing Zhou via wheat9) - - HDFS-8971. Remove guards when calling LOG.debug() and LOG.trace() in client - package. (Mingliang Liu via wheat9) - - HDFS-9175. Change scope of 'AccessTokenProvider.getAccessToken()' and - 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to - public (Santhosh Nayak via cnauroth) - - HDFS-9015. Refactor TestReplicationPolicy to test different block placement - policies. (Ming Ma via lei) - - HDFS-8979. Clean up checkstyle warnings in hadoop-hdfs-client module. - (Mingliang Liu via wheat9) - - HDFS-9155. OEV should treat .XML files as XML even when the file name - extension is uppercase (nijel via cmccabe) - - HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. (wheat9) - - HDFS-8164. cTime is 0 in VERSION file for newly formatted NameNode. - (Xiao Chen via Yongjun Zhang) - - HDFS-9181. Better handling of exceptions thrown during upgrade shutdown. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9110. Use Files.walkFileTree in NNUpgradeUtil#doPreUpgrade for - better efficiency. (Charlie Helin via wang) - - HDFS-8988. Use LightWeightHashSet instead of LightWeightLinkedSet in - BlockManager#excessReplicateMap. (yliu) - - HDFS-9006. Provide BlockPlacementPolicy that supports upgrade domain. - (Ming Ma via lei) - - HDFS-1172. Blocks in newly completed files are considered under-replicated - too quickly. (Masatake Iwasaki via jing9) - - HDFS-9238. Update TestFileCreation.testLeaseExpireHardLimit() to avoid using - DataNodeTestUtils.getFile(). (Tony Wu via lei) - - HDFS-9223. Code cleanup for DatanodeDescriptor and HeartbeatManager. (jing9) - - HDFS-9188. Make block corruption related tests FsDataset-agnostic. (lei) - - HDFS-9205. Do not schedule corrupt blocks for replication. (szetszwo) - - HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. - (Xiao Chen via wang) - - HDFS-9251. Refactor TestWriteToReplica and TestFsDatasetImpl to avoid - explicitly creating Files in the tests code. (lei) - - HDFS-8647. Abstract BlockManager's rack policy into BlockPlacementPolicy. - (Brahma Reddy Battula via mingma) - - HDFS-9225. Fix intermittent test failure of - TestBlockManager.testBlocksAreNotUnderreplicatedInSingleRack. - (Masatake Iwasaki via wang) - - HDFS-7087. Ability to list /.reserved. (Xiao Chen via wang) - - HDFS-9264. Minor cleanup of operations on FsVolumeList#volumes. - (Walter Su via lei) - - HDFS-8808. dfs.image.transfer.bandwidthPerSec should not apply to - -bootstrapStandby (zhz) - - HDFS-4015. Safemode should count and report orphaned blocks. - (Anu Engineer via Arpit Agarwal) - - HDFS-7284. Add more debug info to - BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9284. fsck command should not print exception trace when file not - found. (Jagadesh Kiran N via wang) - - HDFS-9291. Fix TestInterDatanodeProtocol to be FsDataset-agnostic. (lei) - - HDFS-8945. Update the description about replica placement in HDFS - Architecture documentation. (Masatake Iwasaki via wang) - - HDFS-9292. Make TestFileConcorruption independent to underlying FsDataset - Implementation. (lei) - - HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs - write scenario. (Mingliang Liu via mingma) - - HDFS-9299. Give ReplicationMonitor a readable thread name (Staffan Friberg - via Colin P. McCabe) - - HDFS-9307. fuseConnect should be private to fuse_connect.c (Mingliang Liu - via Colin P. McCabe) - - HDFS-9311. Support optional offload of NameNode HA service health checks to - a separate RPC server. (cnauroth) - - HDFS-9255. Consolidate block recovery related implementation into a single - class. (Walter Su via zhz) - - HDFS-9295. Add a thorough test of the full KMS code path. - (Daniel Templeton via zhz) - - HDFS-8545. Refactor FS#getUsed() to use ContentSummary and add an API to fetch - the total file length from a specific path (J.Andreina via vinayakumarb) - - HDFS-9229. Expose size of NameNode directory as a metric. - (Surendra Singh Lilhore via zhz) - - HDFS-9339. Extend full test of KMS ACLs. (Daniel Templeton via zhz) - - HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) - - HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for - filesystem entirely allocated for DFS use. (Tony Wu via lei) - - HDFS-9363. Add fetchReplica() to FsDatasetTestUtils to return FsDataset-agnostic - replica. (Tony Wu via lei) - - HDFS-9377. Fix findbugs warnings in FSDirSnapshotOp. - (Mingliang Liu via Yongjun Zhang) - - HDFS-9236. Missing sanity check for block size during block recovery. - (Tony Wu via Yongjun Zhang) - - HDFS-9379. Make NNThroughputBenchmark$BlockReportStats support more than 10 - datanodes. (Mingliang Liu via Arpit Agarwal) - - HDFS-9398. Make ByteArraryManager log message in one-line format. - (Mingliang Liu via szetszwo) - - HDFS-2261. AOP unit tests are not getting compiled or run. (wheat9) - - HDFS-9369. Use ctest to run tests for hadoop-hdfs-native-client. (wheat9) - - HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block - file size and genstamp. (Lei (Eddy) Xu via cmccabe) - - HDFS-8056. Decommissioned dead nodes should continue to be counted as dead - after NN restart. (mingma) - - HDFS-9439. Include status of closeAck into exception message in DataNode#run. - (Xiao Chen via Yongjun Zhang) - - HDFS-9402. Switch DataNode.LOG to use slf4j. (Walter Su via wheat9) - - HDFS-9153. Pretty-format the output for DFSIO. (Kai Zheng via wheat9) - - HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. - (Walter Su via wheat9) - - HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess - replicas. (Xiao Chen via mingma) - - HDFS-8807. dfs.datanode.data.dir does not handle spaces between - storageType and URI correctly. (Anu Engineer via szetszwo) - - HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. - (John Zhuge via Yongjun Zhang) - - HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage - type info. (xyao) - - HDFS-9269. Update the documentation and wrapper for fuse-dfs. - (Wei-Chiu Chuang via zhz) - - HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept BlockInfo - instead of Block. (Mingliang Liu via jing9) - - HDFS-9490. MiniDFSCluster should change block generation stamp via - FsDatasetTestUtils. (Tony Wu via lei) - - HDFS-9474. TestPipelinesFailover should not fail when printing debug - message. (John Zhuge via Yongjun Zhang) - - HDFS-9214. Support reconfiguring dfs.datanode.balance.max.concurrent.moves - without DN restart. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9527. The return type of FSNamesystem.getBlockCollection should be - changed to INodeFile. (szetszwo) - - HDFS-9472. concat() API does not give proper exception messages on ./reserved - relative path (Rakesh R via umamahesh) - - HDFS-9528. Cleanup namenode audit/log/exception messages. (szetszwo via umamahesh) - - HDFS-9532. Detailed exception info is lost in reportTo methods of - ErrorReportAction and ReportBadBlockAction. (Yongjun Zhang) - - HDFS-9519. Some coding improvement in SecondaryNameNode#main. - (Xiao Chen via Yongjun Zhang) - - HDFS-9514. TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; - exception being swallowed. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9535. Newly completed blocks in IBR should not be considered - under-replicated too quickly. (Mingliang Liu via jing9) - - HDFS-8894. Set SO_KEEPALIVE on DN server sockets. - (Kanaka Kumar Avvaru via wang) - - HDFS-9557. Reduce object allocation in PB conversion - (Daryn Sharp via cnauroth) - - HDFS-9552. Document types of permission checks performed for HDFS - operations. (cnauroth) - - HDFS-7779. Support changing ownership, group and replication in HDFS Web - UI. (Ravi Prakash via wheat9) - - HDFS-9498. Move code that tracks blocks with future generation stamps - to BlockManagerSafeMode. (Mingliang Liu via Arpit Agarwal) - - HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. - (Xiao Chen via zhz) - - HDFS-9630. DistCp minor refactoring and clean up. (Kai Zheng via zhz) - - HDFS-9415. Document dfs.cluster.administrators and - dfs.permissions.superusergroup. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9655. NN should start JVM pause monitor before loading fsimage. - (John Zhuge via Lei (Eddy) Xu) - - HDFS-8898. Create API and command-line argument to get quota and quota - usage without detailed content summary. (Ming Ma via kihwal) - - HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. - (Xiao Chen via Yongjun Zhang) - - HDFS-9638. Improve DistCp Help and documentation. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains - INodeReference. (Xiao Chen via lei) - - HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of - -namenode. (Mingliang Liu via shv) - - HDFS-9777. Fix typos in DFSAdmin command line and documentation. - (Wei-Chiu Chuang via umamahesh) - - HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets - for DataTransferProtocol (Gary Helmling via iwasakims) - - HDFS-9644. Update encryption documentation to reflect nested EZs. (zhz) - - HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider - at DEBUG Level (Inigo Goiri via asuresh) - - HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8923. Add -source flag to balancer usage message. (Chris Trezzo - via szetszwo) - - HDFS-9764. DistCp doesn't print value for several arguments including - -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9084. Pagination, sorting and filtering of files/directories in the - HDFS Web UI. (Ravi Prakash via ozawa) - - HDFS-9844. Correct path creation in getTrashRoot to handle root dir. (zhz) - - HDFS-9854. Log cipher suite negotiation more verbosely - (Wei-Chiu Chuang via cnauroth) - - HDFS-9843. Document distcp options required for copying between encrypted - locations. (Xiaoyu Yao via cnauroth) - - HDFS-9831. Document webhdfs retry configuration keys introduced by - HDFS-5219/HDFS-5122. (Xiaobing Zhou via xyao) - - HDFS-9887. WebHdfs socket timeouts should be configurable. - (Austin Donnelly and Chris Douglas via xyao) - - OPTIMIZATIONS - - HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than - DFSOutputStream#writeChunk (cmccabe) - - HDFS-7433. Optimize performance of DatanodeManager's node map. - (daryn via kihwal) - - HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a - LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) - - HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via - Colin P. McCabe) - - HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) - - HDFS-8929. Add a metric to expose the timestamp of the last journal - (surendra singh lilhore via vinayakumarb) - - HDFS-8829. Make SO_RCVBUF and SO_SNDBUF size configurable for - DataTransferProtocol sockets and allow configuring auto-tuning (He Tianyi - via Colin P. McCabe) - - HDFS-9139. Enable parallel JUnit tests for HDFS Pre-commit - (Chris Nauroth and Vinayakumar B via vinayakumarb) - - HDFS-9145. Tracking methods that hold FSNamesytemLock for too long. - (Mingliang Liu via wheat9) - - HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead - of hdfs. (Mingliang Liu via wheat9) - - HDFS-9253. Refactor tests of libhdfs into a directory. (wheat9) - - HDFS-9280. Document NFS gateway export point parameter. (Xiao Chen via zhz) - - HDFS-9297. Update TestBlockMissingException to use corruptBlockOnDataNodesByDeletingBlockFile(). - (Tony Wu via lei) - - HDFS-9168. Move client side unit test to hadoop-hdfs-client. (wheat9) - - HDFS-9312. Fix TestReplication to be FsDataset-agnostic. (lei) - - HDFS-9308. Add truncateMeta() and deleteMeta() to MiniDFSCluster. (Tony Wu via lei) - - HDFS-9372. Remove dead code in DataStorage.recoverTransitionRead. - (Duo Zhang via wheat9) - - HDFS-9282. Make data directory count and storage raw capacity related tests - FsDataset-agnostic. (Tony Wu via lei) - - HDFS-9318. considerLoad factor can be improved. (Kuhu Shukla via kihwal) - - HDFS-8335. FSNamesystem should construct FSPermissionChecker only if - permission is enabled. (Gabor Liptak via wheat9) - - HDFS-9436. Make NNThroughputBenchmark$BlockReportStats run with 10 - datanodes by default. (Mingliang Liu via shv) - - HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE - blocks. (szetszwo) - - HDFS-9715. Check storage ID uniqueness on datanode startup - (Lei (Eddy) Xu via vinayakumarb) - - HDFS-9726. Refactor IBR code to a new class. (szetszwo) - - HDFS-9686. Remove useless boxing/unboxing code. - (Kousuke Saruta via aajisaka) - - HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. - (Lin Yiqun via aajisaka) - - HDFS-9425. Expose number of blocks per volume as a metric - (Brahma Reddy Battula via vinayakumarb) - - HDFS-9710. DN can be configured to send block receipt IBRs in batches. - (szetszwo) - - BUG FIXES - - HDFS-7501. TransactionsSinceLastCheckpoint can be negative on SBNs. - (Gautam Gopalakrishnan via harsh) - - HDFS-5356. MiniDFSCluster should close all open FileSystems when shutdown() - (Rakesh R via vinayakumarb) - - HDFS-7867. Update action param from "start" to "prepare" in rolling upgrade - javadoc (J.Andreina via vinayakumarb) - - HDFS-3325. When configuring "dfs.namenode.safemode.threshold-pct" to a value - greater or equal to 1 there is mismatch in the UI report - (J.Andreina via vinayakumarb) - - HDFS-8002. Website refers to /trash directory. (Brahma Reddy Battula via - aajisaka) - - HDFS-7261. storageMap is accessed without synchronization in - DatanodeDescriptor#updateHeartbeatState() (Brahma Reddy Battula via Colin - P. McCabe) - - HDFS-7997. The first non-existing xattr should also throw IOException. - (zhouyingchao via yliu) - - HDFS-7922. ShortCircuitCache#close is not releasing - ScheduledThreadPoolExecutors (Rakesh R via Colin P. McCabe) - - HDFS-5215. dfs.datanode.du.reserved is not considered while computing - available space ( Brahma Reddy Battula via Yongjun Zhang) - - HDFS-7725. Incorrect "nodes in service" metrics caused all writes to fail. - (Ming Ma via wang) - - HDFS-8096. DatanodeMetrics#blocksReplicated will get incremented early and - even for failed transfers (vinayakumarb) - - HDFS-7939. Two fsimage_rollback_* files are created which are not deleted - after rollback. (J.Andreina via vinayakumarb) - - HDFS-8111. NPE thrown when invalid FSImage filename given for - 'hdfs oiv_legacy' cmd ( surendra singh lilhore via vinayakumarb ) - - HDFS-7701. Support reporting per storage type quota and usage - with hadoop/hdfs shell. (Peter Shi via Arpit Agarwal) - - HDFS-6666. Abort NameNode and DataNode startup if security is enabled but - block access token is not enabled. (Vijay Bhat via cnauroth) - - HDFS-8055. NullPointerException when topology script is missing. - (Anu Engineer via cnauroth) - - HDFS-8142. DistributedFileSystem encryption zone commands should resolve - relative paths. (Rakesh R via wang) - - HDFS-7863. Missing description of some methods and parameters in javadoc of - FSDirDeleteOp. (Brahma Reddy Battula via ozawa) - - HDFS-8043. NPE in MiniDFSCluster teardown. (Brahma Reddy Battula via ozawa) - - HDFS-8173. NPE thrown at DataNode shutdown when HTTP server was not able to - create (surendra singh lilhore via vinayakumarb) - - HDFS-7993. Provide each Replica details in fsck (J.Andreina via vinayakumarb) - - HDFS-8217. During block recovery for truncate Log new Block Id in case of - copy-on-truncate is true. (vinayakumarb) - - HDFS-8231. StackTrace displayed at client while QuotaByStorageType exceeds - (J.Andreina and Xiaoyu Yao via vinayakumarb) - - HDFS-8191. Fix byte to integer casting in SimulatedFSDataset#simulatedByte. - (Zhe Zhang via wang) - - HDFS-8211. DataNode UUID is always null in the JMX counter. (Anu Engineer - via Arpit Agarwal) - - HDFS-8247. TestDiskspaceQuotaUpdate#testAppendOverTypeQuota is failing. - (Xiaoyu Yao via cnauroth) - - HDFS-8206. Fix the typos in hadoop-hdfs-httpfs. (Brahma Reddy Battula via xyao) - - HDFS-8205. CommandFormat#parse() should not parse option as - value of option. (Peter Shi and Xiaoyu Yao via Arpit Agarwal) - - HDFS-8232. Missing datanode counters when using Metrics2 sink interface. - (Anu Engineer via cnauroth) - - HDFS-8214. Secondary NN Web UI shows wrong date for Last Checkpoint. (clamb via wang) - - HDFS-8300. Fix unit test failures and findbugs warning caused by HDFS-8283. - (jing9) - - HDFS-8276. LazyPersistFileScrubber should be disabled if scrubber interval - configured zero. (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8229. LAZY_PERSIST file gets deleted after NameNode restart. - (Surendra Singh Lilhore via Arpit Agarwal) - - HDFS-8309. Skip unit test using DataNodeTestUtils#injectDataDirFailure() on Windows. - (xyao) - - HDFS-8290. WebHDFS calls before namesystem initialization can cause - NullPointerException. (cnauroth) - - HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. - (Kiran Kumar M R via Xiaoyu Yao) - - HDFS-2484. checkLease should throw FileNotFoundException when file does - not exist. (Rakesh R via shv) - - HDFS-7833. DataNode reconfiguration does not recalculate valid volumes - required, based on configured failed volumes tolerated. - (Lei (Eddy) Xu via cnauroth) - - HDFS-8325. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - HDFS-8321. CacheDirectives and CachePool operations should throw - RetriableException in safemode. (wheat9) - - HDFS-8037. CheckAccess in WebHDFS silently accepts malformed FsActions - parameters. (wheat9) - - HDFS-8257. Namenode rollingUpgrade option is incorrect in document - (J.Andreina via vinayakumarb) - - HDFS-8067. haadmin prints out stale help messages (Ajith S via vinayakumarb) - - HDFS-8174. Update replication count to live rep count in fsck report. (J.Andreina) - - HDFS-6291. FSImage may be left unclosed in BootstrapStandby#doRun() - (Sanghyun Yun via vinayakumarb) - - HDFS-7998. HDFS Federation : Command mentioned to add a NN to existing - federated cluster is wrong (Ajith S via vinayakumarb) - - HDFS-8222. Remove usage of "dfsadmin -upgradeProgress" from document which - is no longer supported. (J.Andreina via aajisaka) - - HDFS-8108. Fsck should provide the info on mandatory option to be used along with "-blocks , - -locations and -racks" (J.Andreina via umamahesh) - - HDFS-8187. Remove usage of "-setStoragePolicy" and "-getStoragePolicy" using - dfsadmin cmd (as it is not been supported) (J.Andreina via vinayakumarb) - - HDFS-8175. Provide information on snapshotDiff for supporting the comparison - between snapshot and current status (J.Andreina via vinayakumarb) - - HDFS-8209. Support different number of datanode directories in MiniDFSCluster. - (surendra singh lilhore via vinayakumarb) - - HDFS-6576. Datanode log is generating at root directory in security mode - (surendra singh lilhore via vinayakumarb) - - HDFS-3384. DataStreamer thread should be closed immediatly when failed to - setup a PipelineForAppendOrRecovery (Uma Maheswara Rao G via vinayakumarb) - - HDFS-6285. tidy an error log inside BlockReceiver. (Liang Xie via umamahesh) - - HDFS-8346. libwebhdfs build fails during link due to unresolved external - symbols. (Chris Nauroth via wheat9) - - HDFS-8274. NFS configuration nfs.dump.dir not working. (Ajith S via - Arpit Agarwal) - - HDFS-8340. Fix NFS documentation of nfs.wtmax. (Ajith S via Arpit Agarwal) - - HDFS-8311. DataStreamer.transfer() should timeout the socket InputStream. - (Esteban Gutierrez via Yongjun Zhang) - - HDFS-8326. Documentation about when checkpoints are run is out of date. - (Misty Stanley-Jones via xyao) - - HDFS-8097. TestFileTruncate is failing intermittently. (Rakesh R via - Arpit Agarwal) - - HDFS-8351. Remove namenode -finalize option from document. (aajisaka) - - HDFS-8362. Java Compilation Error in TestHdfsConfigFields.java - (Arshad Mohammad via vinayakumarb) - - HDFS-8358. TestTraceAdmin fails (Masatake Iwasaki via kihwal) - - HDFS-8380. Always call addStoredBlock on blocks which have been shifted - from one storage to another (cmccabe) - - HDFS-8243. Files written by TestHostsFiles and TestNameNodeMXBean are - causing Release Audit Warnings. (Ruth Wisniewski via Arpit Agarwal) - - HDFS-7728. Avoid updating quota usage while loading edits. - (Jing Zhao via wheat9) - - HDFS-8150. Make getFileChecksum fail for blocks under construction - (J.Andreina via vinayakumarb) - - HDFS-8371. Fix test failure in TestHdfsConfigFields for spanreceiver - properties. (Ray Chiang via aajisaka) - - HDFS-8403. Eliminate retries in TestFileCreation - #testOverwriteOpenForWrite. (Arpit Agarwal via wheat9) - - HDFS-6348. SecondaryNameNode not terminating properly on runtime exceptions - (Rakesh R via vinayakumarb) - - HDFS-8454. Remove unnecessary throttling in TestDatanodeDeath. - (Arpit Agarwal) - - HDFS-8268. Port conflict log for data node server is not sufficient - (Mohammad Shahid Khan via vinayakumarb) - - HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-7401. Add block info to DFSInputStream' WARN message when it adds - node to deadNodes (Arshad Mohammad via vinayakumarb) - - HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS. - (Archana T via ozawa) - - HDFS-8256. "-storagepolicies , -blockId ,-replicaDetails " options are missed - out in usage and from documentation (J.Andreina via vinayakumarb) - - HDFS-8470. fsimage loading progress should update inode, delegation token and - cache pool count. (surendra singh lilhore via vinayakumarb) - - HDFS-3716. Purger should remove stale fsimage ckpt files - (J.Andreina via vinayakumarb) - - HDFS-8463. Calling DFSInputStream.seekToNewSource just after stream creation - causes NullPointerException (Masatake Iwasaki via kihwal) - - HDFS-8539. Hdfs doesnt have class 'debug' in windows. - (Anu Engineer via cnauroth) - - HDFS-8554. TestDatanodeLayoutUpgrade fails on Windows. (cnauroth) - - HDFS-8593. Calculation of effective layout version mishandles comparison to - current layout version in storage. (cnauroth) - - HDFS-8607. TestFileCorruption doesn't work as expected. (Walter Su via - Arpit Agarwal) - - HDFS-8592. SafeModeException never get unwrapped. (wheat9) - - HDFS-8548. Minicluster throws NPE on shutdown. - (surendra singh lilhore via xyao) - - HDFS-8551. Fix hdfs datanode CLI usage message. - (Brahma Reddy Battula via xyao) - - HDFS-8337. Accessing httpfs via webhdfs doesn't work from a jar with - kerberos. (Yongjun Zhang) - - HDFS-4366. Block Replication Policy Implementation May Skip Higher-Priority - Blocks for Lower-Priority Blocks (Derek Dagit via kihwal) - - HDFS-8542. WebHDFS getHomeDirectory behavior does not match specification. - (Kanaka Kumar Avvaru via jghoman) - - HDFS-8546. Prune cached replicas from DatanodeDescriptor state on replica - invalidation. (wang) - - HDFS-8586. Dead Datanode is allocated for write when client is from deadnode - (Brahma Reddy Battula via vinayakumarb) - - HDFS-8628. Update missing command option for fetchdt - (J.Andreina via vinayakumarb) - - HDFS-8687. Remove the duplicate usage message from Dfsck.java. (Brahma - Reddy Battula via Arpit Agarwal) - - HDFS-8579. Update HDFS usage with missing options - (J.Andreina via vinayakumarb) - - HDFS-8706. Fix typo in datanode startup options in HDFSCommands.html. - (Brahma Reddy Battula via Arpit Agarwal) - - HDFS-8577. Avoid retrying to recover lease on a file which does not exist - (J.Andreina via vinayakumarb) - - HDFS-8686. WebHdfsFileSystem#getXAttr(Path p, final String name) doesn't - work if namespace is in capitals (kanaka kumar avvaru via vinayakumarb) - - HDFS-8642. Make TestFileTruncate more reliable. (Rakesh R via - Arpit Agarwal) - - HDFS-8729. Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately - which occasionally failed. (Walter Su via jing9) - - HDFS-8749. Fix findbugs warnings in BlockManager.java. - (Brahma Reddy Battula via aajisaka) - - HDFS-2956. calling fetchdt without a --renewer argument throws NPE - (vinayakumarb) - - HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout (Xiaoyu Yao - via Colin P. McCabe) - - HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. - (Arpit Agarwal) - - HDFS-7582. Enforce maximum number of ACL entries separately per access - and default. (vinayakumarb) - - HDFS-8773. Few FSNamesystem metrics are not documented in the Metrics page. - (Rakesh R via cnauroth) - - HDFS-8810. Correct assertions in TestDFSInotifyEventInputStream class. - (Surendra Singh Lilhore via aajisaka) - - HDFS-8785. TestDistributedFileSystem is failing in trunk. (Xiaoyu Yao) - - HDFS-8670. Better to exclude decommissioned nodes for namenode NodeUsage JMX - (J.Andreina via vinayakumarb) - - HDFS-8847. change TestHDFSContractAppend to not override - testRenameFileBeingAppended method. (Zhihai Xu) - - HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake - Iwasaki via Colin P. McCabe) - - HDFS-8866. Typo in docs: Rumtime -> Runtime. (Gabor Liptak via jghoman) - - HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode - restart. (xyao) - - HDFS-8565. Typo in dfshealth.html - Decomissioning. (nijel via xyao) - - HDFS-8908. TestAppendSnapshotTruncate may fail with IOException: Failed to - replace a bad datanode. (Tsz Wo Nicholas Sze via yliu) - - HDFS-8922. Link the native_mini_dfs test library with libdl, since IBM Java - requires it (Ayappan via Colin P. McCabe) - - HDFS-8809. HDFS fsck reports under construction blocks as "CORRUPT". (jing9) - - HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture - documentation. (Masatake Iwasaki via aajisaka) - - HDFS-8930. Block report lease may leak if the 2nd full block report comes - when NN is still in safemode (Colin P. McCabe via Jing Zhao) - - HDFS-8948. Use GenericTestUtils to set log levels in TestPread and - TestReplaceDatanodeOnFailure. (Mingliang Liu via wheat9) - - HDFS-8932. NPE thrown in NameNode when try to get TotalSyncCount metric - before editLogStream initialization. (Surendra Singh Lilhore via xyao) - - HDFS-8682. Should not remove decommissioned node,while calculating the - number of live/dead decommissioned node. (J. Andreina via vinayakumarb) - - HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit. - DfsClientShmManager.EndpointShmManager. (Mingliang Liu via wheat9) - - HDFS-8969. Clean up findbugs warnings for HDFS-8823 and HDFS-8932. - (Anu Engineer via wheat9) - - HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. - (Colin Patrick McCabe via yliu) - - HDFS-8950. NameNode refresh doesn't remove DataNodes that are no longer in - the allowed list (Daniel Templeton) - - HDFS-8388. Time and Date format need to be in sync in NameNode UI page. - (Surendra Singh Lilhore via aajisaka) - - HDFS-9003. ForkJoin thread pool leaks. (Kihwal Lee via jing9) - - HDFS-8885. ByteRangeInputStream used in webhdfs does not override - available(). (Shradha Revankar via aajisaka) - - HDFS-9009. Send metrics logs to NullAppender by default. (Arpit Agarwal) - - HDFS-8964. When validating the edit log, do not read at or beyond the file - offset that is being written (Zhe Zhang via Colin P. McCabe) - - HDFS-8939. Test(S)WebHdfsFileContextMainOperations failing on branch-2. - (Chris Nauroth via jghoman) - - HDFS-8581. ContentSummary on / skips further counts on yielding lock - (J.Andreina via vinayakumarb) - - HDFS-9036. In BlockPlacementPolicyWithNodeGroup#chooseLocalStorage , random - node is selected eventhough fallbackToLocalRack is true. - (J.Andreina via vinayakumarb) - - HDFS-9041. Move entries in META-INF/services/o.a.h.fs.FileSystem to - hdfs-client. (Mingliang Liu via wheat9) - - HDFS-9069. TestNameNodeMetricsLogger failing -port in use. - (stevel) - - HDFS-9067. o.a.h.hdfs.server.datanode.fsdataset.impl.TestLazyWriter - is failing in trunk (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9072. Fix random failures in TestJMXGet. - (J.Andreina via stevel) - - HDFS-9073. Fix failures in TestLazyPersistLockedMemory - testReleaseOnEviction(). (J.Andreina via stevel) - - HDFS-9063. Correctly handle snapshot path for getContentSummary. (jing9) - - HDFS-8780. Fetching live/dead datanode list with arg true for remove- - DecommissionNode,returns list with decom node. (J.Andreina via vinayakumarb) - - HDFS-9013. Deprecate NameNodeMXBean#getNNStarted in branch2 and remove from - trunk (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9128. TestWebHdfsFileContextMainOperations and - TestSWebHdfsFileContextMainOperations fail due to invalid HDFS path on - Windows. (Chris Nauroth via wheat9) - - HDFS-9076. Log full path instead of inodeId in DFSClient - #closeAllFilesBeingWritten() (Surendra Singh Lilhore via vinayakumarb) - - HDFS-9123. Copying from the root to a subdirectory should be forbidden. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9107. Prevent NN's unrecoverable death spiral after full GC (Daryn - Sharp via Colin P. McCabe) - - HDFS-9147. Fix the setting of visibleLength in ExternalBlockReader. (Colin - P. McCabe via Lei (Eddy) Xu) - - HDFS-9092. Nfs silently drops overlapping write requests and causes data - copying to fail. (Yongjun Zhang) - - HDFS-9141. Thread leak in Datanode#refreshVolumes. (Uma Maheswara Rao G - via yliu) - - HDFS-9174. Fix findbugs warnings in FSOutputSummer.tracer and - DirectoryScanner$ReportCompiler.currentThread. (Yi Liu via wheat9) - - HDFS-9001. DFSUtil.getNsServiceRpcUris() can return too many entries in a - non-HA, non-federated cluster. (Daniel Templeton via atm) - - HDFS-9100. HDFS Balancer does not respect dfs.client.use.datanode.hostname. - (Casey Brotherton via Yongjun Zhang) - - HDFS-9191. Typo in Hdfs.java. NoSuchElementException is misspelled. - (Catherine Palmer via jghoman) - - HDFS-9193. Fix incorrect references the usages of the DN in dfshealth.js. - (Chang Li via wheat9) - - HADOOP-11098. [JDK8] Max Non Heap Memory default changed between JDK7 - and 8 (ozawa). - - HDFS-9151. Mover should print the exit status/reason on console like - balancer tool. (Surendra singh lilhore via vinayakumarb) - - HDFS-9154. [OEV-Doc] : Document does not mention about "-f" and "-r" options - (nijel via vinayakumarb) - - HDFS-7899. Improve EOF error message (Jagadesh Kiran N via vinayakumarb) - - HDFS-9196. Fix TestWebHdfsContentLength. (Masatake Iwasaki via jing9) - - HDFS-9159. [OIV] : return value of the command is not correct if invalid - value specified in "-p (processor)" option (nijel via vinayakumarb) - - HDFS-9176. Fix TestDirectoryScanner#testThrottling often fails. - (Daniel Templeton via lei) - - HDFS-9137. DeadLock between DataNode#refreshVolumes and - BPOfferService#registrationSucceeded. (Uma Maheswara Rao G via yliu) - - HDFS-9142. Separating Configuration object for namenode(s) in - MiniDFSCluster. (Siqi Li via mingma) - - HDFS-8941. DistributedFileSystem listCorruptFileBlocks API should - resolve relative path. (Rakesh R via wang) - - HDFS-9215. Suppress the RAT warnings in hdfs-native-client module. (wheat9) - - HDFS-9222. Add hadoop-hdfs-client as a dependency of - hadoop-hdfs-native-client. (Mingliang Liu via wheat9) - - HDFS-9224. TestFileTruncate fails intermittently with BindException - (Brahma Reddy Battula via vinayakumarb) - - HDFS-9160. [OIV-Doc] : Missing details of 'delimited' for processor options - (nijel via vinayakumarb) - - HDFS-9235. hdfs-native-client build getting errors when built with cmake - 2.6. (Eric Payne via wheat9) - - HDFS-8779. WebUI fails to display block IDs that are larger than 2^53 - 1. - (wheat9) - - HDFS-9187. Fix null pointer error in Globber when FS was not constructed - via FileSystem#createFileSystem (cmccabe) - - HDFS-9157. [OEV and OIV] : Unnecessary parsing for mandatory arguements if - '-h' option is specified as the only option (nijel via vinayakumarb) - - HDFS-9237. NPE at TestDataNodeVolumeFailureToleration#tearDown. - (Brahma Reddy Battula via ozawa) - - HDFS-9208. Disabling atime may fail clients like distCp. (Kihwal Lee via - yliu) - - HDFS-9270. TestShortCircuitLocalRead should not leave socket after unit - test (Masatake Iwasaki via Colin P. McCabe) - - HDFS-3059. ssl-server.xml causes NullPointer. (Xiao Chen via wang) - - HDFS-9274. Default value of dfs.datanode.directoryscan.throttle.limit.ms.per.sec - should be consistent. (Yi Liu via zhz) - - HDFS-9286. HttpFs does not parse ACL syntax correctly for operation - REMOVEACLENTRIES. (Wei-Chiu Chuang via cnauroth) - - HDFS-9301. HDFS clients can't construct HdfsConfiguration instances. - (Mingliang Liu via wheat9) - - HDFS-9304. Add HdfsClientConfigKeys class to TestHdfsConfigFields - #configurationClasses. (Mingliang Liu via wheat9) - - HDFS-9268. fuse_dfs chown crashes when uid is passed as -1 (cmccabe) - - HDFS-9231. fsck doesn't list correct file path when Bad Replicas/Blocks - are in a snapshot. (Xiao Chen via Yongjun Zhang) - - HDFS-9302. WebHDFS throws NullPointerException if newLength is not - provided. (Jagadesh Kiran N via yliu) - - HDFS-9297. Decomissioned capacity should not be considered for - configured/used capacity (Contributed by Kuhu Shukla) - - HDFS-9044. Give Priority to FavouredNodes , before selecting - nodes from FavouredNode's Node Group (J.Andreina via vinayakumarb) - - HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while - saving namespace. (wang) - - HDFS-9343. Empty caller context considered invalid. (Mingliang Liu via - Arpit Agarwal) - - HDFS-9329. TestBootstrapStandby#testRateThrottling is flaky because fsimage - size is smaller than IO buffer size. (zhz) - - HDFS-9313. Possible NullPointerException in BlockManager if no excess - replica can be chosen. (mingma) - - HDFS-9354. Fix TestBalancer#testBalancerWithZeroThreadsForMove on Windows. - (Xiaoyu Yao via cnauroth) - - HDFS-9362. TestAuditLogger#testAuditLoggerWithCallContext assumes Unix line - endings, fails on Windows. (cnauroth) - - HDFS-9351. checkNNStartup() need to be called when fsck calls - FSNamesystem.getSnapshottableDirs(). (Xiao Chen via Yongjun Zhang) - - HDFS-9357. NN UI renders icons of decommissioned DN incorrectly. - (Surendra Singh Lilhore via wheat9) - - HDFS-9360. Storage type usage isn't updated properly after file deletion. - (Ming Ma via xyao) - - HDFS-9378. hadoop-hdfs-client tests do not write logs. (cnauroth) - - HDFS-9384. TestWebHdfsContentLength intermittently hangs and fails due to - TCP conversation mismatch between client and server. (cnauroth) - - HDFS-9394. branch-2 hadoop-hdfs-client fails during FileSystem ServiceLoader - initialization, because HftpFileSystem is missing. - (Mingliang Liu via cnauroth) - - HDFS-9249. NPE is thrown if an IOException is thrown in NameNode constructor. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9401. Fix findbugs warnings in BlockRecoveryWorker. - (Brahma Reddy Battula via waltersu4549) - - HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. - (Xiao Chen via zhz) - - HDFS-9245. Fix findbugs warnings in hdfs-nfs/WriteCtx. - (Mingliang Liu via xyao) - - HDFS-9396. Total files and directories on jmx and web UI on standby is - uninitialized. (kihwal) - - HDFS-9410. Some tests should always reset sysout and syserr. - (Xiao Chen via waltersu4549) - - HDFS-9413. getContentSummary() on standby should throw StandbyException. - (Brahma Reddy Battula via mingma) - - HDFS-9387. Fix namenodeUri parameter parsing in NNThroughputBenchmark. - (Mingliang Liu via xyao) - - HDFS-9421. NNThroughputBenchmark replication test NPE with -namenode option. - (Mingliang Liu via xyao) - - HDFS-9358. TestNodeCount#testNodeCount timed out. - (Masatake Iwasaki via waltersu4549) - - HDFS-9397. Fix typo for readChecksum() LOG.warn in BlockSender.java. - (Enrique Flores via Arpit Agarwal) - - HDFS-9400. TestRollingUpgradeRollback fails on branch-2. - (Brahma Reddy Battula via cnauroth) - - HDFS-9443. Disabling HDFS client socket cache causes logging message - printed to console for CLI commands. (Chris Nauroth via wheat9) - - HDFS-6885. Fix wrong use of BytesWritable in FSEditLogOp#RenameOp. - (Yi Liu via wheat9) - - HDFS-7897. Shutdown metrics when stopping JournalNode. - (zhouyingchao via wheat9) - - HDFS-9356. Decommissioning node does not have Last Contact value in the UI. - (Surendra Singh Lilhore via wheat9) - - HDFS-9024. Deprecate the TotalFiles metric. (Akira Ajisaka via wheat9) - - HDFS-9428. Fix intermittent failure of - TestDNFencing.testQueueingWithAppend. (Masatake Iwasaki via waltersu4549) - - HDFS-9435. TestBlockRecovery#testRBWReplicas is failing intermittently. - (Rakesh R via waltersu4549) - - HDFS-9433. DFS getEZForPath API on a non-existent file should throw - FileNotFoundException (Rakesh R via umamahesh) - - HDFS-6101. TestReplaceDatanodeOnFailure fails occasionally. - (Wei-Chiu Chuang via cnauroth) - - HDFS-8855. Webhdfs client leaks active NameNode connections. - (Xiaobing Zhou via xyao) - - HDFS-8335. FSNamesystem should construct FSPermissionChecker only if - permission is enabled. (Gabor Liptak via wheat9) - - HDFS-6694. TestPipelinesFailover.testPipelineRecoveryStress tests fail - intermittently with various symptoms - debugging patch. (Yongjun Zhang via - Arpit Agarwal) - - HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after - transition to ctest. (Chris Nauroth via wheat9) - - HDFS-9407. TestFileTruncate should not use fixed NN port. - (Brahma Reddy Battula via shv) - - HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in FSNamesystem. - (Mingliang Liu via jing9) - - HDFS-9336. deleteSnapshot throws NPE when snapshotname is null. - (Brahma Reddy Battula via aajisaka) - - HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails - intermittently. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9429. Tests in TestDFSAdminWithHA intermittently fail with - EOFException (Xiao Chen via Colin P. McCabe) - - HDFS-9484. NNThroughputBenchmark$BlockReportStats should not send empty - block reports. (Mingliang Liu via shv) - - HDFS-9430. Remove waitForLoadingFSImage since checkNNStartup has ensured - image loaded and namenode started. (Brahma Reddy Battula via mingma) - - HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh) - - HDFS-9565. TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes - is flaky due to race condition. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9570. Minor typos, grammar, and case sensitivity cleanup in - HdfsPermissionsGuide.md's (Travis Campbell via aw) - - HDFS-9515. NPE when MiniDFSCluster#shutdown is invoked on uninitialized - reference. (Wei-Chiu Chuang via Arpit Agarwal) - - HDFS-9572. Prevent DataNode log spam if a client connects on the data - transfer port but sends no data. (cnauroth) - - HDFS-9571. Fix ASF Licence warnings in Jenkins reports - (Brahma Reddy Battula via cnauroth) - - HDFS-9393. After choosing favored nodes, choosing nodes for remaining - replicas should go through BlockPlacementPolicy - (J.Andreina via vinayakumarb) - - HDFS-9589. Block files which have been hardlinked should be duplicated - before the DataNode appends to the them (cmccabe) - - HDFS-9458. TestBackupNode always binds to port 50070, which can cause bind - failures. (Xiao Chen via cnauroth) - - HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. - (Xiao Chen via cnauroth) - - HDFS-9605. Add links to failed volumes to explorer.html in HDFS Web UI. - (Archana T via wheat9) - - HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the - correct namenode (Wei-Chiu Chuang via vinayakumarb) - - HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. - (Tony Wu via lei) - - HDFS-9639. Inconsistent Logging in BootstrapStandby. (Xiaobing Zhou via - Arpit Agarwal) - - HDFS-9584. NPE in distcp when ssl configuration file does not exist in - class path. (Surendra Singh Lilhore via Xiaoyu Yao) - - HDFS-9612. DistCp worker threads are not terminated after jobs are done. - (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9623. Update example configuration of block state change log in - log4j.properties. (Masatake Iwasaki via aajisaka) - - HDFS-6054. MiniQJMHACluster should not use static port to avoid binding - failure in unit test. (Yongjun Zhang) - - HDFS-9682. Fix a typo "aplication" in HttpFS document. - (Weiwei Yang via aajisaka) - - HDFS-9566. Remove expensive 'BlocksMap#getStorages(Block b, final - DatanodeStorage.State state)' method (Daryn Sharp via vinayakumarb) - - HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames - (Mingliang Liu via stevel) - - HDFS-9210. Fix some misuse of %n in VolumeScanner#printStats. - (Xiaoyu Yao) - - HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei) - - HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys - before initialize (DENG FEI via vinayakumarb) - - HDFS-9739. DatanodeStorage.isValidStorageId() is broken - (Mingliang Liu via vinayakumarb) - - HDFS-9748. Avoid duplication in pendingReplications when - addExpectedReplicasToPending is called twice. (Walter Su via jing9) - - HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle - NotReplicatedYetException on adding block (iwasakims) - - HDFS-9761. Rebalancer sleeps too long between iterations - (Mingliang Liu via cnauroth) - - HDFS-9713. DataXceiver#copyBlock should return if block is pinned. - (umamahesh) - - HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos - auth handler (Ryan Sasson via aw) - - HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value - (Kuhu Shukla via umamahesh) - - HDFS-9788. Incompatible tag renumbering in HeartbeatResponseProto. (wang) - - HDFS-9790. HDFS Balancer should exit with a proper message if upgrade is - not finalized. (Xiaobing Zhou via Arpit Agarwal) - - HDFS-9801. ReconfigurableBase should update the cached configuration. - (Arpit Agarwal) - - HDFS-9456. BlockPlacementPolicyWithNodeGroup should override - verifyBlockPlacement(). (Xiaobing Zhou via junping_du) - - HDFS-9815. Move o.a.h.fs.Hdfs to hadoop-hdfs-client. - (Vinayakumar B via wheat9) - - HDFS-9799. Reimplement getCurrentTrashDir to remove incompatibility. (zhz) - - HDFS-6832. Fix the usage of 'hdfs namenode' command. - (Manjunath Ballur via aajisaka) - - HDFS-9839. Reduce verbosity of processReport logging. (Arpit Agarwal) - - HDFS-7452. skip StandbyException log for getCorruptFiles() - (Bramha Reddy Battula via vinayakumarb) - - HDFS-9842. dfs.datanode.balance.bandwidthPerSec should accept friendly - size units. (Lin Yiqun via Arpit Agarwal) - - HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via - cmccabe) - - HDFS-9870. Remove unused imports from DFSUtil. - (Brahma Reddy Battula via cnauroth) - - HDFS-9881. DistributedFileSystem#getTrashRoot returns incorrect path for - encryption zones. (wang) - - HDFS-9886. Configuration properties for hedged read is broken. - (Akira AJISAKA via zhz) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-7163. WebHdfsFileSystem should retry reads according to the configured - retry policy. (Eric Payne via kihwal) - - HDFS-9574. Reduce client failures during datanode restart (kihwal) - - HDFS-9569. Log the name of the fsimage being loaded for better - supportability. (Yongjun Zhang) - - HDFS-9634. webhdfs client side exceptions don't provide enough details - (Eric Payne via kihwal) - - HDFS-9654. Code refactoring for HDFS-8578. (szetszwo) - - HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size - (Elliot Clark via cmccabe) - - HDFS-4946. Allow preferLocalNode in BlockPlacementPolicyDefault to be - configurable (James Kinley and Nathan Roberts via kihwal) - - HDFS-8578. On upgrade, Datanode should process all storage/data dirs in - parallel. (vinayakumarb and szetszwo via szetszwo) - - HDFS-9395. Make HDFS audit logging consistant (Kuhu Shukla via kihwal) - - HDFS-8791. block ID-based DN storage layout can be very slow for datanode - on ext4 (Chris Trezzo via kihwal) - - HDFS-9048. DistCp documentation is out-of-dated - (Daisuke Kobayashi via iwasakims) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in - commitBlock. (Chang Li via zhz) - - HDFS-4937. ReplicationMonitor can infinite-loop in - BlockPlacementPolicyDefault#chooseRandom(). (kihwal) - - HDFS-9383. TestByteArrayManager#testByteArrayManager fails. - (szetszwo via kihwal) - - HDFS-9516. Truncate file fails with data dirs on multiple disks. - (Plamen Jeliazkov via shv) - - HDFS-9533. seen_txid in the shared edits directory is modified during - bootstrapping (kihwal) - - HDFS-9505. HDFS Architecture documentation needs to be refreshed. - (Masatake Iwasaki via aajisaka) - - HDFS-8914. Document HA support in the HDFS HdfsDesign.md. - (Lars Francke via wheat9) - - HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message - change. (Wei-Chiu Chuang via Yongjun Zhang) - - HDFS-9661. Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and - createRbw (ade via vinayakumarb) - - HDFS-9625. set replication for empty file failed when set storage policy - (DENG FEI via vinayakumarb) - - HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently (Mingliang Liu - via jitendra) - - HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. - (szetszwo) - - HDFS-9406. FSImage may get corrupted after deleting snapshot. - (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang) - - HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() - (Chang Li via vinayakumarb) - - HDFS-9730. Storage ID update does not happen when there is a layout change - (Tsz Wo Nicholas Sze via kihwal) - - HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse - ObjectMapper. (Akira AJISAKA via wheat9) - - HDFS-9752. Permanent write failures may happen to slow writers during - datanode rolling upgrades (Walter Su via kihwal) - - HDFS-9784. Example usage is not correct in Transparent Encryption document. - (Takashi Ohnishi via aajisaka) - - HDFS-9765. TestBlockScanner#testVolumeIteratorWithCaching fails - intermittently. (aajisaka) - - HDFS-9864. Correct reference for RENEWDELEGATIONTOKEN and - CANCELDELEGATIONTOKEN in webhdfs doc. (Brahma Reddy Battula via aajisaka) - - HDFS-9880. TestDatanodeRegistration fails occasionally (kihwal) - - HDFS-9766. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently. - (Xiao Chen via aajisaka) - - HDFS-9851. NameNode throws NPE when setPermission is called on a path that - does not exist. (Brahma Reddy Battula via aajisaka) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-8659. Block scanner INFO message is spamming logs. (Yongjun Zhang) - - HDFS-8099. Change "DFSInputStream has been closed already" message to - debug log level (Charles Lamb via Colin P. McCabe) - - HDFS-9221. HdfsServerConstants#ReplicaState#getState should avoid calling - values() since it creates a temporary array. (Staffan Friberg via yliu) - - OPTIMIZATIONS - - HDFS-8722. Optimize datanode writes for small writes and flushes (kihwal) - - BUG FIXES - - HDFS-6945. BlockManager should remove a block from excessReplicateMap and - decrement ExcessBlocks metric when the block is removed. (aajisaka) - - HDFS-8806. Inconsistent metrics: number of missing blocks with replication - factor 1 not properly cleared. (Zhe Zhang via aajisaka) - - HDFS-8852. HDFS architecture documentation of version 2.x is outdated - about append write support. (Ajith S via aajisaka) - - HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) - - HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) - - HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. - (Kihwal Lee via yliu) - - HDFS-9033. dfsadmin -metasave prints "NaN" for cache used%. - (Brahma Reddy Battula via aajisaka) - - HDFS-9042. Update document for the Storage policy name - (J.Andreina via vinayakumarb) - - HDFS-9043. Doc updation for commands in HDFS Federation - (J.Andreina via vinayakumab) - - HDFS-9106. Transfer failure during pipeline recovery causes permanent - write failures (kihwal) - - HDFS-8850. VolumeScanner thread exits with exception if there is no block - pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe - via yliu) - - HDFS-9178. Slow datanode I/O can cause a wrong node to be marked bad - (kihwal) - - HDFS-8676. Delayed rolling upgrade finalization can cause heartbeat - expiration. (Walter Su via kihwal) - - HDFS-9220. Reading small file (< 512 bytes) that is open for append fails - due to incorrect checksum (Jing Zhao via kihwal) - - HDFS-9290. DFSClient#callAppend() is not backward compatible for slightly - older NameNodes. (Tony Wu via kihwal) - - HDFS-9305. Delayed heartbeat processing causes storm of subsequent - heartbeats. (Arpit Agarwal) - - HDFS-9317. Document fsck -blockId and -storagepolicy options in branch-2.7. - (aajisaka) - - HDFS-6481. DatanodeManager#getDatanodeStorageInfos() should check the - length of storageIDs. (szetszwo via Arpit Agarwal) - - HDFS-9426. Rollingupgrade finalization is not backward compatible - (Kihwal Lee via vinayakumarb) - - HDFS-9445. Datanode may deadlock while handling a bad volume. - (Wlater Su via Kihwal) - - HDFS-8767. RawLocalFileSystem.listStatus() returns null for UNIX pipefile. - (Kanaka Kumar Avvaru via wheat9) - -Release 2.7.1 - 2015-07-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-8081. Split getAdditionalBlock() into two methods. (shv) - - HDFS-7931. DistributedFileSystem should not look for keyProvider in - cache if Encryption is disabled (asuresh) - - HDFS-8204. Mover/Balancer should not schedule two replicas to the same - datanode. (Walter Su via szetszwo) - - HDFS-7770. Need document for storage type label of data node storage - locations under dfs.data.dir. (Xiaoyu Yao via aajisaka) - - HDFS-8213. DFSClient should use hdfs.client.htrace HTrace configuration - prefix rather than hadoop.htrace (cmccabe) - - HDFS-8521. Add VisibleForTesting annotation to - BlockPoolSlice#selectReplicaToDelete. (cmccabe) - - HDFS-8361. Choose SSD over DISK in block placement. (szetszwo) - - HDFS-7546. Document, and set an accepting default for - dfs.namenode.kerberos.principal.pattern (Harsh J via aw) - - HDFS-7164. Feature documentation for HDFS-6581. (Arpit Agarwal) - - HDFS-8143. Mover should exit after some retry when failed to move blocks. - (Surendra Singh Lilhore via szetszwo) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-8151. Always use snapshot path as source when invalid snapshot names - are used for diff based distcp. (jing9) - - HDFS-7934. Update RollingUpgrade rollback documentation: should use - bootstrapstandby for standby NN. (J. Andreina via jing9) - - HDFS-8149. The footer of the Web UI "Hadoop, 2014" is old. - (Brahma Reddy Battula via aajisaka) - - HDFS-8153. Error Message points to wrong parent directory in case of - path component name length error (Anu Engineer via jitendra) - - HDFS-8179. DFSClient#getServerDefaults returns null within 1 - hour of system start. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-8163. Using monotonicNow for block report scheduling causes - test failures on recently restarted systems. (Arpit Agarwal) - - HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. - (surendra singh lilhore via szetszwo) - - HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the - lock. (wheat9) - - HDFS-8269. getBlockLocations() does not resolve the .reserved path and - generates incorrect edit logs when updating the atime. (wheat9) - - HDFS-8091: ACLStatus and XAttributes should be presented to - INodeAttributesProvider before returning to client (asuresh) - - HDFS-8305: HDFS INotify: the destination field of RenameOp should always - end with the file name (cmccabe) - - HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) - - HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor - goes for infinite loop (Rushabh S Shah via kihwal) - - HDFS-6300. Prevent multiple balancers from running simultaneously - (Rakesh R via vinayakumarb) - - HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) - - HDFS-8451. DFSClient probe for encryption testing interprets empty URI - property for "enabled". (Steve Loughran via xyao) - - HDFS-8523. Remove usage information on unsupported operation - "fsck -showprogress" from branch-2 (J.Andreina via vinayakumarb) - - HDFS-8522. Change heavily recorded NN logs from INFO to DEBUG level. (xyao) - - HDFS-8566. HDFS documentation about debug commands wrongly identifies them - as "hdfs dfs" commands (Surendra Singh Lilhore via Colin P. McCabe) - - HDFS-8583. Document that NFS gateway does not work with rpcbind - on SLES 11. (Arpit Agarwal) - - HDFS-8572. DN always uses HTTP/localhost@REALM principals in SPNEGO. - (wheat9) - - HDFS-8596. TestDistributedFileSystem et al tests are broken in branch-2 - due to incorrect setting of "datanode" attribute. (Yongjun Zhang) - - HDFS-8595. TestCommitBlockSynchronization fails in branch-2.7. (Patch - applies to all branches). (Arpit Agarwal) - - HDFS-8576. Lease recovery should return true if the lease can be released - and the file can be closed. (J.Andreina via szetszwo) - - HDFS-8597. Fix TestFSImage#testZeroBlockSize on Windows. (Xiaoyu Yao) - - HDFS-4660. Block corruption can happen during pipeline recovery (kihwal) - - HDFS-8633. Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml - to match DFSConfigKeys. (Ray Chiang via Yongjun Zhang) - - HDFS-8626. Reserved RBW space is not released if creation of RBW File - fails. (kanaka kumar avvaru via Arpit Agarwal) - - HDFS08656. Preserve compatibility of ClientProtocol#rollingUpgrade after - finalization. (wang) - - HDFS-8681. BlockScanner is incorrectly disabled by default. - (Arpit Agarwal) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-6663. Admin command to track file and locations from block id. - (Chen He via kihwal) - - HDFS-6982. nntop: top­-like tool for name node users. - (Maysam Yabandeh via wang) - - HDFS-7424. Add web UI for NFS gateway (brandonli) - - HDFS-7449. Add metrics to NFS gateway (brandonli) - - HDFS-3107. Introduce truncate. (Plamen Jeliazkov via shv) - - HDFS-7056. Snapshot support for truncate. (Plamen Jeliazkov and shv) - - HDFS-6673. Add delimited format support to PB OIV tool. (Eddy Xu via wang) - - HDFS-7655. Expose truncate API for Web HDFS. (yliu) - - HDFS-6133. Add a feature for replica pinning so that a pinned replica - will not be moved by Balancer/Mover. (zhaoyunjiong via szetszwo) - - HDFS-3689. Add support for variable length block. (jing9) - - HDFS-7584. Enable Quota Support for Storage Types (See breakdown of - tasks below) - - HDFS-7656. Expose truncate API for HDFS httpfs. (yliu) - - HDFS-6488. Support HDFS superuser in NFS gateway. (brandonli) - - HDFS-7838. Expose truncate API for libhdfs. (yliu) - - HDFS-6826. Plugin interface to enable delegation of HDFS authorization - assertions. (Arun Suresh via jitendra) - - IMPROVEMENTS - - HDFS-7752. Improve description for - "dfs.namenode.num.extra.edits.retained" - and "dfs.namenode.num.checkpoints.retained" properties on - hdfs-default.xml (Wellington Chevreuil via harsh) - - HDFS-7055. Add tracing to DFSInputStream (cmccabe) - - HDFS-7186. Document the "hadoop trace" command. (Masatake Iwasaki via Colin - P. McCabe) - - HDFS-7202. Should be able to omit package name of SpanReceiver on "hadoop - trace -add" (iwasakims via cmccabe) - - HDFS-7026. Introduce a string constant for "Failed to obtain user group - info...". (Yongjun Zhang via atm) - - HDFS-7209. Populate EDEK cache when creating encryption zone. (Yi Liu via wang) - - HDFS-6252. Phase out the old web UI in HDFS. (wheat9) - - HDFS-7266. HDFS Peercache enabled check should not lock on object (awang - via cmccabe) - - HDFS-7254. Add documentation for hot swaping DataNode drives (Lei Xu via - Colin P. McCabe) - - HDFS-6877. Avoid calling checkDisk when an HDFS volume is removed during a - write. (Lei Xu via Colin P. McCabe) - - HDFS-2486. Remove unnecessary priority level checks in - UnderReplicatedBlocks. (Uma Maheswara Rao G via szetszwo) - - HDFS-6824. Additional user documentation for HDFS encryption. (wang) - - HDFS-7165. Separate block metrics for files with replication count 1. - (Zhe Zhang via wang) - - HDFS-7222. Expose DataNode network errors as a metric. (Charles Lamb via wang) - - HDFS-7257. Add the time of last HA state transition to NN's /jmx page. - (Charles Lamb via wheat9) - - HDFS-7223. Tracing span description of IPC client is too long (iwasakims - via cmccabe) - - HDFS-7283. Bump DataNode OOM log from WARN to ERROR. - (Stephen Chu via wheat9) - - HDFS-5928. Show namespace and namenode ID on NN dfshealth page. - (Siqi Li via wheat9) - - HDFS-7280. Use netty 4 in WebImageViewer. (wheat9) - - HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could - have a better error message. (Yongjun Zhang via wang) - - HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, - etc. (cmccabe) - - HDFS-7356. Use DirectoryListing.hasMore() directly in nfs. (Li Lu via jing9) - - HDFS-7357. FSNamesystem.checkFileProgress should log file path. - (Tsz Wo Nicholas Sze via wheat9) - - HDFS-7335. Redundant checkOperation() in FSN.analyzeFileState(). - (Milan Desai via shv) - - HDFS-7333. Improve logging in Storage.tryLock(). (shv) - - HDFS-7361. TestCheckpoint fails after change of log message related to - locking violation. (shv) - - HDFS-7329. Improve logging when MiniDFSCluster fails to start. - (Byron Wong via shv) - - HDFS-7336. Unused member DFSInputStream.buffersize. (Milan Desai via shv) - - HDFS-7365. Remove hdfs.server.blockmanagement.MutableBlockCollection. - (Li Lu via wheat9) - - HDFS-7381. Decouple the management of block id and gen stamps from - FSNamesystem. (wheat9) - - HDFS-7375. Move FSClusterStats to o.a.h.h.hdfs.server.blockmanagement. - (wheat9) - - HDFS-7386. Replace check "port number < 1024" with shared isPrivilegedPort - method. (Yongjun Zhang via cnauroth) - - HDFS-7394. Log at INFO level, not WARN level, when InvalidToken is seen in - ShortCircuitCache (Keith Pak via Colin P. McCabe) - - HDFS-7279. Use netty to implement DatanodeWebHdfsMethods. (wheat9) - - HDFS-7404. Remove o.a.h.hdfs.server.datanode.web.resources. - (Li Lu via wheat9) - - HDFS-7398. Reset cached thread-local FSEditLogOp's on every - FSEditLog#logEdit. (Gera Shegalov via cnauroth) - - HDFS-7409. Allow dead nodes to finish decommissioning if all files are - fully replicated. (wang) - - HDFS-7413. Some unit tests should use NameNodeProtocols instead of - FSNameSystem. (wheat9) - - HDFS-7415. Move FSNameSystem.resolvePath() to FSDirectory. (wheat9) - - HDFS-7420. Delegate permission checks to FSDirectory. (wheat9) - - HDFS-7331. Add Datanode network counts to datanode jmx page. (Charles Lamb - via atm) - - HDFS-7412. Move RetryCache to NameNodeRpcServer. (wheat9) - - HDFS-7419. Improve error messages for DataNode hot swap drive feature (Lei - Xu via Colin P. Mccabe) - - HDFS-7436. Consolidate implementation of concat(). (wheat9) - - HDFS-7440. Consolidate snapshot related operations in a single class. - (wheat9) - - HDFS-6803 Document DFSClient#DFSInputStream expectations reading and preading - in concurrent context. (stack via stevel) - - HDFS-7310. Mover can give first priority to local DN if it has target storage type - available in local DN. (Vinayakumar B via umamahesh) - - HDFS-7210. Avoid two separate RPC's namenode.append() and namenode.getFileInfo() - for an append call from DFSClient. (Vinayakumar B via umamahesh) - - HDFS-7450. Consolidate the implementation of GetFileInfo(), GetListings() and - GetContentSummary() into a single class. (wheat9) - - HDFS-7438. Consolidate the implementation of rename() into a single class. - (wheat9) - - HDFS-7462. Consolidate implementation of mkdirs() into a single class. - (wheat9) - - HDFS-6735. A minor optimization to avoid pread() be blocked by read() - inside the same DFSInputStream (Lars Hofhansl via stack) - - HDFS-7458. Add description to the nfs ports in core-site.xml used by nfs - test to avoid confusion (Yongjun Zhang via brandonli) - - HDFS-7468. Moving verify* functions to corresponding classes. - (Li Lu via wheat9) - - HDFS-7478. Move org.apache.hadoop.hdfs.server.namenode.NNConf to - FSNamesystem. (Li Lu via wheat9) - - HDFS-7474. Avoid resolving path in FSPermissionChecker. (jing9) - - HDFS-7459. Consolidate cache-related implementation in FSNamesystem into - a single class. (wheat9) - - HDFS-7476. Consolidate ACL-related operations to a single class. - (wheat9 via cnauroth) - - HDFS-7384. 'getfacl' command and 'getAclStatus' output should be in sync. - (Vinayakumar B via cnauroth) - - HDFS-7486. Consolidate XAttr-related implementation into a single class. - (wheat9) - - HDFS-7498. Simplify the logic in INodesInPath. (jing9) - - HDFS-7463. Simplify FSNamesystem#getBlockLocationsUpdateTimes. (wheat9) - - HDFS-7509. Avoid resolving path multiple times. (jing9) - - HDFS-7426. Change nntop JMX format to be a JSON blob. (wang) - - HDFS-7513. HDFS inotify: add defaultBlockSize to CreateEvent (cmccabe) - - HDFS-7536. Remove unused CryptoCodec in org.apache.hadoop.fs.Hdfs. - (Yi Liu via wheat9) - - HDFS-7528. Consolidate symlink-related implementation into a single class. - (wheat9) - - HDFS-7373. Clean up temporary files after fsimage transfer failures. - (kihwal) - - HDFS-7543. Avoid path resolution when getting FileStatus for audit logs. - (wheat9) - - HDFS-7530. Allow renaming of encryption zone roots. (Charles Lamb via wang) - - HDFS-7484. Make FSDirectory#addINode take existing INodes as its parameter. - (jing9) - - HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch - (ozawa) - - HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop - codebase. (Sangjin Lee via Colin P. McCabe) - - HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. - (jing9 via yliu) - - HDFS-7326: Add documentation for hdfs debug commands (Vijay Bhat via Colin - P. McCabe) - - HDFS-7598. Remove dependency on old version of guava in - TestDFSClientCache#testEviction. (Sangjin Lee via Colin P. McCabe) - - HDFS-7600. Refine hdfs admin classes to reuse common code. (jing9) - - HDFS-2219. Change fsck to support fully qualified paths so that a - particular namenode in a federated cluster with multiple namenodes - can be specified in the path parameter. (szetszwo) - - HDFS-7457. DatanodeID generates excessive garbage. (daryn via kihwal) - - HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. - McCabe via yliu) - - HDFS-7573. Consolidate the implementation of delete() into a single class. - (wheat9) - - HDFS-7640. Print NFS Client in the NFS log. (Brandon Li via wheat9) - - HDFS-7623. Add htrace configuration properties to core-default.xml and - update user doc about how to enable htrace. (yliu) - - HDFS-7224. Allow reuse of NN connections via webhdfs (Eric Payne via - kihwal) - - HDFS-7683. Combine usages and percent stats in NameNode UI. - (Vinayakumar B via wheat9) - - HDFS-7675. Remove unused member DFSClient#spanReceiverHost (cmccabe) - - HDFS-7603. The background replication queue initialization may not let - others run (kihwal) - - HDFS-7706. Switch BlockManager logging to use slf4j. (wang) - - HDFS-5631. Change BlockMetadataHeader.readHeader(..), ChunkChecksum - class and constructor to public; and fix FsDatasetSpi to use generic type - instead of FsVolumeImpl. (David Powell and Joe Pallas via szetszwo) - - HDFS-5782. Change BlockListAsLongs constructor to take Replica as parameter - type instead of concrete classes Block and ReplicaInfo. (David Powell - and Joe Pallas via szetszwo) - - HDFS-7681. Change ReplicaInputStreams constructor to take InputStream(s) - instead of FileDescriptor(s). (Joe Pallas via szetszwo) - - HDFS-7712. Switch blockStateChangeLog to use slf4j. (wang) - - HDFS-7270. Add congestion signaling capability to DataNode write protocol. - (wheat9) - - HDFS-7732. Fix the order of the parameters in DFSConfigKeys. - (Brahma Reddy Battula via aajisaka) - - HDFS-7710. Remove dead code in BackupImage.java. (Xiaoyu Yao via aajisaka) - - HDFS-7738. Revise the exception message for recover lease; add more truncate - tests such as truncate with HA setup, negative tests, truncate with other - operations and multiple truncates. (szetszwo) - - HDFS-7743. Code cleanup of BlockInfo and rename BlockInfo to - BlockInfoContiguous. (jing9) - - HDFS-7058. Tests for truncate CLI. (Dasha Boudnik via shv) - - HDFS-7760. Document truncate for WebHDFS. (shv) - - HDFS-7761. cleanup unnecssary code logic in LocatedBlock. (yliu) - - HDFS-7703. Support favouredNodes for the append for new blocks - (vinayakumarb) - - HDFS-7694. FSDataInputStream should support "unbuffer" (cmccabe) - - HDFS-7684. The host:port settings of the daemons should be trimmed before - use. (Anu Engineer via aajisaka) - - HDFS-7790. Do not create optional fields in DFSInputStream unless they are - needed (cmccabe) - - HDFS-7430. Refactor the BlockScanner to use O(1) memory and use multiple - threads (cmccabe) - - HDFS-7604. Track and display failed DataNode storage locations in NameNode. - (cnauroth) - - HDFS-7797. Add audit log for setQuota operation (Rakesh R via umamahesh) - - HDFS-4266. BKJM: Separate write and ack quorum (Rakesh R via umamahesh) - - HDFS-7795. Show warning if not all favored nodes were chosen by namenode - (kihwal) - - HDFS-7780. Update use of Iterator to Iterable in DataXceiverServer and - SnapshotDiffInfo. (Ray Chiang via aajisaka) - - HDFS-7804. correct the haadmin command usage in #HDFSHighAvailabilityWithQJM.html - (Brahma Reddy Battula via umamahesh) - - HDFS-7772. Document hdfs balancer -exclude/-include option in - HDFSCommands.html (Xiaoyu Yao via cnauroth) - - HDFS-7773. Additional metrics in HDFS to be accessed via jmx. - (Anu Engineer via cnauroth) - - HDFS-7740. Test truncate with DataNodes restarting. (yliu) - - HDFS-7495. Remove updatePosition argument from DFSInputStream#getBlockAt() - (cmccabe) - - HDFS-7537. Add "UNDER MIN REPL'D BLOCKS" count to fsck. (GAO Rui via - szetszwo) - - HDFS-7832. Show 'Last Modified' in Namenode's 'Browse Filesystem' - (vinayakumarb) - - HDFS-7819. Log WARN message for the blocks which are not in Block ID based - layout (Rakesh R via Colin P. McCabe) - - HDFS-7308. Change the packet chunk size computation in DFSOutputStream in - order to enforce packet size <= 64kB. (Takuya Fukudome via szetszwo) - - HDFS-7685. Document dfs.namenode.heartbeat.recheck-interval in - hdfs-default.xml. (Kai Sasaki via aajisaka) - - HDFS-5853. Add "hadoop.user.group.metrics.percentiles.intervals" to - hdfs-default.xml. (aajisaka) - - HDFS-7439. Add BlockOpResponseProto's message to the exception messages. - (Takanobu Asanuma via szetszwo) - - HDFS-7789. DFSck should resolve the path to support cross-FS symlinks. - (gera) - - HDFS-7535. Utilize Snapshot diff report for distcp. (jing9) - - HDFS-1522. Combine two BLOCK_FILE_PREFIX constants into one. - (Dongming Liang via shv) - - HDFS-7746. Add a test randomly mixing append, truncate and snapshot - operations. (szetszwo) - - HADOOP-11648. Set DomainSocketWatcher thread name explicitly. - (Liang Xie via ozawa) - - HDFS-7855. Separate class Packet from DFSOutputStream. (Li Bo bia jing9) - - HDFS-7411. Change decommission logic to throttle by blocks rather than - nodes in each interval. (Andrew Wang via cdouglas) - - HDFS-7898. Change TestAppendSnapshotTruncate to fail-fast. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-6806. HDFS Rolling upgrade document should mention the versions - available. (J.Andreina via aajisaka) - - HDFS-7491. Add incremental blockreport latency to DN metrics. - (Ming Ma via cnauroth) - - HDFS-7435. PB encoding of block reports is very inefficient. - (Daryn Sharp via kihwal) - - HDFS-2605. Remove redundant "Release 0.21.1" section from CHANGES.txt. - (Allen Wittenauer via shv) - - HDFS-7940. Add tracing to DFSClient#setQuotaByStorageType (Rakesh R via - Colin P. McCabe) - - HDFS-7054. Make DFSOutputStream tracing more fine-grained (cmccabe) - - HDFS.7849. Update documentation for enabling a new feature in rolling - upgrade ( J.Andreina via vinayakumarb ) - - HDFS-7962. Remove duplicated logs in BlockManager. (yliu) - - HDFS-7917. Use file to replace data dirs in test to simulate a disk failure. - (Lei (Eddy) Xu via cnauroth) - - HDFS-7956. Improve logging for DatanodeRegistration. - (Plamen Jeliazkov via shv) - - HDFS-7976. Update NFS user guide for mount option "sync" to minimize or - avoid reordered writes. (brandonli) - - HDFS-7410. Support CreateFlags with append() to support hsync() for - appending streams (Vinayakumar B via Colin P. McCabe) - - HDFS-8008. Support client-side back off when the datanodes are congested. - (wheat9) - - HDFS-7811. Avoid recursive call getStoragePolicyID in - INodeFile#computeQuotaUsage. (Xiaoyu Yao and jing9) - - HDFS-8071. Redundant checkFileProgress() in PART II of getAdditionalBlock(). - (shv) - - OPTIMIZATIONS - - HDFS-7454. Reduce memory footprint for AclEntries in NameNode. - (Vinayakumar B via wheat9) - - HDFS-7615. Remove longReadLock (kihwal) - - BUG FIXES - - HDFS-6741. Improve permission denied message when - FSPermissionChecker#checkOwner fails (Stephen Chu and harsh). - - HDFS-6538. Comment format error in ShortCircuitRegistry javadoc. - (David Luo via harsh). - - HDFS-7194. Fix findbugs "inefficient new String constructor" warning in - DFSClient#PATH (yzhang via cmccabe) - - HDFS-7198. Fix or suppress findbugs "unchecked conversion" warning in - DFSClient#getPathTraceScope (cmccabe) - - HDFS-6657. Remove link to 'Legacy UI' in trunk's Namenode UI. - (Vinayakumar B via wheat) - - HDFS-7201. Fix typos in hdfs-default.xml. (Dawson Choong via wheat9) - - HDFS-7190. Bad use of Preconditions in startFileInternal(). - (Dawson Choong via wheat9) - - HDFS-7242. Code improvement for FSN#checkUnreadableBySuperuser. - (Yi Liu via vinayakumarb) - - HDFS-7252. small refinement to the use of isInAnEZ in FSNamesystem. - (Yi Liu via vinayakumarb) - - HDFS-7277. Remove explicit dependency on netty 3.2 in BKJournal. (wheat9) - - HDFS-7232. Populate hostname in httpfs audit log (Zoran Dimitrijevic - via aw) - - HDFS-7258. CacheReplicationMonitor rescan schedule log should use DEBUG - level instead of INFO level. (Xiaoyu Yao via wheat9) - - HDFS-7282. Fix intermittent TestShortCircuitCache and - TestBlockReaderFactory failures resulting from TemporarySocketDirectory GC. - (Jinghui Wang via Colin Patrick McCabe) - - HDFS-7301. TestMissingBlocksAlert should use MXBeans instead of old web UI. - (Zhe Zhang via wheat9) - - HDFS-7315. DFSTestUtil.readFileBuffer opens extra FSDataInputStream. - (Plamen Jeliazkov via wheat9) - - HDFS-7324. haadmin command usage prints incorrect command name. - (Brahma Reddy Battula via suresh) - - HDFS-7366. BlockInfo should take replication as an short in the constructor. - (Li Lu via wheat9) - - HDFS-7389. Named user ACL cannot stop the user from accessing the FS entity. - (Vinayakumar B via cnauroth) - - HDFS-6938. Cleanup javac warnings in FSNamesystem (Charles Lamb via wheat9) - - HDFS-7358. Clients may get stuck waiting when using ByteArrayManager. - (szetszwo) - - HDFS-7395. BlockIdManager#clear() bails out when resetting the - GenerationStampV1Limit. (wheat9) - - HDFS-7399. Lack of synchronization in - DFSOutputStream#Packet#getLastByteOffsetBlock() (vinayakumarb) - - HDFS-7146. NFS ID/Group lookup requires SSSD enumeration on the server - (Yongjun Zhang via brandonli) - - HDFS-7406. SimpleHttpProxyHandler puts incorrect "Connection: Close" - header. (wheat9) - - HDFS-7374. Allow decommissioning of dead DataNodes. (Zhe Zhang) - - HDFS-7403. Inaccurate javadoc of BlockUCState#COMPLETE state. ( - Yongjun Zhang via yliu) - - HDFS-7303. NN UI fails to distinguish datanodes on the same host. - (Benoy Antony via wheat9) - - HDFS-7097. Allow block reports to be processed during checkpointing on - standby name node. (kihwal via wang) - - HDFS-7444. convertToBlockUnderConstruction should preserve BlockCollection. - (wheat9) - - HDFS-7448 TestBookKeeperHACheckpoints fails in trunk build - (Akira Ajisaka via stevel) - - HDFS-7472. Fix typo in message of ReplicaNotFoundException. - (Masatake Iwasaki via wheat9) - - HDFS-7473. Document setting dfs.namenode.fs-limits.max-directory-items to 0 - is invalid. (Akira AJISAKA via cnauroth) - - HDFS-7481. Add ACL indicator to the "Permission Denied" exception. - (vinayakumarb) - - HDFS-7502. Fix findbugs warning in hdfs-nfs project. - (Brandon Li via wheat9) - - HDFS-5578. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags - in doc comments. (Andrew Purtell via wheat9) - - HDFS-7475. Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved - deterministic. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7515. Fix new findbugs warnings in hadoop-hdfs. (wheat9) - - HDFS-7497. Inconsistent report of decommissioning DataNodes between - dfsadmin and NameNode webui. (Yongjun Zhang via wang) - - HDFS-7517. Remove redundant non-null checks in FSNamesystem# - getBlockLocations. (wheat9) - - HDFS-7514. TestTextCommand fails on Windows. (Arpit Agarwal) - - HDFS-7506. Consolidate implementation of setting inode attributes into a - single class. (wheat9) - - HDFS-7516. Fix findbugs warnings in hdfs-nfs project. (brandonli) - - HDFS-6425. Large postponedMisreplicatedBlocks has impact on blockReport - latency. (Ming Ma via kihwal) - - HDFS-7494. Checking of closed in DFSInputStream#pread() should be protected - by synchronization (Ted Yu via Colin P. McCabe) - - HDFS-7431. log message for InvalidMagicNumberException may be incorrect. - (Yi Liu via cnauroth) - - HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java - (Colin P.McCabe) - - HDFS-7560. ACLs removed by removeDefaultAcl() will be back after NameNode - restart/failover. (Vinayakumar B via cnauroth) - - HDFS-7456. De-duplicate AclFeature instances with same AclEntries do reduce - memory footprint of NameNode (vinayakumarb) - - HDFS-7563. NFS gateway parseStaticMap NumberFormatException - (Yongjun Zhang via brandonli) - - HDFS-7572. TestLazyPersistFiles#testDnRestartWithSavedReplicas is flaky on - Windows. (Arpit Agarwal via cnauroth) - - HDFS-7583. Fix findbug in TransferFsImage.java (vinayakumarb) - - HDFS-7564. NFS gateway dynamically reload UID/GID mapping file /etc/nfs.map - (Yongjun Zhang via brandonli) - - HDFS-7561. TestFetchImage should write fetched-image-dir under target. - (Liang Xie via shv) - - HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs. - (Zhanwei Wang via cnauroth) - - HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader - incorrectly (Jonathan Mace via Colin P. McCabe) - - HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures - with page sizes other than 4096 (Sam Liu via Colin P. McCabe) - - HDFS-7635. Remove TestCorruptFilesJsp from branch-2. (cnauroth) - - HDFS-7632. MiniDFSCluster configures DataNode data directories incorrectly if - using more than 1 DataNode and more than 2 storage locations per DataNode. - (cnauroth) - - HDFS-7637. Fix the check condition for reserved path. (Yi Liu via jing9) - - HDFS-7641. Update archival storage user doc for list/set/get block storage - policies. (yliu) - - HDFS-7496. Fix FsVolume removal race conditions on the DataNode by - reference-counting the volume instances (lei via cmccabe) - - HDFS-7548. Corrupt block reporting delayed until datablock scanner thread - detects it (Rushabh Shah via kihwal) - - HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. - (Ming Ma via cnauroth) - - HDFS-7660. BlockReceiver#close() might be called multiple times, which - causes the fsvolume reference being released incorrectly. (Lei Xu via - yliu) - - HDFS-7644. minor typo in HttpFS doc (Charles Lamb via aw) - - HDFS-7606. Fix potential NPE in INodeFile.getBlocks(). (Byron Wong via shv) - - HDFS-7638: Small fix and few refinements for FSN#truncate. (yliu) - - HDFS-7634. Disallow truncation of Lazy persist files. (Yi Liu via - Arpit Agarwal) - - HDFS-7643. Test case to ensure lazy persist files cannot be truncated. - (Yi Liu via Arpit Agarwal) - - HDFS-7659. truncate should check negative value of the new length. - (Yi Liu via shv) - - HDFS-7676. Fix TestFileTruncate to avoid bug of HDFS-7611. (shv) - - HDFS-49. MiniDFSCluster.stopDataNode will always shut down a node in - the cluster if a matching name is not found. (stevel) - - HDFS-7566. Remove obsolete entries from hdfs-default.xml (Ray Chiang - via aw) - - HDFS-7677. DistributedFileSystem#truncate should resolve symlinks. (yliu) - - HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks - in the blocksMap on NameNode restart. (jing9 and Byron Wong) - - HDFS-7423. various typos and message formatting fixes in nfs daemon and - doc. (Charles Lamb via yliu) - - HDFS-7696. In FsDatasetImpl, the getBlockInputStream(..) and - getTmpInputStreams(..) methods may leak file descriptors. (szetszwo) - - HDFS-6651. Deletion failure can leak inodes permanently. - (Jing Zhao via wheat9) - - HDFS-7734. Class cast exception in NameNode#main. (yliu via wang) - - HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some - in-memory state associated with volumes. (Lei (Eddy) Xu via Colin P. - McCabe) - - HDFS-7709. Fix findbug warnings in httpfs. (Rakesh R via ozawa) - - HDFS-7698. Fix locking on HDFS read statistics and add a method for - clearing them. (Colin P. McCabe via yliu) - - HDFS-7741. Remove unnecessary synchronized in FSDataInputStream and - HdfsDataInputStream. (yliu) - - HDFS-7647. DatanodeManager.sortLocatedBlocks sorts DatanodeInfos - but not StorageIDs. (Milan Desai via Arpit Agarwal) - - HDFS-7756. Restore method signature for LocatedBlock#getLocations(). (Ted - Yu via yliu) - - HDFS-7744. Fix potential NPE in DFSInputStream after setDropBehind or - setReadahead is called (cmccabe) - - HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider - threads when using FileContext (Arun Suresh via Colin P. McCabe) - - HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. - (szetszwo) - - HDFS-7753. Fix Multithreaded correctness Warnings in BackupImage. - (Rakesh R and shv) - - HDFS-7736. Fix typos in dfsadmin/fsck/snapshotDiff usage messages. - (Brahma Reddy Battula via wheat9) - - HDFS-7771. fuse_dfs should permit FILE: on the front of KRB5CCNAME - (cmccabe) - - HDFS-7704. DN heartbeat to Active NN may be blocked and expire if - connection to Standby NN continues to time out (Rushabh Shah via kihwal) - - HDFS-7721. The HDFS BlockScanner may run fast during the first hour - (cmccabe) - - HDFS-7686. Re-add rapid rescan of possibly corrupt block feature to the - block scanner (cmccabe) - - HDFS-7778. Rename FsVolumeListTest to TestFsVolumeList and commit it to - branch-2. (Lei (Eddy) Xu via cnauroth) - - HDFS-4625. BKJM doesn't take advantage of speculative reads. (Rakesh R - via aajisaka) - - HDFS-7798. Checkpointing failure caused by shared KerberosAuthenticator. - (Chengbing Liu via yliu) - - HDFS-6662. WebHDFS cannot open a file if its path contains "%". - (Gerson Carlos via wheat9) - - HDFS-7814. Fix usage string of storageType parameter for - "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth) - - HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) - - HDFS-7805. NameNode recovery prompt should be printed on console (Surendra - Singh Lilhore via Colin P. McCabe) - - HDFS-7008. xlator should be closed upon exit from DFSAdmin#genericRefresh(). - (ozawa) - - HDFS-7831. Fix the starting index and end condition of the loop in - FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9) - - HDFS-7843. A truncated file is corrupted after rollback from a rolling - upgrade. (szetszwo) - - HDFS-7774. Unresolved symbols error while compiling HDFS on Windows 7/32 bit. - (Kiran Kumar M R via cnauroth) - - HDFS-6753. Initialize checkDisk when DirectoryScanner not able to get - files list for scanning (J.Andreina via vinayakumarb) - - HDFS-7785. Improve diagnostics information for HttpPutFailedException. - (Chengbing Liu via wheat9) - - HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula - via Arpit Agarwal) - - HDFS-6565. Use jackson instead jetty json in hdfs-client. - (Akira Ajisaka via wheat9) - - HDFS-7682. {{DistributedFileSystem#getFileChecksum}} of a snapshotted file - includes non-snapshotted content. (Charles Lamb via atm) - - HDFS-7869. Inconsistency in the return information while performing rolling - upgrade ( J.Andreina via vinayakumarb ) - - HDFS-7879. hdfs.dll does not export functions of the public libhdfs API. - (Chris Nauroth via wheat9) - - HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) - - HDFS-7818. OffsetParam should return the default value instead of throwing - NPE when the value is unspecified. (Eric Payne via wheat9) - - HDFS-6833. DirectoryScanner should not register a deleting block with - memory of DataNode. (Shinichi Yamashita via szetszwo) - - HDFS-7926. NameNode implementation of ClientProtocol.truncate(..) is not - idempotent (Tsz Wo Nicholas Sze via brandonli) - - HDFS-7903. Cannot recover block after truncate and delete snapshot. - (Plamen Jeliazkov via shv) - - HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) - - HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao - via Arpit Agarwal) - - HDFS-7953. NN Web UI fails to navigate to paths that contain #. - (kanaka kumar avvaru via wheat9) - - HDFS-7948. TestDataNodeHotSwapVolumes#testAddVolumeFailures failed on - Windows. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7950. Fix TestFsDatasetImpl#testAddVolumes failure on Windows. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7951. Fix NPE for - TestFsDatasetImpl#testAddVolumeFailureReleasesInUseLock on Linux - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7722. DataNode#checkDiskError should also remove Storage when error - is found. (Lei Xu via Colin P. McCabe) - - HDFS-7697. Mark the PB OIV tool as experimental. (Lei Xu via wheat9) - - HDFS-7914. TestJournalNode#testFailToStartWithBadConfig fails when the - default dfs.journalnode.http-address port 8480 is in use. (Xiaoyu Yao via - Arpit Agarwal) - - HDFS-7945. The WebHdfs system on DN does not honor the length parameter. - (wheat9) - - HDFS-7943. Append cannot handle the last block with length greater than - the preferred block size. (jing9) - - HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) - - HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) - - HDFS-7957. Truncate should verify quota before making changes. (jing9) - - HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() - (Vinayakumar B via kihwal) - - HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) - - HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) - - HDFS-7977. NFS couldn't take percentile intervals (brandonli) - - HDFS-7963. Fix expected tracing spans in TestTracing along with HDFS-7054. - (Masatake Iwasaki via kihwal) - - HDFS-7748. Separate ECN flags from the Status in the DataTransferPipelineAck. - (Anu Engineer and Haohui Mai via wheat9) - - HDFS-8036. Use snapshot path as source when using snapshot diff report in - DistCp. (Jing Zhao via wheat9) - - HDFS-8001 RpcProgramNfs3 : wrong parsing of dfs.blocksize - (Remi Catherinot via brandonli) - - HDFS-7954. TestBalancer#testBalancerWithPinnedBlocks should not be executed - on Windows. (Xiaoyu Yao via szetszwo) - - HDFS-8034. Fix - TestDFSClientRetries#testDFSClientConfigurationLocateFollowingBlockInitialDelay - for Windows. (Xiaoyu Yao via cnauroth) - - HDFS-8039. Fix TestDebugAdmin#testRecoverLease and - testVerifyBlockChecksumCommand on Windows. (Xiaoyu Yao via cnauroth) - - HDFS-7996. After swapping a volume, BlockReceiver reports - ReplicaNotFoundException (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8051. FsVolumeList#addVolume should release volume reference if not - put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) - - HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in - platform-specific format. (Xiaoyu Yao via cnauroth) - - HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki - via Colin P. McCabe) - - BREAKDOWN OF HDFS-7584 SUBTASKS AND RELATED JIRAS - - HDFS-7720. Quota by Storage Type API, tools and ClientNameNode - Protocol changes. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7723. Quota By Storage Type namenode implemenation. (Xiaoyu Yao - via Arpit Agarwal) - - HDFS-7751. Fix TestHDFSCLI for quota with storage type. (Xiaoyu Yao - via szetszwo) - - HDFS-7776. Adding additional unit tests for Quota By Storage Type. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7775. Use consistent naming for NN-internal quota related types - and functions. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7806. Refactor: move StorageType from hadoop-hdfs to - hadoop-common. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7824. GetContentSummary API and its namenode implementation for - Storage Type Quota/Usage. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7700. Document quota support for storage types. (Xiaoyu Yao via - Arpit Agarwal) - -Release 2.6.5 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-9629. Update the footer of Web UI to show year 2016 - (Xiao Chen and Vinayakumar B via vinayakumarb) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown() - is wrong. (Wei-Chiu Chuang via zhz) - -Release 2.6.4 - 2016-02-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-9600. do not check replication if the block is under construction - (Phil Yang via vinayakumarb) - - HDFS-9294. DFSClient deadlock when close file and failed to renew lease. - (Brahma Reddy Battula via szetszwo) - -Release 2.6.3 - 2015-12-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-8615. Correct HTTP method in WebHDFS document. - (Brahma Reddy Battula via aajisaka) - - HDFS-9431. DistributedFileSystem#concat fails if the target path is - relative. (Kazuho Fujii via aajisaka) - - HDFS-9434. Recommission a datanode with 500k blocks may pause NN for 30 - seconds for printing info log messags. (szetszwo) - - HDFS-9470. Encryption zone on root not loaded from fsimage after NN - restart. (Xiao Chen via wang) - - HDFS-9273. ACLs on root directory may be lost after NN restart. - (Xiao Chen via cnauroth) - -Release 2.6.2 - 2015-10-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.6.1 - 2015-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-7278. Add a command that allows sysadmins to manually trigger full - block reports from a DN (cmccabe) - - IMPROVEMENTS - - HDFS-7035. Make adding a new data directory to the DataNode an atomic - operation and improve error handling (Lei Xu via Colin P. McCabe) - - HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin - P. McCabe) - - HDFS-7579. Improve log reporting during block report rpc failure. - (Charles Lamb via cnauroth) - - HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) - - HDFS-7596. NameNode should prune dead storages from storageMap. - (Arpit Agarwal via cnauroth) - - HDFS-8046. Allow better control of getContentSummary (kihwal) - - HDFS-8384. Allow NN to startup if there are files having a lease but are not - under construction. (jing9) - - OPTIMIZATIONS - - HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using - hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang - via Colin P. McCabe) - - BUG FIXES - - HDFS-7425. NameNode block deletion logging uses incorrect appender. - (cnauroth) - - HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in - checkLeases (Ravi Prakash via Colin P. McCabe) - - HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes - (Noah Lorang via Colin P. McCabe) - - HDFS-7503. Namenode restart after large deletions can cause slow - processReport. (Arpit Agarwal) - - HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate - block files are present in the same volume (cmccabe) - - HDFS-3443. Fix NPE when namenode transition to active during startup by - adding checkNNStartup() in NameNodeRpcServer. (Vinayakumar B via szetszwo) - - HDFS-7733. NFS: readdir/readdirplus return null directory - attribute on failure. (Arpit Agarwal) - - HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe) - - HDFS-7213. processIncrementalBlockReport performance degradation. - (Eric Payne via kihwal) - - HDFS-7314. When the DFSClient lease cannot be renewed, abort open-for-write - files rather than the entire DFSClient. (mingma) - - HDFS-7235. DataNode#transferBlock should report blocks that don't exist - using reportBadBlock (yzhang via cmccabe) - - HDFS-7263. Snapshot read can reveal future bytes for appended files. - (Tao Luo via shv) - - HDFS-7225. Remove stale block invalidation work when DN re-registers with - different UUID. (Zhe Zhang and Andrew Wang) - - HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade - shutdown command (Eric Payne via kihwal) - - HDFS-7575. Upgrade should generate a unique storage ID for each - volume. (Arpit Agarwal) - - HDFS-7707. Edit log corruption due to delayed block removal again. - (Yongjun Zhang via kihwal) - - HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause - DataNode to register successfully with only one NameNode.(vinayakumarb) - - HDFS-7788. Post-2.6 namenode may not start up with an image containing - inodes created with an old release. (Rushabh Shah via kihwal) - - HDFS-7009. Active NN and standby NN have different live nodes. - (Ming Ma via cnauroth) - - HDFS-7763. fix zkfc hung issue due to not catching exception in a corner - case. (Liang Xie via wang) - - HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" - message. (jing9) - - HDFS-7885. Datanode should not trust the generation stamp provided by - client. (Tsz Wo Nicholas Sze via jing9) - - HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via - Colin P. McCabe) - - HDFS-7830. DataNode does not release the volume lock when adding a volume - fails. (Lei Xu via Colin P. Mccabe) - - HDFS-7587. Edit log corruption can happen if append fails with a quota - violation. (jing9) - - HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade - starts (Zhe Zhang via Colin P. McCabe) - - HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) - - HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp - provided by the client is larger than the one stored in the datanode. - (Brahma Reddy Battula via szetszwo) - - HDFS-7960. The full block report should prune zombie storages even if - they're not empty. (cmccabe and Eddy Xu via wang) - - HDFS-7742. Favoring decommissioning node for replication can cause a block - to stay underreplicated for long periods (Nathan Roberts via kihwal) - - HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl - lock for a very long time (sinago via cmccabe) - - HDFS-8072. Reserved RBW space is not released if client terminates while - writing block. (Arpit Agarwal) - - HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to - finalize upgrade. (jing9) - - HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and - fail to tell the DFSClient about it because of a network error (cmccabe) - - HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on - post-HDFS-7915 DataNode (cmccabe) - - HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. - (surendra singh lilhore via Xiaoyu Yao) - - HDFS-7894. Rolling upgrade readiness is not updated in jmx until query - command is issued. (Brahma Reddy Battula via kihwal) - - HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block - request is in edit log. (Rushabh S Shah via kihwal) - - HDFS-8404. Pending block replication can get stuck using older genstamp - (Nathan Roberts via kihwal) - - HDFS-8431. hdfs crypto class not found in Windows. - (Anu Engineer via cnauroth) - - HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. - (Ming Ma via jing9) - - HDFS-8270. create() always retried with hardcoded timeout when file already - exists with open lease (J.Andreina via vinayakumarb) - - HDFS-7980. Incremental BlockReport will dramatically slow down namenode - startup. (Walter Su via szetszwo) - - HDFS-7446. HDFS inotify should have the ability to determine what txid it - has read up to (cmccabe) - - HDFS-8846. Add a unit test for INotify functionality across a layout - version upgrade (Zhe Zhang via Colin P. McCabe) - - HDFS-7470. SecondaryNameNode need twice memory when calling - reloadFromImageFile. (zhaoyunjiong via cnauroth) - - HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is - flawed. (Kihwal Lee via yliu) - - HDFS-7552. Change FsVolumeList toString() to fix - TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) - -Release 2.6.0 - 2014-11-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-6584. Support archival storage. (See breakdown of tasks below for - features and contributors) - - HDFS-6581. Support writing to a single replica in RAM. (See breakdown of - tasks below for features and contributors). - - IMPROVEMENTS - - HDFS-6613. Improve logging in caching classes. (wang) - - HDFS-6511. BlockManager#computeInvalidateWork() could do nothing. (Juan Yu via wang) - - HDFS-6638. Shorten test run time with a smaller retry timeout setting. - (Liang Xie via cnauroth) - - HDFS-6627. Rename DataNode#checkWriteAccess to checkReadAccess. - (Liang Xie via cnauroth) - - HDFS-6645. Add test for successive Snapshots between XAttr modifications. - (Stephen Chu via jing9) - - HDFS-6643. Refactor INodeWithAdditionalFields.PermissionStatusFormat and - INodeFile.HeaderFormat. (szetszwo) - - HDFS-6640. Syntax for MKDIRS, CREATESYMLINK, and SETXATTR are given wrongly - in WebHdfs document (missed webhdfs/v1). (Stephen Chu via jing9) - - HDFS-5202. Support Centralized Cache Management on Windows. (cnauroth) - - HDFS-2976. Remove unnecessary method (tokenRefetchNeeded) in DFSClient. - (Uma Maheswara Rao G) - - HDFS-3851. DFSOutputStream class code cleanup. (Jing Zhao via suresh) - - HDFS-2856. Fix block protocol so that Datanodes don't require root or jsvc. - (cnauroth) - - HDFS-5624. Add HDFS tests for ACLs in combination with viewfs. - (Stephen Chu via cnauroth) - - HDFS-6655. Add 'header banner' to 'explorer.html' also in Namenode UI - (vinayakumarb) - - HDFS-4120. Add a new "-skipSharedEditsCheck" option for BootstrapStandby - (Liang Xie and Rakesh R via vinayakumarb) - - HDFS-6597. Add a new option to NN upgrade to terminate the process after - upgrade on NN is completed. (Danilo Vunjak via cnauroth) - - HDFS-6700. BlockPlacementPolicy shoud choose storage but not datanode for - deletion. (szetszwo) - - HDFS-6616. Add exclude-datanodes feature to WebHDFS redirection so that it - will not redirect retries to the same datanode. (zhaoyunjiong via szetszwo) - - HDFS-6702. Change DFSClient to pass the StorageType from the namenode to - datanodes and change datanode to write block replicas using the specified - storage type. (szetszwo) - - HDFS-6701. Make seed optional in NetworkTopology#sortByDistance. - (Ashwin Shankar via wang) - - HDFS-6755. There is an unnecessary sleep in the code path where - DFSOutputStream#close gives up its attempt to contact the namenode - (mitdesai21 via cmccabe) - - HDFS-6750. The DataNode should use its shared memory segment to mark - short-circuit replicas that have been unlinked as stale (cmccabe) - - HDFS-6739. Add getDatanodeStorageReport to ClientProtocol. (szetszwo) - - HDFS-6665. Add tests for XAttrs in combination with viewfs. - (Stephen Chu via wang) - - HDFS-6778. The extended attributes javadoc should simply refer to the - user docs. (clamb via wang) - - HDFS-6570. add api that enables checking if a user has certain permissions on - a file. (Jitendra Pandey via cnauroth) - - HDFS-6441. Add ability to exclude/include specific datanodes while - balancing. (Benoy Antony and Yu Li via Arpit Agarwal) - - HDFS-6685. Balancer should preserve storage type of replicas. (szetszwo) - - HDFS-6798. Add test case for incorrect data node condition during - balancing. (Benoy Antony via Arpit Agarwal) - - HDFS-6796. Improve the argument check during balancer command line parsing. - (Benoy Antony via szetszwo) - - HDFS-6794. Update BlockManager methods to use DatanodeStorageInfo - where possible (Arpit Agarwal) - - HDFS-6802. Some tests in TestDFSClientFailover are missing @Test - annotation. (Akira Ajisaka via wang) - - HDFS-6788. Improve synchronization in BPOfferService with read write lock. - (Yongjun Zhang via wang) - - HDFS-6787. Remove duplicate code in FSDirectory#unprotectedConcat. (Yi Liu via umamahesh) - - HDFS-6809. Move Balancer's inner classes MovedBlocks and Matcher as to - standalone classes and separates KeyManager from NameNodeConnector. - (szetszwo) - - HDFS-6812. Remove addBlock and replaceBlock from DatanodeDescriptor. - (szetszwo) - - HDFS-6781. Separate HDFS commands from CommandsManual.apt.vm. (Akira - Ajisaka via Arpit Agarwal) - - HDFS-6728. Dynamically add new volumes to DataStorage, formatted if - necessary. (Lei Xu via atm) - - HDFS-6740. Make FSDataset support adding data volumes dynamically. (Lei - Xu via atm) - - HDFS-6722. Display readable last contact time for dead nodes on NN webUI. - (Ming Ma via wheat9) - - HDFS-6772. Get DN storages out of blockContentsStale state faster after - NN restarts. (Ming Ma via Arpit Agarwal) - - HDFS-573. Porting libhdfs to Windows. (cnauroth) - - HDFS-6828. Separate block replica dispatching from Balancer. (szetszwo via - jing9) - - HDFS-6837. Code cleanup for Balancer and Dispatcher. (szetszwo via - jing9) - - HDFS-6838. Code cleanup for unnecessary INode replacement. - (Jing Zhao via wheat9) - - HDFS-6836. HDFS INFO logging is verbose & uses file appenders. (Xiaoyu - Yao via Arpit Agarwal) - - HDFS-6567. Normalize the order of public final in HdfsFileStatus. - (Tassapol Athiapinya via wheat9) - - HDFS-6849. Replace HttpFS custom proxyuser handling with common - implementation. (tucu) - - HDFS-6850. Move NFS out of order write unit tests into TestWrites class. - (Zhe Zhang via atm) - - HDFS-6188. An ip whitelist based implementation of TrustedChannelResolver. - (Benoy Antony via Arpit Agarwal) - - HDFS-6858. Allow dfs.data.transfer.saslproperties.resolver.class default to - hadoop.security.saslproperties.resolver.class. (Benoy Antony via cnauroth) - - HDFS-6878. Change MiniDFSCluster to support StorageType configuration - for individual directories. (Arpit Agarwal) - - HDFS-6758. block writer should pass the expected block size to - DataXceiverServer. (Arpit Agarwal) - - HDFS-6899. Allow changing MiniDFSCluster volumes per DN and capacity - per volume. (Arpit Agarwal) - - HDFS-4486. Add log category for long-running DFSClient notices (Zhe Zhang - via Colin Patrick McCabe) - - HDFS-6879. Adding tracing to Hadoop RPC (Masatake Iwasaki via Colin Patrick - McCabe) - - HDFS-6774. Make FsDataset and DataStore support removing volumes. (Lei Xu - via atm) - - HDFS-6634. inotify in HDFS. (James Thomas via wang) - - HDFS-4257. The ReplaceDatanodeOnFailure policies could have a forgiving - option (szetszwo via cmccabe) - - HDFS-6959. Make the HDFS home directory location customizable. (yzhang via - cmccabe) - - HDFS-6609. Use DirectorySnapshottableFeature to represent a snapshottable - directory. (Jing Zhao via wheat9) - - HDFS-6886. Use single editlog record for creating file + overwrite. (Yi Liu - via jing9) - - HDFS-6376. Distcp data between two HA clusters requires another configuration. - (Dave Marion and Haohui Mai via jing9) - - HDFS-6943. Improve NN allocateBlock log to include replicas' datanode IPs. - (Ming Ma via wheat9) - - HDFS-6036. Forcibly timeout misbehaving DFSClients that try to do - no-checksum reads that extend too long (cmccabe) - - HDFS-6482. Use block ID-based block layout on datanodes (James Thomas via - Colin Patrick McCabe) - - HDFS-7061. Add test to verify encryption zone creation after NameNode - restart without saving namespace. (Stephen Chu via wang) - - HDFS-7059. HAadmin transtionToActive with forceActive option can show - confusing message. - - HDFS-6880. Adding tracing to DataNode data transfer protocol. (iwasakims - via cmccabe) - - HDFS-7006. Test encryption zones with KMS. (Anthony Young-Garner and tucu) - - HDFS-6851. Refactor EncryptionZoneWithId and EncryptionZone. (clamb via wang) - - HDFS-6705. Create an XAttr that disallows the HDFS admin from accessing a - file. (clamb via wang) - - HDFS-6843. Create FileStatus isEncrypted() method (clamb via cmccabe) - - HDFS-7004. Update KeyProvider instantiation to create by URI. (wang) - - HDFS-7047. Expose FileStatus#isEncrypted in libhdfs (cmccabe) - - HDFS-7003. Add NFS Gateway support for reading and writing to - encryption zones. (clamb via wang) - - HDFS-6727. Refresh data volumes on DataNode based on configuration changes - (Lei Xu via cmccabe) - - HDFS-6970. Move startFile EDEK retries to the DFSClient. (wang) - - HDFS-6948. DN rejects blocks if it has older UC block - (Eric Payne via kihwal) - - HDFS-6987. Move CipherSuite xattr information up to the encryption zone - root. (Zhe Zhang via wang) - - HDFS-7139. Unit test for creating encryption zone on root path. (Zhe Zhang via wang) - - HDFS-7138. Fix hftp to work with encryption. (clamb via wang) - - HDFS-7118. Improve diagnostics on storage directory rename operations by - using NativeIO#renameTo in Storage#rename. (cnauroth) - - HDFS-6808. Add command line option to ask DataNode reload configuration. - (Lei Xu via Colin Patrick McCabe) - - HDFS-7119. Split error checks in AtomicFileOutputStream#close into separate - conditions to improve diagnostics. (cnauroth) - - HDFS-7077. Separate CipherSuite from crypto protocol version. (wang) - - HDFS-6956. Allow dynamically changing the tracing level in Hadoop servers - (cmccabe) - - HDFS-7156. Update fsck documentation. (Masahiro Yamaguch via shv) - - HDFS-7093. Add config key to restrict setStoragePolicy. (Arpit Agarwal) - - HDFS-6519. Document oiv_legacy command (Akira AJISAKA via aw) - - HDFS-4165. Faulty sanity check in FsDirectory.unprotectedSetQuota. - (Binglin Chang via suresh) - - HDFS-7104. Fix and clarify INodeInPath getter functions. (Zhe Zhang via wang) - - HDFS-7124. Remove EncryptionZoneManager.NULL_EZ. (clamb via wang) - - HDFS-6779. Add missing version subcommand for hdfs. - (Sasaki Toru via wheat9) - - HDFS-7153. Add storagePolicy to NN edit log during file creation. - (Arpit Agarwal) - - HDFS-7158. Reduce the memory usage of WebImageViewer. (wheat9) - - HDFS-6894. Add XDR parser method for each NFS response. - (Brandon Li via wheat9) - - HDFS-7169. Add SE_BAD_FIELD to findbugsExcludeFile.xml. (szetszwo) - - HDFS-7217. Better batching of IBRs. (kihwal) - - HDFS-7195. Update user doc of secure mode about Datanodes don't require root - or jsvc. (cnauroth) - - HDFS-7228. Add an SSD policy into the default BlockStoragePolicySuite. - (jing9) - - HDFS-6904. YARN unable to renew delegation token fetched via webhdfs - due to incorrect service port. (jitendra) - - HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin - P. McCabe) - - HDFS-7230. Add rolling downgrade documentation. (szetszwo via jing9) - - HDFS-6385. Show when block deletion will start after NameNode startup in - WebUI. (cnauroth) - - HDFS-7313. Support optional configuration of AES cipher suite on - DataTransferProtocol. (cnauroth) - - HDFS-7276. Limit the number of byte arrays used by DFSOutputStream and - provide a mechanism for recycling arrays. (szetszwo) - - HDFS-7233. NN logs unnecessary org.apache.hadoop.hdfs.protocol.UnresolvedPathException. - (Rushabh S Shah via jing9) - - HDFS-7221. TestDNFencingWithReplication fails consistently. (Charles Lamb via wang) - - OPTIMIZATIONS - - HDFS-6690. Deduplicate xattr names in memory. (wang) - - HDFS-6773. MiniDFSCluster should skip edit log fsync by default (Stephen - Chu via Colin Patrick McCabe) - - HDFS-6865. Byte array native checksumming on client side - (James Thomas via todd) - - HDFS-7122. Use of ThreadLocal results in poor block placement. - (wang) - - HDFS-6606. Optimize HDFS Encrypted Transport performance. (yliu) - - BUG FIXES - - HDFS-7309. XMLUtils.mangleXmlString doesn't seem to handle less than sign - (Colin Patrick McCabe via raviprak) - - HDFS-6823. dfs.web.authentication.kerberos.principal shows up in logs for - insecure HDFS (Allen Wittenauer via raviprak) - - HDFS-6517. Remove hadoop-metrics2.properties from hdfs project (Akira - AJISAKA via aw) - - HDFS-6617. Flake TestDFSZKFailoverController.testManualFailoverWithDFSHAAdmin - due to a long edit log sync op. (Liang Xie via cnauroth) - - HDFS-6646. [ HDFS Rolling Upgrade - Shell ] shutdownDatanode and getDatanodeInfo - usage is missed ( Brahma Reddy Battula via vinayakumarb) - - HDFS-6630. Unable to fetch the block information by Browsing the file system on - Namenode UI through IE9 ( Haohui Mai via vinayakumarb) - - HADOOP-8158. Interrupting hadoop fs -put from the command line - causes a LeaseExpiredException. (daryn via harsh) - - HDFS-6678. MiniDFSCluster may still be partially running after initialization - fails. (cnauroth) - - HDFS-5809. BlockPoolSliceScanner and high speed hdfs appending make - datanode to drop into infinite loop (cmccabe) - - HDFS-6456. NFS should throw error for invalid entry in - dfs.nfs.exports.allowed.hosts (Abhiraj Butala via brandonli) - - HDFS-6689. NFS doesn't return correct lookup access for direcories (brandonli) - - HDFS-6478. RemoteException can't be retried properly for non-HA scenario. - (Ming Ma via jing9) - - HDFS-6693. TestDFSAdminWithHA fails on windows ( vinayakumarb ) - - HDFS-6667. In HDFS HA mode, Distcp/SLive with webhdfs on secure cluster fails - with Client cannot authenticate via:[TOKEN, KERBEROS] error. (jing9) - - HDFS-6704. Fix the command to launch JournalNode in HDFS-HA document. - (Akira AJISAKA via jing9) - - HDFS-6731. Run "hdfs zkfc-formatZK" on a server in a non-namenode will cause - a null pointer exception. (Masatake Iwasaki via brandonli) - - HDFS-6114. Block Scan log rolling will never happen if blocks written - continuously leading to huge size of dncp_block_verification.log.curr - (vinayakumarb via cmccabe) - - HDFS-6455. NFS: Exception should be added in NFS log for invalid separator in - nfs.exports.allowed.hosts. (Abhiraj Butala via brandonli) - - HDFS-6715. Webhdfs wont fail over when it gets java.io.IOException: Namenode - is in startup mode. (jing9) - - HDFS-5919. FileJournalManager doesn't purge empty and corrupt inprogress edits - files (vinayakumarb) - - HDFS-6752. Avoid Address bind errors in TestDatanodeConfig#testMemlockLimit - (vinayakumarb) - - HDFS-6749. FSNamesystem methods should call resolvePath. - (Charles Lamb via cnauroth) - - HDFS-4629. Using com.sun.org.apache.xml.internal.serialize.* in - XmlEditsVisitor.java is JVM vendor specific. Breaks IBM JAVA. - (Amir Sanjar via stevel) - - HDFS-3482. hdfs balancer throws ArrayIndexOutOfBoundsException - if option is specified without values. ( Madhukara Phatak via umamahesh) - - HDFS-6797. DataNode logs wrong layoutversion during upgrade. (Benoy Antony - via Arpit Agarwal) - - HDFS-6810. StorageReport array is initialized with wrong size in - DatanodeDescriptor#getStorageReports. (szetszwo via Arpit Agarwal) - - HDFS-5723. Append failed FINALIZED replica should not be accepted as valid - when that block is underconstruction (vinayakumarb) - - HDFS-5185. DN fails to startup if one of the data dir is full. (vinayakumarb) - - HDFS-6451. NFS should not return NFS3ERR_IO for AccessControlException - (Abhiraj Butala via brandonli) - - HDFS-6717. JIRA HDFS-5804 breaks default nfs-gateway behavior for unsecured config - (brandonli) - - HDFS-6790. DFSUtil Should Use configuration.getPassword for SSL passwords - (Larry McCay via brandonli) - - HDFS-6791. A block could remain under replicated if all of its replicas are on - decommissioned nodes. (Ming Ma via jing9) - - HDFS-6582. Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler) - (Abhiraj Butala via brandonli) - - HDFS-6830. BlockInfo.addStorage fails when DN changes the storage for a - block replica (Arpit Agarwal) - - HDFS-6247. Avoid timeouts for replaceBlock() call by sending intermediate - responses to Balancer (vinayakumarb) - - HDFS-6783. Fix HDFS CacheReplicationMonitor rescan logic. (Yi Liu and Colin Patrick McCabe via umamahesh) - - HDFS-6825. Edit log corruption due to delayed block removal. - (Yongjun Zhang via wang) - - HDFS-6569. OOB message can't be sent to the client when DataNode shuts down for upgrade - (brandonli) - - HDFS-6868. portmap and nfs3 are documented as hadoop commands instead of hdfs - (brandonli) - - HDFS-6870. Blocks and INodes could leak for Rename with overwrite flag. (Yi - Liu via jing9) - - HDFS-6890. NFS readdirplus doesn't return dotdot attributes (brandonli) - - HDFS-6829. DFSAdmin refreshSuperUserGroupsConfiguration failed in - security cluster (zhaoyunjiong via Arpit Agarwal) - - HDFS-4852. libhdfs documentation is out of date. (cnauroth) - - HDFS-6908. Incorrect snapshot directory diff generated by snapshot deletion. - (Juan Yu and jing9 via jing9) - - HDFS-6892. Add XDR packaging method for each NFS request (brandonli) - - HDFS-6902. FileWriter should be closed in finally block in - BlockReceiver#receiveBlock() (Tsuyoshi OZAWA via Colin Patrick McCabe) - - HDFS-6972. TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration - doesn't decode url correctly. (Yongjun Zhang via wang) - - HDFS-6942. Fix typos in log messages. (Ray Chiang via wheat9) - - HDFS-6848. Lack of synchronization on access to datanodeUuid in - DataStorage#format(). (Xiaoyu Yao via Arpit Agarwal) - - HDFS-6996. SnapshotDiff report can hit IndexOutOfBoundsException when there - are nested renamed directory/file. (jing9) - - HDFS-6831. Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-6979. hdfs.dll does not produce .pdb files. (cnauroth) - - HDFS-6862. Add missing timeout annotations to tests. (Xiaoyu Yao via - Arpit Agarwal) - - HDFS-6898. DN must reserve space for a full block when an RBW block is - created. (Arpit Agarwal) - - HDFS-7025. HDFS Credential Provider related Unit Test Failure. - (Xiaoyu Yao via cnauroth) - - HDFS-7005. DFS input streams do not timeout. - - HDFS-6951. Correctly persist raw namespace xattrs to edit log and fsimage. - (clamb via wang) - - HDFS-6800. Support Datanode layout changes with rolling upgrade. - (James Thomas via Arpit Agarwal) - - HDFS-6981. Fix DN upgrade with layout version change. (Arpit Agarwal) - - HDFS-6506. Newly moved block replica been invalidated and deleted in - TestBalancer. (Binglin Chang via cnauroth) - - HDFS-6966. Add additional unit tests for encryption zones. - (Stephen Chu via wang) - - HDFS-6621. Hadoop Balancer prematurely exits iterations. - (Rafal Wojdyla and Benjamin Bowman via wang) - - HDFS-7045. Fix NameNode deadlock when opening file under /.reserved path. - (Yi Liu via wang) - - HDFS-7032. Add WebHDFS support for reading and writing to encryption zones. - (clamb via wang) - - HDFS-6965. NN continues to issue block locations for DNs with full disks. - (Rushabh Shah via kihwal) - - HDFS-6789. TestDFSClientFailover.testFileContextDoesntDnsResolveLogicalURI - and TestDFSClientFailover.testDoesntDnsResolveLogicalURI failing on jdk7. - (Akira Ajisaka via wang) - - HDFS-6912. SharedFileDescriptorFactory should not allocate sparse files - (cmccabe) - - HDFS-7075. hadoop-fuse-dfs fails because it cannot find - JavaKeyStoreProvider$Factory (cmccabe) - - HDFS-7078. Fix listEZs to work correctly with snapshots. (wang) - - HDFS-6840. Clients are always sent to the same datanode when read - is off rack. (wang) - - HDFS-7065. Pipeline close recovery race can cause block corruption (kihwal) - - HDFS-7096. Fix TestRpcProgramNfs3 to use DFS_ENCRYPTION_KEY_PROVIDER_URI - (clamb via cmccabe) - - HDFS-7046. HA NN can NPE upon transition to active. (kihwal) - - HDFS-7106. Reconfiguring DataNode volumes does not release the lock files - in removed volumes. (cnauroth via cmccabe) - - HDFS-7001. Tests in TestTracing depends on the order of execution - (iwasakims via cmccabe) - - HDFS-7132. hdfs namenode -metadataVersion command does not honor - configured name dirs. (Charles Lamb via wang) - - HDFS-7049. TestByteRangeInputStream.testPropagatedClose fails and throw - NPE on branch-2. (Juan Yu via wheat9) - - HDFS-7148. TestEncryptionZones#testIsEncryptedMethod fails on branch-2 - after archival storage merge. (wang) - - HDFS-7157. Using Time.now() for recording start/end time of reconfiguration - tasks (Lei Xu via cmccabe) - - HDFS-6664. HDFS permissions guide documentation states incorrect default - group mapping class. (Ray Chiang via aw) - - HDFS-4227. Document dfs.namenode.resource.* (Daisuke Kobayashi via aw) - - HDFS-6754. TestNamenodeCapacityReport may sometimes fail due to lack of - retry. (Mit Desai via kihwal) - - HDFS-7172. Test data files may be checked out of git with incorrect line - endings, causing test failures in TestHDFSCLI. (Chris Nauroth via wheat9) - - HDFS-7176. The namenode usage message doesn't include "-rollingupgrade - started" (cmccabe) - - HDFS-7178. Additional unit test for replica write with full disk. - (Arpit Agarwal) - - HDFS-7162. Wrong path when deleting through fuse-dfs a file which already - exists in trash (Chengbing Liu via cmccabe) - - HDFS-7179. DFSClient should instantiate a KeyProvider, not a - KeyProviderCryptoExtension. (wang) - - HDFS-7181. Remove incorrect precondition check on key length in - FileEncryptionInfo. (wang) - - HDFS-7203. Concurrent appending to the same file can cause data corruption - (kihwal) - - HDFS-7236. Fix TestOpenFilesWithSnapshot#testOpenFilesWithMultipleSnapshots. - (Yongjun Zhang via jing9) - - HDFS-6544. Broken Link for GFS in package.html. (Suraj Nayak M via wheat9) - - HDFS-7237. The command "hdfs namenode -rollingUpgrade" throws - ArrayIndexOutOfBoundsException. (szetszwo) - - HDFS-7185. The active NameNode will not accept an fsimage sent from the - standby during rolling upgrade. (jing9) - - HDFS-7208. NN doesn't schedule replication when a DN storage fails. - (Ming Ma via szetszwo) - - HDFS-5089. When a LayoutVersion support SNAPSHOT, it must support - FSIMAGE_NAME_OPTIMIZATION. (szetszwo) - - HDFS-7260. Change DFSOutputStream.MAX_PACKETS to be configurable. (szetszwo) - - HDFS-7259. Unresponseive NFS mount point due to deferred COMMIT response. - (brandonli) - - HDFS-7215.Add JvmPauseMonitor to NFS gateway (brandonli) - - HDFS-7180. NFSv3 gateway frequently gets stuck due to GC (brandonli) - - HDFS-7243. HDFS concat operation should not be allowed in Encryption Zone. - (clamb via yliu) - - HADOOP-11233. hadoop.security.kms.client.encrypted.key.cache.expiry - property spelled wrong in core-default. (Stephen Chu via yliu) - - HDFS-7218. FSNamesystem ACL operations should write to audit log on - failure. (clamb via yliu) - - HDFS-7199. DFSOutputStream should not silently drop data if DataStreamer - crashes with an unchecked exception (rushabhs via cmccabe) - - HDFS-7383. DataNode.requestShortCircuitFdsForRead may throw - NullPointerException. (szetszwo via suresh) - - HDFS-7147. Update archival storage user documentation. - (Tsz Wo Nicholas Sze via wheat9) - - HDFS-7340. Make rollingUpgrade start/finalize idempotent. (jing9) - - HDFS-7334. Fix periodic failures of TestCheckpoint - #testTooManyEditReplayFailures. (Charles Lamb via wheat9) - - HDFS-7355. TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure - fails on Windows, because we cannot deny access to the file owner. - (Chris Nauroth via wheat9) - - HDFS-7367. HDFS short-circuit read cannot negotiate shared memory slot and - file descriptors when SASL is enabled on DataTransferProtocol. - (Chris Nauroth via wheat9) - - HDFS-7364. Balancer always shows zero Bytes Already Moved. - (Tsz Wo Nicholas Sze via jing9) - - HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files - incorrectly. (Xiaoyu Yao via wheat9) - - BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS - - HDFS-6387. HDFS CLI admin tool for creating & deleting an - encryption zone. (clamb) - - HDFS-6386. HDFS Encryption Zones (clamb) - - HDFS-6388. HDFS integration with KeyProvider. (clamb) - - HDFS-6473. Protocol and API for Encryption Zones (clamb) - - HDFS-6392. Wire crypto streams for encrypted files in - DFSClient. (clamb and yliu) - - HDFS-6476. Print out the KeyProvider after finding KP successfully on - startup. (Juan Yu via wang) - - HDFS-6391. Get the Key/IV from the NameNode for encrypted files in - DFSClient. (Charles Lamb and wang) - - HDFS-6389. Rename restrictions for encryption zones. (clamb) - - HDFS-6605. Client server negotiation of cipher suite. (wang) - - HDFS-6625. Remove the Delete Encryption Zone function (clamb) - - HDFS-6516. List of Encryption Zones should be based on inodes (clamb) - - HDFS-6629. Not able to create symlinks after HDFS-6516 (umamaheswararao) - - HDFS-6635. Refactor encryption zone functionality into new - EncryptionZoneManager class. (wang) - - HDFS-6474. Namenode needs to get the actual keys and iv from the - KeyProvider. (wang) - - HDFS-6619. Clean up encryption-related tests. (wang) - - HDFS-6405. Test Crypto streams in HDFS. (yliu via wang) - - HDFS-6490. Fix the keyid format for generated keys in - FSNamesystem.createEncryptionZone (clamb) - - HDFS-6716. Update usage of KeyProviderCryptoExtension APIs on NameNode. - (wang) - - HDFS-6718. Remove EncryptionZoneManager lock. (wang) - - HDFS-6720. Remove KeyProvider in EncryptionZoneManager. (wang) - - HDFS-6738. Remove unnecessary getEncryptionZoneForPath call in - EZManager#createEncryptionZone. (clamb) - - HDFS-6724. Decrypt EDEK before creating - CryptoInputStream/CryptoOutputStream. (wang) - - HDFS-6509. Create a special /.reserved/raw directory for raw access to - encrypted data. (clamb via wang) - - HDFS-6771. Require specification of an encryption key when creating - an encryption zone. (wang) - - HDFS-6730. Create a .RAW extended attribute namespace. (clamb) - - HDFS-6692. Add more HDFS encryption tests. (wang) - - HDFS-6780. Batch the encryption zones listing API. (wang) - - HDFS-6394. HDFS encryption documentation. (wang) - - HDFS-6834. Improve the configuration guidance in DFSClient when there - are no Codec classes found in configs. (umamahesh) - - HDFS-6546. Add non-superuser capability to get the encryption zone - for a specific path. (clamb) - - HDFS-6733. Creating encryption zone results in NPE when - KeyProvider is null. (clamb) - - HDFS-6785. Should not be able to create encryption zone using path - to a non-directory file. (clamb) - - HDFS-6807. Fix TestReservedRawPaths. (clamb) - - HDFS-6814. Mistakenly dfs.namenode.list.encryption.zones.num.responses configured - as boolean. (umamahesh) - - HDFS-6817. Fix findbugs and other warnings. (yliu) - - HDFS-6839. Fix TestCLI to expect new output. (clamb) - - HDFS-6954. With crypto, no native lib systems are too verbose. (clamb via wang) - - HDFS-2975. Rename with overwrite flag true can make NameNode to stuck in safemode - on NN (crash + restart). (Yi Liu via umamahesh) - - HDFS-6905. fs-encryption merge triggered release audit failures. (clamb via tucu) - - HDFS-6714. TestBlocksScheduledCounter#testBlocksScheduledCounter should - shutdown cluster (vinayakumarb) - - HDFS-6986. DistributedFileSystem must get delegation tokens from configured - KeyProvider. (zhz via tucu) - - HDFS-6776. Using distcp to copy data between insecure and secure cluster via webdhfs - doesn't work. (yzhangal via tucu) - - HDFS-7042. Upgrade fails for Windows HA cluster due to file locks held during - rename in JournalNode. (cnauroth) - - HDFS-7051. TestDataNodeRollingUpgrade#isBlockFileInPrevious assumes Unix file - path separator. (cnauroth) - - HDFS-7105. Fix TestJournalNode#testFailToStartWithBadConfig to match log - output change. (Ray Chiang via cnauroth) - - HDFS-7105. Allow falling back to a non-SASL connection on - DataTransferProtocol in several edge cases. (cnauroth) - - HDFS-7107. Avoid Findbugs warning for synchronization on - AbstractNNFailoverProxyProvider#fallbackToSimpleAuth. (cnauroth) - - HDFS-7109. TestDataStorage does not release file locks between tests. - (cnauroth) - - HDFS-7110. Skip tests related to short-circuit read on platforms that do not - currently implement short-circuit read. (cnauroth) - - HDFS-7115. TestEncryptionZones assumes Unix path separator for KMS key store - path. (Xiaoyu Yao via cnauroth) - - HDFS-7115. TestEncryptionZonesWithHA assumes Unix path separator for KMS key - store path. (Xiaoyu Yao via cnauroth) - - HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows. - (cnauroth) - - HDFS-6534. Fix build on macosx: HDFS parts (Binglin Chang via aw) - - HDFS-7111. TestSafeMode assumes Unix line endings in safe mode tip. - (cnauroth) - - HDFS-7127. TestLeaseRecovery leaks MiniDFSCluster instances. (cnauroth) - - HDFS-7131. During HA upgrade, JournalNode should create a new committedTxnId - file in the current directory. (jing9) - - HDFS-6995. Block should be placed in the client's 'rack-local' node - if 'client-local' node is not available (vinayakumarb) - - HDFS-7128. Decommission slows way down when it gets towards the end. - (Ming Ma via cnauroth) - - HDFS-7287. The OfflineImageViewer (OIV) can output invalid XML depending on - the filename (Ravi Prakash via Colin P. McCabe) - - HDFS-7300. The getMaxNodesPerRack() method in BlockPlacementPolicyDefault - is flawed (kihwal) - - HDFS-7305. NPE seen in wbhdfs FS while running SLive. (jing9) - - BREAKDOWN OF HDFS-6584 ARCHIVAL STORAGE - - HDFS-6677. Change INodeFile and FSImage to support storage policy ID. - (szetszwo) - - HDFS-6670. Add block storage policy support with default HOT, WARM and COLD - policies. (szetszwo) - - HDFS-6671. Change BlockPlacementPolicy to consider block storage policy - in replicaiton. (szetszwo) - - HDFS-6710. Change BlockPlacementPolicy to consider block storage policy - in replica deletion. (szetszwo) - - HDFS-6679. Bump NameNodeLayoutVersion and update editsStored test files. - (vinayakumarb via szetszwo) - - HDFS-6686. Change BlockPlacementPolicy to use fallback when some storage - types are unavailable. (szetszwo) - - HDFS-6835. Add a new API to set storage policy. (jing9) - - HDFS-6847. Support storage policy on directories and include storage policy - in HdfsFileStatus. (Jing Zhao via szetszwo) - - HDFS-6801. Add a new data migration tool, Mover, for archiving data. - (szetszwo via jing9) - - HDFS-6863. Support migration for snapshot paths. (jing9) - - HDFS-6906. Add more tests for BlockStoragePolicy. (szetszwo via jing9) - - HDFS-6911. check if a block is already scheduled in Mover. - (szetszwo via jing9) - - HDFS-6920. Check the storage type of delNodeHintStorage when deleting - a replica. (szetszwo via jing9) - - HDFS-6944. Add retry and termination logic for Mover. (jing9) - - HDFS-6969. INode#getStoragePolicyID should always return the latest - storage policy. (jing9) - - HDFS-6961. BlockPlacementPolicy#chooseTarget should check each valid - storage type in each choosing round. (jing9) - - HDFS-6876. support set/get storage policy in DFSAdmin. (jing9) - - HDFS-6997. Add more tests for data migration and replicaion. (szetszwo) - - HDFS-6875. Support migration for a list of specified paths. (jing9) - - HDFS-7027. Mover does not terminate when some storage type is out of space. - (szetszwo via jing9) - - HDFS-7029. Fix TestDFSInotifyEventInputStream and TestDistributedFileSystem. - (szetszwo via jing9) - - HDFS-7028. FSDirectory should not get storage policy id from symlinks. - (szetszwo) - - HDFS-7034. Fix TestBlockPlacement and TestStorageMover. (jing9) - - HDFS-7039. Fix Balancer tests. (szetszwo via jing9) - - HDFS-7062. Skip under construction block for migration. (jing9) - - HDFS-7052. Add Mover into hdfs script. (jing9) - - HDFS-7072. Fix TestBlockManager and TestStorageMover. (jing9 via szetszwo) - - HDFS-6864. Archival Storage: add user documentation. (szetszwo via jing9) - - HDFS-7088. Archival Storage: fix TestBalancer and - TestBalancerWithMultipleNameNodes. (szetszwo via jing9) - - HDFS-7095. TestStorageMover often fails in Jenkins. (jing9) - - HDFS-7081. Add new DistributedFileSystem API for getting all the existing - storage policies. (jing9) - - HDFS-7140. Add a tool to list all the existing block storage policies. - (jing9) - - HDFS-7167. NPE while running Mover if the given path is for a file. (jing9) - - HDFS-7154. Fix returning value of starting reconfiguration task (Lei Xu via - Colin P. McCabe) - - BREAKDOWN OF HDFS-6581 SUBTASKS AND RELATED JIRAS - - HDFS-6921. Add LazyPersist flag to FileStatus. (Arpit Agarwal) - - HDFS-6924. Add new RAM_DISK storage type. (Arpit Agarwal) - - HDFS-6922. Add LazyPersist flag to INodeFile, save it in FsImage and - edit logs. (Arpit Agarwal) - - HDFS-6923. Propagate LazyPersist flag to DNs via DataTransferProtocol. - (Arpit Agarwal) - - HDFS-6925. DataNode should attempt to place replicas on transient storage - first if lazyPersist flag is received. (Arpit Agarwal) - - HDFS-6926. DN support for saving replicas to persistent storage and - evicting in-memory replicas. (Arpit Agarwal) - - HDFS-6927. Initial unit tests for lazy persist files. (Arpit Agarwal) - - HDFS-6929. NN periodically unlinks lazy persist files with missing - replicas from namespace. (Arpit Agarwal) - - HDFS-6928. 'hdfs put' command should accept lazyPersist flag for testing. - (Arpit Agarwal) - - HDFS-6960. Bugfix in LazyWriter, fix test case and some refactoring. - (Arpit Agarwal) - - HDFS-6931. Move lazily persisted replicas to finalized directory on DN - startup. (Arpit Agarwal) - - HDFS-6950. Add Additional unit tests for HDFS-6581. (Xiaoyu Yao via - Arpit Agarwal) - - HDFS-6930. Improve replica eviction from RAM disk. (Arpit Agarwal) - - HDFS-6977. Delete all copies when a block is deleted from the block space. - (Arpit Agarwal) - - HDFS-6991. Notify NN of evicted block before deleting it from RAM disk. - (Arpit Agarwal) - - HDFS-6978. Directory scanner should correctly reconcile blocks on RAM - disk. (Arpit Agarwal) - - HDFS-7066. LazyWriter#evictBlocks misses a null check for replicaState. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7064. Fix unit test failures in HDFS-6581 branch. (Xiaoyu Yao via - Arpit Agarwal) - - HDFS-6581. Few more unit test fixes for HDFS-6581. (Arpit Agarwal) - - HDFS-7080. Fix finalize and upgrade unit test failures. (Arpit Agarwal) - - HDFS-7084. FsDatasetImpl#copyBlockFiles debug log can be improved. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7091. Add forwarding constructor for INodeFile for existing callers. - (Arpit Agarwal) - - HDFS-7100. Make eviction scheme pluggable. (Arpit Agarwal) - - HDFS-7108. Fix unit test failures in SimulatedFsDataset. (Arpit Agarwal) - - HDFS-7071. Updated editsStored and editsStored.xml to bump layout - version and add LazyPersist flag. (Xiaoyu Yao and Arpit Agarwal via - Arpit Agarwal) - - HDFS-6990. Add unit test for evict/delete RAM_DISK block with open - handle. (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7143. Fix findbugs warnings in HDFS-6581 branch. (szetszwo via - Arpit Agarwal) - - HDFS-6932. Balancer and Mover tools should ignore replicas on RAM_DISK. - (Xiaoyu Yao via Arpit Agarwal) - - HDFS-7144. Fix findbugs warnings in RamDiskReplicaTracker. (szetszwo via - Arpit Agarwal) - - HDFS-7155. Bugfix in createLocatedFileStatus caused by bad merge. - (Arpit Agarwal) - - HDFS-7153. Add storagePolicy to NN edit log during file creation. - (Arpit Agarwal) - - HDFS-7159. Use block storage policy to set lazy persist preference. - (Arpit Agarwal) - - HDFS-7129. Metrics to track usage of memory for writes. (Xiaoyu Yao - via Arpit Agarwal) - - HDFS-7171. Fix Jenkins failures in HDFS-6581 branch. (Arpit Agarwal) - - HDFS-7112. LazyWriter should use either async IO or one thread per physical - disk. (Xiaoyu Yao via cnauroth) - - HDFS-7090. Use unbuffered writes when persisting in-memory replicas. - (Xiaoyu Yao via cnauroth) - - HDFS-6934. Move checksum computation off the hot path when writing to RAM - disk. (cnauroth) - - HDFS-7291. Persist in-memory replicas with appropriate unbuffered copy API - on POSIX and Windows. (Xiaoyu Yao via cnauroth) - - HDFS-7328. TestTraceAdmin assumes Unix line endings. (cnauroth) - - HDFS-7359. NameNode in secured HA cluster fails to start if - dfs.namenode.secondary.http-address cannot be interpreted as a network - address. (cnauroth) - - HDFS-7226. Fix TestDNFencing.testQueueingWithAppend. (Yongjun Zhang via jing9) - - HDFS-7382. DataNode in secure mode may throw NullPointerException if client - connects before DataNode registers itself with NameNode. (cnauroth) - - HDFS-7387. NFS may only do partial commit due to a race between COMMIT and write - (brandonli) - - HDFS-7391. Renable SSLv2Hello in HttpFS. (rkanter via acmurthy) - - HDFS-7385. ThreadLocal used in FSEditLog class causes FSImage permission mess - up. (jiangyu via cnauroth) - -Release 2.5.2 - 2014-11-19 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-7274. Disable SSLv3 in HttpFS. (Robert Kanter via kasha) - - -Release 2.5.1 - 2014-09-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HADOOP-10957. The globber will sometimes erroneously return a permission - denied exception when there is a non-terminal wildcard (cmccabe) - -Release 2.5.0 - 2014-08-11 - - INCOMPATIBLE CHANGES - - HDFS-6168. Remove a deprecated constructor and the deprecated methods reportChecksumFailure, - getDelegationToken(Text), renewDelegationToken and cancelDelegationToken from - DistributedFileSystem. (szetszwo) - - NEW FEATURES - - HDFS-6281. Provide option to use the NFS Gateway without having to use the - Hadoop portmapper. (atm) - - HDFS-5168. Add cross node dependency support to BlockPlacementPolicy. - (Nikola Vujic via szetszwo) - - HDFS-6334. Client failover proxy provider for IP failover based NN HA. - (kihwal) - - HDFS-6406. Add capability for NFS gateway to reject connections from - unprivileged ports. (atm) - - HDFS-2006. Ability to support storing extended attributes per file. - - HDFS-5978. Create a tool to take fsimage and expose read-only WebHDFS API. - (Akira Ajisaka via wheat9) - - HDFS-6278. Create HTML5-based UI for SNN. (wheat9) - - HDFS-6279. Create new index page for JN / DN. (wheat9) - - IMPROVEMENTS - - HDFS-6007. Update documentation about short-circuit local reads (iwasakims - via cmccabe) - - HDFS-6125. Cleanup unnecessary cast in HDFS code base. (suresh) - - HDFS-5196. Provide more snapshot information in WebUI. - (Shinichi Yamashita via wheat9) - - HDFS-6155. Fix Boxing/unboxing to parse a primitive findbugs warnings. - (suresh) - - HDFS-6119. FSNamesystem code cleanup. (suresh) - - HDFS-6158. Clean up dead code for OfflineImageViewer. (wheat9) - - HDFS-6164. Remove lsr in OfflineImageViewer. (wheat9) - - HDFS-6167. Relocate the non-public API classes in the hdfs.client package. - (szetszwo) - - HDFS-6191. Disable quota checks when replaying edit log. (kihwal) - - HDFS-6170. Support GETFILESTATUS operation in WebImageViewer. - (Akira Ajisaka via wheat9) - - HDFS-6225. Remove the o.a.h.hdfs.server.common.UpgradeStatusReport. - (wheat9) - - HDFS-6224. Add a unit test to TestAuditLogger for file permissions - passed to logAuditEvent. (Charles Lamb via wang) - - HDFS-6194. Create new tests for ByteRangeInputStream. - (Akira Ajisaka via wheat9) - - HDFS-6219. Proxy superuser configuration should use true client IP for - address checks. (daryn via kihwal) - - HDFS-6256. Clean up ImageVisitor and SpotCheckImageVisitor. - (Akira Ajisaka via wheat9) - - HDFS-6265. Prepare HDFS codebase for JUnit 4.11. (cnauroth) - - HDFS-5693. Few NN metrics data points were collected via JMX when NN - is under heavy load. (Ming Ma via jing9) - - HDFS-6273. Config options to allow wildcard endpoints for namenode HTTP - and HTTPS servers. (Arpit Agarwal) - - HDFS-6282. Re-add testIncludeByRegistrationName. (cmccabe) - - HDFS-6266. Identify full path for a given INode. (jing9) - - HDFS-6210. Support GETACLSTATUS operation in WebImageViewer. - (Akira Ajisaka via wheat9) - - HDFS-6269. NameNode Audit Log should differentiate between webHDFS open and - HDFS open. (Eric Payne via jeagles) - - HDFS-6304. Consolidate the logic of path resolution in FSDirectory. - (wheat9) - - HDFS-6295. Add "decommissioning" state and node state filtering to - dfsadmin. (wang) - - HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is - renamed. (cmccabe) - - HDFS-6328. Clean up dead code in FSDirectory. (wheat9) - - HDFS-6230. Expose upgrade status through NameNode web UI. - (Mit Desai via wheat9) - - HDFS-6186. Pause deletion of blocks when the namenode starts up. (jing9) - - HDFS-6293. Issues with OIV processing PB-based fsimages. (kihwal) - - HDFS-2949. Add check to active state transition to prevent operator-induced - split brain. (Rushabh S Shah via kihwal) - - HDFS-6287. Add vecsum test of libhdfs read access times (cmccabe) - - HDFS-5683. Better audit log messages for caching operations. - (Abhiraj Butala via wang) - - HDFS-6345. DFS.listCacheDirectives() should allow filtering based on - cache directive ID. (wang) - - HDFS-6432. Add snapshot related APIs to webhdfs. (jing9) - - HDFS-6396. Remove support for ACL feature from INodeSymlink. - (Charles Lamb via wang) - - HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS - gateway. (atm via wang) - - HDFS-6416. Use Time#monotonicNow in OpenFileCtx and OpenFileCtxCatch to - avoid system clock bugs (Abhiraj Butala via brandonli) - - HDFS-6356. Fix typo in DatanodeLayoutVersion. (Tulasi G via wang) - - HDFS-6447. balancer should timestamp the completion message. - (Juan Yu via wang) - - HDFS-6463. Clarify behavior of AclStorage#createFsPermissionForExtendedAcl - in comments. (cnauroth) - - HDFS-6472. fix typo in webapps/hdfs/explorer.js. (Juan Yu via wang) - - HDFS-6056. Clean up NFS config settings (brandonli) - - HDFS-6110 adding more slow action log in critical write path - (Liang Xie via stack) - - HDFS-6109 let sync_file_range() system call run in background - (Liang Xie via stack) - - HDFS-6268. Better sorting in NetworkTopology#pseudoSortByDistance when - no local node is found. (wang) - - HDFS-6369. Document that BlockReader#available() can return more bytes than - are remaining in the block (Ted Yu via Colin Patrick McCabe) - - HDFS-6487. TestStandbyCheckpoint#testSBNCheckpoints is racy. - (Mit Desai via wang) - - HDFS-6297. Add CLI testcases to reflect new features of dfs and dfsadmin - (Dasha Boudnik via cos) - - HDFS-6399. Add note about setfacl in HDFS permissions guide. - (cnauroth via wang) - - HDFS-6315. Decouple recording edit logs from FSDirectory. (wheat9) - - HDFS-6379. HTTPFS - Implement ACLs support. (yoderme via tucu) - - HDFS-6471. Make moveFromLocal CLI testcases to be non-disruptive - (Dasha Boudnik via cos) - - HDFS-6395. Skip checking xattr limits for non-user-visible namespaces. - (Yi Liu via wang). - - HDFS-3493. Invalidate excess corrupted blocks as long as minimum - replication is satisfied. (Juan Yu and Vinayakumar B via wang) - - HDFS-6330. Move mkdirs() to FSNamesystem. (wheat9) - - HDFS-6470. TestBPOfferService.testBPInitErrorHandling is flaky. - (Ming Ma via wang) - - HDFS-6529. Trace logging for RemoteBlockReader2 to identify remote datanode - and file being read. (Anubhav Dhoot via atm) - - HDFS-6499. Use NativeIO#renameTo instead of File#renameTo in - FileJournalManager. (Yongjun Zhang via atm) - - HDFS-6518. TestCacheDirectives#testExceedsCapacity should - take FSN read lock when accessing pendingCached list. - (wang) - - HDFS-6528. Add XAttrs to TestOfflineImageViewer. (Stephen Chu via wang) - - HDFS-6545. Finalizing rolling upgrade can make NN unavailable for a long - duration. (kihwal) - - HDFS-6530. Fix Balancer documentation. (szetszwo) - - HDFS-6480. Move waitForReady() from FSDirectory to FSNamesystem. (wheat9) - - HDFS-6403. Add metrics for log warnings reported by JVM pauses. (Yongjun - Zhang via atm) - - HDFS-6557. Move the reference of fsimage to FSNamesystem. (wheat9) - - HDFS-4667. Capture renamed files/directories in snapshot diff report. (jing9 - and Binglin Chang via jing9) - - HDFS-6507. Improve DFSAdmin to support HA cluster better. - (Zesheng Wu via vinayakumarb) - - HDFS-6578. add toString method to DatanodeStorage for easier debugging. - (Yongjun Zhang via Arpit Agarwal) - - HDFS-6562. Refactor rename() in FSDirectory. (wheat9) - - HDFS-6486. Add user doc for XAttrs via WebHDFS. (Yi Liu via umamahesh) - - HDFS-6430. HTTPFS - Implement XAttr support. (Yi Liu via tucu) - - HDFS-6593. Move SnapshotDiffInfo out of INodeDirectorySnapshottable. - (Jing Zhao via wheat9) - - HDFS-6595. Allow the maximum threads for balancing on datanodes to be - configurable. (Benoy Antony via szetszwo) - - HDFS-6572. Add an option to the NameNode that prints the software and - on-disk image versions. (Charles Lamb via cnauroth) - - HDFS-6603. Add XAttr with ACL test. (Stephen Chu via cnauroth) - - HDFS-6612. MiniDFSNNTopology#simpleFederatedTopology(int) - always hardcode nameservice ID. (Juan Yu via wang) - - HDFS-6614. shorten TestPread run time with a smaller retry timeout setting. - (Liang Xie via cnauroth) - - HDFS-6610. TestShortCircuitLocalRead tests sometimes timeout on slow - machines. (Charles Lamb via wang) - - HDFS-6620. Snapshot docs should specify about preserve options with cp command - (Stephen Chu via umamahesh) - - HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second - instead of millisecond. (Juan Yu via wang) - - HDFS-6680. BlockPlacementPolicyDefault does not choose favored nodes - correctly. (szetszwo) - - HDFS-6712. Document HDFS Multihoming Settings. (Arpit Agarwal) - - OPTIMIZATIONS - - HDFS-6214. Webhdfs has poor throughput for files >2GB (daryn) - - HDFS-6460. Ignore stale and decommissioned nodes in - NetworkTopology#sortByDistance. (Yongjun Zhang via wang) - - HDFS-6583. Remove clientNode in FileUnderConstructionFeature. (wheat9) - - HDFS-6599. 2.4 addBlock is 10 to 20 times slower compared to 0.23 (daryn) - - BUG FIXES - - HDFS-6112. NFS Gateway docs are incorrect for allowed hosts configuration. - (atm) - - HDFS-3087. Decomissioning on NN restart can complete without blocks being - replicated. (Rushabh S Shah via kihwal) - - HDFS-6162. Format strings should use platform independent line separator. - (suresh) - - HDFS-6156. Simplify the JMX API that provides snapshot information. - (wheat9) - - HDFS-6173. Move the default processor from Ls to Web in OfflineImageViewer. - (Akira Ajisaka via wheat9) - - HDFS-5591. Checkpointing should use monotonic time when calculating period. - (Charles Lamb via wang) - - HDFS-6190. Minor textual fixes in DFSClient. (Charles Lamb via wheat9) - - HDFS-6159. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails if - there is block missing after balancer success (Chen He via kihwal) - - HDFS-6181. Fix the wrong property names in NFS user guide (brandonli) - - HDFS-6180. dead node count / listing is very broken in JMX and old GUI. - (wheat9) - - HDFS-6143. WebHdfsFileSystem open should throw FileNotFoundException for - non-existing paths. (Gera Shegalov via wheat9) - - HDFS-6169. Move the address in WebImageViewer. (Akira Ajisaka via wheat9) - - HDFS-6160. TestSafeMode occasionally fails. (Arpit Agarwal) - - HDFS-5669. Storage#tryLock() should check for null before logging successfull message - (Vinayakumar B via umamahesh) - - HDFS-6237. TestDFSShell#testGet fails on Windows due to invalid file system - path. (cnauroth) - - HDFS-6238. TestDirectoryScanner leaks file descriptors. (cnauroth) - - HDFS-6243. HA NameNode transition to active or shutdown may leave lingering - image transfer thread. (cnauroth) - - HDFS-5409. TestOfflineEditsViewer#testStored fails on Windows due to CRLF - line endings in editsStored.xml from git checkout. (cnauroth) - - HDFS-4909. Avoid protocol buffer RPC namespace clashes. (cmccabe) - - HDFS-6178. Decommission on standby NN couldn't finish. (Ming Ma via jing9) - - HDFS-6213. TestDataNodeConfig failing on Jenkins runs due to DN web port - in use. (wang) - - HDFS-6274. Cleanup javadoc warnings in HDFS code. (suresh) - - HDFS-6275. Fix warnings - type arguments can be inferred and redudant - local variable. (suresh) - - HDFS-6217. Webhdfs PUT operations may not work via a http proxy. - (Daryn Sharp via kihwal) - - HDFS-6276. Remove unnecessary conditions and null check. (suresh) - - HDFS-5865. Update OfflineImageViewer document. (Akira Ajisaka via wheat9) - - HDFS-6270. Secondary namenode status page shows transaction count in bytes. - (Benoy Antony via wheat9) - - HDFS-6218. Audit log should use true client IP for proxied webhdfs - operations. (daryn via kihwal) - - HDFS-6288. DFSInputStream Pread doesn't update ReadStatistics. - (Juan Yu via wang) - - HDFS-6289. HA failover can fail if there are pending DN messages for DNs - which no longer exist. (atm) - - HDFS-6337. Setfacl testcase is failing due to dash character in username - in TestAclCLI (umamahesh) - - HDFS-5381. ExtendedBlock#hashCode should use both blockId and block pool ID - (Benoy Antony via Colin Patrick McCabe) - - HDFS-6240. WebImageViewer returns 404 if LISTSTATUS to an empty directory. - (Akira Ajisaka via wheat9) - - HDFS-6351. Command hdfs dfs -rm -r can't remove empty directory. - (Yongjun Zhang via wang) - - HDFS-5522. Datanode disk error check may be incorrectly skipped. - (Rushabh S Shah via kihwal) - - HDFS-6367. EnumSetParam$Domain#parse fails for parameter containing more than one enum. - (Yi Liu via umamahesh) - - HDFS-6305. WebHdfs response decoding may throw RuntimeExceptions (Daryn - Sharp via jeagles) - - HDFS-6355. Fix divide-by-zero, improper use of wall-clock time in - BlockPoolSliceScanner (cmccabe) - - HDFS-6370. Web UI fails to display in intranet under IE. - (Haohui Mai via cnauroth) - - HDFS-6381. Fix a typo in INodeReference.java. (Binglin Chang via jing9) - - HDFS-6400. Cannot execute hdfs oiv_legacy. (Akira AJISAKA via kihwal) - - HDFS-6250. Fix test failed in TestBalancerWithNodeGroup.testBalancerWithRackLocality - (Binglin Chang and Chen He via junping_du) - - HDFS-4913. Deleting file through fuse-dfs when using trash fails requiring - root permissions (cmccabe) - - HDFS-6421. Fix vecsum.c compile on BSD and some other systems. (Mit Desai - via Colin Patrick McCabe) - - HDFS-6419. TestBookKeeperHACheckpoints#TestSBNCheckpoints fails on trunk. - (Akira AJISAKA via kihwal) - - HDFS-6409. Fix typo in log message about NameNode layout version upgrade. - (Chen He via cnauroth) - - HDFS-6433. Replace BytesMoved class with AtomicLong. - (Benoy Antony via cnauroth) - - HDFS-6438. DeleteSnapshot should be a DELETE request in WebHdfs. (jing9) - - HDFS-6423. Diskspace quota usage should be updated when appending data to - partial block. (jing9) - - HDFS-6443. Fix MiniQJMHACluster related test failures. (Zesheng Wu via - Arpit Agarwal) - - HDFS-6227. ShortCircuitCache#unref should purge ShortCircuitReplicas whose - streams have been closed by java interrupts. (Colin Patrick McCabe via jing9) - - HDFS-6442. Fix TestEditLogAutoroll and TestStandbyCheckpoints failure - caused by port conficts. (Zesheng Wu via Arpit Agarwal) - - HDFS-6448. BlockReaderLocalLegacy should set socket timeout based on - conf.socketTimeout (liangxie via cmccabe) - - HDFS-6453. Use Time#monotonicNow to avoid system clock reset. - (Liang Xie via wang) - - HDFS-6461. Use Time#monotonicNow to compute duration in DataNode#shutDown. - (James Thomas via wang) - - HDFS-6462. NFS: fsstat request fails with the secure hdfs (brandonli) - - HDFS-6404. HttpFS should use a 000 umask for mkdir and create - operations. (yoderme via tucu) - - HDFS-6424. blockReport doesn't need to invalidate blocks on SBN. (Ming Ma - via jing9) - - HDFS-6497. Make TestAvailableSpaceVolumeChoosingPolicy deterministic - (cmccabe) - - HDFS-6500. Snapshot shouldn't be removed silently after renaming to an - existing snapshot. (Nicholas SZE via junping_du) - - HDFS-6257. TestCacheDirectives#testExceedsCapacity fails occasionally - (cmccabe) - - HDFS-6364. Incorrect check for unknown datanode in Balancer. (Benoy - Antony via Arpit Agarwal) - - HDFS-6503. Fix typo of DFSAdmin restoreFailedStorage. - (Zesheng Wu via wheat9) - - HDFS-6464. Support multiple xattr.name parameters for WebHDFS getXAttrs. - (Yi Liu via umamahesh) - - HDFS-6539. test_native_mini_dfs is skipped in hadoop-hdfs/pom.xml - (decstery via cmccabe) - - HDFS-6527. Edit log corruption due to defered INode removal. (kihwal and - jing9 via jing9) - - HDFS-6552. add DN storage to a BlockInfo will not replace the different - storage from same DN. (Amir Langer via Arpit Agarwal) - - HDFS-6551. Rename with OVERWRITE option may throw NPE when the target - file/directory is a reference INode. (jing9) - - HDFS-6439. NFS should not reject NFS requests to the NULL procedure whether - port monitoring is enabled or not. (brandonli) - - HDFS-6559. Fix wrong option "dfsadmin -rollingUpgrade start" in the - document. (Akira Ajisaka via Arpit Agarwal) - - HDFS-6553. Add missing DeprecationDeltas for NFS Kerberos configurations - (Stephen Chu via brandonli) - - HDFS-6563. NameNode cannot save fsimage in certain circumstances when - snapshots are in use. (atm) - - HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class - (Hooman Peiro Sajjad and Chen He via kihwal) - - HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS - client. (atm) - - HDFS-6535. HDFS quota update is wrong when file is appended. (George Wong - via jing9) - - HDFS-6222. Remove background token renewer from webhdfs. - (Rushabh Shah and Daryn Sharp via cnauroth) - - HDFS-6580. FSNamesystem.mkdirsInt should call the getAuditFileInfo() - wrapper. (Zhilei Xu via wheat9) - - HDFS-6587. Bug in TestBPOfferService can cause test failure. (Zhilei Xu - via Arpit Agarwal) - - HDFS-6598. Fix a typo in message issued from explorer.js. (Yongjun Zhang - via wheat9) - - HDFS-6475. WebHdfs clients fail without retry because incorrect handling - of StandbyException. (Yongjun Zhang via atm) - - HADOOP-10701. NFS should not validate the access premission only based on - the user's primary group (Harsh J via atm) - - HDFS-6556. Refine XAttr permissions (umamahesh) - - HDFS-6601. Issues in finalizing rolling upgrade when there is a layout - version change (kihwal) - - HDFS-6418. Regression: DFS_NAMENODE_USER_NAME_KEY missing - (szetszwo via stevel) - - HDFS-6558. Missing newline in the description of dfsadmin -rollingUpgrade. - (Chen He via kihwal) - - HDFS-6591. while loop is executed tens of thousands of times in Hedged Read - (Liang Xie via cnauroth) - - HDFS-6604. The short-circuit cache doesn't correctly time out replicas that - haven't been used in a while (cmccabe) - - HDFS-4286. Changes from BOOKKEEPER-203 broken capability of including - bookkeeper-server jar in hidden package of BKJM (Rakesh R via umamahesh) - - HDFS-4221. Remove the format limitation point from BKJM documentation as HDFS-3810 - closed. (Rakesh R via umamahesh) - - HDFS-5411. Update Bookkeeper dependency to 4.2.3. (Rakesh R via umamahesh) - - HDFS-6631. TestPread#testHedgedReadLoopTooManyTimes fails intermittently. - (Liang Xie via cnauroth) - - HDFS-6647. Edit log corruption when pipeline recovery occurs for deleted - file present in snapshot (kihwal) - - HDFS-6378. NFS registration should timeout instead of hanging when - portmap/rpcbind is not available (Abhiraj Butala via brandonli) - - HDFS-6703. NFS: Files can be deleted from a read-only mount - (Srikanth Upputuri via brandonli) - - HDFS-6422. getfattr in CLI doesn't throw exception or return non-0 return code - when xattr doesn't exist. (Charles Lamb via umamahesh) - - HDFS-6696. Name node cannot start if the path of a file under - construction contains ".snapshot". (wang) - - HDFS-6312. WebHdfs HA failover is broken on secure clusters. - (daryn via tucu) - - HDFS-6618. FSNamesystem#delete drops the FSN lock between removing INodes - from the tree and deleting them from the inode map (kihwal via cmccabe) - - HDFS-6622. Rename and AddBlock may race and produce invalid edits (kihwal - via cmccabe) - - HDFS-6723. New NN webUI no longer displays decommissioned state for dead node. - (Ming Ma via wheat9) - - HDFS-6768. Fix a few unit tests that use hard-coded port numbers. (Arpit - Agarwal) - - BREAKDOWN OF HDFS-2006 SUBTASKS AND RELATED JIRAS - - HDFS-6299. Protobuf for XAttr and client-side implementation. (Yi Liu via umamahesh) - - HDFS-6302. Implement XAttr as a INode feature. (Yi Liu via umamahesh) - - HDFS-6309. Javadocs for Xattrs apis in DFSClient and other minor fixups. (Charles Lamb via umamahesh) - - HDFS-6258. Namenode server-side storage for XAttrs. (Yi Liu via umamahesh) - - HDFS-6303. HDFS implementation of FileContext API for XAttrs. (Yi Liu and Charles Lamb via umamahesh) - - HDFS-6324. Shift XAttr helper code out for reuse. (Yi Liu via umamahesh) - - HDFS-6301. NameNode: persist XAttrs in fsimage and record XAttrs modifications to edit log. - (Yi Liu via umamahesh) - - HDFS-6298. XML based End-to-End test for getfattr and setfattr commands. (Yi Liu via umamahesh) - - HDFS-6314. Test cases for XAttrs. (Yi Liu via umamahesh) - - HDFS-6344. Maximum limit on the size of an xattr. (Yi Liu via umamahesh) - - HDFS-6377. Unify xattr name and value limits into a single limit. (wang) - - HDFS-6373. Remove support for extended attributes on symlinks. (Charles Lamb via wang) - - HDFS-6283. Write end user documentation for xattrs. (wang) - - HDFS-6412. Interface audience and stability annotations missing from - several new classes related to xattrs. (wang) - - HDFS-6259. Support extended attributes via WebHDFS. (yliu) - - HDFS-6346. Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call - (Yi Liu via umamahesh) - - HDFS-6331. ClientProtocol#setXattr should not be annotated idempotent. - (umamahesh via wang) - - HDFS-6335. TestOfflineEditsViewer for XAttr. (Yi Liu via umamahesh) - - HDFS-6343. fix TestNamenodeRetryCache and TestRetryCacheWithHA failures. (umamahesh) - - HDFS-6366. FsImage loading failed with RemoveXattr op (umamahesh) - - HDFS-6357. SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA - (umamahesh) - - HDFS-6372. Handle setXattr rpcIDs for OfflineEditsViewer. (umamahesh) - - HDFS-6410. DFSClient unwraps AclException in xattr methods, but those - methods cannot throw AclException. (wang) - - HDFS-6413. xattr names erroneously handled as case-insensitive. - (Charles Lamb via cnauroth) - - HDFS-6414. xattr modification operations are based on state of latest - snapshot instead of current version of inode. (Andrew Wang via cnauroth) - - HDFS-6374. setXAttr should require the user to be the owner of the file - or directory (Charles Lamb via wang) - - HDFS-6375. Listing extended attributes with the search permission. - (Charles Lamb via wang) - - HDFS-6492. Support create-time xattrs and atomically setting multiple - xattrs. (wang) - -Release 2.4.1 - 2014-06-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-6215. Wrong error message for upgrade. (Kihwal Lee via jeagles) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-6189. Multiple HDFS tests fail on Windows attempting to use a test - root path containing a colon. (cnauroth via szetszwo) - - HDFS-6197. Rolling upgrade rollback on Windows can fail attempting to rename - edit log segment files to a destination that already exists. (cnauroth) - - HDFS-6198. DataNode rolling upgrade does not correctly identify current - block pool directory and replace with trash on Windows. (cnauroth) - - HDFS-6206. Fix NullPointerException in DFSUtil.substituteForWildcardAddress. - (szetszwo) - - HDFS-6204. Fix TestRBWBlockInvalidation: change the last sleep to a loop. - (szetszwo) - - HDFS-6209. TestValidateConfigurationSettings should use random ports. - (Arpit Agarwal via szetszwo) - - HDFS-6208. DataNode caching can leak file descriptors. (cnauroth) - - HDFS-6231. DFSClient hangs infinitely if using hedged reads and all eligible - datanodes die. (cnauroth) - - HDFS-6234. TestDatanodeConfig#testMemlockLimit fails on Windows due to - invalid file path. (cnauroth) - - HDFS-6235. TestFileJournalManager can fail on Windows due to file locking if - tests run out of order. (cnauroth) - - HDFS-6229. Race condition in failover can cause RetryCache fail to work. - (jing9) - - HDFS-6232. OfflineEditsViewer throws a NPE on edits containing ACL - modifications (ajisakaa via cmccabe) - - HDFS-6236. ImageServlet should use Time#monotonicNow to measure latency. - (cnauroth) - - HDFS-6245. datanode fails to start with a bad disk even when failed - volumes is set. (Arpit Agarwal) - - HDFS-2882. DN continues to start up, even if block pool fails to initialize - (vinayakumarb) - - HDFS-6340. DN can't finalize upgrade. (Rahul Singhal via Arpit Agarwal) - - HDFS-6329. WebHdfs does not work if HA is enabled on NN but logical URI is - not configured. (kihwal) - - HDFS-6313. WebHdfs may use the wrong NN when configured for multiple HA NNs - (kihwal) - - HDFS-6326. WebHdfs ACL compatibility is broken. (cnauroth) - - HDFS-6361. TestIdUserGroup.testUserUpdateSetting failed due to out of range - nfsnobody Id. (Yongjun Zhang via brandonli) - - HDFS-6362. InvalidateBlocks is inconsistent in usage of DatanodeUuid and - StorageID. (Arpit Agarwal) - - HDFS-6402. Suppress findbugs warning for failure to override equals and - hashCode in FsAclPermission. (cnauroth) - - HDFS-6325. Append should fail if the last block has insufficient number of - replicas (Keith Pak via cos) - - HDFS-6397. NN shows inconsistent value in deadnode count. - (Mohammad Kamrul Islam via kihwal) - - HDFS-6411. nfs-hdfs-gateway mount raises I/O error and hangs when a - unauthorized user attempts to access it (brandonli) - -Release 2.4.0 - 2014-04-07 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-5698. Use protobuf to serialize / deserialize FSImage. (See breakdown - of tasks below for features and contributors) - - HDFS-5776 Support 'hedged' reads in DFSClient (Liang Xie via stack) - - HDFS-4685. Implementation of ACLs in HDFS. (See breakdown of tasks below for - features and contributors) - - IMPROVEMENTS - - HDFS-5781. Use an array to record the mapping between FSEditLogOpCode and - the corresponding byte value. (jing9) - - HDFS-5153. Datanode should send block reports for each storage in a - separate message. (Arpit Agarwal) - - HDFS-5804. HDFS NFS Gateway fails to mount and proxy when using Kerberos. - (Abin Shahab via jing9) - - HDFS-5859. DataNode#checkBlockToken should check block tokens even if - security is not enabled. (cmccabe) - - HDFS-5746. Add ShortCircuitSharedMemorySegment (cmccabe) - - HDFS-4911. Reduce PeerCache timeout to be commensurate with - dfs.datanode.socket.reuse.keepalive (cmccabe) - - HDFS-4370. Fix typo Blanacer in DataNode. (Chu Tong via shv) - - HDFS-5929. Add blockpool % usage to HDFS federated nn page. - (Siqi Li via suresh) - - HDFS-5810. Unify mmap cache and short-circuit file descriptor cache - (cmccabe) - - HDFS-5940. Minor cleanups to ShortCircuitReplica, FsDatasetCache, and - DomainSocketWatcher (cmccabe) - - HDFS-5531. Combine the getNsQuota() and getDsQuota() methods in INode. - (szetszwo) - - HDFS-5285. Flatten INodeFile hierarchy: Replace INodeFileUnderConstruction - and INodeFileUnderConstructionWithSnapshot with FileUnderContructionFeature. - (jing9 via szetszwo) - - HDFS-5286. Flatten INodeDirectory hierarchy: Replace INodeDirectoryWithQuota - with DirectoryWithQuotaFeature. (szetszwo) - - HDFS-5537. Remove FileWithSnapshot interface. (jing9 via szetszwo) - - HDFS-5554. Flatten INodeFile hierarchy: Replace INodeFileWithSnapshot with - FileWithSnapshotFeature. (jing9 via szetszwo) - - HDFS-5647. Merge INodeDirectory.Feature and INodeFile.Feature. (Haohui Mai - via jing9) - - HDFS-5632. Flatten INodeDirectory hierarchy: Replace - INodeDirectoryWithSnapshot with DirectoryWithSnapshotFeature. - (jing9 via szetszwo) - - HDFS-5715. Use Snapshot ID to indicate the corresponding Snapshot for a - FileDiff/DirectoryDiff. (jing9) - - HDFS-5726. Fix compilation error in AbstractINodeDiff for JDK7. (jing9) - - HDFS-5973. add DomainSocket#shutdown method (cmccabe) - - HDFS-5318. Support read-only and read-write paths to shared replicas. - (Eric Sirianni via Arpit Agarwal) - - HDFS-5868. Make hsync implementation pluggable on the DataNode. - (Buddy Taylor via Arpit Agarwal) - - HDFS-5768. Consolidate the serialization code in DelegationTokenSecretManager - (Haohui Mai via brandonli) - - HDFS-5775. Consolidate the code for serialization in CacheManager - (Haohui Mai via brandonli) - - HDFS-5935. New Namenode UI FS browser should throw smarter error messages. - (Travis Thompson via jing9) - - HDFS-5939. WebHdfs returns misleading error code and logs nothing if trying - to create a file with no DNs in cluster. (Yongjun Zhang via jing9) - - HDFS-6006. Remove duplicate code in FSNameSystem#getFileInfo. - (Akira Ajisaka via cnauroth) - - HDFS-6018. Exception recorded in LOG when IPCLoggerChannel#close is called. - (jing9) - - HDFS-3969. Small bug fixes and improvements for disk locations API. - (Todd Lipcon and Andrew Wang) - - HDFS-6025. Update findbugsExcludeFile.xml. (szetszwo) - - HDFS-6030. Remove an unused constructor in INode.java. (yzhang via - cmccabe) - - HDFS-4200. Reduce the size of synchronized sections in PacketResponder. - (suresh) - - HDFS-5950. The DFSClient and DataNode should use shared memory segments to - communicate short-circuit information. (cmccabe) - - HDFS-6046. add dfs.client.mmap.enabled (cmccabe) - - HDFS-5321. Clean up the HTTP-related configuration in HDFS (wheat9) - - HDFS-5167. Add metrics about the NameNode retry cache. (Tsuyoshi OZAWA via - jing9) - - HDFS-6043. Give HDFS daemons NFS3 and Portmap their own OPTS (brandonli) - - HDFS-6044. Add property for setting the NFS look up time for users - (brandonli) - - HDFS-6061. Allow dfs.datanode.shared.file.descriptor.path to contain - multiple entries and fall back when needed (cmccabe) - - HDFS-5986. Capture the number of blocks pending deletion on namenode webUI. - (cnauroth) - - HDFS-6070. Cleanup use of ReadStatistics in DFSInputStream. (wang) - - HDFS-6055. Change default configuration to limit file name length in HDFS. - (cnauroth) - - HDFS-3405. Checkpointing should use HTTP POST or PUT instead of GET-GET - to send merged fsimages. (Vinayakumar B via wang) - - HDFS-6085. Improve CacheReplicationMonitor log messages a bit (cmccabe) - - HDFS-6072. Clean up dead code of FSImage. (wheat9) - - HDFS-6080. Improve NFS gateway performance by making rtmax and wtmax - configurable. (Abin Shahab via brandonli) - - HDFS-6084. Namenode UI - "Hadoop" logo link shouldn't go to hadoop - homepage. (Travis Thompson via wheat9) - - HDFS-6106. Reduce default for - dfs.namenode.path.based.cache.refresh.interval.ms (cmccabe) - - HDFS-6090. Use MiniDFSCluster.Builder instead of deprecated constructors. - (Akira AJISAKA via jing9) - - HDFS-6068. Disallow snapshot names that are also invalid directory names. - (sathish via szetszwo) - - HDFS-6123. Do not log stack trace for ReplicaAlreadyExistsException and - SocketTimeoutException. (szetszwo) - - HDFS-6129. When a replica is not found for deletion, do not throw an - exception. (szetszwo) - - HDFS-6138. Add a user guide for how to use viewfs with federation. - (sanjay and szetszwo via szetszwo) - - HDFS-6120. Fix and improve safe mode log messages. (Arpit Agarwal) - - HDFS-6050. NFS does not handle exceptions correctly in a few places - (brandonli) - - HDFS-5138. Support HDFS upgrade in HA. (atm via todd) - - HDFS-6124. Add final modifier to class members. (Suresh Srinivas via - Arpit Agarwal) - - HDFS-5910. Enhance DataTransferProtocol to allow per-connection choice - of encryption/plain-text. (Benoy Antony via Arpit Agarwal) - - HDFS-6150. Add inode id information in the logs to make debugging easier. - (suresh) - - OPTIMIZATIONS - - HDFS-5790. LeaseManager.findPath is very slow when many leases need recovery - (todd) - - BUG FIXES - - HDFS-5492. Port HDFS-2069 (Incorrect default trash interval in the - docs) to trunk. (Akira Ajisaka via Arpit Agarwal) - - HDFS-5843. DFSClient.getFileChecksum() throws IOException if checksum is - disabled. (Laurent Goujon via jing9) - - HDFS-5856. DataNode.checkDiskError might throw NPE. - (Josh Elser via suresh) - - HDFS-5828. BlockPlacementPolicyWithNodeGroup can place multiple replicas on - the same node group when dfs.namenode.avoid.write.stale.datanode is true. - (Buddy via junping_du) - - HDFS-5767. NFS implementation assumes userName userId mapping to be unique, - which is not true sometimes (Yongjun Zhang via brandonli) - - HDFS-5791. TestHttpsFileSystem should use a random port to avoid binding - error during testing (Haohui Mai via brandonli) - - HDFS-5709. Improve NameNode upgrade with existing reserved paths and path - components. (Andrew Wang via atm) - - HDFS-5881. Fix skip() of the short-circuit local reader(legacy). (kihwal) - - HDFS-5895. HDFS cacheadmin -listPools has exit_code of 1 when the command - returns 0 result. (Tassapol Athiapinya via cnauroth) - - HDFS-5807. TestBalancerWithNodeGroup.testBalancerWithNodeGroup fails - intermittently. (Chen He via kihwal) - - HDFS-5882. TestAuditLogs is flaky (jxiang via cmccabe) - - HDFS-5900. Cannot set cache pool limit of "unlimited" via CacheAdmin. - (wang) - - HDFS-5886. Potential null pointer deference in RpcProgramNfs3#readlink() - (brandonli) - - HDFS-4858. HDFS DataNode to NameNode RPC should timeout. - (Henry Wang via shv) - - HDFS-5879. Some TestHftpFileSystem tests do not close streams. - (Gera Shegalov via suresh) - - HDFS-5938. Make BlockReaderFactory#BlockReaderPeer a static class to avoid - a findbugs warning. (cmccabe) - - HDFS-5891. webhdfs should not try connecting the DN during redirection - (Haohui Mai via brandonli) - - HDFS-5904. TestFileStatus fails intermittently. (Mit Desai via kihwal) - - HDFS-5941. add dfs.namenode.secondary.https-address and - dfs.namenode.secondary.https-address in hdfs-default.xml. - (Haohui Mai via cnauroth) - - HDFS-5913. Nfs3Utils#getWccAttr() should check attr parameter against null - (brandonli) - - HDFS-5934. New Namenode UI back button doesn't work as expected - (Travis Thompson via brandonli) - - HDFS-5901. NameNode new UI doesn't support IE8 and IE9 on windows 7 - (Vinayakumar B via brandonli) - - HDFS-5943. 'dfs.namenode.https-address' property is not loaded from - configuration in federation setup. (suresh) - - HDFS-3128. Unit tests should not use a test root in /tmp. (wang) - - HDFS-5948. TestBackupNode flakes with port in use error. (Haohui Mai - via Arpit Agarwal) - - HDFS-5949. New Namenode UI when trying to download a file, the browser - doesn't know the file name. (Haohui Mai via brandonli) - - HDFS-5716. Allow WebHDFS to use pluggable authentication filter - (Haohui Mai via brandonli) - - HDFS-5953. TestBlockReaderFactory fails if libhadoop.so has not been built. - (Akira Ajisaka via wang) - - HDFS-5759. Web UI does not show up during the period of loading FSImage. - (Haohui Mai via Arpit Agarwal) - - HDFS-5942. Fix javadoc in OfflineImageViewer. (Akira Ajisaka via cnauroth) - - HDFS-5780. TestRBWBlockInvalidation times out intemittently. (Mit Desai - via kihwal) - - HDFS-5803. TestBalancer.testBalancer0 fails. (Chen He via kihwal) - - HDFS-5893. HftpFileSystem.RangeHeaderUrlOpener uses the default - URLConnectionFactory which does not import SSL certificates. (Haohui Mai via - jing9) - - HDFS-5961. OIV cannot load fsimages containing a symbolic link. (kihwal) - - HDFS-5483. NN should gracefully handle multiple block replicas on same DN. - (Arpit Agarwal) - - HDFS-5742. DatanodeCluster (mini cluster of DNs) fails to start. - (Arpit Agarwal) - - HDFS-5979. Typo and logger fix for fsimage PB code. (wang) - - HDFS-5962. Mtime and atime are not persisted for symbolic links. (Akira - Ajisaka via kihwal) - - HDFS-5944. LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ - and cause SecondaryNameNode failed do checkpoint (Yunjiong Zhao via brandonli) - - HDFS-5982. Need to update snapshot manager when applying editlog for deleting - a snapshottable directory. (jing9) - - HDFS-5988. Bad fsimage always generated after upgrade. (wang) - - HDFS-5922. DN heartbeat thread can get stuck in tight loop. (Arpit Agarwal) - - HDFS-6008. Namenode dead node link is giving HTTP error 500. - (Benoy Antony via cnauroth) - - HDFS-5936. MiniDFSCluster does not clean data left behind by - SecondaryNameNode. (Binglin Chang via cnauroth) - - HDFS-5339. WebHDFS URI does not accept logical nameservices when security is - enabled. (Haohui Mai via jing9) - - HDFS-6033. PBImageXmlWriter incorrectly handles processing cache - directives. (atm) - - HDFS-5821. TestHDFSCLI fails for user names with the dash character. - (Gera Shegalov via Arpit Agarwal) - - HDFS-5956. A file size is multiplied by the replication factor in 'hdfs oiv - -p FileDistribution' option. (Akira Ajisaka via wheat9) - - HDFS-5866. '-maxSize' and '-step' option fail in OfflineImageViewer. - (Akira Ajisaka via wheat9) - - HDFS-6040. fix DFSClient issue without libhadoop.so and some other - ShortCircuitShm cleanups (cmccabe) - - HDFS-6047 TestPread NPE inside in DFSInputStream hedgedFetchBlockByteRange - (stack) - - HDFS-6051. HDFS cannot run on Windows since short-circuit shared memory - segment changes. (cmccabe) - - HDFS-5857. TestWebHDFS#testNamenodeRestart fails intermittently with NPE. - (Mit Desai via wheat9) - - HDFS-5898. Allow NFS gateway to login/relogin from its kerberos keytab. - (Abin Shahab via atm) - - HDFS-6057. DomainSocketWatcher.watcherThread should be marked as daemon - thread (cmccabe) - - HDFS-6058. Fix TestHDFSCLI failures after HADOOP-8691 change. - (Akira Ajisaka via wheat9) - - HDFS-6062. TestRetryCacheWithHA#testConcat is flaky. (Jing Zhao via wheat9) - - HDFS-6059. TestBlockReaderLocal fails if native library is not available. - (Akira AJISAKA via Colin Patrick McCabe) - - HDFS-6084. DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_DEFAULT is - not updated with latest block report interval of 6 hrs. - (Vinayakumar B via wheat9) - - HDFS-6067. TestPread.testMaxOutHedgedReadPool is flaky (cmccabe) - - HDFS-6065. HDFS zero-copy reads should return null on EOF when doing ZCR - (cmccabe) - - HDFS-5064. Standby checkpoints should not block concurrent readers. - (atm via wang) - - HDFS-6078. TestIncrementalBlockReports is flaky. (Arpit Agarwal) - - HDFS-6071. BlockReaderLocal doesn't return -1 on EOF when doing a - zero-length read on a short file (cmccabe) - - HDFS-5892. TestDeleteBlockPool fails in branch-2. (Ted Yu via wheat9) - - HDFS-6077. Running slive with webhdfs on secure HA cluster fails with unkown - host exception. (jing9) - - HDFS-6086. Fix a case where zero-copy or no-checksum reads were not allowed - even when the block was cached (cmccabe) - - HDFS-6079. Timeout for getFileBlockStorageLocations does not work. (wang) - - HDFS-5705. TestSecondaryNameNodeUpgrade#testChangeNsIDFails may fail due - to ConcurrentModificationException. (Ted Yu via brandonli) - - HDFS-6096. TestWebHdfsTokens may timeout. (szetszwo via Arpit Agarwal) - - HDFS-5244. TestNNStorageRetentionManager#testPurgeMultipleDirs fails. - (Jinghui Wang via suresh) - - HDFS-6097. zero-copy reads are incorrectly disabled on file offsets above - 2GB (cmccabe) - - HDFS-6102. Lower the default maximum items per directory to fix PB fsimage - loading. (wang) - - HDFS-6094. The same block can be counted twice towards safe mode - threshold. (Arpit Agarwal) - - HDFS-6107. When a block can't be cached due to limited space on the - DataNode, that block becomes uncacheable (cmccabe) - - HDFS-5516. WebHDFS does not require user name when anonymous http requests - are disallowed. (Miodrag Radulovic via cnauroth) - - HDFS-6117. Print file path information in FileNotFoundException on INode - ID mismatch. (suresh) - - HDFS-6099. HDFS file system limits not enforced on renames. (cnauroth) - - HDFS-6100. DataNodeWebHdfsMethods does not failover in HA mode. (Haohui Mai - via jing9) - - HDFS-6105. NN web UI for DN list loads the same jmx page multiple times. - (wheat9) - - HDFS-6127. WebHDFS tokens cannot be renewed in HA setup. (wheat9) - - HDFS-6140. WebHDFS cannot create a file with spaces in the name after HA - failover changes. (cnauroth) - - HDFS-6135. In HDFS upgrade with HA setup, JournalNode cannot handle layout - version bump when rolling back. (jing9) - - HDFS-5846. Assigning DEFAULT_RACK in resolveNetworkLocation method can break - data resiliency. (Nikola Vujic via cnauroth) - - HDFS-5840. Follow-up to HDFS-5138 to improve error handling during partial - upgrade failures. (atm, jing9 and suresh via jing9) - - HDFS-6130. NPE when upgrading namenode from fsimages older than -32. - (wheat9) - - HDFS-6115. Call flush() for every append on block scan verification log. - (Vinayakumar B via szetszwo) - - HDFS-5672. TestHASafeMode#testSafeBlockTracking fails in trunk. (jing9) - - HDFS-6157. Fix the entry point of OfflineImageViewer for hdfs.cmd. (wheat9) - - HDFS-6163. Fix a minor bug in the HA upgrade document. (Fengdong Yu via - jing9) - - HDFS-6166. Change Balancer socket read timeout to 20 minutes and add - 10 seconds delay after error. (Nathan Roberts via szetszwo) - - BREAKDOWN OF HDFS-5698 SUBTASKS AND RELATED JIRAS - - HDFS-5717. Save FSImage header in protobuf. (Haohui Mai via jing9) - - HDFS-5738. Serialize INode information in protobuf. (Haohui Mai via jing9) - - HDFS-5772. Serialize under-construction file information in FSImage. (jing9) - - HDFS-5783. Compute the digest before loading FSImage. (Haohui Mai via jing9) - - HDFS-5785. Serialize symlink in protobuf. (Haohui Mai via jing9) - - HDFS-5793. Optimize the serialization of PermissionStatus. (Haohui Mai via - jing9) - - HDFS-5743. Use protobuf to serialize snapshot information. (jing9) - - HDFS-5774. Serialize CachePool directives in protobuf. (Haohui Mai via jing9) - - HDFS-5744. Serialize information for token managers in protobuf. (Haohui Mai - via jing9) - - HDFS-5824. Add a Type field in Snapshot DiffEntry's protobuf definition. - (jing9) - - HDFS-5808. Implement cancellation when saving FSImage. (Haohui Mai via jing9) - - HDFS-5826. Update the stored edit logs to be consistent with the changes in - HDFS-5698 branch. (Haohui Mai via jing9) - - HDFS-5797. Implement offline image viewer. (Haohui Mai via jing9) - - HDFS-5771. Track progress when loading fsimage. (Haohui Mai via cnauroth) - - HDFS-5871. Use PBHelper to serialize CacheDirectiveInfoExpirationProto. - (Haohui Mai via jing9) - - HDFS-5884. LoadDelegator should use IOUtils.readFully() to read the magic - header. (Haohui Mai via jing9) - - HDFS-5885. Add annotation for repeated fields in the protobuf definition. - (Haohui Mai via jing9) - - HDFS-5906. Fixing findbugs and javadoc warnings in the HDFS-5698 branch. - (Haohui Mai via jing9) - - HDFS-5911. The id of a CacheDirective instance does not get serialized in - the protobuf-fsimage. (Haohui Mai via jing9) - - HDFS-5915. Refactor FSImageFormatProtobuf to simplify cross section reads. - (Haohui Mai via cnauroth) - - HDFS-5847. Consolidate INodeReference into a separate section. (jing9) - - HDFS-5959. Fix typo at section name in FSImageFormatProtobuf.java. - (Akira Ajisaka via suresh) - - HDFS-5981. PBImageXmlWriter generates malformed XML. - (Haohui Mai via cnauroth) - - HDFS-6089. Standby NN while transitioning to active throws a connection - refused error when the prior active NN process is suspended. - (Jing Zhao via Andrew Wang) - - BREAKDOWN OF HDFS-4685 SUBTASKS AND RELATED JIRAS - - HDFS-5596. Implement RPC stubs. (Haohui Mai via cnauroth) - - HDFS-5685. Implement ACL as a INode feature. (Haohui Mai via cnauroth) - - HDFS-5618. NameNode: persist ACLs in fsimage. (Haohui Mai via cnauroth) - - HDFS-5619. NameNode: record ACL modifications to edit log. - (Haohui Mai via cnauroth) - - HDFS-5673. Implement logic for modification of ACLs. (cnauroth) - - HDFS-5758. NameNode: complete implementation of inode modifications for - ACLs. (Chris Nauroth via wheat9) - - HDFS-5612. NameNode: change all permission checks to enforce ACLs in - addition to permissions. (Chris Nauroth via wheat9) - - HDFS-5613. NameNode: implement handling of ACLs in combination with - symlinks. (Chris Nauroth via wheat9) - - HDFS-5615. NameNode: implement handling of ACLs in combination with sticky - bit. (Chris Nauroth via wheat9) - - HDFS-5702. FsShell Cli: Add XML based End-to-End test for getfacl and - setfacl commands. (Vinay via cnauroth) - - HDFS-5608. WebHDFS: implement ACL APIs. - (Sachin Jose and Renil Joseph via cnauroth) - - HDFS-5614. NameNode: implement handling of ACLs in combination with - snapshots. (cnauroth) - - HDFS-5858. Refactor common ACL test cases to be run through multiple - FileSystem implementations. (cnauroth) - - HDFS-5860. Refactor INodeDirectory getDirectoryXFeature methods to use - common getFeature helper method. (Jing Zhao via cnauroth) - - HDFS-5861. Add CLI test for Ls output for extended ACL marker. - (Vinay via cnauroth) - - HDFS-5616. NameNode: implement default ACL handling. (cnauroth) - - HDFS-5899. Add configuration flag to disable/enable support for ACLs. - (cnauroth) - - HDFS-5914. Incorporate ACLs with the changes from HDFS-5698. - (Haohui Mai via cnauroth) - - HDFS-5625. Write end user documentation for HDFS ACLs. (cnauroth) - - HDFS-5925. ACL configuration flag must only reject ACL API calls, not ACLs - present in fsimage or edits. (cnauroth) - - HDFS-5923. Do not persist the ACL bit in the FsPermission. - (Haohui Mai via cnauroth) - - HDFS-5933. Optimize the FSImage layout for ACLs (Haohui Mai via cnauroth) - - HDFS-5932. Ls should display the ACL bit (Chris Nauroth via wheat9) - - HDFS-5937. Fix TestOfflineEditsViewer on HDFS-4685 branch. (cnauroth) - - HDFS-5737. Replacing only the default ACL can fail to copy unspecified base - entries from the access ACL. (cnauroth) - - HDFS-5739. ACL RPC must allow null name or unspecified permissions in ACL - entries. (cnauroth) - - HDFS-5799. Make audit logging consistent across ACL APIs. (cnauroth) - - HDFS-5849. Removing ACL from an inode fails if it has only a default ACL. - (cnauroth) - - HDFS-5623. NameNode: add tests for skipping ACL enforcement when permission - checks are disabled, user is superuser or user is member of supergroup. - (cnauroth) - - HDFS-5908. Change AclFeature to capture list of ACL entries in an - ImmutableList. (cnauroth) - - HDFS-6028. Print clearer error message when user attempts to delete required - mask entry from ACL. (cnauroth) - - HDFS-6039. Uploading a File under a Dir with default acls throws "Duplicated - ACLFeature". (cnauroth) - - HDFS-6063. TestAclCLI fails intermittently when running test 24: - copyFromLocal. (cnauroth) - - HDFS-6069. Quash stack traces when ACLs are disabled. (cnauroth) - - HDFS-5638. HDFS implementation of FileContext API for ACLs. - (Vinayakumar B via cnauroth) - -BREAKDOWN OF HDFS-5535 ROLLING UPGRADE SUBTASKS AND RELATED JIRAS - - HDFS-5496. Make replication queue initialization asynchronous. (Vinay via - jing9) - - HDFS-5645. Support upgrade marker in editlog streams. (szetszwo) - - HDFS-5752. Add a new DFSAdmin command to query, start and finalize rolling - upgrade. (szetszwo) - - HDFS-5786. Support QUERY and FINALIZE actions of rolling upgrade. (szetszwo) - - HDFS-5753. Add new Namenode startup options for downgrade and rollback using - upgrade marker. (szetszwo) - - HDFS-5835. Add a new option for starting Namenode when rolling upgrade is - in progress. (szetszwo) - - HDFS-5754. Split LayoutVerion into NameNodeLayoutVersion and - DataNodeLayoutVersion. (Brandon Li via szetszwo) - - HDFS-5848. Add rolling upgrade status to heartbeat response. (szetszwo) - - HDFS-5890. Avoid NPE in Datanode heartbeat. (Vinay via brandonli) - - HDFS-5869. When starting rolling upgrade or NN restarts, NN should create - a checkpoint right before the upgrade marker. (szetszwo) - - HDFS-5874. Should not compare DataNode current layout version with that of - NameNode in DataStrorage. (brandonli) - - HDFS-5889. When starting rolling upgrade, create a fs image for rollback - so that the standby namenode can create checkpoints during upgrade. - (szetszwo & jing9) - - HDFS-5907. Add BlockPoolSliceStorage 'trash' to handle block deletions - during rolling upgrades. (Arpit Agarwal) - - HDFS-5494. Merge Protobuf-based-FSImage code from trunk - fix build - break after the merge. (Jing Zhao via Arpit Agarwal) - - HDFS-5585. Provide admin commands for data node upgrade (kihwal) - - HDFS-5920. Support rollback of rolling upgrade in NameNode and JournalNodes. - (jing9) - - HDFS-5945. Add rolling upgrade information to fsimage; and disallow upgrade - and rolling upgrade to be started simultaneously. (szetszwo & jing9) - - HDFS-5966. Fix rollback of rolling upgrade in NameNode HA setup. (jing9 - via szetszwo) - - HDFS-5974. Fix compilation error, NameNodeLayoutVersion and - DataNodeLayoutVersion after merge from trunk. (szetszwo) - - HDFS-5963. TestRollingUpgrade#testSecondaryNameNode causes subsequent - tests to fail. (szetszwo via Arpit Agarwal) - - HDFS-5976. Create unit tests for downgrade and finalize rolling upgrade. - (Haohui Mai via Arpit Agarwal) - - HDFS-5980. Rollback does not need to load edits. (jing9 via szetszwo) - - HDFS-5984. Fix TestEditLog and TestStandbyCheckpoints. (jing9 via szetszwo) - - HDFS-5985. SimulatedFSDataset#disableAndPurgeTrashStorage should not throw - UnsupportedOperationException. (jing9 via kihwal) - - HDFS-5987. Fix findbugs warnings in Rolling Upgrade branch. (seztszwo via - Arpit Agarwal) - - HDFS-5992. Fix NPE in MD5FileUtils and update editsStored for - TestOfflineEditsViewer. (szetszwo) - - HDFS-5994. Fix TestDataNodeRollingUpgrade. (Arpit Agarwal via szetszwo) - - HDFS-5999. Do not create rollback fsimage when it already exists. (jing9) - - HDFS-6005. Simplify Datanode rollback and downgrade. (Suresh Srinivas via - Arpit Agarwal) - - HDFS-6004. Change DFSAdmin for rolling upgrade commands. (szetszwo via - Arpit Agarwal) - - HDFS-5583. Make DN send an OOB Ack on shutdown before restarting. (kihwal) - - HDFS-5778. Add rolling upgrade user document. (szetszwo) - - HDFS-6003. Add the new -rollingUpgrade startup option to the namenode - usage message. (Vinayakumar B via szetszwo) - - HDFS-6014. Fix findbug warnings introduced by HDFS-5583. (kihwal) - - HDFS-6015. Fix TestBlockRecovery - #testRaceBetweenReplicaRecoveryAndFinalizeBlock. (kihwal) - - HDFS-5924. Utilize OOB upgrade message processing for writes. (kihwal) - - HDFS-5498. Improve datanode startup time. (kihwal) - - HDFS-6000. Avoid saving namespace when starting rolling upgrade. (jing9) - - HDFS-6017. Query the status of rolling upgrade in the preparation stage in - TestRollingUpgrade and TestRollingUpgradeRollback. (Haohui Mai via - Arpit Agarwal) - - HDFS-6020. Fix the five findbugs warnings. (kihwal) - - HDFS-6019. Standby NN might not checkpoint when processing the rolling - upgrade marker. (Haohui Mai via jing9) - - HDFS-6023. Test whether the standby NN continues to checkpoint after the - prepare stage. (Haohui Mai via jing9) - - HDFS-6024. Test whether the NN will reject the downgrade if it has a - fsimage from a newer release. (Haohui Mai via jing9) - - HDFS-6026. Fix TestDFSUpgrade and TestDataNodeRollingUpgrade. - (jing9 via szetszwo) - - HDFS-6029. Secondary NN fails to checkpoint after -rollingUpgrade prepare. - (jing9) - - HDFS-6032. -rollingUpgrade query hits NPE after the NN restarts. (Haohui Mai - via jing9) - - HDFS-6031. Add back the "-rollingUpgrade started" namenode startup option; - otherwise, namenode cannot start when the layout version is changed. - (szetszwo) - - HDFS-6034. Use DataNodeLayoutVersion for DN registration check and do not - verify layout version if there is a rolling upgrade in progress. (szetszwo) - - HDFS-6013. add rollingUpgrade information to latest UI. - (Vinayakumar B via wheat9) - - HDFS-6042. Fix rolling upgrade documentation and error messages. (szetszwo - via Arpit Agarwal) - - HDFS-6041. Downgrade/Finalize should rename the rollback image instead of - purging it. (jing9) - - HDFS-6060. NameNode should not check DataNode layout version (brandonli) - - HDFS-6076. DataNode with SimulatedDataSet should not create - DatanodeRegistration with namenode layout version and namenode node type. - (szetszwo) - - HDFS-6038. Allow JournalNode to handle editlog produced by new release with - future layoutversion. (jing9) - - HDFS-4564. Ensure webhdfs returns correct HTTP response codes for denied - operations. (daryn via acmurthy) - -Release 2.3.1 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.3.0 - 2014-02-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-5122. Support failover and retry in WebHdfsFileSystem for NN HA. - (Haohui Mai via jing9) - - HDFS-4953. Enable HDFS local reads via mmap. - (Colin Patrick McCabe via wang). - - HDFS-5342. Provide more information in the FSNamesystem JMX interfaces. - (Haohui Mai via jing9) - - HDFS-5334. Implement dfshealth.jsp in HTML pages. (Haohui Mai via jing9) - - HDFS-5379. Update links to datanode information in dfshealth.html. (Haohui - Mai via jing9) - - HDFS-5382. Implement the UI of browsing filesystems in HTML 5 page. (Haohui - Mai via jing9) - - HDFS-3987. Support webhdfs over HTTPS. (Haohui Mai via jing9) - - HDFS-5444. Choose default web UI based on browser capabilities. (Haohui Mai - via jing9) - - HDFS-5514. FSNamesystem's fsLock should allow custom implementation (daryn) - - HDFS-2832. Heterogeneous Storages support in HDFS phase 1 - treat DataNode - as a collection of storages (see breakdown of tasks below for features and - contributors). - - HDFS-5784. reserve space in edit log header and fsimage header for feature - flag section (cmccabe) - - HDFS-5703. Add support for HTTPS and swebhdfs to HttpFS. (tucu) - - HDFS-4949. Centralized cache management in HDFS. (wang and cmccabe) - - IMPROVEMENTS - - HDFS-5360. Improvement of usage message of renameSnapshot and - deleteSnapshot. (Shinichi Yamashita via wang) - - HDFS-5331. make SnapshotDiff.java to a o.a.h.util.Tool interface implementation. - (Vinayakumar B via umamahesh) - - HDFS-4657. Limit the number of blocks logged by the NN after a block - report to a configurable value. (Aaron T. Myers via Colin Patrick - McCabe) - - HDFS-5344. Make LsSnapshottableDir as Tool interface implementation. (Sathish via umamahesh) - - HDFS-5544. Adding Test case For Checking dfs.checksum type as NULL value. (Sathish via umamahesh) - - HDFS-5568. Support includeSnapshots option with Fsck command. (Vinayakumar B via umamahesh) - - HDFS-4983. Numeric usernames do not work with WebHDFS FS. (Yongjun Zhang via - jing9) - - HDFS-5592. statechangeLog of completeFile should be logged only in case of success. - (Vinayakumar via umamahesh) - - HDFS-5662. Can't decommission a DataNode due to file's replication factor - larger than the rest of the cluster size. (brandonli) - - HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. - (shv) - - HDFS-5675. Add Mkdirs operation to NNThroughputBenchmark. - (Plamen Jeliazkov via shv) - - HDFS-5677. Need error checking for HA cluster configuration. - (Vincent Sheffer via cos) - - HDFS-5825. Use FileUtils.copyFile() to implement DFSTestUtils.copyFile(). - (Haohui Mai via Arpit Agarwal) - - HDFS-5267. Remove volatile from LightWeightHashSet. (Junping Du via llu) - - HDFS-4278. Log an ERROR when DFS_BLOCK_ACCESS_TOKEN_ENABLE config is - disabled but security is turned on. (Kousuke Saruta via harsh) - - HDFS-5004. Add additional JMX bean for NameNode status data - (Trevor Lorimer via cos) - - HDFS-4994. Audit log getContentSummary() calls. (Robert Parker via kihwal) - - HDFS-5144. Document time unit to NameNodeMetrics. (Akira Ajisaka via - suresh) - - HDFS-4491. Parallel testing HDFS. (Andrey Klochkov via cnauroth) - - HDFS-4879. Add "blocked ArrayList" collection to avoid CMS full GCs - (Todd Lipcon via Colin Patrick McCabe) - - HDFS-4096. Add snapshot information to namenode WebUI. (Haohui Mai via - jing9) - - HDFS-5188. In BlockPlacementPolicy, reduce the number of chooseTarget(..) - methods; replace HashMap with Map in parameter declarations and cleanup - some related code. (szetszwo) - - HDFS-5207. In BlockPlacementPolicy.chooseTarget(..), change the writer - and the excludedNodes parameter types respectively to Node and Set. - (Junping Du via szetszwo) - - HDFS-5240. Separate formatting from logging in the audit logger API (daryn) - - HDFS-5191. Revisit zero-copy API in FSDataInputStream to make it more - intuitive. (Contributed by Colin Patrick McCabe) - - HDFS-5260. Merge zero-copy memory-mapped HDFS client reads to trunk and - branch-2. (cnauroth) - - HDFS-4517. Cover class RemoteBlockReader with unit tests. (Vadim Bondarev - and Dennis Y via kihwal) - - HDFS-4512. Cover package org.apache.hadoop.hdfs.server.common with tests. - (Vadim Bondarev via kihwal) - - HDFS-4510. Cover classes ClusterJspHelper/NamenodeJspHelper with unit - tests. (Andrey Klochkov via kihwal) - - HDFS-5323. Remove some deadcode in BlockManager (Colin Patrick McCabe) - - HDFS-5338. Add a conf to disable hostname check in datanode registration. - (szetszwo) - - HDFS-5130. Add test for snapshot related FsShell and DFSAdmin commands. - (Binglin Chang via jing9) - - HDFS-5374. Remove deadcode in DFSOutputStream. (suresh) - - HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test - (Andrey Klochkov via jeagles) - - HDFS-4885. Improve the verifyBlockPlacement() API in BlockPlacementPolicy. - (Junping Du via szetszwo) - - HDFS-5363. Refactor WebHdfsFileSystem: move SPENGO-authenticated connection - creation to URLConnectionFactory. (Haohui Mai via jing9) - - HDFS-5436. Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web - (Haohui Mai via Arpit Agarwal) - - HDFS-5371. Let client retry the same NN when - "dfs.client.test.drop.namenode.response.number" is enabled. (jing9) - - HDFS-5467. Remove tab characters in hdfs-default.xml. - (Shinichi Yamashita via Andrew Wang) - - HDFS-5495. Remove further JUnit3 usages from HDFS. - (Jarek Jarcec Cecho via wang) - - HDFS-5325. Remove WebHdfsFileSystem#ConnRunner. (Haohui Mai via jing9) - - HDFS-5488. Clean up TestHftpURLTimeout. (Haohui Mai via jing9) - - HDFS-5440. Extract the logic of handling delegation tokens in HftpFileSystem - to the TokenAspect class. (Haohui Mai via jing9) - - HDFS-5487. Introduce unit test for TokenAspect. (Haohui Mai via jing9) - - HDFS-4995. Make getContentSummary less expensive. (kihwal) - - HDFS-5506. Use URLConnectionFactory in DelegationTokenFetcher. (Haohui Mai - via jing9) - - HDFS-5489. Use TokenAspect in WebHDFSFileSystem. (Haohui Mai via jing9) - - HDFS-5393. Serve bootstrap and jQuery locally. (Haohui Mai via jing9) - - HDFS-5073. TestListCorruptFileBlocks fails intermittently. (Arpit Agarwal) - - HDFS-1386. TestJMXGet fails in jdk7 (jeagles) - - HDFS-5532. Enable the webhdfs by default to support new HDFS web UI. (Vinay - via jing9) - - HDFS-5525. Inline dust templates for new Web UI. (Haohui Mai via jing9) - - HDFS-5561. FSNameSystem#getNameJournalStatus() in JMX should return plain - text instead of HTML. (Haohui Mai via jing9) - - HDFS-5581. NameNodeFsck should use only one instance of - BlockPlacementPolicy. (vinay via cmccabe) - - HDFS-5633. Improve OfflineImageViewer to use less memory. (jing9) - - HDFS-5023. TestSnapshotPathINodes.testAllowSnapshot is failing with jdk7 - (Mit Desai via jeagles) - - HDFS-5637. Try to refeatchToken while local read InvalidToken occurred. - (Liang Xie via junping_du) - - HDFS-5652. Refactor invalid block token exception handling in DFSInputStream. - (Liang Xie via junping_du) - - HDFS-5350. Name Node should report fsimage transfer time as a metric. - (Jimmy Xiang via wang) - - HDFS-5538. URLConnectionFactory should pick up the SSL related configuration - by default. (Haohui Mai via jing9) - - HDFS-5545. Allow specifying endpoints for listeners in HttpServer. (Haohui - Mai via jing9) - - HDFS-5536. Implement HTTP policy for Namenode and DataNode. (Haohui Mai via - jing9) - - HDFS-5312. Generate HTTP / HTTPS URL in DFSUtil#getInfoServer() based on the - configured http policy. (Haohui Mai via jing9) - - HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode. - (Haohui Mai via jing9) - - HDFS-5674. Editlog code cleanup: remove @SuppressWarnings("deprecation") in - FSEditLogOp; change FSEditLogOpCodes.fromByte(..) to be more efficient; and - change Some fields in FSEditLog to final. (szetszwo) - - HDFS-5634. Allow BlockReaderLocal to switch between checksumming and not - (cmccabe) - - HDFS-5663 make the retry time and interval value configurable in openInfo() - (Liang Xie via stack) - - HDFS-5540. Fix intermittent failure in TestBlocksWithNotEnoughRacks. - (Binglin Chang via junping_du) - - HDFS-2933. Improve DataNode Web UI Index Page. (Vivek Ganesan via - Arpit Agarwal) - - HDFS-5695. Clean up TestOfflineEditsViewer and OfflineEditsViewerHelper. - (Haohui Mai via jing9) - - HDFS-5220. Expose group resolution time as metric (jxiang via cmccabe) - - HDFS-5762. BlockReaderLocal doesn't return -1 on EOF when doing zero-length - reads (Colin Patrick McCabe) - - HDFS-5766. In DFSInputStream, do not add datanode to deadNodes after - InvalidEncryptionKeyException in fetchBlockByteRange (Liang Xie via Colin - Patrick McCabe) - - HDFS-5704. Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK. (jing9) - - HDFS-5434. Change block placement policy constructors from package private - to protected. (Buddy Taylor via Arpit Agarwal) - - HDFS-5788. listLocatedStatus response can be very large. (Nathan Roberts - via kihwal) - - HDFS-5841. Update HDFS caching documentation with new changes. (wang) - - OPTIMIZATIONS - - HDFS-5239. Allow FSNamesystem lock fairness to be configurable (daryn) - - HDFS-5341. Reduce fsdataset lock duration during directory scanning. - (Qus-Jiawei via kihwal) - - HDFS-5681. renewLease should not hold fsn write lock. (daryn via Kihwal) - - HDFS-5241. Provide alternate queuing audit logger to reduce logging - contention (daryn) - - BUG FIXES - - HDFS-5307. Support both HTTP and HTTPS in jsp pages (Haohui Mai via - brandonli) - - HDFS-5291. Standby namenode after transition to active goes into safemode. - (jing9) - - HDFS-5317. Go back to DFS Home link does not work on datanode webUI - (Haohui Mai via brandonli) - - HDFS-5316. Namenode ignores the default https port (Haohui Mai via - brandonli) - - HDFS-5281. COMMIT request should not block. (brandonli) - - HDFS-5337. should do hsync for a commit request even there is no pending - writes (brandonli) - - HDFS-5335. Hive query failed with possible race in dfs output stream. - (Haohui Mai via suresh) - - HDFS-5322. HDFS delegation token not found in cache errors seen on secure HA - clusters. (jing9) - - HDFS-5329. Update FSNamesystem#getListing() to handle inode path in startAfter - token. (brandonli) - - HDFS-5330. fix readdir and readdirplus for large directories (brandonli) - - HDFS-5370. Typo in Error Message: different between range in condition - and range in error message. (Kousuke Saruta via suresh) - - HDFS-5365. Fix libhdfs compile error on FreeBSD9. (Radim Kolar via cnauroth) - - HDFS-5347. Add HDFS NFS user guide. (brandonli) - - HDFS-5403. WebHdfs client cannot communicate with older WebHdfs servers - post HDFS-5306. (atm) - - HDFS-5171. NFS should create input stream for a file and try to share it - with multiple read requests. (Haohui Mai via brandonli) - - HDFS-5413. hdfs.cmd does not support passthrough to any arbitrary class. - (cnauroth) - - HDFS-5433. When reloading fsimage during checkpointing, we should clear - existing snapshottable directories. (Aaron T. Myers via wang) - - HDFS-5432. TestDatanodeJsp fails on Windows due to assumption that loopback - address resolves to host name localhost. (cnauroth) - - HDFS-5065. TestSymlinkHdfsDisable fails on Windows. (ivanmi) - - HDFS-4633 TestDFSClientExcludedNodes fails sporadically if excluded nodes - cache expires too quickly (Chris Nauroth via Sanjay) - - HDFS-5037. Active NN should trigger its own edit log rolls (wang) - - HDFS-5035. getFileLinkStatus and rename do not correctly check permissions - of symlinks. (Andrew Wang via Colin Patrick McCabe) - - HDFS-5456. NameNode startup progress creates new steps if caller attempts to - create a counter for a step that doesn't already exist. (cnauroth) - - HDFS-5458. Datanode failed volume threshold ignored if exception is thrown - in getDataDirsFromURIs. (Mike Mellenthin via wang) - - HDFS-5252. Stable write is not handled correctly in someplace. (brandonli) - - HDFS-5364. Add OpenFileCtx cache. (brandonli) - - HDFS-5469. Add configuration property for the sub-directroy export path - (brandonli) - - HDFS-5519. COMMIT handler should update the commit status after sync - (brandonli) - - HDFS-5372. In FSNamesystem, hasReadLock() returns false if the current thread - holds the write lock (VinayaKumar B via umamahesh) - - HDFS-4516. Client crash after block allocation and NN switch before lease recovery for - the same file can cause readers to fail forever (VinaayKumar B via umamahesh) - - HDFS-5014. Process register commands with out holding BPOfferService lock. - (Vinaykumar B via umamahesh) - - HDFS-5288. Close idle connections in portmap (Haohui Mai via brandonli) - - HDFS-5407. Fix typos in DFSClientCache (Haohui Mai via brandonli) - - HDFS-5548. Use ConcurrentHashMap in portmap (Haohui Mai via brandonli) - - HDFS-5577. NFS user guide update (brandonli) - - HDFS-5563. NFS gateway should commit the buffered data when read request comes - after write to the same file (brandonli) - - HDFS-4997. libhdfs doesn't return correct error codes in most cases (cmccabe) - - HDFS-5587. add debug information when NFS fails to start with duplicate user - or group names (brandonli) - - HDFS-5590. Block ID and generation stamp may be reused when persistBlocks is - set to false. (jing9) - - HDFS-5353. Short circuit reads fail when dfs.encrypt.data.transfer is - enabled. (Colin Patrick McCabe via jing9) - - HDFS-5283. Under construction blocks only inside snapshots should not be - counted in safemode threshhold. (Vinay via szetszwo) - - HDFS-5257. addBlock() retry should return LocatedBlock with locations else client - will get AIOBE. (Vinay via jing9) - - HDFS-5427. Not able to read deleted files from snapshot directly under - snapshottable dir after checkpoint and NN restart. (Vinay via jing9) - - HDFS-5443. Delete 0-sized block when deleting an under-construction file that - is included in snapshot. (jing9) - - HDFS-5476. Snapshot: clean the blocks/files/directories under a renamed - file/directory while deletion. (jing9) - - HDFS-5425. Renaming underconstruction file with snapshots can make NN failure on - restart. (jing9 and Vinay) - - HDFS-5474. Deletesnapshot can make Namenode in safemode on NN restarts. - (Sathish via jing9) - - HDFS-5504. In HA mode, OP_DELETE_SNAPSHOT is not decrementing the safemode threshold, - leads to NN safemode. (Vinay via jing9) - - HDFS-5428. Under construction files deletion after snapshot+checkpoint+nn restart - leads nn safemode. (jing9) - - HDFS-5074. Allow starting up from an fsimage checkpoint in the middle of a - segment. (Todd Lipcon via atm) - - HDFS-4201. NPE in BPServiceActor#sendHeartBeat. (jxiang via cmccabe) - - HDFS-5666. Fix inconsistent synchronization in BPOfferService (jxiang via cmccabe) - - HDFS-5657. race condition causes writeback state error in NFS gateway (brandonli) - - HDFS-5661. Browsing FileSystem via web ui, should use datanode's fqdn instead of ip - address. (Benoy Antony via jing9) - - HDFS-5582. hdfs getconf -excludeFile or -includeFile always failed (sathish - via cmccabe) - - HDFS-5671. Fix socket leak in DFSInputStream#getBlockReader. (JamesLi via umamahesh) - - HDFS-5649. Unregister NFS and Mount service when NFS gateway is shutting down. - (brandonli) - - HDFS-5789. Some of snapshot APIs missing checkOperation double check in fsn. (umamahesh) - - HDFS-5343. When cat command is issued on snapshot files getting unexpected result. - (Sathish via umamahesh) - - HDFS-5297. Fix dead links in HDFS site documents. (Akira Ajisaka via - Arpit Agarwal) - - HDFS-5830. WebHdfsFileSystem.getFileBlockLocations throws - IllegalArgumentException when accessing another cluster. (Yongjun Zhang via - Colin Patrick McCabe) - - HDFS-5833. Fix SecondaryNameNode javadoc. (Bangtao Zhou via Arpit Agarwal) - - HDFS-5844. Fix broken link in WebHDFS.apt.vm. (Akira Ajisaka via - Arpit Agarwal) - - HDFS-5034. Remove debug prints from GetFileLinkInfo (Andrew Wang via Colin - Patrick McCabe) - - HDFS-4816. transitionToActive blocks if the SBN is doing checkpoint image - transfer. (Andrew Wang) - - HDFS-5164. deleteSnapshot should check if OperationCategory.WRITE is - possible before taking write lock. (Colin Patrick McCabe) - - HDFS-5170. BlockPlacementPolicyDefault uses the wrong classname when - alerting to enable debug logging. (Andrew Wang) - - HDFS-5031. BlockScanner scans the block multiple times. (Vinay via Arpit - Agarwal) - - HDFS-5266. ElasticByteBufferPool#Key does not implement equals. (cnauroth) - - HDFS-5352. Server#initLog() doesn't close InputStream in httpfs. (Ted Yu via - jing9) - - HDFS-4376. Fix race conditions in Balancer. (Junping Du via szetszwo) - - HDFS-5375. hdfs.cmd does not expose several snapshot commands. (cnauroth) - - HDFS-5336. DataNode should not output 'StartupProgress' metrics. - (Akira Ajisaka via cnauroth) - - HDFS-5400. DFS_CLIENT_MMAP_CACHE_THREAD_RUNS_PER_TIMEOUT constant is set - to the wrong value. (Colin Patrick McCabe) - - HDFS-5075. httpfs-config.sh calls out incorrect env script name - (Timothy St. Clair via stevel) - - HDFS-5438. Flaws in block report processing can cause data loss. (kihwal) - - HDFS-5502. Fix HTTPS support in HsftpFileSystem. (Haohui Mai via jing9) - - HDFS-5552. Fix wrong information of "Cluster summay" in dfshealth.html. - (Haohui Mai via jing9) - - HDFS-5533. Symlink delete/create should be treated as DELETE/CREATE in snapshot diff - report. (Binglin Chang via jing9) - - HDFS-5580. Fix infinite loop in Balancer.waitForMoveCompletion. - (Binglin Chang via junping_du) - - HDFS-5676. fix inconsistent synchronization of CachingStrategy (cmccabe) - - HDFS-5691. Fix typo in ShortCircuitLocalRead document. - (Akira Ajisaka via suresh) - - HDFS-5690. DataNode fails to start in secure mode when dfs.http.policy equals to - HTTP_ONLY. (Haohui Mai via jing9) - - HDFS-5449. WebHdfs compatibility broken between 2.2 and 1.x / 23.x (kihwal) - - HDFS-5756. hadoopRzOptionsSetByteBufferPool does not accept NULL argument, - contrary to docs. (cmccabe via wang) - - HDFS-5747. Fix NPEs in BlockManager. (Arpit Agarwal) - - HDFS-5710. FSDirectory#getFullPathName should check inodes against null. - (Uma Maheswara Rao G via jing9) - - HDFS-5579. Under construction files make DataNode decommission take very long - hours. (zhaoyunjiong via jing9) - - HDFS-5777. Update LayoutVersion for the new editlog op OP_ADD_BLOCK. (jing9) - - HDFS-5800. Fix a typo in DFSClient.renewLease(). (Kousuke Saruta - via szetszwo) - - HDFS-5748. Too much information shown in the dfs health page. - (Haohui Mai via brandonli) - - HDFS-5806. balancer should set SoTimeout to avoid indefinite hangs. - (Nathan Roberts via Andrew Wang). - - HDFS-5728. Block recovery will fail if the metafile does not have crc - for all chunks of the block (Vinay via kihwal) - - HDFS-5845. SecondaryNameNode dies when checkpointing with cache pools. - (wang) - - HDFS-5842. Cannot create hftp filesystem when using a proxy user ugi and a doAs - on a secure cluster. (jing9) - - HDFS-5399. Revisit SafeModeException and corresponding retry policies. - (Jing Zhao via todd) - - HDFS-5876. SecureDataNodeStarter does not pick up configuration in - hdfs-site.xml. (Haohui Mai via jing9) - - HDFS-5873. dfs.http.policy should have higher precedence over dfs.https.enable. - (Haohui Mai via jing9) - - HDFS-5837. dfs.namenode.replication.considerLoad should consider - decommissioned nodes. (Tao Luo via shv) - - HDFS-5921. Cannot browse file system via NN web UI if any directory has - the sticky bit set. (atm) - - BREAKDOWN OF HDFS-2832 SUBTASKS AND RELATED JIRAS - - HDFS-4985. Add storage type to the protocol and expose it in block report - and block locations. (Arpit Agarwal) - - HDFS-5115. Make StorageID a UUID. (Arpit Agarwal) - - HDFS-5000. DataNode configuration should allow specifying storage type. - (Arpit Agarwal) - - HDFS-4987. Namenode changes to track multiple storages per datanode. - (szetszwo) - - HDFS-5154. Fix TestBlockManager and TestDatanodeDescriptor after HDFS-4987. - (Junping Du via szetszwo) - - HDFS-5009. Include storage information in the LocatedBlock. (szetszwo) - - HDFS-5134. Move blockContentsStale, heartbeatedSinceFailover and - firstBlockReport from DatanodeDescriptor to DatanodeStorageInfo; and - fix a synchronization problem in DatanodeStorageInfo. (szetszwo) - - HDFS-5157. Add StorageType to FsVolume. (Junping Du via szetszwo) - - HDFS-4990. Change BlockPlacementPolicy to choose storages instead of - datanodes. (szetszwo) - - HDFS-5232. Protocol changes to transmit StorageUuid. (Arpit Agarwal) - - HDFS-5233. Use Datanode UUID to identify Datanodes. (Arpit Agarwal) - - HDFS-5222. Move block schedule information from DatanodeDescriptor to - DatanodeStorageInfo. (szetszwo) - - HDFS-4988. Datanode must support all the volumes as individual storages. - (Arpit Agarwal) - - HDFS-5377. Heartbeats from Datandode should include one storage report - per storage directory. (Arpit Agarwal) - - HDFS-5398. NameNode changes to process storage reports per storage - directory. (Arpit Agarwal) - - HDFS-5390. Send one incremental block report per storage directory. - (Arpit Agarwal) - - HDFS-5401. Fix NPE in Directory Scanner. (Arpit Agarwal) - - HDFS-5417. Fix storage IDs in PBHelper and UpgradeUtilities. (szetszwo) - - HDFS-5214. Fix NPEs in BlockManager and DirectoryScanner. (Arpit Agarwal) - - HDFS-5435. File append fails to initialize storageIDs. (Junping Du via - Arpit Agarwal) - - HDFS-5437. Fix TestBlockReport and TestBPOfferService failures. (Arpit - Agarwal) - - HDFS-5447. Fix TestJspHelper. (Arpit Agarwal) - - HDFS-5452. Fix TestReplicationPolicy and TestBlocksScheduledCounter. - - HDFS-5448. Datanode should generate its ID on first registration. (Arpit - Agarwal) - - HDFS-5448. Fix break caused by previous checkin for HDFS-5448. (Arpit - Agarwal) - - HDFS-5455. NN should update storageMap on first heartbeat. (Arpit Agarwal) - - HDFS-5457. Fix TestDatanodeRegistration, TestFsck and TestAddBlockRetry. - (Contributed by szetszwo) - - HDFS-5466. Update storage IDs when the pipeline is updated. (Contributed - by szetszwo) - - HDFS-5439. Fix TestPendingReplication. (Contributed by Junping Du, Arpit - Agarwal) - - HDFS-5470. Add back trunk's reportDiff algorithm to the branch. - (Contributed by szetszwo) - - HDFS-5472. Fix TestDatanodeManager, TestSafeMode and - TestNNThroughputBenchmark (Contributed by szetszwo) - - HDFS-5475. NN incorrectly tracks more than one replica per DN. (Arpit - Agarwal) - - HDFS-5481. Fix TestDataNodeVolumeFailure in branch HDFS-2832. (Contributed - by Junping Du) - - HDFS-5480. Update Balancer for HDFS-2832. (Contributed by szetszwo) - - HDFS-5486. Fix TestNameNodeMetrics for HDFS-2832. (Arpit Agarwal) - - HDFS-5491. Update editsStored for HDFS-2832. (Arpit Agarwal) - - HDFS-5494. Fix findbugs warnings for HDFS-2832. (Arpit Agarwal) - - HDFS-5508. Fix compilation error after merge. (Contributed by szetszwo) - - HDFS-5501. Fix pendingReceivedRequests tracking in BPServiceActor. (Arpit - Agarwal) - - HDFS-5510. Fix a findbug warning in DataStorage.java on HDFS-2832 branch. - (Junping Du via Arpit Agarwal) - - HDFS-5515. Fix TestDFSStartupVersions for HDFS-2832. (Arpit Agarwal) - - HDFS-5527. Fix TestUnderReplicatedBlocks on branch HDFS-2832. (Arpit - Agarwal) - - HDFS-5547. Fix build break after merge from trunk to HDFS-2832. (Arpit - Agarwal) - - HDFS-5542. Fix TODO and clean up the code in HDFS-2832. (Contributed by - szetszwo) - - HDFS-5559. Fix TestDatanodeConfig in HDFS-2832. (Contributed by szetszwo) - - HDFS-5484. StorageType and State in DatanodeStorageInfo in NameNode is - not accurate. (Eric Sirianni via Arpit Agarwal) - - HDFS-5648. Get rid of FsDatasetImpl#perVolumeReplicaMap. (Arpit Agarwal) - - HDFS-5406. Send incremental block reports for all storages in a - single call. (Arpit Agarwal) - - HDFS-5454. DataNode UUID should be assigned prior to FsDataset - initialization. (Arpit Agarwal) - - HDFS-5667. Include DatanodeStorage in StorageReport. (Arpit Agarwal) - - BREAKDOWN OF HDFS-4949 SUBTASKS AND RELATED JIRAS - - HDFS-5049. Add JNI mlock support. (Andrew Wang via Colin Patrick McCabe) - - HDFS-5051. Propagate cache status information from the DataNode to the - NameNode (Andrew Wang via Colin Patrick McCabe) - - HDFS-5052. Add cacheRequest/uncacheRequest support to NameNode. - (Contributed by Colin Patrick McCabe.) - - HDFS-5050. Add DataNode support for mlock and munlock (contributed by - Andrew Wang) - - HDFS-5141. Add cache status information to datanode heartbeat. (Contributed - by Andrew Wang) - - HDFS-5121. Add RPCs for creating and manipulating cache pools. - (Contributed by Colin Patrick McCabe) - - HDFS-5163. Miscellaneous cache pool RPC fixes (Contributed by Colin Patrick - McCabe) - - HDFS-5169. hdfs.c: translateZCRException: null pointer deref when - translating some exceptions (Contributed by Colin Patrick McCabe) - - HDFS-5120. Add command-line support for manipulating cache pools. (cmccabe) - - HDFS-5158. Add command-line support for manipulating cache directives. - (cmccabe) - - HDFS-5198. NameNodeRpcServer must not send back DNA_FINALIZE in reply to a - cache report. (cmccabe) - - HDFS-5195. Prevent passing null pointer to mlock and munlock. Contributed - by Chris Nauroth. - - HDFS-5053. NameNode should invoke DataNode APIs to coordinate caching. - (Andrew Wang) - - HDFS-5201. NativeIO: consolidate getrlimit into NativeIO#getMemlockLimit. - (Contributed by Colin Patrick McCabe) - - HDFS-5197. Document dfs.cachereport.intervalMsec in hdfs-default.xml. - Contributed by Chris Nauroth. - - HDFS-5210. Fix some failing unit tests on HDFS-4949 branch. (Contributed by - Andrew Wang) - - HDFS-5213. Separate PathBasedCacheEntry and PathBasedCacheDirectiveWithId. - Contributed by Colin Patrick McCabe. - - HDFS-5236. Change PathBasedCacheDirective APIs to be a single value rather - than batch. (Contributed by Andrew Wang) - - HDFS-5119. Persist CacheManager state in the edit log. (Contributed by - Andrew Wang) - - HDFS-5190. Move cache pool related CLI commands to CacheAdmin. (Contributed - by Andrew Wang) - - HDFS-5309. Fix failing caching unit tests. (Andrew Wang) - - HDFS-5314. Do not expose CachePool type in AddCachePoolOp (Colin Patrick - McCabe) - - HDFS-5304. Expose if a block replica is cached in getFileBlockLocations. - (Contributed by Andrew Wang) - - HDFS-5224. Refactor PathBasedCache* methods to use a Path rather than a - String. Contributed by Chris Nauroth. - - HDFS-5348. Fix error message when dfs.datanode.max.locked.memory is - improperly configured. (Contributed by Colin Patrick McCabe) - - HDFS-5349. DNA_CACHE and DNA_UNCACHE should be by blockId only (cmccabe) - - HDFS-5358. Add replication field to PathBasedCacheDirective. (Contributed - by Colin Patrick McCabe) - - HDFS-5359. Allow LightWeightGSet#Iterator to remove elements. (Contributed - by Colin Patrick McCabe) - - HDFS-5373. hdfs cacheadmin -addDirective short usage does not mention - -replication parameter. Contributed by Chris Nauroth. - - HDFS-5096. Automatically cache new data added to a cached path (contributed - by Colin Patrick McCabe) - - HDFS-5383. fix broken caching unit tests (Andrew Wang) - - HDFS-5388. Loading fsimage fails to find cache pools during namenode - startup (Chris Nauroth via Colin Patrick McCabe) - - HDFS-5203. Concurrent clients that add a cache directive on the same path - may prematurely uncache each other. (Chris Nauroth via Colin Patrick McCabe) - - HDFS-5378. In CacheReport, don't send genstamp and length on the wire - (Contributed by Colin Patrick McCabe) - - HDFS-5385. Caching RPCs are AtMostOnce, but do not persist client ID and - call ID to edit log. (Chris Nauroth via Colin Patrick McCabe) - - HDFS-5404 Resolve regressions in Windows compatibility on HDFS-4949 branch. - Contributed by Chris Nauroth. - - HDFS-5405. Fix possible RetryCache hang for caching RPC handlers in - FSNamesystem. (Contributed by Andrew Wang) - - HDFS-5419. Fixup test-patch.sh warnings on HDFS-4949 branch. (wang) - - HDFS-5386. Add feature documentation for datanode caching. Contributed by - Colin Patrick McCabe. - - HDFS-5468. CacheAdmin help command does not recognize commands (Stephen - Chu via Colin Patrick McCabe) - - HDFS-5326. add modifyDirective to cacheAdmin (cmccabe) - - HDFS-5394: Fix race conditions in DN caching and uncaching (cmccabe) - - HDFS-5320. Add datanode caching metrics. Contributed by Andrew Wang. - - HDFS-5482. DistributedFileSystem#listPathBasedCacheDirectives must support - relative paths. Contributed by Colin Patrick McCabe. - - HDFS-5471. CacheAdmin -listPools fails when user lacks permissions to view - all pools (Andrew Wang via Colin Patrick McCabe) - - HDFS-5450. better API for getting the cached blocks locations. Contributed - by Andrew Wang. - - HDFS-5485. add command-line support for modifyDirective (cmccabe) - - HDFS-5366. recaching improvements (cmccabe) - - HDFS-5520. loading cache path directives from edit log doesnt update - nextEntryId (cmccabe) - - HDFS-5512. CacheAdmin -listPools fails with NPE when user lacks permissions - to view all pools (awang via cmccabe) - - HDFS-5513. CacheAdmin commands fail when using . as the path. Contributed - by Andrew Wang. - - HDFS-5511. improve CacheManipulator interface to allow better unit testing - (cmccabe) - - HDFS-5451. Add byte and file statistics to PathBasedCacheEntry. Contributed - by Colin Patrick McCabe. - - HDFS-5473. Consistent naming of user-visible caching classes and methods - (cmccabe) - - HDFS-5543. Fix narrow race condition in TestPathBasedCacheRequests - (cmccabe) - - HDFS-5565. CacheAdmin help should match against non-dashed commands (wang - via cmccabe) - - HDFS-5556. Add some more NameNode cache statistics, cache pool stats - (cmccabe) - - HDFS-5562. TestCacheDirectives and TestFsDatasetCache should stub out - native mlock. Contributed by Colin Patrick McCabe and Akira Ajisaka. - - HDFS-5430. Support TTL on CacheDirectives. Contributed by Andrew Wang. - - HDFS-5555. CacheAdmin commands fail when first listed NameNode is in - Standby (jxiang via cmccabe) - - HDFS-5626. dfsadmin report shows incorrect values (cmccabe) - - HDFS-5630. Hook up cache directive and pool usage statistics. (wang) - - HDFS-5665. Remove the unnecessary writeLock while initializing CacheManager - in FsNameSystem Ctor. (Uma Maheswara Rao G via Andrew Wang) - - HDFS-5431. Support cachepool-based limit management in path-based caching. - (awang via cmccabe) - - HDFS-5679. TestCacheDirectives should handle the case where native code is - not available. (wang) - - HDFS-5636. Enforce a max TTL per cache pool (awang via cmccabe) - - HDFS-5701. Fix the CacheAdmin -addPool -maxTtl option name. Contributed by - Stephen Chu. - - HDFS-5708. The CacheManager throws a NPE in the DataNode logs when - processing cache reports that refer to a block not known to the BlockManager. - Contributed by Colin Patrick McCabe. - - HDFS-5659. dfsadmin -report doesn't output cache information properly. - Contributed by Andrew Wang. - - HDFS-5651. Remove dfs.namenode.caching.enabled and improve CRM locking. - Contributed by Colin Patrick McCabe. - - HDFS-5589. Namenode loops caching and uncaching when data should be - uncached. (awang via cmccabe) - - HDFS-5724. modifyCacheDirective logging audit log command wrongly as - addCacheDirective (Uma Maheswara Rao G via Colin Patrick McCabe) - -Release 2.2.0 - 2013-10-13 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-4817. Make HDFS advisory caching configurable on a per-file basis. - (Colin Patrick McCabe) - - HDFS-5230. Introduce RpcInfo to decouple XDR classes from the RPC API. - (Haohui Mai via brandonli) - - IMPROVEMENTS - - HDFS-5246. Make Hadoop nfs server port and mount daemon port - configurable. (Jinghui Wang via brandonli) - - HDFS-5256. Use guava LoadingCache to implement DFSClientCache. (Haohui Mai - via brandonli) - - HDFS-5308. Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS - JSP. (Haohui Mai via jing9) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-5139. Remove redundant -R option from setrep. - - HDFS-5251. Race between the initialization of NameNode and the http - server. (Haohui Mai via suresh) - - HDFS-5258. Skip tests in TestHDFSCLI that are not applicable on Windows. - (Chuan Liu via cnauroth) - - HDFS-5186. TestFileJournalManager fails on Windows due to file handle leaks. - (Chuan Liu via cnauroth) - - HDFS-5268. NFS write commit verifier is not set in a few places (brandonli) - - HDFS-5265. Namenode fails to start when dfs.https.port is unspecified. - (Haohui Mai via jing9) - - HDFS-5255. Distcp job fails with hsftp when https is enabled in insecure - cluster. (Arpit Agarwal) - - HDFS-5279. Guard against NullPointerException in NameNode JSP pages before - initialization of FSNamesystem. (cnauroth) - - HDFS-5289. Race condition in TestRetryCacheWithHA#testCreateSymlink causes - spurious test failure. (atm) - - HDFS-5300. FSNameSystem#deleteSnapshot() should not check owner in case of - permissions disabled. (Vinay via jing9) - - HDFS-5306. Datanode https port is not available at the namenode. (Suresh - Srinivas via brandonli) - - HDFS-5299. DFS client hangs in updatePipeline RPC when failover happened. - (Vinay via jing9) - - HDFS-5259. Support client which combines appended data with old data - before sends it to NFS server. (brandonli) - -Release 2.1.1-beta - 2013-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-4962 Use enum for nfs constants (Nicholas SZE via jing9) - - HDFS-5071 Change hdfs-nfs parent project to hadoop-project (brandonli) - - HDFS-4763 Add script changes/utility for starting NFS gateway (brandonli) - - HDFS-5076 Add MXBean methods to query NN's transaction information and - JournalNode's journal status. (jing9) - - HDFS-5104 Support dotdot name in NFS LOOKUP operation (brandonli) - - HDFS-5107 Fix array copy error in Readdir and Readdirplus responses - (brandonli) - - HDFS-5110 Change FSDataOutputStream to HdfsDataOutputStream for opened - streams to fix type cast error. (brandonli) - - HDFS-5069 Include hadoop-nfs and hadoop-hdfs-nfs into hadoop dist for - NFS deployment (brandonli) - - HDFS-4947 Add NFS server export table to control export by hostname or - IP range (Jing Zhao via brandonli) - - HDFS-5078 Support file append in NFSv3 gateway to enable data streaming - to HDFS (brandonli) - - HDFS-5136 MNT EXPORT should give the full group list which can mount the - exports (brandonli) - - HDFS-5118. Provide testing support for DFSClient to drop RPC responses. - (jing9) - - HDFS-5085. Refactor o.a.h.nfs to support different types of - authentications. (jing9) - - HDFS-5067 Support symlink operations in NFS gateway. (brandonli) - - HDFS-5199 Add more debug trace for NFS READ and WRITE. (brandonli) - - HDFS-5234 Move RpcFrameDecoder out of the public API. - (Haohui Mai via brandonli) - - IMPROVEMENTS - - HDFS-4513. Clarify in the WebHDFS REST API that all JSON respsonses may - contain additional properties. (szetszwo) - - HDFS-5061. Make FSNameSystem#auditLoggers an unmodifiable list. - (Arpit Agarwal via suresh) - - HDFS-4905. Add appendToFile command to "hdfs dfs". (Arpit Agarwal via - cnauroth) - - HDFS-4926. Namenode webserver's page has a tooltip that is inconsistent - with the datanode HTML link. (Vivek Ganesan via jing9) - - HDFS-5047. Supress logging of full stack trace of quota and lease - exceptions. (Robert Parker via kihwal) - - HDFS-5111. Remove duplicated error message for snapshot commands when - processing invalid arguments. (jing9) - - HDFS-5045. Add more unit tests for retry cache to cover all AtMostOnce - methods. (jing9) - - HDFS-3245. Add metrics and web UI for cluster version summary. (Ravi - Prakash via kihwal) - - HDFS-5128. Allow multiple net interfaces to be used with HA namenode RPC - server. (kihwal) - - HDFS-5150. Allow per NN SPN for internal SPNEGO. (kihwal) - - HDFS-4680. Audit logging of delegation tokens for MR tracing. (Andrew Wang) - - HDFS-5212. Refactor RpcMessage and NFS3Response to support different - types of authentication information. (jing9) - - HDFS-4971. Move IO operations out of locking in OpenFileCtx. (brandonli and - jing9) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-5028. LeaseRenewer throws ConcurrentModificationException when timeout. - (zhaoyunjiong via szetszwo) - - HDFS-5043. For HdfsFileStatus, set default value of childrenNum to -1 - instead of 0 to avoid confusing applications. (brandonli) - - HDFS-4993. Fsck can fail if a file is renamed or deleted. (Robert Parker - via kihwal) - - HDFS-5091. Support for spnego keytab separate from the JournalNode keytab - for secure HA. (jing9) - - HDFS-5055. nn fails to download checkpointed image from snn in some - setups. (Vinay and suresh via suresh) - - HDFS-4898. BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to - properly fallback to local rack. (szetszwo) - - HDFS-4632. globStatus using backslash for escaping does not work on Windows. - (Chuan Liu via cnauroth) - - HDFS-5093. TestGlobPaths should re-use the MiniDFSCluster to avoid failure - on Windows. (Chuan Liu via cnauroth) - - HDFS-5080. BootstrapStandby not working with QJM when the existing NN is - active. (jing9) - - HDFS-5099. Namenode#copyEditLogSegmentsToSharedDir should close - EditLogInputStreams upon finishing. (Chuan Liu via cnauroth) - - HDFS-2994. If lease soft limit is recovered successfully - the append can fail. (Tao Luo via shv) - - HDFS-5100. TestNamenodeRetryCache fails on Windows due to incorrect cleanup. - (Chuan Liu via cnauroth) - - HDFS-5103. TestDirectoryScanner fails on Windows. (Chuan Liu via cnauroth) - - HDFS-5102. Snapshot names should not be allowed to contain slash characters. - (jing9) - - HDFS-5105. TestFsck fails on Windows. (Chuan Liu via arp) - - HDFS-5106. TestDatanodeBlockScanner fails on Windows due to incorrect path - format. (Chuan Liu via cnauroth) - - HDFS-4594. WebHDFS open sets Content-Length header to what is specified by - length parameter rather than how much data is actually returned. (cnauroth) - - HDFS-5124. DelegationTokenSecretManager#retrievePassword can cause deadlock - in NameNode. (Daryn Sharp via jing9) - - HDFS-5132. Deadlock in NameNode between SafeModeMonitor#run and - DatanodeManager#handleHeartbeat. (kihwal) - - HDFS-5077. NPE in FSNamesystem.commitBlockSynchronization(). - (Plamen Jeliazkov via shv) - - HDFS-5140. Too many safemode monitor threads being created in the standby - namenode causing it to fail with out of memory error. (jing9) - - HDFS-5159. Secondary NameNode fails to checkpoint if error occurs - downloading edits on first checkpoint. (atm) - - HDFS-5192. NameNode may fail to start when - dfs.client.test.drop.namenode.response.number is set. (jing9) - - HDFS-5219. Add configuration keys for retry policy in WebHDFSFileSystem. - (Haohui Mai via jing9) - - HDFS-5231. Fix broken links in the document of HDFS Federation. (Haohui Mai - via jing9) - - HDFS-5249. Fix dumper thread which may die silently. (brandonli) - -Release 2.1.0-beta - 2013-08-22 - - INCOMPATIBLE CHANGES - - HDFS-4053. Increase the default block size. (eli) - - HDFS-4305. Add a configurable limit on number of blocks per file, and min - block size. (Andrew Wang via atm) - - HDFS-4434. Provide a mapping from INodeId to INode. (suresh) - - HDFS-2802. Add HDFS Snapshot feature. (See breakdown of tasks below for - subtasks and contributors) - - HDFS-4866. Protocol buffer support cannot compile under C. (Arpit Agarwal via - cnauroth) - - NEW FEATURES - - HDFS-1804. Add a new block-volume device choosing policy that looks at - free space. (atm) - - HDFS-4296. Reserve layout version for release 1.2.0. (suresh) - - HDFS-4334. Add a unique id to INode. (Brandon Li via szetszwo) - - HDFS-4339. Persist inode id in fsimage and editlog. (Brandon Li via - suresh) - - HDFS-4340. Update addBlock() to inculde inode id as additional argument. - (Brandon Li via suresh) - - HDFS-4502. JsonUtil.toFileStatus(..) should check if the fileId property - exists. (Brandon Li via suresh) - - HDFS-2576. Enhances the DistributedFileSystem's create API so that clients - can specify favored datanodes for a file's blocks. (ddas) - - HDFS-347. DFS read performance suboptimal when client co-located on nodes - with data. (Colin Patrick McCabe via todd and atm) - - HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows - Azure environments. (See breakdown of tasks below for subtasks and - contributors) - - HDFS-3601. Add BlockPlacementPolicyWithNodeGroup to support block placement - with 4-layer network topology. (Junping Du via szetszwo) - - HDFS-3495. Update Balancer to support new NetworkTopology with NodeGroup. - (Junping Du via szetszwo) - - HDFS-4659 Support setting execution bit for regular files (Brandon Li via sanjay) - - HDFS-4762 Provide HDFS based NFSv3 and Mountd implementation (brandonli) - - HDFS-4372. Track NameNode startup progress. (cnauroth) - - HDFS-4373. Add HTTP API for querying NameNode startup progress. (cnauroth) - - HDFS-4374. Display NameNode startup progress in UI. (cnauroth) - - HDFS-4974. Add Idempotent and AtMostOnce annotations to namenode - protocol methods. (suresh) - - HDFS-4979. Implement retry cache on Namenode. (suresh) - - HDFS-5025. Record ClientId and CallId in EditLog to enable rebuilding - retry cache in case of HA failover. (Jing Zhao via suresh) - - IMPROVEMENTS - - HDFS-4461. DirectoryScanner: volume path prefix takes up memory for every - block that is scanned (Colin Patrick McCabe) - - HDFS-4222. NN is unresponsive and loses heartbeats from DNs when - configured to use LDAP and LDAP has issues. (Xiaobo Peng, suresh) - - HDFS-4304. Make FSEditLogOp.MAX_OP_SIZE configurable. (Colin Patrick - McCabe via atm) - - HDFS-4518. Finer grained metrics for HDFS capacity. - (Arpit Agarwal via suresh) - - HDFS-4519. Support overriding jsvc binary and log file locations - when launching secure datanode. (Chris Nauroth via suresh) - - HDFS-4569. Small image transfer related cleanups. - (Andrew Wang via suresh) - - HDFS-4521. Invalid network toploogies should not be cached. (Colin Patrick - McCabe via atm) - - HDFS-4246. The exclude node list should be more forgiving, for each output - stream. (harsh via atm) - - HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. (suresh) - - HDFS-4621. Additional logging to help diagnose slow QJM syncs. (todd) - - HDFS-4618. Default transaction interval for checkpoints is too low. (todd) - - HDFS-4525. Provide an API for knowing that whether file is closed or not. - (SreeHari via umamahesh) - - HDFS-3940. Add Gset#clear method and clear the block map when namenode is - shutdown. (suresh) - - HDFS-4679. Namenode operation checks should be done in a consistent - manner. (suresh) - - HDFS-4693. Some test cases in TestCheckpoint do not clean up after - themselves. (Arpit Agarwal, suresh via suresh) - - HDFS-3817. Avoid printing SafeModeException stack trace. - (Brandon Li via suresh) - - HDFS-4124. Refactor INodeDirectory#getExistingPathINodes() to enable - returning more than INode array. (Jing Zhao via suresh) - - HDFS-4151. Change the methods in FSDirectory to pass INodesInPath instead - of INode[] as a parameter. (szetszwo) - - HDFS-4129. Add utility methods to dump NameNode in memory tree for - testing. (szetszwo via suresh) - - HDFS-4152. Add a new class BlocksMapUpdateInfo for the parameter in - INode.collectSubtreeBlocksAndClear(..). (Jing Zhao via szetszwo) - - HDFS-4206. Change the fields in INode and its subclasses to private. - (szetszwo) - - HDFS-4215. Remove locking from addToParent(..) since it is used in image - loading, and add INode.isFile(). (szetszwo) - - HDFS-4243. When replacing an INodeDirectory, the parent pointers of the - children of the child have to be updated to the new child. (Jing Zhao - via szetszwo) - - HDFS-4209. Clean up the addNode/addChild/addChildNoQuotaCheck methods in - FSDirectory and INodeDirectory. (szetszwo) - - HDFS-4346. Add SequentialNumber as a base class for INodeId and - GenerationStamp. (szetszwo) - - HDFS-4721. Speed up lease recovery by avoiding stale datanodes and choosing - the datanode with the most recent heartbeat as the primary. (Varun Sharma - via szetszwo) - - HDFS-4804. WARN when users set the block balanced preference percent below - 0.5 or above 1.0. (Stephen Chu via atm) - - HDFS-4698. Provide client-side metrics for remote reads, local reads, and - short-circuit reads. (Colin Patrick McCabe via atm) - - HDFS-3498. Support replica removal in BlockPlacementPolicy and make - BlockPlacementPolicyDefault extensible for reusing code in subclasses. - (Junping Du via szetszwo) - - HDFS-4234. Use generic code for choosing datanode in Balancer. (szetszwo) - - HDFS-4880. Print the image and edits file loaded by the namenode in the - logs. (Arpit Agarwal via suresh) - - HDFS-4914. Use DFSClient.Conf instead of Configuration. (szetszwo) - - HDFS-4883. complete() should verify fileId. (Tao Luo via shv) - - HDFS-4772. Add number of children in HdfsFileStatus. (brandonli) - - HDFS-4932. Avoid a wide line on the name node webUI if we have more Journal - nodes. (Fengdong Yu via cnauroth) - - HDFS-4908. Reduce snapshot inode memory usage. (szetszwo) - - HDFS-4645. Move from randomly generated block ID to sequentially generated - block ID. (Arpit Agarwal via szetszwo) - - HDFS-4912. Cleanup FSNamesystem#startFileInternal. (suresh) - - HDFS-4903. Print trash configuration and trash emptier state in - namenode log. (Arpit Agarwal via suresh) - - HDFS-4992. Make balancer's mover thread count and dispatcher thread count - configurable. (Max Lapan via szetszwo) - - HDFS-4996. ClientProtocol#metaSave can be made idempotent by overwriting the - output file instead of appending to it. (cnauroth) - - HADOOP-9418. Add symlink support to DistributedFileSystem (Andrew Wang via - Colin Patrick McCabe) - - HDFS-5007. Replace hard-coded property keys with DFSConfigKeys fields. - (Kousuke Saruta via jing9) - - HDFS-5008. Make ClientProtocol#abandonBlock() idempotent. (jing9) - - HADOOP-9760. Move GSet and related classes to common from HDFS. - (suresh) - - HDFS-5020. Make DatanodeProtocol#blockReceivedAndDeleted idempotent. - (jing9) - - HDFS-5024. Make DatanodeProtocol#commitBlockSynchronization idempotent. - (Arpit Agarwal via jing9) - - HDFS-3880. Use Builder to build RPC server in HDFS. - (Brandon Li via suresh) - - OPTIMIZATIONS - - HDFS-4465. Optimize datanode ReplicasMap and ReplicaInfo. (atm) - - HDFS-5027. On startup, DN should scan volumes in parallel. (atm) - - BUG FIXES - - HDFS-4626. ClientProtocol#getLinkTarget should throw an exception for - non-symlink and non-existent paths. (Andrew Wang via cmccabe) - - HDFS-4470. Several HDFS tests attempt file operations on invalid HDFS - paths when running on Windows. (Chris Nauroth via suresh) - - HDFS-4471. Namenode WebUI file browsing does not work with wildcard - addresses configured. (Andrew Wang via atm) - - HDFS-4342. Directories configured in dfs.namenode.edits.dir.required - but not in dfs.namenode.edits.dir are silently ignored. (Arpit Agarwal - via szetszwo) - - HDFS-4482. ReplicationMonitor thread can exit with NPE due to the race - between delete and replication of same file. (umamahesh) - - HDFS-4269. Datanode rejects all datanode registrations from localhost - in single-node developer setup on Windows. (Chris Nauroth via suresh) - - HDFS-4235. When outputting XML, OfflineEditsViewer can't handle some edits - containing non-ASCII strings. (Colin Patrick McCabe via atm) - - HDFS-4541. Set hadoop.log.dir and hadoop.id.str when starting secure - datanode to write the logs to right dir by default. (Arpit Gupta via - suresh) - - HDFS-4540. Namenode http server should use the web authentication - keytab for spnego principal. (Arpit Gupta via suresh) - - HDFS-4544. Error in deleting blocks should not do check disk, for - all types of errors. (Arpit Agarwal via suresh) - - HDFS-4565. Use DFSUtil.getSpnegoKeytabKey() to get the spnego keytab key - in secondary namenode and namenode http server. (Arpit Gupta via suresh) - - HDFS-4571. WebHDFS should not set the service hostname on the server side. - (tucu) - - HDFS-4013. TestHftpURLTimeouts throws NPE. (Chao Shi via suresh) - - HDFS-4592. Default values for access time precision are out of sync between - hdfs-default.xml and the code. (atm) - - HDFS-4522. LightWeightGSet expects incrementing a volatile to be atomic. - (Colin Patrick McCabe via atm) - - HDFS-4484. libwebhdfs compilation broken with gcc 4.6.2. (Colin Patrick - McCabe via atm) - - HDFS-4595. When short circuit read is fails, DFSClient does not fallback - to regular reads. (suresh) - - HDFS-4583. TestNodeCount fails. (Ivan Mitic via suresh) - - HDFS-4591. HA clients can fail to fail over while Standby NN is performing - long checkpoint. (atm) - - HDFS-3277. fail over to loading a different FSImage if the first one we - try to load is corrupt. (Colin Patrick McCabe and Andrew Wang via atm) - - HDFS-4596. Shutting down namenode during checkpointing can lead to md5sum - error. (Andrew Wang via atm) - - HDFS-4614. FSNamesystem#getContentSummary should use getPermissionChecker - helper method. (atm) - - HDFS-4620. Documentation for dfs.namenode.rpc-address specifies wrong - format. (Sandy Ryza via atm) - - HDFS-4609. TestAuditLogs should release log handles between tests. - (Ivan Mitic via szetszwo) - - HDFS-4598. Fix the default value of ConcatSourcesParam and the WebHDFS doc. - (szetszwo) - - HDFS-4655. DNA_FINALIZE is logged as being an unknown command by the DN - when received from the standby NN. (atm) - - HDFS-4656. DN heartbeat loop can be briefly tight. (atm) - - HDFS-4658. Standby NN will log that it has received a block report "after - becoming active" (atm) - - HDFS-4646. createNNProxyWithClientProtocol ignores configured timeout - value (Jagane Sundar via cos) - - HDFS-3981. Fix handling of FSN lock in getBlockLocations. (Xiaobo Peng - and todd via todd) - - HDFS-4676. TestHDFSFileSystemContract should set MiniDFSCluster variable - to null to free up memory. (suresh) - - HDFS-4669. TestBlockPoolManager fails using IBM java. (Tian Hong Wang via - suresh) - - HDFS-4643. Fix flakiness in TestQuorumJournalManager. (todd) - - HDFS-4639. startFileInternal() should not increment generation stamp. - (Plamen Jeliazkov via shv) - - HDFS-4695. TestEditLog leaks open file handles between tests. - (Ivan Mitic via suresh) - - HDFS-4737. JVM path embedded in fuse binaries. (Sean Mackrory via atm) - - HDFS-4739. NN can miscalculate the number of extra edit log segments to - retain. (atm) - - HDFS-4745. TestDataTransferKeepalive#testSlowReader has race condition that - causes sporadic failure. (Chris Nauroth via suresh) - - HDFS-4768. File handle leak in datanode when a block pool is removed. - (Chris Nauroth via suresh) - - HDFS-4748. MiniJournalCluster#restartJournalNode leaks resources, which - causes sporadic test failures. (Chris Nauroth via suresh) - - HDFS-4733. Make HttpFS username pattern configurable. (tucu via atm) - - HDFS-4778. Fixes some issues that the first patch on HDFS-2576 missed. - (ddas) - - HDFS-4785. Concat operation does not remove concatenated files from - InodeMap. (suresh) - - HDFS-4784. NPE in FSDirectory.resolvePath(). (Brandon Li via suresh) - - HDFS-4810. several HDFS HA tests have timeouts that are too short. (Chris - Nauroth via atm) - - HDFS-4799. Corrupt replica can be prematurely removed from - corruptReplicas map. (todd via kihwal) - - HDFS-4751. TestLeaseRenewer#testThreadName flakes. (Andrew Wang via atm) - - HDFS-4533. start-dfs.sh ignores additional parameters besides -upgrade. - (Fengdong Yu via suresh) - - HDFS-4765. Permission check of symlink deletion incorrectly throws - UnresolvedLinkException. (Andrew Wang via atm) - - HDFS-4300. TransferFsImage.downloadEditsToStorage should use a tmp file for - destination. (Andrew Wang via atm) - - HDFS-4813. Add volatile to BlocksMap.blocks so that the replication thread - can see the updated value. (Jing Zhao via szetszwo) - - HDFS-3180. Add socket timeouts to WebHdfsFileSystem. (Chris Nauroth via - szetszwo) - - HDFS-4787. Create a new HdfsConfiguration before each TestDFSClientRetries - testcases. (Tian Hong Wang via atm) - - HDFS-4830. Typo in config settings for AvailableSpaceVolumeChoosingPolicy - in hdfs-default.xml. (atm) - - HDFS-4824. FileInputStreamCache.close leaves dangling reference to - FileInputStreamCache.cacheCleaner. (Colin Patrick McCabe via todd) - - HDFS-4298. StorageRetentionManager spews warnings when used with QJM. (atm) - - HDFS-4725. Fix HDFS file handle leaks in FSEditLog, NameNode, - OfflineEditsBinaryLoader and some tests. (Chris Nauroth via szetszwo) - - HDFS-4825. webhdfs / httpfs tests broken because of min block size change. - (Andrew Wang via suresh) - - HDFS-4780. Use the correct relogin method for services. (Robert Parker via - kihwal) - - HDFS-4827. Slight update to the implementation of API for handling favored - nodes in DFSClient (ddas) - - HDFS-4865. Remove sub resource warning from httpfs log at startup time. - (ywskycn via tucu) - - HDFS-4240. For nodegroup-aware block placement, when a node is excluded, - the nodes in the same nodegroup should also be excluded. (Junping Du - via szetszwo) - - HDFS-4261. Fix bugs in Balaner causing infinite loop and - TestBalancerWithNodeGroup timeing out. (Junping Du via szetszwo) - - HDFS-4382. Fix typo MAX_NOT_CHANGED_INTERATIONS. (Ted Yu via suresh) - - HDFS-4840. ReplicationMonitor gets NPE during shutdown. (kihwal) - - HDFS-4815. TestRBWBlockInvalidation: Double call countReplicas() to fetch - corruptReplicas and liveReplicas is not needed. (Tian Hong Wang via atm) - - HADOOP-8957 HDFS tests for AbstractFileSystem#IsValidName should be overridden for - embedded file systems like ViewFs (Chris Nauroth via Sanjay Radia) - - HDFS-4586. TestDataDirs.testGetDataDirsFromURIs fails with all directories - in dfs.datanode.data.dir are invalid. (Ivan Mitic via atm) - - HDFS-4845. FSNamesystem.deleteInternal should acquire write-lock before - changing the inode map. (Arpit Agarwal via szetszwo) - - HDFS-4906. HDFS Output streams should not accept writes after being - closed. (atm) - - HDFS-4917. Start-dfs.sh cannot pass the parameters correctly. - (Fengdong Yu via suresh) - - HDFS-4205. fsck fails with symlinks. (jlowe) - - HDFS-4927. CreateEditsLog creates inodes with an invalid inode ID, which then - cannot be loaded by a namenode. (cnauroth) - - HDFS-4944. WebHDFS cannot create a file path containing characters that must - be URI-encoded, such as space. (cnauroth) - - HDFS-4888. Refactor and fix FSNamesystem.getTurnOffTip. (Ravi Prakash via - kihwal) - - HDFS-4943. WebHdfsFileSystem does not work when original file path has - encoded chars. (Jerry He via szetszwo) - - HDFS-4948. mvn site for hadoop-hdfs-nfs fails. (brandonli) - - HDFS-4887. TestNNThroughputBenchmark exits abruptly. (kihwal) - - HDFS-4980. Incorrect logging.properties file for hadoop-httpfs. - (Mark Grover via suresh) - - HDFS-5003. TestNNThroughputBenchmark failed caused by existing directories. - (Xi Fang via cnauroth) - - HDFS-5018. Misspelled DFSConfigKeys#DFS_NAMENODE_STALE_DATANODE_INTERVAL_DEFAULT - in javadoc of DatanodeInfo#isStale(). (Ted Yu via jing9) - - HDFS-4602. TestBookKeeperHACheckpoints fails. (umamahesh) - - HDFS-5016. Deadlock in pipeline recovery causes Datanode to be marked dead. - (suresh) - - HDFS-5228. The RemoteIterator returned by DistributedFileSystem.listFiles - may throw NullPointerException. (szetszwo and cnauroth via szetszwo) - - BREAKDOWN OF HDFS-347 SUBTASKS AND RELATED JIRAS - - HDFS-4353. Encapsulate connections to peers in Peer and PeerServer classes. - (Colin Patrick McCabe via todd) - - HDFS-4354. Create DomainSocket and DomainPeer and associated unit tests. - (Colin Patrick McCabe via todd) - - HDFS-4356. BlockReaderLocal should use passed file descriptors rather than paths. - (Colin Patrick McCabe via todd) - - HDFS-4388. DomainSocket should throw AsynchronousCloseException when appropriate. - (Colin Patrick McCabe via todd) - - HDFS-4390. Bypass UNIX domain socket unit tests when they cannot be run. - (Colin Patrick McCabe via todd) - - HDFS-4400. DFSInputStream#getBlockReader: last retries should ignore the cache - (Colin Patrick McCabe via todd) - - HDFS-4401. Fix bug in DomainSocket path validation - (Colin Patrick McCabe via todd) - - HDFS-4402. Some small DomainSocket fixes: avoid findbugs warning, change - log level, etc. (Colin Patrick McCabe via todd) - - HDFS-4418. increase default FileInputStreamCache size (todd) - - HDFS-4416. Rename dfs.datanode.domain.socket.path to dfs.domain.socket.path - (Colin Patrick McCabe via todd) - - HDFS-4417. Fix case where local reads get disabled incorrectly - (Colin Patrick McCabe and todd via todd) - - HDFS-4433. Make TestPeerCache not flaky (Colin Patrick McCabe via todd) - - HDFS-4438. TestDomainSocket fails when system umask is set to 0002. (Colin - Patrick McCabe via atm) - - HDFS-4440. Avoid annoying log message when dfs.domain.socket.path is not - set. (Colin Patrick McCabe via atm) - - HDFS-4473. Don't create domain socket unless we need it. (Colin Patrick McCabe via atm) - - HDFS-4485. DN should chmod socket path a+w. (Colin Patrick McCabe via atm) - - HDFS-4453. Make a simple doc to describe the usage and design of the - shortcircuit read feature. (Colin Patrick McCabe via atm) - - HDFS-4496. DFSClient: don't create a domain socket unless we need it (Colin - Patrick McCabe via todd) - - HDFS-347: style cleanups (Colin Patrick McCabe via atm) - - HDFS-4538. Allow use of legacy blockreader (Colin Patrick McCabe via todd) - - HDFS-4661. A few little code cleanups of some HDFS-347-related code. (Colin - Patrick McCabe via atm) - - BREAKDOWN OF HADOOP-8562 and HDFS-3602 SUBTASKS AND RELATED JIRAS - - HDFS-4145. Merge hdfs cmd line scripts from branch-1-win. (David Lao, - Bikas Saha, Lauren Yang, Chuan Liu, Thejas M Nair and Ivan Mitic via suresh) - - HDFS-4163. HDFS distribution build fails on Windows. (Chris Nauroth via - suresh) - - HDFS-4316. branch-trunk-win contains test code accidentally added during - work on fixing tests on Windows. (Chris Nauroth via suresh) - - HDFS-4297. Fix issues related to datanode concurrent reading and writing on - Windows. (Arpit Agarwal, Chuan Liu via suresh) - - HDFS-4573. Fix TestINodeFile on Windows. (Arpit Agarwal via suresh) - - HDFS-4572. Fix TestJournal failures on Windows. (Arpit Agarwal via suresh) - - HDFS-4287. HTTPFS tests fail on Windows. (Chris Nauroth via suresh) - - HDFS-4593. TestSaveNamespace fails on Windows. (Arpit Agarwal via suresh) - - HDFS-4582. TestHostsFiles fails on Windows. (Ivan Mitic via suresh) - - HDFS-4603. TestMiniDFSCluster fails on Windows. (Ivan Mitic via suresh) - - HDFS-4604. TestJournalNode fails on Windows. (Ivan Mitic via suresh) - - HDFS-4607. In TestGetConf.testGetSpecificKey(), use a platform-specific - line separator; otherwise, it fails on Windows. (Ivan Mitic via szetszwo) - - HDFS-4625. Make TestNNWithQJM#testNewNamenodeTakesOverWriter work on - Windows. (Ivan Mitic via suresh) - - HDFS-4674. TestBPOfferService fails on Windows due to failure parsing - datanode data directory as URI. (Chris Nauroth via suresh) - - HDFS-4615. Fix TestDFSShell failures on Windows. (Arpit Agarwal - via szetszwo) - - HDFS-4584. Skip TestNNWithQJM.testNewNamenodeTakesOverWriter() on Windows. - (Arpit Agarwal via szetszwo) - - HDFS-4732. Fix TestDFSUpgradeFromImage which fails on Windows due to - failure to unpack old image tarball that contains hard links. - (Chris Nauroth via szetszwo) - - HDFS-4741. TestStorageRestore#testStorageRestoreFailure fails on Windows. - (Arpit Agarwal via suresh) - - HDFS-4743. TestNNStorageRetentionManager fails on Windows. - (Chris Nauroth via suresh) - - HDFS-4740. Fixes for a few test failures on Windows. - (Arpit Agarwal via suresh) - - HDFS-4722. TestGetConf#testFederation times out on Windows. - (Ivan Mitic via suresh) - - HDFS-4705. Address HDFS test failures on Windows because of invalid - dfs.namenode.name.dir. (Ivan Mitic via suresh) - - HDFS-4734. HDFS Tests that use ShellCommandFencer are broken on Windows. - (Arpit Agarwal via suresh) - - HDFS-4610. Use common utils FileUtil#setReadable/Writable/Executable and - FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) - - HDFS-4677. Editlog should support synchronous writes. (ivanmi) - - HDFS-4752. TestRBWBlockInvalidation fails on Windows due to file locking. - (Chris Nauroth via suresh) - - HDFS-4783. TestDelegationTokensWithHA#testHAUtilClonesDelegationTokens fails - on Windows. (cnauroth) - - HDFS-4818. Several HDFS tests that attempt to make directories unusable do - not work correctly on Windows. (cnauroth) - - BREAKDOWN OF HDFS-2802 HDFS SNAPSHOT SUBTASKS AND RELATED JIRAS - - HDFS-4076. Support snapshot of single files. (szetszwo) - - HDFS-4082. Add editlog opcodes for snapshot create and delete operations. - (suresh via szetszwo) - - HDFS-4086. Add editlog opcodes to allow and disallow snapshots on a - directory. (Brandon Li via suresh) - - HDFS-4083. Protocol changes for snapshots. (suresh) - - HDFS-4077. Add support for Snapshottable Directory. (szetszwo via suresh) - - HDFS-4087. Protocol changes for listSnapshots functionality. - (Brandon Li via suresh) - - HDFS-4079. Add SnapshotManager which maintains a list for all the - snapshottable directories and supports snapshot methods such as setting a - directory to snapshottable and creating a snapshot. (szetszwo) - - HDFS-4078. Handle replication in snapshots. (szetszwo) - - HDFS-4084. Provide CLI support to allow and disallow snapshot - on a directory. (Brondon Li via suresh) - - HDFS-4091. Add snapshot quota to limit the number of snapshots allowed. - (szetszwo) - - HDFS-4097. Provide CLI support for createSnapshot. (Brandon Li via suresh) - - HDFS-4092. Update file deletion logic for snapshot so that the current inode - is removed from the circular linked list; and if some blocks at the end of - the block list no longer belong to any other inode, collect them and update - the block list. (szetszwo) - - HDFS-4111. Support snapshot of subtrees. (szetszwo via suresh) - - HDFS-4119. Complete the allowSnapshot code and add a test for it. (szetszwo) - - HDFS-4133. Add testcases for testing basic snapshot functionalities. - (Jing Zhao via suresh) - - HDFS-4116. Add auditlog for some snapshot operations. (Jing Zhao via suresh) - - HDFS-4095. Add some snapshot related metrics. (Jing Zhao via suresh) - - HDFS-4141. Support directory diff - the difference between the current state - and a previous snapshot of an INodeDirectory. (szetszwo) - - HDFS-4146. Use getter and setter in INodeFileWithLink to access blocks and - initialize root directory as snapshottable. (szetszwo) - - HDFS-4149. Implement the disallowSnapshot(..) in FSNamesystem and add - resetSnapshottable(..) to SnapshotManager. (szetszwo) - - HDFS-4147. When there is a snapshot in a subtree, deletion of the subtree - should fail. (Jing Zhao via szetszwo) - - HDFS-4150. Update the inode in the block map when a snapshotted file or a - snapshot file is deleted. (Jing Zhao via szetszwo) - - HDFS-4159. Rename should fail when the destination directory is - snapshottable and has snapshots. (Jing Zhao via szetszwo) - - HDFS-4170. Add snapshot information to INodesInPath. (szetszwo) - - HDFS-4177. Add a snapshot parameter to INodeDirectory.getChildrenList() for - selecting particular snapshot children list views. (szetszwo) - - HDFS-4148. Disallow write/modify operations on files and directories in a - snapshot. (Brandon Li via suresh) - - HDFS-4188. Add Snapshot.ID_COMPARATOR for comparing IDs and fix a bug in - ReadOnlyList.Util.binarySearch(..). (szetszwo) - - HDFS-4187. Add tests for replication handling in snapshots. (Jing Zhao via - szetszwo) - - HDFS-4196. Support renaming of snapshots. (Jing Zhao via szetszwo) - - HDFS-4175. Additional snapshot tests for more complicated directory - structure and modifications. (Jing Zhao via suresh) - - HDFS-4293. Fix TestSnapshot failure. (Jing Zhao via suresh) - - HDFS-4317. Change INode and its subclasses to support HDFS-4103. (szetszwo) - - HDFS-4103. Support O(1) snapshot creation. (szetszwo) - - HDFS-4330. Support snapshots up to the snapshot limit. (szetszwo) - - HDFS-4357. Fix a bug that if an inode is replaced, further INode operations - should apply to the new inode. (Jing Zhao via szetszwo) - - HDFS-4230. Support listing of all the snapshottable directories. (Jing Zhao - via szetszwo) - - HDFS-4244. Support snapshot deletion. (Jing Zhao via szetszwo) - - HDFS-4245. Include snapshot related operations in TestOfflineEditsViewer. - (Jing Zhao via szetszwo) - - HDFS-4395. In INodeDirectorySnapshottable's constructor, the passed-in dir - could be an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo) - - HDFS-4397. Fix a bug in INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) - that it may put the wrong node into the deleted list. (szetszwo) - - HDFS-4407. Change INodeDirectoryWithSnapshot.Diff.combinePostDiff(..) to - merge-sort like and keep the postDiff parameter unmodified. (szetszwo) - - HDFS-4098. Add FileWithSnapshot, INodeFileUnderConstructionWithSnapshot and - INodeFileUnderConstructionSnapshot for supporting append to snapshotted - files. (szetszwo) - - HDFS-4126. Add reading/writing snapshot information to FSImage. - (Jing Zhao via suresh) - - HDFS-4436. Change INode.recordModification(..) to return only the current - inode and remove the updateCircularList parameter from some methods in - INodeDirectoryWithSnapshot.Diff. (szetszwo) - - HDFS-4429. When the latest snapshot exists, INodeFileUnderConstruction - should be replaced with INodeFileWithSnapshot but not INodeFile. - (Jing Zhao via szetszwo) - - HDFS-4441. Move INodeDirectoryWithSnapshot.Diff and the related classes to a - package. (szetszwo) - - HDFS-4432. Support INodeFileUnderConstructionWithSnapshot in FSImage - saving/loading. (Jing Zhao via suresh) - - HDFS-4131. Add capability to namenode to get snapshot diff. (Jing Zhao via - suresh) - - HDFS-4447. Refactor INodeDirectoryWithSnapshot for supporting general INode - diff lists. (szetszwo) - - HDFS-4189. Renames the getMutableXxx methods to getXxx4Write and fix a bug - that some getExistingPathINodes calls should be getINodesInPath4Write. - (szetszwo) - - HDFS-4361. When listing snapshottable directories, only return those - where the user has permission to take snapshots. (Jing Zhao via szetszwo) - - HDFS-4464. Combine collectSubtreeBlocksAndClear with deleteDiffsForSnapshot - and rename it to destroySubtreeAndCollectBlocks. (szetszwo) - - HDFS-4414. Add support for getting snapshot diff from DistributedFileSystem. - (Jing Zhao via suresh) - - HDFS-4446. Support file snapshots with diff lists. (szetszwo) - - HDFS-4480. Eliminate the file snapshot circular linked list. (szetszwo) - - HDFS-4481. Change fsimage to support snapshot file diffs. (szetszwo) - - HDFS-4500. Refactor snapshot INode methods. (szetszwo) - - HDFS-4487. Fix snapshot diff report for HDFS-4446. (Jing Zhao via szetszwo) - - HDFS-4431. Support snapshot in OfflineImageViewer. (Jing Zhao via szetszwo) - - HDFS-4503. Update computeContentSummary(..), spaceConsumedInTree(..) and - diskspaceConsumed(..) in INode for snapshot. (szetszwo) - - HDFS-4499. Fix file/directory/snapshot deletion for file diff. (Jing Zhao - via szetszwo) - - HDFS-4524. Update SnapshotManager#snapshottables when loading fsimage. - (Jing Zhao via szetszwo) - - HDFS-4520. Support listing snapshots under a snapshottable directory using - ls. (Jing Zhao via szetszwo) - - HDFS-4514. Add CLI for supporting snapshot rename, diff report, and - snapshottable directory listing. (Jing Zhao via szetszwo) - - HDFS-4523. Fix INodeFile replacement, TestQuota and javac errors from trunk - merge. (szetszwo) - - HDFS-4507. Update quota verification for snapshots. (szetszwo) - - HDFS-4545. With snapshots, FSDirectory.unprotectedSetReplication(..) always - changes file replication but it may or may not changes block replication. - (szetszwo) - - HDFS-4557. Fix FSDirectory#delete when INode#cleanSubtree returns 0. - (Jing Zhao via szetszwo) - - HDFS-4579. Annotate snapshot tests. (Arpit Agarwal via suresh) - - HDFS-4574. Move Diff to the util package. (szetszwo) - - HDFS-4563. Update namespace/diskspace usage after deleting snapshots. - (Jing Zhao via szetszwo) - - HDFS-4144. Create test for all snapshot-related metrics. - (Jing Zhao via suresh) - - HDFS-4556. Add snapshotdiff and LsSnapshottableDir tools to hdfs script. - (Arpit Agarwal via szetszwo) - - HDFS-4534. Add INodeReference in order to support rename with snapshots. - (szetszwo) - - HDFS-4616. Update the FilesDeleted metric while deleting file/dir in the - current tree. (Jing Zhao via szetszwo) - - HDFS-4627. Fix FSImageFormat#Loader NPE and synchronization issues. - (Jing Zhao via suresh) - - HDFS-4612. Not to use INode.getParent() when generating snapshot diff - report. (Jing Zhao via szetszwo) - - HDFS-4636. Update quota usage when deleting files/dirs that were created - after taking the latest snapshot. (Jing Zhao via szetszwo) - - HDFS-4648. For snapshot deletion, when merging the diff from to-delete - snapshot to the prior snapshot, make sure files/directories created after - the prior snapshot get deleted. (Jing Zhao via szetszwo) - - HDFS-4637. INodeDirectory#replaceSelf4Quota may incorrectly convert a newly - created directory to an INodeDirectoryWithSnapshot. (Jing Zhao via szetszwo) - - HDFS-4611. Update FSImage for INodeReference. (szetszwo) - - HDFS-4647. Rename should call setLocalName after an inode is removed from - snapshots. (Arpit Agarwal via szetszwo) - - HDFS-4684. Use INode id for image serialization when writing INodeReference. - (szetszwo) - - HDFS-4675. Fix rename across snapshottable directories. (Jing Zhao via - szetszwo) - - HDFS-4692. Use timestamp as default snapshot names. (szetszwo) - - HDFS-4666. Define ".snapshot" as a reserved inode name so that users cannot - create a file/directory with ".snapshot" as the name. If ".snapshot" is - used in a previous version of HDFS, it must be renamed before upgrade; - otherwise, upgrade will fail. (szetszwo) - - HDFS-4700. Fix the undo section of rename with snapshots. (Jing Zhao via - szetszwo) - - HDFS-4529. Disallow concat when one of the src files is in some snapshot. - (szetszwo) - - HDFS-4550. Refactor INodeDirectory.INodesInPath to a standalone class. - (szetszwo) - - HDFS-4707. Add snapshot methods to FilterFileSystem and fix findbugs - warnings. (szetszwo) - - HDFS-4706. Do not replace root inode for disallowSnapshot. (szetszwo) - - HDFS-4717. Change the path parameter type of the snapshot methods in - HdfsAdmin from String to Path. (szetszwo) - - HDFS-4708. Add snapshot user documentation. (szetszwo) - - HDFS-4726. Fix test failures after merging the INodeId-INode mapping - from trunk. (Jing Zhao via szetszwo) - - HDFS-4727. Update inodeMap after deleting files/directories/snapshots. - (Jing Zhao via szetszwo) - - HDFS-4719. Remove AbstractINodeDiff.Factory and move its methods to - AbstractINodeDiffList. (Arpit Agarwal via szetszwo) - - HDFS-4735. DisallowSnapshot throws IllegalStateException for nested - snapshottable directories. (Jing Zhao via szetszwo) - - HDFS-4738. Changes AbstractINodeDiff to implement Comparable, and - fix javadoc and other warnings. (szetszwo) - - HDFS-4686. Update quota computation for rename and INodeReference. - (Jing Zhao via szetszwo) - - HDFS-4729. Fix OfflineImageViewer and permission checking for snapshot - operations. (Jing Zhao via szetszwo) - - HDFS-4749. Use INodeId to identify the corresponding directory node in - FSImage saving/loading. (Jing Zhao via szetszwo) - - HDFS-4742. Fix appending to a renamed file with snapshot. (Jing Zhao via - szetszwo) - - HDFS-4755. Fix AccessControlException message and moves "implements - LinkedElement" from INode to INodeWithAdditionalFields. (szetszwo) - - HDFS-4650. Fix a bug in FSDirectory and add more unit tests for rename with - existence of snapshottable directories and snapshots. (Jing Zhao via - szetszwo) - - HDFS-4650. When passing two non-existing snapshot names to snapshotDiff, it - returns success if the names are the same. (Jing Zhao via szetszwo) - - HDFS-4767. If a directory is snapshottable, do not replace it when clearing - quota. (Jing Zhao via szetszwo) - - HDFS-4578. Restrict snapshot IDs to 24-bit wide. (Arpit Agarwal via - szetszwo) - - HDFS-4773. Fix bugs in quota usage computation and OfflineImageViewer. - (Jing Zhao via szetszwo) - - HDFS-4760. Update inodeMap after node replacement. (Jing Zhao via szetszwo) - - HDFS-4758. Disallow nested snapshottable directories and unwrap - RemoteException. (szetszwo) - - HDFS-4781. Fix a NullPointerException when listing .snapshot under - a non-existing directory. (szetszwo) - - HDFS-4791. Update and fix deletion of reference inode. (Jing Zhao via - szetszwo) - - HDFS-4798. Update computeContentSummary() for the reference nodes in - snapshots. (szetszwo) - - HDFS-4800. Fix INodeDirectoryWithSnapshot#cleanDeletedINode. (Jing Zhao via - szetszwo) - - HDFS-4801. lsSnapshottableDir throws IllegalArgumentException when root is - snapshottable. (Jing Zhao via szetszwo) - - HDFS-4802. Disallowing snapshot on / twice should throw SnapshotException - but not IllegalStateException. (Jing Zhao via szetszwo) - - HDFS-4806. In INodeDirectoryWithSnapshot, use isInLatestSnapshot() to - determine if an added/removed child should be recorded in the snapshot diff. - (Jing Zhao via szetszwo) - - HDFS-4809. When a QuotaExceededException is thrown during rename, the quota - usage should be subtracted back. (Jing Zhao via szetszwo) - - HDFS-4842. Identify the correct prior snapshot when deleting a - snapshot under a renamed subtree. (jing9) - - HDFS-4846. Clean up snapshot CLI commands output stacktrace for invalid - arguments. (Jing Zhao via brandonli) - - HDFS-4857. Snapshot.Root and AbstractINodeDiff#snapshotINode should not be - put into INodeMap when loading FSImage. (jing9) - - HDFS-4863. The root directory should be added to the snapshottable - directory list while loading fsimage. (jing9) - - HDFS-4848. copyFromLocal and renaming a file to ".snapshot" should output - that ".snapshot" is a reserved name. (Jing Zhao via brandonli) - - HDFS-4826. TestNestedSnapshots times out due to repeated slow edit log - flushes when running on virtualized disk. (Chris Nauroth via szetszwo) - - HDFS-4876. Fix the javadoc of FileWithSnapshot and move FileDiffList to - FileWithSnapshot. (szetszwo) - - HDFS-4850. Fix OfflineImageViewer to work on fsimages with empty files or - snapshots. (jing9) - - HDFS-4877. Snapshot: fix the scenario where a directory is renamed under - its prior descendant. (jing9) - - HDFS-4873. callGetBlockLocations returns incorrect number of blocks for - snapshotted files. (jing9) - - HDFS-4819. Update Snapshot doc to clarify that nested snapshots are not - allowed. (szetszwo) - - HDFS-4902. DFSClient.getSnapshotDiffReport should use string path rather - than o.a.h.fs.Path. (Binglin Chang via jing9) - - HDFS-4875. Add a test for testing snapshot file length. - (Arpit Agarwal via jing9) - - HDFS-4841. FsShell commands using secure webhfds fail ClientFinalizer - shutdown hook. (rkanter via tucu) - - HDFS-4951. FsShell commands using secure httpfs throw exceptions due - to missing TokenRenewer. (rknater via tucu) - - HDFS-4969. WebhdfsFileSystem expects non-standard WEBHDFS Json element. - (rkanter via tucu) - - HDFS-4797. BlockScanInfo does not override equals(..) and hashCode() - consistently. (szetszwo) - - HDFS-4978. Make disallowSnapshot idempotent. (jing9) - - HDFS-5005. Move SnapshotException and SnapshotAccessControlException - to o.a.h.hdfs.protocol. (jing9) - - HDFS-4982. JournalNode should relogin from keytab before fetching logs - from other JNs (todd) - -Release 2.0.6-alpha - 08/22/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - -Release 2.0.5-alpha - 06/06/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.0.4-alpha - 2013-04-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.0.3-alpha - 2013-02-06 - - INCOMPATIBLE CHANGES - - HDFS-4122. Cleanup HDFS logs and reduce the size of logged messages. - (suresh) - - HDFS-4362. GetDelegationTokenResponseProto does not handle null token. - (suresh) - - HDFS-4367. GetDataEncryptionKeyResponseProto does not handle null - response. (suresh) - - HDFS-4364. GetLinkTargetResponseProto does not handle null path. (suresh) - - HDFS-4369. GetBlockKeysResponseProto does not handle null response. - (suresh) - - HDFS-4451. hdfs balancer command returns exit code 1 on success instead - of 0. (Joshua Blatt via suresh) - - HDFS-4350. Make enabling of stale marking on read and write paths - independent. (Andrew Wang via suresh) - - - NEW FEATURES - - HDFS-2656. Add libwebhdfs, a pure C client based on WebHDFS. - (Jaimin D Jetly and Jing Zhao via szetszwo) - - HDFS-3912. Detect and avoid stale datanodes for writes. - (Jing Zhao via suresh) - - HDFS-4059. Add number of stale DataNodes to metrics. (Jing Zhao via suresh) - - HDFS-4155. libhdfs implementation of hsync API (Liang Xie via todd) - - HDFS-4213. Add an API to hsync for updating the last block length at the - namenode. (Jing Zhao via szetszwo) - - HDFS-3077. Implement QuorumJournalManager, a distributed mechanism for - reliably storing HDFS edit logs. See dedicated section below for breakdown - of subtasks. - - IMPROVEMENTS - - HDFS-3925. Prettify PipelineAck#toString() for printing to a log - (Andrew Wang via todd) - - HDFS-3939. NN RPC address cleanup. (eli) - - HDFS-3373. Change DFSClient input stream socket cache to global static and - add a thread to cleanup expired cache entries. (John George via szetszwo) - - HDFS-3896. Add descriptions for dfs.namenode.rpc-address and - dfs.namenode.servicerpc-address to hdfs-default.xml. (Jeff Lord via atm) - - HDFS-3996. Add debug log removed in HDFS-3873 back. (eli) - - HDFS-3916. libwebhdfs (C client) code cleanups. - (Colin Patrick McCabe via eli) - - HDFS-3813. Log error message if security and WebHDFS are enabled but - principal/keytab are not configured. (Stephen Chu via atm) - - HDFS-3483. Better error message when hdfs fsck is run against a ViewFS - config. (Stephen Fritz via atm) - - HDFS-3682. MiniDFSCluster#init should provide more info when it fails. - (todd via eli) - - HDFS-4008. TestBalancerWithEncryptedTransfer needs a timeout. (eli) - - HDFS-4007. Rehabilitate bit-rotted unit tests under - hadoop-hdfs-project/hadoop-hdfs/src/test/unit/ - (Colin Patrick McCabe via todd) - - HDFS-4041. Hadoop HDFS Maven protoc calls must not depend on external - sh script. (Chris Nauroth via suresh) - - HADOOP-8911. CRLF characters in source and text files. - (Raja Aluri via suresh) - - HDFS-4037. Rename the getReplication() method in BlockCollection to - getBlockReplication(). (szetszwo) - - HDFS-4036. Remove "throws UnresolvedLinkException" from - FSDirectory.unprotectedAddFile(..). (Jing Zhao via szetszwo) - - HDFS-2946. HA: Put a cap on the number of completed edits files retained - by the NN. (atm) - - HDFS-4029. GenerationStamp should use an AtomicLong. (eli) - - HDFS-4068. DatanodeID and DatanodeInfo member should be private. (eli) - - HDFS-4073. Two minor improvements to FSDirectory. (Jing Zhao via szetszwo) - - HDFS-4074. Remove the unused default constructor from INode. (Brandon Li - via szetszwo) - - HDFS-4088. Remove "throws QuotaExceededException" from an - INodeDirectoryWithQuota constructor. (szetszwo) - - HDFS-4099. Clean up replication code and add more javadoc. (szetszwo) - - HDFS-4107. Add utility methods for casting INode to INodeFile and - INodeFileUnderConstruction. (szetszwo) - - HDFS-4112. A few improvements on INodeDirectory include adding a utility - method for casting; avoiding creation of new empty lists; cleaning up - some code and rewriting some javadoc. (szetszwo) - - HDFS-4121. Add namespace declarations in hdfs .proto files for languages - other than java. (Binglin Chang via suresh) - - HDFS-3573. Supply NamespaceInfo when instantiating JournalManagers. - (todd and ivank via umamahesh) - - HDFS-3695. Genericize format() to non-file JournalManagers. - (todd via umamahesh) - - HDFS-3789. JournalManager#format() should be able to throw IOException. - (Ivan Kelly via umamahesh) - - HDFS-3809. Make BKJM use protobufs for all serialization with ZK. - (Ivan Kelly via umamhesh) - - HDFS-3916. libwebhdfs testing code cleanup. (Jing Zhao via suresh) - - HDFS-4143. Change blocks to private in INodeFile and renames isLink() to - isSymlink() in INode. (szetszwo) - - HDFS-4046. Rename ChecksumTypeProto enum NULL since it is illegal in - C/C++. (Binglin Chang via suresh) - - HDFS-4048. Use ERROR instead of INFO for volume failure logs. - (Stephen Chu via eli) - - HDFS-1322. Document umask in DistributedFileSystem#mkdirs javadocs. - (Colin Patrick McCabe via eli) - - HDFS-4038. Override toString() for BookKeeperEditLogInputStream. - (Vinay via umamahesh) - - HDFS-4214. OfflineEditsViewer should print out the offset at which it - encountered an error. (Colin Patrick McCabe via atm) - - HDFS-4199. Provide test for HdfsVolumeId. (Ivan A. Veselovsky via atm) - - HDFS-3049. During the normal NN startup process, fall back on a different - edit log if we see one that is corrupt (Colin Patrick McCabe via todd) - - HDFS-3571. Allow EditLogFileInputStream to read from a remote URL (todd) - - HDFS-4110. Refine a log printed in JNStorage. (Liang Xie via suresh) - - HDFS-4153. Add START_MSG/SHUTDOWN_MSG for JournalNode. (liang xie via atm) - - HDFS-3935. Add JournalNode to the start/stop scripts (Andy Isaacson via todd) - - HDFS-4268. Remove redundant enum NNHAStatusHeartbeat.State. (shv) - - HDFS-3680. Allow customized audit logging in HDFS FSNamesystem. (Marcelo - Vanzin via atm) - - HDFS-4130. BKJM: The reading for editlog at NN starting using bkjm is not efficient. - (Han Xiao via umamahesh) - - HDFS-4326. bump up Tomcat version for HttpFS to 6.0.36. (tucu via acmurthy) - - HDFS-4270. Introduce soft and hard limits for max replication so that - replications of the highest priority are allowed to choose a source datanode - that has reached its soft limit but not the hard limit. (Derek Dagit via - szetszwo) - - HADOOP-9173. Add security token protobuf definition to common and - use it in hdfs. (suresh) - - HDFS-4030. BlockManager excessBlocksCount and - postponedMisreplicatedBlocksCount should be AtomicLongs. (eli) - - HDFS-4031. Update findbugsExcludeFile.xml to include findbugs 2 - exclusions. (eli) - - HDFS-4033. Miscellaneous findbugs 2 fixes. (eli) - - HDFS-4034. Remove redundant null checks. (eli) - - HDFS-4035. LightWeightGSet and LightWeightHashSet increment a - volatile without synchronization. (eli) - - HDFS-4032. Specify the charset explicitly rather than rely on the - default. (eli) - - HDFS-4363. Combine PBHelper and HdfsProtoUtil and remove redundant - methods. (suresh) - - HDFS-4377. Some trivial DN comment cleanup. (eli) - - HDFS-4381. Document fsimage format details in FSImageFormat class javadoc. - (Jing Zhao via suresh) - - HDFS-4375. Use token request messages defined in hadoop common. - (suresh) - - HDFS-4392. Use NetUtils#getFreeSocketPort in MiniDFSCluster. - (Andrew Purtell via suresh) - - HDFS-4393. Make empty request and responses in protocol translators can be - static final members. (Brandon Li via suresh) - - HDFS-4403. DFSClient can infer checksum type when not provided by reading - first byte (todd) - - HDFS-4259. Improve pipeline DN replacement failure message (harsh) - - HDFS-3598. WebHDFS support for file concat. (Plamen Jeliazkov via shv) - - HDFS-4456. Add concat to HttpFS and WebHDFS REST API docs. (plamenj2003 via tucu) - - OPTIMIZATIONS - - HDFS-3429. DataNode reads checksums even if client does not need them (todd) - - BUG FIXES - - HDFS-3919. MiniDFSCluster:waitClusterUp can hang forever. - (Andy Isaacson via eli) - - HDFS-3924. Multi-byte id in HdfsVolumeId. (Andrew Wang via atm) - - HDFS-3936. MiniDFSCluster shutdown races with BlocksMap usage. (eli) - - HDFS-3951. datanode web ui does not work over HTTPS when datanode is started in secure mode. (tucu) - - HDFS-3949. NameNodeRpcServer#join should join on both client and - server RPC servers. (eli) - - HDFS-3932. NameNode Web UI broken if the rpc-address is set to the wildcard. - (Colin Patrick McCabe via eli) - - HDFS-3931. TestDatanodeBlockScanner#testBlockCorruptionPolicy2 is broken. - (Andy Isaacson via eli) - - HDFS-3964. Make NN log of fs.defaultFS debug rather than info. (eli) - - HDFS-3992. Method org.apache.hadoop.hdfs.TestHftpFileSystem.tearDown() - sometimes throws NPEs. (Ivan A. Veselovsky via atm) - - HDFS-3753. Tests don't run with native libraries. - (Colin Patrick McCabe via eli) - - HDFS-4000. TestParallelLocalRead fails with "input ByteBuffers - must be direct buffers". (Colin Patrick McCabe via eli) - - HDFS-3999. HttpFS OPEN operation expects len parameter, it should be length. (tucu) - - HDFS-4006. TestCheckpoint#testSecondaryHasVeryOutOfDateImage - occasionally fails due to unexpected exit. (todd via eli) - - HDFS-4018. testMiniDFSClusterWithMultipleNN is missing some - cluster cleanup. (eli) - - HDFS-4020. TestRBWBlockInvalidation may time out. (eli) - - HDFS-4021. Misleading error message when resources are low on the NameNode. - (Christopher Conner via atm) - - HDFS-4044. Duplicate ChecksumType definition in HDFS .proto files. - (Binglin Chang via suresh) - - HDFS-4049. Fix hflush performance regression due to nagling delays - (todd) - - HDFS-3678. Edit log files are never being purged from 2NN. (atm) - - HDFS-4058. DirectoryScanner may fail with IOOB if the directory - scanning threads return out of volume order. (eli) - - HDFS-3985. Add timeouts to TestMulitipleNNDataBlockScanner. (todd via eli) - - HDFS-4061. TestBalancer and TestUnderReplicatedBlocks need timeouts. (eli) - - HDFS-3997. OfflineImageViewer incorrectly passes value of imageVersion when - visiting IS_COMPRESSED element. (Mithun Radhakrishnan via atm) - - HDFS-4055. TestAuditLogs is flaky. (Binglin Chang via eli) - - HDFS-4072. On file deletion remove corresponding blocks pending - replications. (Jing Zhao via suresh) - - HDFS-4022. Replication not happening for appended block. - (Vinay via umamahesh) - - HDFS-3948. Do not use hflush in TestWebHDFS.testNamenodeRestart() since the - out stream returned by WebHdfsFileSystem does not support it. (Jing Zhao - via szetszwo) - - HDFS-3616. Fix a ConcurrentModificationException bug that BP actor threads - may not be shutdown properly in DataNode. (Jing Zhao via szetszwo) - - HDFS-4127. Log message is not correct in case of short of replica. - (Junping Du via suresh) - - HADOOP-8994. TestDFSShell creates file named "noFileHere", making further - tests hard to understand (Andy Isaacson via daryn) - - HDFS-3804. TestHftpFileSystem fails intermittently with JDK7 - (Trevor Robinson via daryn) - - HDFS-4132. When libwebhdfs is not enabled, nativeMiniDfsClient frees - uninitialized memory (Colin Patrick McCabe via todd) - - HDFS-1331. dfs -test should work like /bin/test (Andy Isaacson via daryn) - - HDFS-3979. For hsync, datanode should wait for the local sync to complete - before sending ack. (Lars Hofhansl via szetszwo) - - HDFS-3810. Implement format() for BKJM (Ivan Kelly via umamahesh) - - HDFS-3625. Fix TestBackupNode by properly initializing edit log during - startup. (Junping Du via todd) - - HDFS-4138. BackupNode startup fails due to uninitialized edit log. - (Kihwal Lee via shv) - - HDFS-4162. Some malformed and unquoted HTML strings are returned from - datanode web ui. (Darek Dagit via suresh) - - HDFS-4164. fuse_dfs: add -lrt to the compiler command line on Linux. - (Colin Patrick McCabe via eli) - - HDFS-3921. NN will prematurely consider blocks missing when entering active - state while still in safe mode. (atm) - - HDFS-4106. BPServiceActor#lastHeartbeat, lastBlockReport and - lastDeletedReport should be volatile. (Jing Zhao via suresh) - - HDFS-4139. fuse-dfs RO mode still allows file truncation. - (Colin Patrick McCabe via eli) - - HDFS-4104. dfs -test -d prints inappropriate error on nonexistent directory - (Andy Isaacson via daryn) - - HDFS-3623. BKJM: zkLatchWaitTimeout hard coded to 6000. Make use of ZKSessionTimeout instead. - (umamahesh) - - HDFS-4100. Fix all findbug security warings. (Liang Xie via eli) - - HDFS-3507. DFS#isInSafeMode needs to execute only on Active NameNode. - (Vinay via atm) - - HDFS-4105. The SPNEGO user for secondary namenode should use the web - keytab. (Arpit Gupta via jitendra) - - HDFS-4156. Seeking to a negative position should throw an IOE. - (Eli Reisman via eli) - - HDFS-4171. WebHDFS and HttpFs should accept only valid Unix user - names. (tucu) - - HDFS-4178. Shell scripts should not close stderr (Andy Isaacson via daryn) - - HDFS-4179. BackupNode: allow reads, fix checkpointing, safeMode. (shv) - - HDFS-4216. Do not ignore QuotaExceededException when adding symlinks. - (szetszwo) - - HDFS-4242. Map.Entry is incorrectly used in LeaseManager since the behavior - of it is undefined after the iteration or modifications of the map. - (szetszwo) - - HDFS-4231. BackupNode: Introduce BackupState. (shv) - - HDFS-4238. Standby namenode should not do purging of shared - storage edits. (todd) - - HDFS-4282. TestEditLog.testFuzzSequences FAILED in all pre-commit test - (todd) - - HDFS-4236. Remove artificial limit on username length introduced in - HDFS-4171. (tucu via suresh) - - HDFS-4279. NameNode does not initialize generic conf keys when started - with -recover. (Colin Patrick McCabe via atm) - - HDFS-4291. edit log unit tests leave stray test_edit_log_file around - (Colin Patrick McCabe via todd) - - HDFS-4292. Sanity check not correct in RemoteBlockReader2.newBlockReader - (Binglin Chang via todd) - - HDFS-4295. Using port 1023 should be valid when starting Secure DataNode - (Stephen Chu via todd) - - HDFS-4294. Backwards compatibility is not maintained for TestVolumeId. - (Ivan A. Veselovsky and Robert Parker via atm) - - HDFS-2264. NamenodeProtocol has the wrong value for clientPrincipal in - KerberosInfo annotation. (atm) - - HDFS-4307. SocketCache should use monotonic time. (Colin Patrick McCabe - via atm) - - HDFS-4315. DNs with multiple BPs can have BPOfferServices fail to start - due to unsynchronized map access. (atm) - - HDFS-4140. fuse-dfs handles open(O_TRUNC) poorly. (Colin Patrick McCabe - via atm) - - HDFS-4308. addBlock() should persist file blocks once. - (Plamen Jeliazkov via shv) - - HDFS-4347. Avoid infinite waiting checkpoint to complete in TestBackupNode. - (Plamen Jeliazkov via shv) - - HDFS-4349. Add test for reading files from BackupNode. (shv) - - HDFS-4302. Fix fatal exception when starting NameNode with DEBUG logs - (Eugene Koontz via todd) - - HDFS-3970. Fix bug causing rollback of HDFS upgrade to result in bad - VERSION file. (Vinay and Andrew Wang via atm) - - HDFS-4306. PBHelper.convertLocatedBlock miss convert BlockToken. (Binglin - Chang via atm) - - HDFS-4384. test_libhdfs_threaded gets SEGV if JNIEnv cannot be - initialized. (Colin Patrick McCabe via eli) - - HDFS-4328. TestLargeBlock#testLargeBlockSize is timing out. (Chris Nauroth - via atm) - - HDFS-4274. BlockPoolSliceScanner does not close verification log during - shutdown. (Chris Nauroth via suresh) - - HDFS-1245. Pluggable block id generation. (shv) - - HDFS-4415. HostnameFilter should handle hostname resolution failures and - continue processing. (Robert Kanter via atm) - - HDFS-4359. Slow RPC responses from NN can prevent metrics collection on - DNs. (liang xie via atm) - - HDFS-4444. Add space between total transaction time and number of - transactions in FSEditLog#printStatistics. (Stephen Chu via suresh) - - HDFS-4428. FsDatasetImpl should disclose what the error is when a rename - fails. (Colin Patrick McCabe via atm) - - HDFS-4452. getAdditionalBlock() can create multiple blocks if the client - times out and retries. (shv) - - HDFS-4445. All BKJM ledgers are not checked while tailing, So failover will fail. - (Vinay via umamahesh) - - HDFS-4462. 2NN will fail to checkpoint after an HDFS upgrade from a - pre-federation version of HDFS. (atm) - - HDFS-4404. Create file failure when the machine of first attempted NameNode - is down. (Todd Lipcon via atm) - - HDFS-4344. dfshealth.jsp throws NumberFormatException when - dfs.hosts/dfs.hosts.exclude includes port number. (Andy Isaacson via atm) - - HDFS-4468. Use the new StringUtils methods added by HADOOP-9252 and fix - TestHDFSCLI and TestQuota. (szetszwo) - - HDFS-4458. In DFSUtil.getNameServiceUris(..), convert default fs URI using - NetUtils.createSocketAddr(..) for being consistent with other addresses. - (Binglin Chang via szetszwo) - - BREAKDOWN OF HDFS-3077 SUBTASKS - - HDFS-3077. Quorum-based protocol for reading and writing edit logs. - (todd, Brandon Li, and Hari Mankude via todd) - - HDFS-3694. Fix getEditLogManifest to fetch httpPort if necessary (todd) - - HDFS-3692. Support purgeEditLogs() call to remotely purge logs on JNs - (todd) - - HDFS-3693. JNStorage should read its storage info even before a writer - becomes active (todd) - - HDFS-3725. Fix QJM startup when individual JNs have gaps (todd) - - HDFS-3741. Exhaustive failure injection test for skipped RPCs (todd) - - HDFS-3773. TestNNWithQJM fails after HDFS-3741. (atm) - - HDFS-3793. Implement genericized format() in QJM (todd) - - HDFS-3795. QJM: validate journal dir at startup (todd) - - HDFS-3798. Avoid throwing NPE when finalizeSegment() is called on invalid - segment (todd) - - HDFS-3799. QJM: handle empty log segments during recovery (todd) - - HDFS-3797. QJM: add segment txid as a parameter to journal() RPC (todd) - - HDFS-3800. improvements to QJM fault testing (todd) - - HDFS-3823. QJM: TestQJMWithFaults fails occasionally because of missed - setting of HTTP port. (todd and atm) - - HDFS-3826. QJM: Some trivial logging / exception text improvements. (todd - and atm) - - HDFS-3839. QJM: hadoop-daemon.sh should be updated to accept "journalnode" - (eli) - - HDFS-3845. Fixes for edge cases in QJM recovery protocol (todd) - - HDFS-3877. QJM: Provide defaults for dfs.journalnode.*address (eli) - - HDFS-3863. Track last "committed" txid in QJM (todd) - - HDFS-3869. Expose non-file journal manager details in web UI (todd) - - HDFS-3884. Journal format() should reset cached values (todd) - - HDFS-3870. Add metrics to JournalNode (todd) - - HDFS-3891. Make selectInputStreams throw IOE instead of RTE (todd) - - HDFS-3726. If a logger misses an RPC, don't retry that logger until next - segment (todd) - - HDFS-3893. QJM: Make QJM work with security enabled. (atm) - - HDFS-3897. QJM: TestBlockToken fails after HDFS-3893. (atm) - - HDFS-3898. QJM: enable TCP_NODELAY for IPC (todd) - - HDFS-3885. QJM: optimize log sync when JN is lagging behind (todd) - - HDFS-3900. QJM: avoid validating log segments on log rolls (todd) - - HDFS-3901. QJM: send 'heartbeat' messages to JNs even when they are - out-of-sync (todd) - - HDFS-3899. QJM: Add client-side metrics (todd) - - HDFS-3914. QJM: acceptRecovery should abort current segment (todd) - - HDFS-3915. QJM: Failover fails with auth error in secure cluster (todd) - - HDFS-3906. QJM: quorum timeout on failover with large log segment (todd) - - HDFS-3840. JournalNodes log JournalNotFormattedException backtrace error - before being formatted (todd) - - HDFS-3894. QJM: testRecoverAfterDoubleFailures can be flaky due to IPC - client caching (todd) - - HDFS-3926. QJM: Add user documentation for QJM. (atm) - - HDFS-3943. QJM: remove currently-unused md5sum field (todd) - - HDFS-3950. QJM: misc TODO cleanup, improved log messages, etc. (todd) - - HDFS-3955. QJM: Make acceptRecovery() atomic. (todd) - - HDFS-3956. QJM: purge temporary files when no longer within retention - period (todd) - - HDFS-4004. TestJournalNode#testJournal fails because of test case execution - order (Chao Shi via todd) - - HDFS-4017. Unclosed FileInputStream in GetJournalEditServlet - (Chao Shi via todd) - - HDFS-4351. In BlockPlacementPolicyDefault.chooseTarget(..), numOfReplicas - needs to be updated when avoiding stale nodes. (Andrew Wang via szetszwo) - - HDFS-2908. Add apache license header for StorageReport.java. (Brandon Li - via tgraves) - - HDFS-4399. Fix RAT warnings by excluding images sub-dir in docs. (Thomas - Graves via acmurthy) - -Release 2.0.2-alpha - 2012-09-07 - - INCOMPATIBLE CHANGES - - HDFS-3446. HostsFileReader silently ignores bad includes/excludes - (Matthew Jacobs via todd) - - HDFS-3755. Creating an already-open-for-write file with overwrite=true fails - (todd) - - NEW FEATURES - - HDFS-744. Support hsync in HDFS. (Lars Hofhansl via szetszwo) - - HDFS-3042. Automatic failover support for NameNode HA (todd) - (see dedicated section below for breakdown of subtasks) - - HDFS-3518. Add a utility method HdfsUtils.isHealthy(uri) for checking if - the given HDFS is healthy. (szetszwo) - - HDFS-3113. httpfs does not support delegation tokens. (tucu) - - HDFS-3513. HttpFS should cache filesystems. (tucu) - - HDFS-3637. Add support for encrypting the DataTransferProtocol. (atm) - - HDFS-3150. Add option for clients to contact DNs via hostname. (eli) - - HDFS-2793. Add an admin command to trigger an edit log roll. (todd) - - HDFS-3703. Datanodes are marked stale if heartbeat is not received in - configured timeout and are selected as the last location to read from. - (Jing Zhao via suresh) - - IMPROVEMENTS - - HDFS-3040. TestMulitipleNNDataBlockScanner is misspelled. (Madhukara Phatak - via atm) - - HDFS-3390. DFSAdmin should print full stack traces of errors when DEBUG - logging is enabled. (atm) - - HDFS-3341. Change minimum RPC versions to respective SNAPSHOTs instead of - final releases. (todd) - - HDFS-3369. Rename {get|set|add}INode(..) methods in BlockManager and - BlocksMap to {get|set|add}BlockCollection(..). (John George via szetszwo) - - HDFS-3134. harden edit log loader against malformed or malicious input. - (Colin Patrick McCabe via eli) - - HDFS-3230. Cleanup DatanodeID creation in the tests. (eli) - - HDFS-3401. Cleanup DatanodeDescriptor creation in the tests. (eli) - - HDFS-3400. DNs should be able start with jsvc even if security is disabled. - (atm via eli) - - HDFS-3404. Make putImage in GetImageServlet infer remote address to fetch - from request. (atm) - - HDFS-3335. check for edit log corruption at the end of the log - (Colin Patrick McCabe via todd) - - HDFS-3417. Rename BalancerDatanode#getName to getDisplayName to be - consistent with Datanode. (eli) - - HDFS-3416. Cleanup DatanodeID and DatanodeRegistration - constructors used by testing. (eli) - - HDFS-3419. Cleanup LocatedBlock. (eli) - - HDFS-3440. More effectively limit stream memory consumption when reading - corrupt edit logs (Colin Patrick McCabe via todd) - - HDFS-3438. BootstrapStandby should not require a rollEdits on active node - (todd) - - HDFS-2885. Remove "federation" from the nameservice config options. - (Tsz Wo (Nicholas) Sze via eli) - - HDFS-3394. Do not use generic in INodeFile.getLastBlock(): the run-time - ClassCastException check is useless since generic type information is only - available in compile-time. (szetszwo) - - HDFS-3454. Balancer unconditionally logs InterruptedException at - INFO level on shutdown if security is enabled. (eli) - - HDFS-1013. Miscellaneous improvements to HTML markup for web UIs - (Eugene Koontz via todd) - - HDFS-3052. Change INodeFile and INodeFileUnderConstruction to package - private. (szetszwo) - - HDFS-3520. Add transfer rate logging to TransferFsImage. (eli) - - HDFS-3504. Support configurable retry policy in DFSClient for RPC - connections and RPC calls, and add MultipleLinearRandomRetry, a new retry - policy. (szetszwo) - - HDFS-3372. offlineEditsViewer should be able to read a binary - edits file with recovery mode. (Colin Patrick McCabe via eli) - - HDFS-3516. Check content-type in WebHdfsFileSystem. (szetszwo) - - HDFS-3535. Audit logging should log denied accesses. (Andy Isaacson via eli) - - HDFS-3481. Refactor HttpFS handling of JAX-RS query string parameters (tucu) - - HDFS-3572. Cleanup code which inits SPNEGO in HttpServer (todd) - - HDFS-3475. Make the replication monitor multipliers configurable. - (harsh via eli) - - HDFS-3343. Improve metrics for DN read latency (Andrew Wang via todd) - - HDFS-3170. Add more useful metrics for write latency (Matthew Jacobs via - todd) - - HDFS-3604. Add dfs.webhdfs.enabled to hdfs-default.xml. (eli) - - HDFS-2988. Improve error message when storage directory lock fails - (Miomir Boljanovic via harsh) - - HDFS-2391. Newly set BalancerBandwidth value is not displayed anywhere. - (harsh) - - HDFS-3067. NPE in DFSInputStream.readBuffer if read is repeated on - corrupted block. (Henry Robinson via atm) - - HDFS-3555. idle client socket triggers DN ERROR log - (should be INFO or DEBUG). (Andy Isaacson via harsh) - - HDFS-3568. fuse_dfs: add support for security. (Colin McCabe via atm) - - HDFS-3629. Fix the typo in the error message about inconsistent - storage layout version. (Brandon Li via harsh) - - HDFS-3613. GSet prints some INFO level values, which aren't - really very useful to all (Andrew Wang via harsh) - - HDFS-3611. NameNode prints unnecessary WARNs about edit log normally skipping - a few bytes. (Colin Patrick McCabe via harsh) - - HDFS-3582. Hook daemon process exit for testing. (eli) - - HDFS-3641. Move server Util time methods to common and use now - instead of System#currentTimeMillis. (eli) - - HDFS-3633. libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE. - (Colin Patrick McCabe via eli) - - HDFS-799. libhdfs must call DetachCurrentThread when a thread is destroyed. - (Colin Patrick McCabe via eli) - - HDFS-3306. fuse_dfs: don't lock release operations. - (Colin Patrick McCabe via eli) - - HDFS-3612. Single namenode image directory config warning can - be improved. (Andy Isaacson via harsh) - - HDFS-3606. libhdfs: create self-contained unit test. - (Colin Patrick McCabe via eli) - - HDFS-3539. libhdfs code cleanups. (Colin Patrick McCabe via eli) - - HDFS-3610. fuse_dfs: Provide a way to use the default (configured) NN URI. - (Colin Patrick McCabe via eli) - - HDFS-3663. MiniDFSCluster should capture the code path that led to - the first ExitException. (eli) - - HDFS-3659. Add missing @Override to methods across the hadoop-hdfs - project. (Brandon Li via harsh) - - HDFS-3537. Move libhdfs and fuse-dfs source to native subdirectories. - (Colin Patrick McCabe via eli) - - HDFS-3665. Add a test for renaming across file systems via a symlink. (eli) - - HDFS-3666. Plumb more exception messages to terminate. (eli) - - HDFS-3673. libhdfs: fix some compiler warnings. (Colin Patrick McCabe via eli) - - HDFS-3675. libhdfs: follow documented return codes. (Colin Patrick McCabe via eli) - - HDFS-1249. With fuse-dfs, chown which only has owner (or only group) - argument fails with Input/output error. (Colin Patrick McCabe via eli) - - HDFS-3583. Convert remaining tests to Junit4. (Andrew Wang via atm) - - HDFS-3711. Manually convert remaining tests to JUnit4. (Andrew Wang via atm) - - HDFS-3650. Use MutableQuantiles to provide latency histograms for various - operations. (Andrew Wang via atm) - - HDFS-3667. Add retry support to WebHdfsFileSystem. (szetszwo) - - HDFS-3291. add test that covers HttpFS working w/ a non-HDFS Hadoop - filesystem (tucu) - - HDFS-3634. Add self-contained, mavenized fuse_dfs test. (Colin Patrick - McCabe via atm) - - HDFS-3190. Simple refactors in existing NN code to assist - QuorumJournalManager extension. (todd) - - HDFS-3276. initializeSharedEdits should have a -nonInteractive flag (todd) - - HDFS-3765. namenode -initializeSharedEdits should be able to initialize - all shared storages. (Vinay and todd via todd) - - HDFS-3723. Add support -h, -help to all the commands. (Jing Zhao via - suresh) - - HDFS-3803. Change BlockPoolSliceScanner chatty INFO log to DEBUG. - (Andrew Purtell via suresh) - - HDFS-3802. StartupOption.name in HdfsServerConstants should be final. - (Jing Zhao via szetszwo) - - HDFS-3796. Speed up edit log tests by avoiding fsync() (todd) - - HDFS-2963. Console Output is confusing while executing metasave - (dfsadmin command). (Andrew Wang via eli) - - HDFS-3672. Expose disk-location information for blocks to enable better - scheduling. (Andrew Wang via atm) - - HDFS-2727. libhdfs should get the default block size from the server. - (Colin Patrick McCabe via eli) - - HDFS-3832. Remove protocol methods related to DistributedUpgrade. (suresh) - - HDFS-3819. Should check whether invalidate work percentage default value is - not greater than 1.0f. (Jing Zhao via jitendra) - - HDFS-3177. Update DFSClient and DataXceiver to handle different checkum - types in file checksum computation. (Kihwal Lee via szetszwo) - - HDFS-3844. Add @Override and remove {@inheritdoc} and unnecessary - imports. (Jing Zhao via suresh) - - HDFS-3853. Port MiniDFSCluster enableManagedDfsDirsRedundancy - option to branch-2. (Colin Patrick McCabe via eli) - - HDFS-3871. Change NameNodeProxies to use RetryUtils. (Arun C Murthy - via szetszwo) - - HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy. - (Jing Zhao via szetszwo) - - HDFS-3888. Clean up BlockPlacementPolicyDefault. (Jing Zhao via szetszwo) - - HDFS-3907. Allow multiple users for local block readers. (eli) - - HDFS-3510. Editlog pre-allocation is performed prior to writing edits - to avoid partial edits case disk out of space. (Colin McCabe via todd) - - HDFS-3910. DFSTestUtil#waitReplication should timeout. (eli) - - HDFS-3920. libwebdhfs string processing and using strerror consistently - to handle all errors. (Jing Zhao via suresh) - - OPTIMIZATIONS - - HDFS-2982. Startup performance suffers when there are many edit log - segments. (Colin Patrick McCabe via todd) - - HDFS-2834. Add a ByteBuffer-based read API to DFSInputStream. - (Henry Robinson via todd) - - HDFS-3110. Use directRead API to reduce the number of buffer copies in - libhdfs (Henry Robinson via todd) - - HDFS-3697. Enable fadvise readahead by default. (todd) - - HDFS-2421. Improve the concurrency of SerialNumberMap in NameNode. - (Jing Zhao and Weiyan Wang via szetszwo) - - HDFS-3866. HttpFS POM should have property where to download tomcat from (zero45 via tucu) - - BUG FIXES - - HDFS-3385. The last block of INodeFileUnderConstruction is not - necessarily a BlockInfoUnderConstruction, so do not cast it in - FSNamesystem.recoverLeaseInternal(..). (szetszwo) - - HDFS-3414. Balancer does not find NameNode if rpc-address or - servicerpc-address are not set in client configs. (atm) - - HDFS-3031. Fix complete() and getAdditionalBlock() RPCs to be idempotent - (todd) - - HDFS-2759. Pre-allocate HDFS edit log files after writing version number. - (atm) - - HDFS-3413. TestFailureToReadEdits timing out. (atm) - - HDFS-3422. TestStandbyIsHot timeouts too aggressive (todd) - - HDFS-3433. GetImageServlet should allow administrative requestors when - security is enabled. (atm) - - HDFS-1153. dfsnodelist.jsp should handle invalid input parameters. - (Ravi Phulari via eli) - - HDFS-3434. InvalidProtocolBufferException when visiting DN - browseDirectory.jsp (eli) - - HDFS-2800. Fix cancellation of checkpoints in the standby node to be more - reliable. (todd) - - HDFS-3391. Fix InvalidateBlocks to compare blocks including their - generation stamps. (todd) - - HDFS-3444. hdfs groups command doesn't work with security enabled. (atm) - - HDFS-2717. BookKeeper Journal output stream doesn't check addComplete rc. - (Ivan Kelly via umamahesh) - - HDFS-3415. Make sure all layout versions are the same for all storage - directories in the Namenode. (Brandon Li via szetszwo) - - HDFS-3436. In DataNode.transferReplicaForPipelineRecovery(..), it should - use the stored generation stamp to check if the block is valid. (Vinay - via szetszwo) - - HDFS-3460. HttpFS proxyuser validation with Kerberos ON uses full - principal name. (tucu) - - HDFS-3058. HA: Bring BookKeeperJournalManager up to date with HA changes. - (Ivan Kelly via umamahesh) - - HDFS-3368. Missing blocks due to bad DataNodes coming up and down. (shv) - - HDFS-3452. BKJM:Switch from standby to active fails and NN gets shut down - due to delay in clearing of lock. (umamahesh) - - HDFS-3398. Client will not retry when primaryDN is down once it's just got pipeline. - (Amith D K via umamahesh) - - HDFS-3474. Cleanup Exception handling in BookKeeper journal manager. - (Ivan Kelly via umamahesh) - - HDFS-3468. Make BKJM-ZK session timeout configurable. (umamahesh) - - HDFS-3423. BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments() - a bad inProgress_ ZNodes. (Ivan Kelly and Uma via umamahesh) - - HDFS-3441. Race condition between rolling logs at active NN and purging at standby. - (Rakesh R via umamahesh) - - HDFS-3484. hdfs fsck doesn't work if NN HTTP address is set to - 0.0.0.0 even if NN RPC address is configured. (atm via eli) - - HDFS-3486. offlineimageviewer can't read fsimage files that contain - persistent delegation tokens. (Colin Patrick McCabe via eli) - - HDFS-3487. offlineimageviewer should give byte offset information - when it encounters an exception. (Colin Patrick McCabe via eli) - - HDFS-3442. Incorrect count for Missing Replicas in FSCK report. (Andrew - Wang via atm) - - HDFS-2025. Go Back to File View link is not working in tail.jsp. - (Ashish and Sravan via umamahesh) - - HDFS-3501. Checkpointing with security enabled will stop working - after ticket lifetime expires. (atm via eli) - - HDFS-3266. DFSTestUtil#waitCorruptReplicas doesn't sleep between checks. - (Madhukara Phatak via atm) - - HDFS-3505. DirectoryScanner does not join all threads in shutdown. - (Colin Patrick McCabe via eli) - - HDFS-3485. DataTransferThrottler will over-throttle when currentTimeMillis - jumps (Andy Isaacson via todd) - - HDFS-2914. HA: Standby should not enter safemode when resources are low. - (Vinay via atm) - - HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option. - (Henry Robinson via atm) - - HDFS-3514. Add missing TestParallelLocalRead. (Henry Robinson via atm) - - HDFS-3243. TestParallelRead timing out on jenkins. (Henry Robinson via todd) - - HDFS-3490. DatanodeWebHdfsMethods throws NullPointerException if - NamenodeRpcAddressParam is not set. (szetszwo) - - HDFS-2797. Fix misuses of InputStream#skip in the edit log code. - (Colin Patrick McCabe via eli) - - HDFS-3517. TestStartup should bind ephemeral ports. (eli) - - HDFS-3522. If a namenode is in safemode, it should throw SafeModeException - when getBlockLocations has zero locations. (Brandon Li via szetszwo) - - HDFS-3408. BKJM : Namenode format fails, if there is no BK root. (Rakesh R via umamahesh) - - HDFS-3389. Document the BKJM usage in Namenode HA. (umamahesh and Ivan Kelly via umamahesh) - - HDFS-3531. EditLogFileOutputStream#preallocate should check for - incomplete writes. (Colin Patrick McCabe via eli) - - HDFS-766. Error message not clear for set space quota out of boundary - values. (Jon Zuanich via atm) - - HDFS-3480. Multiple SLF4J binding warning. (Vinay via eli) - - HDFS-3524. Update TestFileLengthOnClusterRestart for HDFS-3522. (Brandon - Li via szetszwo) - - HDFS-3559. DFSTestUtil: use Builder class to construct DFSTestUtil - instances. (Colin Patrick McCabe via atm) - - HDFS-3551. WebHDFS CREATE should use client location for HTTP redirection. - (szetszwo) - - HDFS-3157. Fix a bug in the case that the generation stamps of the stored - block in a namenode and the reported block from a datanode do not match. - (Ashish Singhi via szetszwo) - - HDFS-3575. HttpFS does not log Exception Stacktraces (brocknoland via tucu) - - HDFS-3574. Fix small race and do some cleanup in GetImageServlet (todd) - - HDFS-3581. FSPermissionChecker#checkPermission sticky bit check - missing range check. (eli) - - HDFS-3541. Deadlock between recovery, xceiver and packet responder. - (Vinay via umamahesh) - - HDFS-3428. Move DelegationTokenRenewer to common (tucu) - - HDFS-3491. HttpFs does not set permissions correctly (tucu) - - HDFS-3580. incompatible types; no instance(s) of type variable(s) V exist - so that V conforms to boolean compiling HttpFSServer.java with OpenJDK - (adi2 via tucu) - - HDFS-3603. Decouple TestHDFSTrash from TestTrash. (Jason Lowe via eli) - - HDFS-711. hdfsUtime does not handle atime = 0 or mtime = 0 correctly. - (Colin Patrick McCabe via eli) - - HDFS-3548. NamenodeFsck.copyBlock fails to create a Block Reader. - (Colin Patrick McCabe via eli) - - HDFS-3615. Two BlockTokenSecretManager findbugs warnings. (atm) - - HDFS-470. libhdfs should handle 0-length reads from FSInputStream - correctly. (Colin Patrick McCabe via eli) - - HDFS-3492. fix some misuses of InputStream#skip. - (Colin Patrick McCabe via eli) - - HDFS-3609. libhdfs: don't force the URI to look like hdfs://hostname:port. - (Colin Patrick McCabe via eli) - - HDFS-2966 TestNameNodeMetrics tests can fail under load. (stevel) - - HDFS-3605. Block mistakenly marked corrupt during edit log catchup - phase of failover. (todd and Brahma Reddy Battula via todd) - - HDFS-3690. BlockPlacementPolicyDefault incorrectly casts LOG. (eli) - - HDFS-3597. SNN fails to start after DFS upgrade. (Andy Isaacson via todd) - - HDFS-3608. fuse_dfs: detect changes in UID ticket cache. (Colin Patrick - McCabe via atm) - - HDFS-3709. TestStartup tests still binding to the ephemeral port. (eli) - - HDFS-3720. hdfs.h must get packaged. (Colin Patrick McCabe via atm) - - HDFS-3626. Creating file with invalid path can corrupt edit log (todd) - - HDFS-3679. fuse_dfs notrash option sets usetrash. (Conrad Meyer via suresh) - - HDFS-3732. fuse_dfs: incorrect configuration value checked for connection - expiry timer period. (Colin Patrick McCabe via atm) - - HDFS-3738. TestDFSClientRetries#testFailuresArePerOperation sets incorrect - timeout config. (atm) - - HDFS-3756. DelegationTokenFetcher creates 2 HTTP connections, the second - one not properly configured. (tucu) - - HDFS-3579. libhdfs: fix exception handling. (Colin Patrick McCabe via atm) - - HDFS-3754. BlockSender doesn't shutdown ReadaheadPool threads. (eli) - - HDFS-3760. primitiveCreate is a write, not a read. (Andy Isaacson via atm) - - HDFS-3710. libhdfs misuses O_RDONLY/WRONLY/RDWR. (Andy Isaacson via atm) - - HDFS-3721. hsync support broke wire compatibility. (todd and atm) - - HDFS-3758. TestFuseDFS test failing. (Colin Patrick McCabe via eli) - - HDFS-2330. In NNStorage and FSImagePreTransactionalStorageInspector, - IOExceptions of stream closures can mask root exceptions. (Uma Maheswara - Rao G via szetszwo) - - HDFS-3790. test_fuse_dfs.c doesn't compile on centos 5. (Colin Patrick - McCabe via atm) - - HDFS-3658. Fix bugs in TestDFSClientRetries and add more tests. (szetszwo) - - HDFS-3794. WebHDFS OPEN returns the incorrect Content-Length in the HTTP - header when offset is specified and length is omitted. - (Ravi Prakash via szetszwo) - - HDFS-3048. Small race in BlockManager#close. (Andy Isaacson via eli) - - HDFS-3194. DataNode block scanner is running too frequently. - (Andy Isaacson via eli) - - HDFS-3808. fuse_dfs: postpone libhdfs intialization until after fork. - (Colin Patrick McCabe via atm) - - HDFS-3788. ByteRangeInputStream should not expect HTTP Content-Length header - when chunked transfer-encoding is used. (szetszwo) - - HDFS-3816. Invalidate work percentage default value should be 0.32f - instead of 32. (Jing Zhao via suresh) - - HDFS-3707. TestFSInputChecker: improper use of skip. - (Colin Patrick McCabe via eli) - - HDFS-3830. test_libhdfs_threaded: use forceNewInstance. - (Colin Patrick McCabe via eli) - - HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is - enabled and the NN restarts with outstanding delegation tokens. (atm) - - HDFS-3715. Fix TestFileCreation#testFileCreationNamenodeRestart. - (Andrew Whang via eli) - - HDFS-3683. Edit log replay progress indicator shows >100% complete. (Plamen - Jeliazkov via atm) - - HDFS-3731. Release upgrade must handle blocks being written from 1.0. - (Colin Patrick McCabe via eli) - - HDFS-3856. TestHDFSServerPorts failure is causing surefire fork failure. - (eli) - - HDFS-3860. HeartbeatManager#Monitor may wrongly hold the writelock of - namesystem. (Jing Zhao via atm) - - HDFS-3849. When re-loading the FSImage, we should clear the existing - genStamp and leases. (Colin Patrick McCabe via atm) - - HDFS-3864. NN does not update internal file mtime for OP_CLOSE when reading - from the edit log. (atm) - - HDFS-3837. Fix DataNode.recoverBlock findbugs warning. (eli) - - HDFS-3733. Audit logs should include WebHDFS access. (Andy Isaacson via - eli) - - HDFS-2686. Remove DistributedUpgrade related code. (suresh) - - HDFS-3833. TestDFSShell fails on windows due to concurrent file - read/write. (Brandon Li via suresh) - - HDFS-3466. Get HTTP kerberos principal from the web authentication keytab. - (omalley) - - HDFS-3879. Fix findbugs warning in TransferFsImage on branch-2. (eli) - - HDFS-3469. start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly. - (Vinay via umamahesh) - - HDFS-1490. TransferFSImage should timeout (Dmytro Molkov and Vinay via todd) - - HDFS-3828. Block Scanner rescans blocks too frequently. - (Andy Isaacson via eli) - - HDFS-3895. hadoop-client must include commons-cli (tucu) - - HDFS-2757. Cannot read a local block that's being written to when - using the local read short circuit. (Jean-Daniel Cryans via eli) - - HDFS-3664. BlockManager race when stopping active services. - (Colin Patrick McCabe via eli) - - HDFS-3928. MiniDFSCluster should reset the first ExitException on shutdown. (eli) - - HDFS-3938. remove current limitations from HttpFS docs. (tucu) - - HDFS-3944. Httpfs resolveAuthority() is not resolving host correctly. (tucu) - - HDFS-3972. Trash emptier fails in secure HA cluster. (todd via eli) - - HDFS-4443. Remove a trailing '`' character from the HTML code generated by - NamenodeJspHelper.generateNodeData(..). (Christian Rohling via szetszwo) - - BREAKDOWN OF HDFS-3042 SUBTASKS - - HDFS-2185. HDFS portion of ZK-based FailoverController (todd) - - HDFS-3200. Scope all ZKFC configurations by nameservice (todd) - - HDFS-3223. add zkfc to hadoop-daemon.sh script (todd) - - HDFS-3261. TestHASafeMode fails on HDFS-3042 branch (todd) - - HDFS-3159. Document NN auto-failover setup and configuration (todd) - - HDFS-3412. Fix findbugs warnings in auto-HA branch (todd) - - HDFS-3432. TestDFSZKFailoverController tries to fail over too early (todd) - - HDFS-3902. TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken. - (Andy Isaacson via eli) - -Release 2.0.0-alpha - 05-23-2012 - - INCOMPATIBLE CHANGES - - HDFS-2676. Remove Avro RPC. (suresh) - - HDFS-2303. Unbundle jsvc. (Roman Shaposhnik and Mingjie Lai via eli) - - HDFS-3137. Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16. (eli) - - HDFS-3138. Move DatanodeInfo#ipcPort to DatanodeID. (eli) - - HDFS-3164. Move DatanodeInfo#hostName to DatanodeID. (eli) - - NEW FEATURES - - HDFS-2978. The NameNode should expose name dir statuses via JMX. (atm) - - HDFS-395. DFS Scalability: Incremental block reports. (Tomasz Nykiel - via hairong) - - HDFS-2517. Add protobuf service for JounralProtocol. (suresh) - - HDFS-2518. Add protobuf service for NamenodeProtocol. (suresh) - - HDFS-2520. Add protobuf service for InterDatanodeProtocol. (suresh) - - HDFS-2519. Add protobuf service for DatanodeProtocol. (suresh) - - HDFS-2581. Implement protobuf service for JournalProtocol. (suresh) - - HDFS-2618. Implement protobuf service for NamenodeProtocol. (suresh) - - HDFS-2629. Implement protobuf service for InterDatanodeProtocol. (suresh) - - HDFS-2636. Implement protobuf service for ClientDatanodeProtocol. (suresh) - - HDFS-2642. Protobuf translators for DatanodeProtocol. (jitendra) - - HDFS-2647. Used protobuf based RPC for InterDatanodeProtocol, - ClientDatanodeProtocol, JournalProtocol, NamenodeProtocol. (suresh) - - HDFS-2661. Enable protobuf RPC for DatanodeProtocol. (jitendra) - - HDFS-2697. Move RefreshAuthPolicy, RefreshUserMappings, GetUserMappings - protocol to protocol buffers. (jitendra) - - HDFS-2880. Protobuf changes in DatanodeProtocol to add multiple storages. - (suresh) - - HDFS-2899. Service protocol changes in DatanodeProtocol to add multiple - storages. (suresh) - - HDFS-2430. The number of failed or low-resource volumes the NN can tolerate - should be configurable. (atm) - - HDFS-1623. High Availability Framework for HDFS NN. Contributed by Todd - Lipcon, Aaron T. Myers, Eli Collins, Uma Maheswara Rao G, Bikas Saha, - Suresh Srinivas, Jitendra Nath Pandey, Hari Mankude, Brandon Li, Sanjay - Radia, Mingjie Lai, and Gregory Chanan - - HDFS-2941. Add an administrative command to download a copy of the fsimage - from the NN. (atm) - - HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change - DistributedFileSystem to @InterfaceAudience.LimitedPrivate. - (harsh via szetszwo) - - HDFS-3167. CLI-based driver for MiniDFSCluster. (Henry Robinson via atm) - - HDFS-3148. The client should be able to use multiple local interfaces - for data transfer. (eli) - - HDFS-3000. Add a public API for setting quotas. (atm) - - HDFS-3102. Add CLI tool to initialize the shared-edits dir. (atm) - - HDFS-3004. Implement Recovery Mode. (Colin Patrick McCabe via eli) - - HDFS-3282. Add HdfsDataInputStream as a public API. (umamahesh) - - HDFS-3298. Add HdfsDataOutputStream as a public API. (szetszwo) - - HDFS-234. Integration with BookKeeper logging system. (Ivan Kelly - via jitendra) - - IMPROVEMENTS - - HDFS-2018. Move all journal stream management code into one place. - (Ivan Kelly via jitendra) - - HDFS-2223. Untangle depencencies between NN components (todd) - - HDFS-2351. Change Namenode and Datanode to register each of their protocols - seperately (sanjay) - - HDFS-2337. DFSClient shouldn't keep multiple RPC proxy references (atm) - - HDFS-2181. Separate HDFS Client wire protocol data types (sanjay) - - HDFS-2459. Separate datatypes for Journal Protocol. (suresh) - - HDFS-2480. Separate datatypes for NamenodeProtocol. (suresh) - - HDFS-2489. Move Finalize and Register to separate file out of - DatanodeCommand.java. (suresh) - - HDFS-2488. Separate datatypes for InterDatanodeProtocol. (suresh) - - HDFS-2496. Separate datatypes for DatanodeProtocol. (suresh) - - HDFS-2479. HDFS Client Data Types in Protocol Buffers (sanjay) - - HADOOP-7862. Hdfs changes to work with HADOOP-7862: Move the support for - multiple protocols to lower layer so that Writable, PB and Avro can all - use it. (sanjay) - - HDFS-2597. ClientNameNodeProtocol in Protocol Buffers. (sanjay) - - HDFS-2651. ClientNameNodeProtocol Translators for Protocol Buffers. (sanjay) - - HDFS-2650. Replace @inheritDoc with @Override. (Hari Mankude via suresh). - - HDFS-2669. Enable protobuf rpc for ClientNamenodeProtocol. (sanjay) - - HDFS-2801. Provide a method in client side translators to check for a - methods supported in underlying protocol. (jitendra) - - HDFS-2895. Remove Writable wire protocol types and translators to - complete transition to protocol buffers. (suresh) - - HDFS-2992. Edit log failure trace should include transaction ID of - error. (Colin Patrick McCabe via eli) - - HDFS-2507. Allow saveNamespace operations to be canceled. (todd) - - HDFS-2410. Further cleanup of hardcoded configuration keys and values. - (suresh) - - HDFS-208. name node should warn if only one dir is listed in dfs.name.dir. - (Uma Maheswara Rao G via eli) - - HDFS-3003. Remove getHostPortString() from NameNode, replace it with - NetUtils.getHostPortString(). (Brandon Li via atm) - - HDFS-3014. FSEditLogOp and its subclasses should have toString() method. - (Sho Shimauchi via atm) - - HDFS-3036. Remove unused method DFSUtil#isDefaultNamenodeAddress. (atm) - - HDFS-2158. Add JournalSet to manage the set of journals. (jitendra) - - HDFS-2334. Add Closeable to JournalManager. (Ivan Kelly via jitendra) - - HDFS-1580. Add interface for generic Write Ahead Logging mechanisms. - (Ivan Kelly via jitendra) - - HDFS-3060. Bump TestDistributedUpgrade#testDistributedUpgrade timeout (eli) - - HDFS-3021. Use generic type to declare FSDatasetInterface. (szetszwo) - - HDFS-3056. Add a new interface RollingLogs for DataBlockScanner logging. - (szetszwo) - - HDFS-2731. Add command to bootstrap the Standby Node's name directories - from the Active NameNode. (todd) - - HDFS-3082. Clean up FSDatasetInterface and change DataNode.data to package - private. (szetszwo) - - HDFS-3057. httpfs and hdfs launcher scripts should honor CATALINA_HOME - and HADOOP_LIBEXEC_DIR (rvs via tucu) - - HDFS-3088. Move FSDatasetInterface inner classes to a package. (szetszwo) - - HDFS-3105. Add DatanodeStorage information to block recovery. (szetszwo) - - HDFS-3086. Change Datanode not to send storage list in registration. - (szetszwo) - - HDFS-3044. fsck move should be non-destructive by default. - (Colin Patrick McCabe via eli) - - HDFS-3071. haadmin failover command does not provide enough detail when - target NN is not ready to be active. (todd) - - HDFS-3089. Move FSDatasetInterface and the related classes to a package. - (szetszwo) - - HDFS-3129. NetworkTopology: add test that getLeaf should check for - invalid topologies (Colin Patrick McCabe via eli) - - HDFS-3155. Clean up FSDataset implemenation related code. (szetszwo) - - HDFS-3158. LiveNodes member of NameNodeMXBean should list non-DFS used - space and capacity per DN. (atm) - - HDFS-3172. dfs.upgrade.permission is dead code. (eli) - - HDFS-3171. The DatanodeID "name" field is overloaded. (eli) - - HDFS-3144. Refactor DatanodeID#getName by use. (eli) - - HDFS-3130. Move fsdataset implementation to a package. (szetszwo) - - HDFS-3120. Enable hsync and hflush by default. (eli) - - HDFS-3187. Upgrade guava to 11.0.2 (todd) - - HDFS-3168. Remove unnecessary "throw IOException" and change fields to - final in FSNamesystem and BlockManager. (szetszwo) - - HDFS-2564. Cleanup unnecessary exceptions thrown and unnecessary casts. - (Hari Mankude via eli) - - HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass - namenodeId as well as host:port (todd) - - HDFS-3050. rework OEV to share more code with the NameNode. - (Colin Patrick McCabe via eli) - - HDFS-3226. Allow GetConf tool to print arbitrary keys (todd) - - HDFS-3204. Minor modification to JournalProtocol.proto to make - it generic. (suresh) - - HDFS-2505. Add a test to verify getFileChecksum(..) with ViewFS. (Ravi - Prakash via szetszwo) - - HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG - (todd) - - HDFS-3238. ServerCommand and friends don't need to be writables. (eli) - - HDFS-3094. add -nonInteractive and -force option to namenode -format - command (Arpit Gupta via todd) - - HDFS-3244. Remove dead writable code from hdfs/protocol. (eli) - - HDFS-3247. Improve bootstrapStandby behavior when original NN is not active - (todd) - - HDFS-3249. Use ToolRunner.confirmPrompt in NameNode (todd) - - HDFS-3179. Improve the exception message thrown by DataStreamer when - it failed to add a datanode. (szetszwo) - - HDFS-2983. Relax the build version check to permit rolling upgrades within - a release. (atm) - - HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir - with edit log segments. (atm) - - HDFS-2708. Stats for the # of blocks per DN. (atm) - - HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to - TestEditLog. (Arpit Gupta via szetszwo) - - HDFS-3294. Fix code indentation in NamenodeWebHdfsMethods and - DatanodeWebHdfsMethods. (szetszwo) - - HDFS-3263. HttpFS should read HDFS config from Hadoop site.xml files (tucu) - - HDFS-3206. Miscellaneous xml cleanups for OEV. - (Colin Patrick McCabe via eli) - - HDFS-3169. TestFsck should test multiple -move operations in a row. - (Colin Patrick McCabe via eli) - - HDFS-3258. Test for HADOOP-8144 (pseudoSortByDistance in - NetworkTopology for first rack local node). (Junping Du via eli) - - HDFS-3322. Use HdfsDataInputStream and HdfsDataOutputStream in Hdfs. - (szetszwo) - - HDFS-3339. Change INode to package private. (John George via szetszwo) - - HDFS-3303. Remove Writable implementation from RemoteEditLogManifest. - (Brandon Li via szetszwo) - - HDFS-2617. Replaced Kerberized SSL for image transfer and fsck - with SPNEGO-based solution. (jghoman, omalley, tucu, and atm via eli) - - HDFS-3365. Enable users to disable socket caching in DFS client - configuration (todd) - - HDFS-3375. Put client name in DataXceiver thread name for readBlock - and keepalive (todd) - - HDFS-3363. Define BlockCollection and MutableBlockCollection interfaces - so that INodeFile and INodeFileUnderConstruction do not have to be used in - block management. (John George via szetszwo) - - HDFS-3211. Add fence(..) and replace NamenodeRegistration with JournalInfo - and epoch in JournalProtocol. (suresh via szetszwo) - - HDFS-3418. Rename BlockWithLocationsProto datanodeIDs field to storageIDs. - (eli) - - OPTIMIZATIONS - - HDFS-3024. Improve performance of stringification in addStoredBlock (todd) - - HDFS-2477. Optimize computing the diff between a block report and the - namenode state. (Tomasz Nykiel via hairong) - - HDFS-2495. Increase granularity of write operations in ReplicationMonitor - thus reducing contention for write lock. (Tomasz Nykiel via hairong) - - HDFS-2476. More CPU efficient data structure for under-replicated, - over-replicated, and invalidated blocks. (Tomasz Nykiel via todd) - - HDFS-3378. Remove DFS_NAMENODE_SECONDARY_HTTPS_PORT_KEY and DEFAULT. (eli) - - BUG FIXES - - HDFS-2481. Unknown protocol: org.apache.hadoop.hdfs.protocol.ClientProtocol. - (sanjay) - - HDFS-2497. Fix TestBackupNode failure. (suresh) - - HDFS-2499. RPC client is created incorrectly introduced in HDFS-2459. - (suresh) - - HDFS-2526. (Client)NamenodeProtocolTranslatorR23 do not need to keep a - reference to rpcProxyWithoutRetry (atm) - - HDFS-2532. TestDfsOverAvroRpc timing out in trunk (Uma Maheswara Rao G - via todd) - - HDFS-2666. Fix TestBackupNode failure. (suresh) - - HDFS-2663. Optional protobuf parameters are not handled correctly. (suresh) - - HDFS-2694. Removal of Avro broke non-PB NN services. (atm) - - HDFS-2687. Tests failing with ClassCastException post protobuf RPC - changes. (suresh) - - HDFS-2700. Fix failing TestDataNodeMultipleRegistrations in trunk - (Uma Maheswara Rao G via todd) - - HDFS-2739. SecondaryNameNode doesn't start up. (jitendra) - - HDFS-2768. BackupNode stop can not close proxy connections because - it is not a proxy instance. (Uma Maheswara Rao G via eli) - - HDFS-2968. Protocol translator for BlockRecoveryCommand broken when - multiple blocks need recovery. (todd) - - HDFS-3020. Fix editlog to automatically sync when buffer is full. (todd) - - HDFS-3038. Add FSEditLog.metrics to findbugs exclude list. (todd via atm) - - HDFS-2285. BackupNode should reject requests to modify namespace. - (shv and Uma Maheswara Rao) - - HDFS-2764. TestBackupNode is racy. (atm) - - HDFS-2188. Make FSEditLog create its journals from a list of URIs rather - than NNStorage. (Ivan Kelly via jitendra) - - HDFS-1765. Block Replication should respect under-replication - block priority. (Uma Maheswara Rao G via eli) - - HDFS-3093. Fix bug where namenode -format interpreted the -force flag in - reverse. (todd) - - HDFS-3005. FSVolume.decDfsUsed(..) should be synchronized. (szetszwo) - - HDFS-3099. SecondaryNameNode does not properly initialize metrics system. - (atm) - - HDFS-3062. Fix bug which prevented MR job submission from creating - delegation tokens on an HA cluster. (Mingjie Lai via todd) - - HDFS-3083. Cannot run an MR job with HA and security enabled when - second-listed NN active. (atm) - - HDFS-3100. In BlockSender, throw an exception when it needs to verify - checksum but the meta data does not exist. (Brandon Li via szetszwo) - - HDFS-3132. Fix findbugs warning on HDFS trunk. (todd) - - HDFS-3156. TestDFSHAAdmin is failing post HADOOP-8202. (atm) - - HDFS-3143. TestGetBlocks.testGetBlocks is failing. (Arpit Gupta via atm) - - HDFS-3142. TestHDFSCLI.testAll is failing. (Brandon Li via atm) - - HDFS-3070. HDFS balancer doesn't ensure that hdfs-site.xml is loaded. (atm) - - HDFS-2995. start-dfs.sh should only start the 2NN for namenodes - with dfs.namenode.secondary.http-address configured. (eli) - - HDFS-3174. Fix assert in TestPendingDataNodeMessages. (eli) - - HDFS-3199. TestValidateConfigurationSettings is failing. (todd via eli) - - HDFS-3202. NamespaceInfo PB translation drops build version. (atm) - - HDFS-3109. Remove hsqldf exclusions from pom.xml. (Ravi Prakash - via suresh) - - HDFS-3210. JsonUtil#toJsonMap for for a DatanodeInfo should use - "ipAddr" instead of "name". (eli) - - HDFS-3208. Bogus entries in hosts files are incorrectly displayed - in the report. (eli) - - HDFS-3136. Remove SLF4J dependency as HDFS does not need it to fix - unnecessary warnings. (Jason Lowe via suresh) - - HDFS-3214. InterDatanodeProtocolServerSideTranslatorPB doesn't handle - null response from initReplicaRecovery (todd) - - HDFS-3234. Accidentally left log message in GetConf after HDFS-3226 (todd) - - HDFS-3236. NameNode does not initialize generic conf keys when started - with -initializeSharedEditsDir (atm) - - HDFS-3248. bootstrapStandby repeated twice in hdfs namenode usage message - (Colin Patrick McCabe via todd) - - HDFS-2696. Fix the fuse-fds build. (Bruno Mahé via eli) - - HDFS-3260. TestDatanodeRegistration should set minimum DN version in - addition to minimum NN version. (atm) - - HDFS-3255. HA DFS returns wrong token service (Daryn Sharp via todd) - - HDFS-3256. HDFS considers blocks under-replicated if topology script is - configured with only 1 rack. (atm) - - HDFS-2799. Trim fs.checkpoint.dir values. (Amith D K via eli) - - HDFS-2765. TestNameEditsConfigs is incorrectly swallowing IOE. (atm) - - HDFS-3268. FileContext API mishandles token service and incompatible with - HA (Daryn Sharp via todd) - - HDFS-3284. bootstrapStandby fails in secure cluster (todd) - - HDFS-3165. HDFS Balancer scripts are refering to wrong path of - hadoop-daemon.sh (Amith D K via eli) - - HDFS-891. DataNode no longer needs to check for dfs.network.script. - (harsh via eli) - - HDFS-3305. GetImageServlet should consider SBN a valid requestor in a - secure HA setup. (atm) - - HDFS-3314. HttpFS operation for getHomeDirectory is incorrect. (tucu) - - HDFS-3319. Change DFSOutputStream to not to start a thread in constructors. - (szetszwo) - - HDFS-3181. Fix a test case in TestLeaseRecovery2. (szetszwo) - - HDFS-3309. HttpFS (Hoop) chmod not supporting octal and sticky bit - permissions. (tucu) - - HDFS-3326. Append enabled log message uses the wrong variable. - (Matthew Jacobs via eli) - - HDFS-3336. hdfs launcher script will be better off not special casing - namenode command with regards to hadoop.security.logger (rvs via tucu) - - HDFS-3330. If GetImageServlet throws an Error or RTE, response should not - have HTTP "OK" status. (todd) - - HDFS-3351. NameNode#initializeGenericKeys should always set fs.defaultFS - regardless of whether HA or Federation is enabled. (atm) - - HDFS-3359. DFSClient.close should close cached sockets. (todd) - - HDFS-3350. In INode, add final to compareTo(..), equals(..) and hashCode(), - and remove synchronized from updatePermissionStatus(..). (szetszwo) - - HDFS-3357. DataXceiver reads from client socket with incorrect/no timeout - (todd) - - HDFS-3376. DFSClient fails to make connection to DN if there are many - unusable cached sockets (todd) - - HDFS-3328. NPE in DataNode.getIpcPort. (eli) - - HDFS-3396. FUSE build fails on Ubuntu 12.04. (Colin Patrick McCabe via eli) - - HDFS-3395. NN doesn't start with HA+security enabled and HTTP address - set to 0.0.0.0. (atm) - - HDFS-3026. HA: Handle failure during HA state transition. (atm) - - HDFS-860. fuse-dfs truncate behavior causes issues with scp. - (Brian Bockelman via eli) - - BREAKDOWN OF HDFS-1623 SUBTASKS - - HDFS-2179. Add fencing framework and mechanisms for NameNode HA. (todd) - - HDFS-1974. Introduce active and standy states to the namenode. (suresh) - - HDFS-2407. getServerDefaults and getStats don't check operation category (atm) - - HDFS-1973. HA: HDFS clients must handle namenode failover and switch over to - the new active namenode. (atm) - - HDFS-2301. Start/stop appropriate namenode services when transition to active - and standby states. (suresh) - - HDFS-2231. Configuration changes for HA namenode. (suresh) - - HDFS-2418. Change ConfiguredFailoverProxyProvider to take advantage of - HDFS-2231. (atm) - - HDFS-2393. Mark appropriate methods of ClientProtocol with the idempotent - annotation. (atm) - - HDFS-2523. Small NN fixes to include HAServiceProtocol and prevent NPE on - shutdown. (todd) - - HDFS-2577. NN fails to start since it tries to start secret manager in - safemode. (todd) - - HDFS-2582. Scope dfs.ha.namenodes config by nameservice (todd) - - HDFS-2591. MiniDFSCluster support to mix and match federation with HA (todd) - - HDFS-1975. Support for sharing the namenode state from active to standby. - (jitendra, atm, todd) - - HDFS-1971. Send block report from datanode to both active and standby - namenodes. (sanjay, todd via suresh) - - HDFS-2616. Change DatanodeProtocol#sendHeartbeat() to return HeartbeatResponse. - (suresh) - - HDFS-2622. Fix TestDFSUpgrade in HA branch. (todd) - - HDFS-2612. Handle refreshNameNodes in federated HA clusters (todd) - - HDFS-2623. Add test case for hot standby capability (todd) - - HDFS-2626. BPOfferService.verifyAndSetNamespaceInfo needs to be synchronized - (todd) - - HDFS-2624. ConfiguredFailoverProxyProvider doesn't correctly stop - ProtocolTranslators (todd) - - HDFS-2625. TestDfsOverAvroRpc failing after introduction of HeartbeatResponse - type (todd) - - HDFS-2627. Determine DN's view of which NN is active based on heartbeat - responses (todd) - - HDFS-2634. Standby needs to ingest latest edit logs before transitioning to - active (todd) - - HDFS-2671. NN should throw StandbyException in response to RPCs in STANDBY - state (todd) - - HDFS-2680. DFSClient should construct failover proxy with exponential backoff - (todd) - - HDFS-2683. Authority-based lookup of proxy provider fails if path becomes - canonicalized (todd) - - HDFS-2689. HA: BookKeeperEditLogInputStream doesn't implement isInProgress() - (atm) - - HDFS-2602. NN should log newly-allocated blocks without losing BlockInfo (atm) - - HDFS-2667. Fix transition from active to standby (todd) - - HDFS-2684. Fix up some failing unit tests on HA branch (todd) - - HDFS-2679. Add interface to query current state to HAServiceProtocol (eli via - todd) - - HDFS-2677. Web UI should indicate the NN state. (eli via todd) - - HDFS-2678. When a FailoverProxyProvider is used, DFSClient should not retry - connection ten times before failing over (atm via todd) - - HDFS-2682. When a FailoverProxyProvider is used, Client should not retry for 45 - times if it is timing out to connect to server. (Uma Maheswara Rao G via todd) - - HDFS-2693. Fix synchronization issues around state transition (todd) - - HDFS-1972. Fencing mechanism for block invalidations and replications (todd) - - HDFS-2714. Fix test cases which use standalone FSNamesystems (todd) - - HDFS-2692. Fix bugs related to failover from/into safe mode. (todd) - - HDFS-2716. Configuration needs to allow different dfs.http.addresses for each - HA NN (todd) - - HDFS-2720. Fix MiniDFSCluster HA support to work properly on Windows. (Uma - Maheswara Rao G via todd) - - HDFS-2291. Allow the StandbyNode to make checkpoints in an HA setup. (todd) - - HDFS-2709. Appropriately handle error conditions in EditLogTailer (atm via - todd) - - HDFS-2730. Refactor shared HA-related test code into HATestUtil class (todd) - - HDFS-2762. Fix TestCheckpoint timing out on HA branch. (Uma Maheswara Rao G via - todd) - - HDFS-2724. NN web UI can throw NPE after startup, before standby state is - entered. (todd) - - HDFS-2753. Fix standby getting stuck in safemode when blocks are written while - SBN is down. (Hari Mankude and todd via todd) - - HDFS-2773. Reading edit logs from an earlier version should not leave blocks in - under-construction state. (todd) - - HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing - intermittently. (todd) - - HDFS-2766. Test for case where standby partially reads log and then performs - checkpoint. (atm) - - HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress streams - even when non-in-progress are requested. (atm) - - HDFS-2789. TestHAAdmin.testFailover is failing (eli) - - HDFS-2747. Entering safe mode after starting SBN can NPE. (Uma Maheswara Rao G - via todd) - - HDFS-2772. On transition to active, standby should not swallow ELIE. (atm) - - HDFS-2767. ConfiguredFailoverProxyProvider should support NameNodeProtocol. - (Uma Maheswara Rao G via todd) - - HDFS-2795. Standby NN takes a long time to recover from a dead DN starting up. - (todd) - - HDFS-2592. Balancer support for HA namenodes. (Uma Maheswara Rao G via todd) - - HDFS-2367. Enable the configuration of multiple HA cluster addresses. (atm) - - HDFS-2812. When becoming active, the NN should treat all leases as freshly - renewed. (todd) - - HDFS-2737. Automatically trigger log rolls periodically on the active NN. (todd - and atm) - - HDFS-2820. Add a simple sanity check for HA config (todd) - - HDFS-2688. Add tests for quota tracking in an HA cluster. (todd) - - HDFS-2804. Should not mark blocks under-replicated when exiting safemode (todd) - - HDFS-2807. Service level authorizartion for HAServiceProtocol. (jitendra) - - HDFS-2809. Add test to verify that delegation tokens are honored after - failover. (jitendra and atm) - - HDFS-2838. NPE in FSNamesystem when in safe mode. (Gregory Chanan via eli) - - HDFS-2805. Add a test for a federated cluster with HA NNs. (Brandon Li via - jitendra) - - HDFS-2841. HAAdmin does not work if security is enabled. (atm) - - HDFS-2691. Fixes for pipeline recovery in an HA cluster: report RBW replicas - immediately upon pipeline creation. (todd) - - HDFS-2824. Fix failover when prior NN died just after creating an edit log - segment. (atm via todd) - - HDFS-2853. HA: NN fails to start if the shared edits dir is marked required - (atm via eli) - - HDFS-2845. SBN should not allow browsing of the file system via web UI. (Bikas - Saha via atm) - - HDFS-2742. HA: observed dataloss in replication stress test. (todd via eli) - - HDFS-2870. Fix log level for block debug info in processMisReplicatedBlocks - (todd) - - HDFS-2859. LOCAL_ADDRESS_MATCHER.match has NPE when called from - DFSUtil.getSuffixIDs when the host is incorrect (Bikas Saha via todd) - - HDFS-2861. checkpointing should verify that the dfs.http.address has been - configured to a non-loopback for peer NN (todd) - - HDFS-2860. TestDFSRollback#testRollback is failing. (atm) - - HDFS-2769. HA: When HA is enabled with a shared edits dir, that dir should be - marked required. (atm via eli) - - HDFS-2863. Failures observed if dfs.edits.dir and shared.edits.dir have same - directories. (Bikas Saha via atm) - - HDFS-2874. Edit log should log to shared dirs before local dirs. (todd) - - HDFS-2890. DFSUtil#getSuffixIDs should skip unset configurations. (atm) - - HDFS-2792. Make fsck work. (atm) - - HDFS-2808. HA: haadmin should use namenode ids. (eli) - - HDFS-2819. Document new HA-related configs in hdfs-default.xml. (eli) - - HDFS-2752. HA: exit if multiple shared dirs are configured. (eli) - - HDFS-2894. HA: automatically determine the nameservice Id if only one - nameservice is configured. (eli) - - HDFS-2733. Document HA configuration and CLI. (atm) - - HDFS-2794. Active NN may purge edit log files before standby NN has a chance to - read them (todd) - - HDFS-2901. Improvements for SBN web UI - not show under-replicated/missing - blocks. (Brandon Li via jitendra) - - HDFS-2905. HA: Standby NN NPE when shared edits dir is deleted. (Bikas Saha via - jitendra) - - HDFS-2579. Starting delegation token manager during safemode fails. (todd) - - HDFS-2510. Add HA-related metrics. (atm) - - HDFS-2924. Standby checkpointing fails to authenticate in secure cluster. - (todd) - - HDFS-2915. HA: TestFailureOfSharedDir.testFailureOfSharedDir() has race - condition. (Bikas Saha via jitendra) - - HDFS-2912. Namenode not shutting down when shared edits dir is inaccessible. - (Bikas Saha via atm) - - HDFS-2917. HA: haadmin should not work if run by regular user (eli) - - HDFS-2939. TestHAStateTransitions fails on Windows. (Uma Maheswara Rao G via - atm) - - HDFS-2947. On startup NN throws an NPE in the metrics system. (atm) - - HDFS-2942. TestActiveStandbyElectorRealZK fails if build dir does not exist. - (atm) - - HDFS-2948. NN throws NPE during shutdown if it fails to startup (todd) - - HDFS-2909. HA: Inaccessible shared edits dir not getting removed from FSImage - storage dirs upon error. (Bikas Saha via jitendra) - - HDFS-2934. Allow configs to be scoped to all NNs in the nameservice. (todd) - - HDFS-2935. Shared edits dir property should be suffixed with nameservice and - namenodeID (todd) - - HDFS-2928. ConfiguredFailoverProxyProvider should not create a NameNode proxy - with an underlying retry proxy. (Uma Maheswara Rao G via atm) - - HDFS-2955. IllegalStateException during standby startup in getCurSegmentTxId. - (Hari Mankude via atm) - - HDFS-2937. TestDFSHAAdmin needs tests with MiniDFSCluster. (Brandon Li via - suresh) - - HDFS-2586. Add protobuf service and implementation for HAServiceProtocol. - (suresh via atm) - - HDFS-2952. NN should not start with upgrade option or with a pending an - unfinalized upgrade. (atm) - - HDFS-2974. MiniDFSCluster does not delete standby NN name dirs during format. - (atm) - - HDFS-2929. Stress test and fixes for block synchronization (todd) - - HDFS-2972. Small optimization building incremental block report (todd) - - HDFS-2973. Re-enable NO_ACK optimization for block deletion. (todd) - - HDFS-2922. HA: close out operation categories (eli) - - HDFS-2993. HA: BackupNode#checkOperation should permit CHECKPOINT operations - (eli) - - HDFS-2904. Client support for getting delegation tokens. (todd) - - HDFS-3013. HA: NameNode format doesn't pick up - dfs.namenode.name.dir.NameServiceId configuration (Mingjie Lai via todd) - - HDFS-3019. Fix silent failure of TestEditLogJournalFailures (todd) - - HDFS-2958. Sweep for remaining proxy construction which doesn't go through - failover path. (atm) - - HDFS-2920. fix remaining TODO items. (atm and todd) - - HDFS-3027. Implement a simple NN health check. (atm) - - HDFS-3023. Optimize entries in edits log for persistBlocks call. (todd) - - HDFS-2979. Balancer should use logical uri for creating failover proxy with HA - enabled. (atm) - - HDFS-3035. Fix failure of TestFileAppendRestart due to OP_UPDATE_BLOCKS (todd) - - HDFS-3039. Address findbugs and javadoc warnings on branch. (todd via atm) - -Release 0.23.10 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - HDFS-5010. Reduce the frequency of getCurrentUser() calls from namenode - (kihwal) - - HDFS-5346. Avoid unnecessary call to getNumLiveDataNodes() for each block - during IBR processing (Ravi Prakash via kihwal) - - OPTIMIZATIONS - - BUG FIXES - - HDFS-4998. TestUnderReplicatedBlocks fails intermittently (kihwal) - - HDFS-4329. DFSShell issues with directories with spaces in name (Cristina - L. Abad via jeagles) - - HDFS-5526. Datanode cannot roll back to previous layout version (kihwal) - - HDFS-5557. Write pipeline recovery for the last packet in the block may - cause rejection of valid replicas. (kihwal) - - HDFS-5558. LeaseManager monitor thread can crash if the last block is - complete but another block is not. (kihwal) - -Release 0.23.9 - 2013-07-08 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-4867. metaSave NPEs when there are invalid blocks in repl queue. - (Plamen Jeliazkov and Ravi Prakash via shv) - - HDFS-4862. SafeModeInfo.isManual() returns true when resources are low even - if it wasn't entered into manually (Ravi Prakash via kihwal) - - HDFS-4832. Namenode doesn't change the number of missing blocks in - safemode when DNs rejoin or leave (Ravi Prakash via kihwal) - - HDFS-4878. On Remove Block, block is not removed from neededReplications - queue. (Tao Luo via shv) - - HDFS-4205. fsck fails with symlinks. (jlowe) - -Release 0.23.8 - 2013-06-05 - - INCOMPATIBLE CHANGES - - HDFS-4714. Log short messages in Namenode RPC server for exceptions - meant for clients. (kihwal) - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-4477. Secondary namenode may retain old tokens (daryn via kihwal) - - HDFS-4699. TestPipelinesFailover#testPipelineRecoveryStress fails - sporadically (Chris Nauroth via kihwal) - - HDFS-4805. Webhdfs client is fragile to token renewal errors - (daryn via kihwal) - - HDFS-3875. Issue handling checksum errors in write pipeline. (kihwal) - - HDFS-4807. createSocketForPipeline() should not include timeout extension - on connect. (Cristina L. Abad via kihwal) - -Release 0.23.7 - 2013-04-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - HDFS-4532. RPC call queue may fill due to current user lookup (daryn) - - BUG FIXES - - HDFS-4288. NN accepts incremental BR as IBR in safemode (daryn via kihwal) - - HDFS-4495. Allow client-side lease renewal to be retried beyond soft-limit - (kihwal) - - HDFS-4128. 2NN gets stuck in inconsistent state if edit log replay fails - in the middle (kihwal via daryn) - - HDFS-4542. Webhdfs doesn't support secure proxy users (Daryn Sharp via - kihwal) - - HDFS-4560. Webhdfs cannot use tokens obtained by another user (daryn) - - HDFS-4566. Webdhfs token cancelation should use authentication (daryn) - - HDFS-4567. Webhdfs does not need a token for token operations (daryn via - kihwal) - - HDFS-4577. Webhdfs operations should declare if authentication is required - (daryn via kihwal) - - HDFS-3344. Unreliable corrupt blocks counting in TestProcessCorruptBlocks - (kihwal) - - HDFS-3367. WebHDFS doesn't use the logged in user when opening - connections (daryn) - - HDFS-4581. checkDiskError should not be called on network errors (Rohit - Kochar via kihwal) - - HDFS-4649. Webhdfs cannot list large directories (daryn via kihwal) - - HDFS-4548. Webhdfs doesn't renegotiate SPNEGO token (daryn via kihwal) - -Release 0.23.6 - 2013-02-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-4247. saveNamespace should be tolerant of dangling lease (daryn) - - HDFS-4248. Renaming directories may incorrectly remove the paths in leases - under the tree. (daryn via szetszwo) - - HDFS-4385. Maven RAT plugin is not checking all source files (tgraves) - - HDFS-4426. Secondary namenode shuts down immediately after startup. - (Arpit Agarwal via suresh) - -Release 0.23.5 - 2012-11-28 - - INCOMPATIBLE CHANGES - - HDFS-4080. Add a separate logger for block state change logs to enable turning - off those logs. (Kihwal Lee via suresh) - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - HDFS-4075. Reduce recommissioning overhead (Kihwal Lee via daryn) - - HDFS-3990. NN's health report has severe performance problems (daryn) - - HDFS-4181. LeaseManager tries to double remove and prints extra messages - (Kihwal Lee via daryn) - - BUG FIXES - - HDFS-3829. TestHftpURLTimeouts fails intermittently with JDK7 (Trevor - Robinson via tgraves) - - HDFS-3824. TestHftpDelegationToken fails intermittently with JDK7 (Trevor - Robinson via tgraves) - - HDFS-3224. Bug in check for DN re-registration with different storage ID - (jlowe) - - HDFS-4090. getFileChecksum() result incompatible when called against - zero-byte files. (Kihwal Lee via daryn) - - HDFS-4172. namenode does not URI-encode parameters when building URI for - datanode request (Derek Dagit via bobby) - - HDFS-4182. SecondaryNameNode leaks NameCache entries (bobby) - - HDFS-4186. logSync() is called with the write lock held while releasing - lease (Kihwal Lee via daryn) - -Release 0.23.4 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-3831. Failure to renew tokens due to test-sources left in classpath - (jlowe via bobby) - -Release 0.23.3 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - HDFS-3166. Add timeout to Hftp connections. (Daryn Sharp via szetszwo) - - HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . (Kihwal - Lee via szetszwo) - - HDFS-2652. Add support for host-based delegation tokens. (Daryn Sharp via - szetszwo) - - HDFS-3308. Uses canonical URI to select delegation tokens in HftpFileSystem - and WebHdfsFileSystem. (Daryn Sharp via szetszwo) - - HDFS-3312. In HftpFileSystem, the namenode URI is non-secure but the - delegation tokens have to use secure URI. (Daryn Sharp via szetszwo) - - HDFS-3318. Use BoundedInputStream in ByteRangeInputStream, otherwise, it - hangs on transfers >2 GB. (Daryn Sharp via szetszwo) - - HDFS-3321. Fix safe mode turn off tip message. (Ravi Prakash via szetszwo) - - HDFS-3334. Fix ByteRangeInputStream stream leakage. (Daryn Sharp via - szetszwo) - - HDFS-3331. In namenode, check superuser privilege for setBalancerBandwidth - and acquire the write lock for finalizeUpgrade. (szetszwo) - - HDFS-3577. In DatanodeWebHdfsMethods, use MessageBodyWriter instead of - StreamingOutput, otherwise, it will fail to transfer large files. - (szetszwo) - - HDFS-3646. LeaseRenewer can hold reference to inactive DFSClient - instances forever. (Kihwal Lee via daryn) - - HDFS-3696. Set chunked streaming mode in WebHdfsFileSystem write operations - to get around a Java library bug causing OutOfMemoryError. (szetszwo) - - HDFS-3553. Hftp proxy tokens are broken (daryn) - - HDFS-3718. Datanode won't shutdown because of runaway DataBlockScanner - thread (Kihwal Lee via daryn) - - HDFS-3861. Deadlock in DFSClient (Kihwal Lee via daryn) - - HDFS-3873. Hftp assumes security is disabled if token fetch fails (daryn) - - HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) - - HDFS-3890. filecontext mkdirs doesn't apply umask as expected - (Tom Graves via daryn) - -Release 0.23.2 - UNRELEASED - - INCOMPATIBLE CHANGES - - HDFS-2887. FSVolume, is a part of FSDatasetInterface implementation, should - not be referred outside FSDataset. A new FSVolumeInterface is defined. - The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also - updated. (szetszwo) - - NEW FEATURES - - HDFS-2943. Expose last checkpoint time and transaction stats as JMX - metrics. (atm) - - IMPROVEMENTS - - HDFS-2931. Switch DataNode's BlockVolumeChoosingPolicy to private-audience. - (harsh via szetszwo) - - HDFS-2725. hdfs script usage information is missing the information - about "dfs" command (Prashant Sharma via stevel) - - HDFS-2907. Add a conf property dfs.datanode.fsdataset.factory to make - FSDataset in Datanode pluggable. (szetszwo) - - HDFS-2985. Improve logging when replicas are marked as corrupt. (todd) - - HDFS-3098. Update and add tests for HADOOP-8173. (Daryn Sharp via szetszwo) - - HDFS-3104. Add tests for HADOOP-8175. (Daryn Sharp via szetszwo) - - HDFS-3066. Cap space usage of default log4j rolling policy. - (Patrick Hunt via eli) - - OPTIMIZATIONS - - BUG FIXES - HDFS-2923. Namenode IPC handler count uses the wrong configuration key - (todd) - - HDFS-2869. Fix an error in the webhdfs docs for the mkdir op (harsh) - - HDFS-776. Fix exception handling in Balancer. (Uma Maheswara Rao G - via szetszwo) - - HDFS-2815. Namenode sometimes oes not come out of safemode during - NN crash + restart. (Uma Maheswara Rao via suresh) - - HDFS-2950. Secondary NN HTTPS address should be listed as a - NAMESERVICE_SPECIFIC_KEY. (todd) - - HDFS-2525. Race between BlockPoolSliceScanner and append. (Brandon Li - via jitendra) - - HDFS-2938. Recursive delete of a large directory make namenode - unresponsive. (Hari Mankude via suresh) - - HDFS-2969. ExtendedBlock.equals is incorrectly implemented (todd) - - HDFS-2944. Typo in hdfs-default.xml causes - dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly - disabled. (atm) - - HDFS-2981. In hdfs-default.xml, the default value of - dfs.client.block.write.replace-datanode-on-failure.enable should be true. - (szetszwo) - - HDFS-3008. Negative caching of local addrs doesn't work. (eli) - - HDFS-3006. In WebHDFS, when the return body is empty, set the Content-Type - to application/octet-stream instead of application/json. (szetszwo) - - HDFS-2991. Fix case where OP_ADD would not be logged in append(). (todd) - - HDFS-3012. Exception while renewing delegation token. (Bobby Evans via - jitendra) - - HDFS-3032. Change DFSClient.renewLease() so that it only retries up to the - lease soft-limit. (Kihwal Lee via szetszwo) - - HDFS-2038. Update TestHDFSCLI to handle relative paths with globs. - (Kihwal Lee via szetszwo) - - HDFS-3101. Cannot read empty file using WebHDFS. (szetszwo) - - HDFS-3160. httpfs should exec catalina instead of forking it. - (Roman Shaposhnik via eli) - -Release 0.23.1 - 2012-02-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - HDFS-2316. [umbrella] WebHDFS: a complete FileSystem implementation for - accessing HDFS over HTTP (szetszwo) - - HDFS-2594. Support getDelegationTokens and createSymlink in WebHDFS. - (szetszwo) - - HDFS-2545. Change WebHDFS to support multiple namenodes in federation. - (szetszwo) - - HDFS-2178. Contributing Hoop to HDFS, replacement for HDFS proxy with - read/write capabilities. (tucu) - - IMPROVEMENTS - HDFS-2560. Refactor BPOfferService to be a static inner class (todd) - - HDFS-2544. Hadoop scripts unconditionally source - "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) - - HDFS-2543. HADOOP_PREFIX cannot be overridden. (Bruno Mahé via tomwhite) - - HDFS-2562. Refactor DN configuration variables out of DataNode class - (todd) - - HDFS-2563. Some cleanup in BPOfferService. (todd) - - HDFS-2568. Use a set to manage child sockets in XceiverServer. - (harsh via eli) - - HDFS-2454. Move maxXceiverCount check to before starting the - thread in dataXceiver. (harsh via eli) - - HDFS-2570. Add descriptions for dfs.*.https.address in hdfs-default.xml. - (eli) - - HDFS-2536. Remove unused imports. (harsh via eli) - - HDFS-2566. Move BPOfferService to be a non-inner class. (todd) - - HDFS-2552. Add Forrest doc for WebHDFS REST API. (szetszwo) - - HDFS-2587. Add apt doc for WebHDFS REST API. (szetszwo) - - HDFS-2604. Add a log message to show if WebHDFS is enabled and a - configuration section in the forrest doc. (szetszwo) - - HDFS-2511. Add dev script to generate HDFS protobufs. (tucu) - - HDFS-2654. Make BlockReaderLocal not extend RemoteBlockReader2. (eli) - - HDFS-2675. Reduce warning verbosity when double-closing edit logs - (todd) - - HDFS-2335. DataNodeCluster and NNStorage always pull fresh entropy. - (Uma Maheswara Rao G via eli) - - HDFS-2574. Remove references to some deprecated properties in conf - templates and defaults files. (Joe Crobak via harsh) - - HDFS-2722. HttpFs should not be using an int for block size. (harsh) - - HDFS-2710. Add HDFS tests related to HADOOP-7933. (sid via suresh) - - HDFS-2349. Corruption detected during block transfers between DNs - should log a WARN instead of INFO. (harsh) - - HDFS-2729. Update BlockManager's comments regarding the invalid block - set (harsh) - - HDFS-2726. Fix a logging issue under DFSClient's createBlockOutputStream - method (harsh) - - HDFS-554. Use System.arraycopy in BlockInfo.ensureCapacity. (harsh) - - HDFS-1314. Make dfs.blocksize accept size-indicating prefixes. - (Sho Shimauchi via harsh) - - HDFS-69. Improve the 'dfsadmin' commandline help. (harsh) - - HDFS-2788. HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code. (eli) - - HDFS-362. FSEditLog should not writes long and short as UTF8, and should - not use ArrayWritable for writing non-array items. (Uma Maheswara Rao G - via szetszwo) - - HDFS-2803. Add logging to LeaseRenewer for better lease expiration debugging. - (Jimmy Xiang via todd) - - HDFS-2817. Combine the two TestSafeMode test suites. (todd) - - HDFS-2818. Fix a missing space issue in HDFS webapps' title tags. - (Devaraj K via harsh) - - HDFS-2397. Undeprecate SecondaryNameNode. (eli) - - HDFS-2814 NamenodeMXBean does not account for svn revision in the version - information. (Hitesh Shah via jitendra) - - HDFS-2784. Update hftp and hdfs for host-based token support. - (Kihwal Lee via jitendra) - - HDFS-2785. Update webhdfs and httpfs for host-based token support. - (Robert Joseph Evans via jitendra) - - HDFS-2868. Expose xceiver counts via the DataNode MXBean. (harsh) - - HDFS-3139. Minor Datanode logging improvement. (eli) - - OPTIMIZATIONS - - HDFS-2130. Switch default checksum to CRC32C. (todd) - - HDFS-2533. Remove needless synchronization on some FSDataSet methods. - (todd) - - HDFS-2129. Simplify BlockReader to not inherit from FSInputChecker. - (todd) - - HDFS-2246. Enable reading a block directly from local file system - for a client on the same node as the block file. (Andrew Purtell, - Suresh Srinivas and Jitendra Nath Pandey via szetszwo) - - HDFS-2825. Add test hook to turn off the writer preferring its local - DN. (todd) - - HDFS-2826. Add test case for HDFS-1476 (safemode can initialize - replication queues before exiting) (todd) - - HDFS-2864. Remove some redundant methods and the constant METADATA_VERSION - from FSDataset. (szetszwo) - - HDFS-2879. Change FSDataset to package private. (szetszwo) - - BUG FIXES - - HDFS-2541. For a sufficiently large value of blocks, the DN Scanner - may request a random number with a negative seed value. (harsh via eli) - - HDFS-2502. hdfs-default.xml should include dfs.name.dir.restore. - (harsh via eli) - - HDFS-2567. When 0 DNs are available, show a proper error when - trying to browse DFS via web UI. (harsh via eli) - - HDFS-2575. DFSTestUtil may create empty files (todd) - - HDFS-2588. hdfs jsp pages missing DOCTYPE. (Dave Vronay via mattf) - - HDFS-2590. Fix the missing links in the WebHDFS forrest doc. (szetszwo) - - HDFS-2596. TestDirectoryScanner doesn't test parallel scans. (eli) - - HDFS-2606. webhdfs client filesystem impl must set the content-type - header for create/append. (tucu) - - HDFS-2614. hadoop dist tarball is missing hdfs headers. (tucu) - - HDFS-2653. DFSClient should cache whether addrs are non-local when - short-circuiting is enabled. (eli) - - HDFS-2649. eclipse:eclipse build fails for hadoop-hdfs-httpfs. - (Jason Lowe via eli) - - HDFS-2640. Javadoc generation hangs. (tomwhite) - - HDFS-2553. Fix BlockPoolSliceScanner spinning in a tight loop (Uma - Maheswara Rao G via todd) - - HDFS-2658. HttpFS introduced 70 javadoc warnings. (tucu) - - HDFS-2706. Use configuration for blockInvalidateLimit if it is set. - (szetszwo) - - HDFS-2646. Hadoop HttpFS introduced 4 findbug warnings. (tucu) - - HDFS-2657. TestHttpFSServer and TestServerWebApp are failing on trunk. - (tucu) - - HDFS-2705. HttpFS server should check that upload requests have correct - content-type. (tucu) - - HDFS-2707. HttpFS should read the hadoop-auth secret from a file - instead inline from the configuration. (tucu) - - HDFS-2790. FSNamesystem.setTimes throws exception with wrong - configuration name in the message. (Arpit Gupta via eli) - - HDFS-2810. Leases not getting renewed properly by clients (todd) - - HDFS-2751. Datanode may incorrectly drop OS cache behind reads - even for short reads. (todd) - - HDFS-2816. Fix missing license header in httpfs findbugsExcludeFile.xml. - (hitesh via tucu) - - HDFS-2822. processMisReplicatedBlock incorrectly identifies - under-construction blocks as under-replicated. (todd) - - HDFS-442. dfsthroughput in test jar throws NPE (harsh) - - HDFS-2836. HttpFSServer still has 2 javadoc warnings in trunk. - (revans2 via tucu) - - HDFS-2837. mvn javadoc:javadoc not seeing LimitedPrivate class - (revans2 via tucu) - - HDFS-2840. TestHostnameFilter should work with localhost or - localhost.localdomain (tucu) - - HDFS-2791. If block report races with closing of file, replica is - incorrectly marked corrupt. (todd) - - HDFS-2827. When the parent of a directory is the root, renaming the - directory results in leases updated incorrectly. (Uma Maheswara Rao G - via szetszwo) - - HDFS-2835. Fix findbugs and javadoc issue with GetConf.java. - (suresh) - - HDFS-2889. getNumCurrentReplicas is package private but should be public on - 0.23 (see HDFS-2408). (Gregory Chanan via atm) - - HDFS-2893. The start/stop scripts don't start/stop the 2NN when - using the default configuration. (eli) - -Release 0.23.0 - 2011-11-01 - - INCOMPATIBLE CHANGES - - HDFS-1526. Dfs client name for a map/reduce task should be unique - among threads. (hairong) - - HDFS-1536. Improve HDFS WebUI. (hairong) - - HDFS-2210. Remove hdfsproxy. (eli) - - HDFS-1073. Redesign the NameNode's storage layout for image checkpoints - and edit logs to introduce transaction IDs and be more robust. - Please see HDFS-1073 section below for breakout of individual patches. - - NEW FEATURES - - HDFS-1359. Add BlockPoolID to Block. (suresh) - - HDFS-1365. Federation: propose ClusterID and BlockPoolID format - (Tanping via boryas) - - HDFS-1394. Federation: modify -format option for namenode to generated - new blockpool id and accept newcluster (boryas) - - HDFS-1400. Federation: DataTransferProtocol uses ExtendedBlockPool to - include BlockPoolID in the protocol. (suresh) - - HDFS-1428. Federation : add cluster ID and block pool ID into - Name node web UI(Tanping via boryas) - - HDFS-1450. Federation: Introduce block pool ID into FSDatasetInterface. - (suresh) - - HDFS-1632. Federation: data node storage structure changes and - introduce block pool storage. (Tanping via suresh) - - HDFS-1634. Federation: Convert single threaded DataNode into - per BlockPool thread model.(boryas) - - HDFS-1637. Federation: FSDataset in Datanode should be created after - initial handshake with namenode. (boryas and jitendra) - - HDFS-1653. Federation: Block received message from datanode sends invalid - DatanodeRegistration. (Tanping via suresh) - - HDFS-1645. Federation: DatanodeCommond.Finalize needs to include - BlockPoolId. (suresh) - - HDFS-1638. Federation: DataNode.handleDiskError needs to inform - ALL namenodes if a disk failed (boryas) - - HDFS-1647. Federation: Multiple namenode configuration. (jitendra) - - HDFS-1639. Federation: Add block pool management to FSDataset. (suresh) - - HDFS-1648. Federation: Only DataStorage must be locked using in_use.lock - and no locks must be associated with BlockPoolStorage. (Tanping via suresh) - - HDFS-1641. Federation: Datanode fields that are no longer used should - be removed (boryas) - - HDFS-1642. Federation: add Datanode.getDNRegistration(String bpid) - method (boryas) - - HDFS-1643. Federation: remove namenode argument from DataNode - constructor (boryas) - - HDFS-1657. Federation: Tests that corrupt block files fail due to changed - file path in federation. (suresh) - - HDFS-1661. Federation: Remove unnecessary TODO:FEDERATION comments. - (jitendra) - - HDFS-1660. Federation: Datanode doesn't start with two namenodes (boryas) - - HDFS-1650. Federation: TestReplication fails. (Tanping via suresh) - - HDFS-1651. Federation: Tests fail due to null pointer exception in - Datnode#shutdown() method. (Tanping via suresh) - - HDFS-1649. Federation: Datanode command to refresh namenode list at - the datanode. (jitendra) - - HDFS-1646. Federation: MiniDFSClsuter#waitActive() waits for ever - with the introduction of BPOfferService in datanode. (suresh) - - HDFS-1659. Federation: BPOfferService exits after one iteration - incorrectly. (Tanping via suresh) - - HDFS-1654. Federation: Fix TestDFSUpgrade and TestDFSRollback failures. - (suresh) - - HDFS-1668. Federation: Datanodes sends block pool usage information - to the namenode in heartbeat. (suresh) - - HDFS-1669. Federation: Fix TestHftpFileSystem failure. (suresh) - - HDFS-1670. Federation: remove dnRegistration from Datanode (boryas) - - HDFS-1662. Federation: fix unit test case, TestCheckpoint - and TestDataNodeMXBean (tanping via boryas) - - HDFS-1671. Federation: shutdown in DataNode should be able to - shutdown individual BP threads as well as the whole DN (boryas). - - HDFS-1663. Federation: Rename getPoolId() everywhere to - getBlockPoolId() (tanping via boryas) - - HDFS-1652. FederationL Add support for multiple namenodes in - MiniDFSCluster. (suresh) - - HDFS-1672. Federation: refactor stopDatanode(name) to work - with multiple Block Pools (boryas) - - HDFS-1687. Federation: DirectoryScanner changes for - federation (Matt Foley via boryas) - - HDFS-1626. Make BLOCK_INVALIDATE_LIMIT configurable. (szetszwo) - - HDFS-1655. Federation: DatablockScanner should scan blocks for - all the block pools. (jitendra) - - HDFS-1664. Federation: Add block pool storage usage to Namenode WebUI. - (Tanping via suresh) - - HDFS-1674. Federation: Rename BlockPool class to BlockPoolSlice. - (jghoman, Tanping via suresh) - - HDFS-1673. Federation: Datanode changes to track block token secret per - namenode. (suresh) - - HDFS-1677. Federation: Fix TestFsck and TestListCorruptFileBlocks - failures. (Tanping via suresh) - - HDFS-1678. Federation: Remove unnecessary #getBlockpool() - for NameNodeMXBean in FSNameSystem. (Tanping via Suresh) - - HDFS-1688. Federation: Fix failures in fault injection tests, - TestDiskError, TestDatanodeRestart and TestDFSTartupVersions. (suresh) - - HDFS-1696. Federation: when build version doesn't match - - datanode should wait (keep connecting) untill NN comes up - with the right version (boryas) - - HDFS-1681. Balancer: support per pool and per node policies. (szetszwo) - - HDFS-1695. Federation: Fix testOIV and TestDatanodeUtils - (jhoman and tanping via boryas) - - HDFS:1699. Federation: Fix failure of TestBlockReport. - (Matt Foley via suresh) - - HDFS-1698. Federation: TestOverReplicatedBlocks and TestWriteToReplica - failing. (jhoman and jitendra) - - HDFS-1701. Federation: Fix TestHeartbeathandling. - (Erik Steffl and Tanping Wang via suresh) - - HDFS-1693. Federation: Fix TestDFSStorageStateRecovery failure. (suresh) - - HDFS-1694. Federation: SimulatedFSDataset changes to work with - federation and multiple block pools. (suresh) - - HDFS-1689. Federation: Configuration for namenodes. (suresh and jitendra) - - HDFS-1682. Change Balancer CLI for multiple namenodes and balancing - policy. (szetszwo) - - HDFS-1697. Federation: fix TestBlockRecovery (boryas) - - HDFS-1702. Federation: fix TestBackupNode and TestRefreshNamendoes - failures. (suresh) - - HDFS-1706. Federation: TestFileAppend2, TestFileAppend3 and - TestBlockTokenWithDFS failing. (jitendra) - - HDFS-1704. Federation: Add a tool that lists namenodes, secondary and - backup from configuration file. (suresh) - - HDFS-1711. Federation: create method for updating machine name in - DataNode.java (boryas) - - HDFS-1712. Federation: when looking up datanode we should use machineNmae - (in testOverReplicatedBlocks) (boryas) - - HDFS-1709. Federation: Error "nnaddr url param is null" when clicking on a - node from NN Live Node Link. (jitendra) - - HDFS-1714. Federation: refactor upgrade object in DataNode (boryas) - - HDFS-1715. Federation: warning/error not generated when datanode sees - inconsistent/different Cluster ID between namenodes (boryas) - - HDFS-1715. Federation: warning/error not generated when datanode sees - inconsistent/different Cluster ID between namenodes (boryas) - - HDFS-1716. Federation: Add decommission tests for federated namenodes. - (suresh) - - HDFS-1713. Federation: Prevent DataBlockScanner from running in tight loop. - (jitendra) - - HDFS-1721. Federation: Configuration for principal names should not be - namenode specific. (jitendra) - - HDFS-1717. Federation: FSDataset volumeMap access is not synchronized - correctly. (suresh) - - HDFS-1722. Federation: Add flag to MiniDFSCluser to differentiate between - federation and non-federation modes. (boryas via suresh) - - HDFS-1718. Federation: MiniDFSCluster#waitActive() bug causes some tests - to fail. (suresh) - - HDFS-1719. Federation: Fix TestDFSRemove that fails intermittently. - (suresh) - - HDFS-1720. Federation: FSVolumeSet volumes is not synchronized correctly. - (suresh) - - HDFS-1700. Federation: fsck needs to work with federation changes. - (Matt Foley via suresh) - - HDFS-1482. Add listCorruptFileBlocks to DistributedFileSystem. - (Patrick Kling via hairong) - - HDFS-1448. Add a new tool Offline Edits Viewer (oev). (Erik Steffl - via szetszwo) - - HDFS-1735. Federation: merge FSImage change in federation to - FSImage+NNStorage refactoring in trunk. (suresh) - - HDFS-1737. Federation: Update the layout version for federation - changes. (suresh) - - HDFS-1744. Federation: Add new layout version to offline image viewer - and edits viewer. (suresh) - - HDFS-1745. Federation: Fix fault injection test failures. (suresh) - - HDFS-1746. Federation: TestFileAppend3 fails intermittently. (jitendra) - - HDFS-1703. Improve start/stop scripts and add decommission tool for - federation. (Tanping Wang, Erik Steffl via suresh) - - HDFS-1749. Federation: TestListCorruptFileBlocks failing in federation - branch. (jitendra) - - HDFS-1754. Federation: testFsck fails. (boryas) - - HDFS-1755. Federation: The BPOfferService must always connect to namenode - as the login user. (jitendra) - - HDFS-1675. Support transferring RBW between datanodes. (szetszwo) - - HDFS-1791. Federation: Add command to delete block pool directories - from a datanode. (jitendra) - - HDFS-1761. Add a new DataTransferProtocol operation, Op.TRANSFER_BLOCK, - for transferring RBW/Finalized with acknowledgement and without using RPC. - (szetszwo) - - HDFS-1813. Federation: Authentication using BlockToken in RPC to datanode - fails. (jitendra) - - HDFS-1630. Support fsedits checksum. (hairong) - - HDFS-1606. Provide a stronger data guarantee in the write pipeline by - adding a new datanode when an existing datanode failed. (szetszwo) - - HDFS-1442. Api to get delegation token in Hdfs class. (jitendra) - - HDFS-1070. Speedup namenode image loading and saving by storing only - local file names. (hairong) - - HDFS-1751. Intrinsic limits for HDFS files, directories (daryn via boryas). - - HDFS-1873. Federation: Add cluster management web console. - (Tanping Wang via suresh) - - HDFS 1911 HDFS tests for the newly added viewfs - - HDFS-1814. Add "hdfs groups" command to query the server-side groups - resolved for a user. (Aaron T. Myers via todd) - - HDFS-1914. Federation: namenode storage directories must be configurable - specific to name service. (suresh) - - HDFS-1963. Create RPM and Debian packages for HDFS. Changes deployment - layout to be consistent across the binary tgz, rpm, and deb. - (Eric Yang via omalley) - - HDFS-2058. Change Data Transfer wire protocol to use protocol buffers. - (todd) - - HDFS-2055. Add hflush support to libhdfs. (Travis Crawford via eli) - - HDFS-2083. Query JMX statistics over http via JMXJsonServlet. (tanping) - - HDFS-2156. Make hdfs and mapreduce rpm only depend on the same major - version for common and hdfs. (eyang via omalley) - - HDFS-2202. Add a new DFSAdmin command to set balancer bandwidth of - datanodes without restarting. (Eric Payne via szetszwo) - - HDFS-2284. Add a new FileSystem, webhdfs://, for supporting write Http - access to HDFS. (szetszwo) - - HDFS-2317. Support read access to HDFS in WebHDFS. (szetszwo) - - HDFS-2338. Add configuration option to enable/disable WebHDFS. - (jitendra via szetszwo) - - HDFS-2318. Provide authentication to WebHDFS using SPNEGO and delegation - tokens. (szetszwo) - - HDFS-2340. Support getFileBlockLocations and getDelegationToken in WebHDFS. - (szetszwo) - - HDFS-2348. Support getContentSummary and getFileChecksum in WebHDFS. - (szetszwo) - - HDFS-2385. Support renew and cancel delegation tokens in WebHDFS. - (szetszwo) - - HDFS-2539. Support doAs and GETHOMEDIRECTORY in WebHDFS. - (szetszwo) - - IMPROVEMENTS - - HDFS-1875. MiniDFSCluster hard-codes dfs.datanode.address to localhost - (Eric Payne via mattf) - - HDFS-2019. Fix all the places where Java method File.list is used with - FileUtil.list API (Bharath Mundlapudi via mattf) - - HDFS-1934. Fix NullPointerException when certain File APIs return null - (Bharath Mundlapudi via mattf) - - HDFS-1510. Added test-patch.properties required by test-patch.sh (nigel) - - HDFS-1628. Display full path in AccessControlException. (John George - via szetszwo) - - HDFS-1707. Federation: Failure in browsing data on new namenodes. - (jitendra) - - HDFS-1683. Test Balancer with multiple NameNodes. (szetszwo) - - HDFS-1547. Improve decommission mechanism. (suresh) - - HDFS-2143. Federation: In cluster web console, add link to namenode page - that displays live and dead datanodes. (Ravi Prakash via suresh) - - HDFS-1588. Remove hardcoded strings for configuration keys, "dfs.hosts" - and "dfs.hosts.exlude". (Erik Steffl via suresh) - - HDFS-1481. NameNode should validate fsimage before rolling. (hairong) - - HDFS-1506. Refactor fsimage loading code. (hairong) - - HDFS-1533. A more elegant FileSystem#listCorruptFileBlocks API - (HDFS portion) (Patrick Kling via hairong) - - HDFS-1476. listCorruptFileBlocks should be functional while the - name node is in safe mode. (Patrick Kling via hairong) - - HDFS-1534. Fix some incorrect logs in FSDirectory. (eli) - - HDFS-1539. A config option for the datanode to fsycn a block file - when block is completely written. (dhruba) - - HDFS-1335. HDFS side change of HADDOP-6904: RPC compatibility. (hairong) - - HDFS-1557. Separate Storage from FSImage. (Ivan Kelly via jitendra) - - HDFS-560 Enhancements/tuning to hadoop-hdfs/build.xml - - HDFS-1629. Add a method to BlockPlacementPolicy for keeping the chosen - nodes in the output array. (szetszwo) - - HDFS-1731. Allow using a file to exclude certain tests from build (todd) - - HDFS-1736. Remove the dependency from DatanodeJspHelper to FsShell. - (Daryn Sharp via szetszwo) - - HDFS-780. Revive TestFuseDFS. (eli) - - HDFS-1445. Batch the calls in DataStorage to FileUtil.createHardLink(). - (Matt Foley via jghoman) - - HDFS-1763. Replace hard-coded option strings with variables from - DFSConfigKeys. (eli) - - HDFS-1541. Not marking datanodes dead when namenode in safemode. - (hairong) - - HDFS-1120. Make DataNode's block-to-device placement policy pluggable - (Harsh J Chouraria via todd) - - HDFS-1785. In BlockReceiver and DataXceiver, clientName.length() is used - multiple times for determining whether the source is a client or a - datanode. (szetszwo) - - HDFS-1789. Refactor frequently used codes from DFSOutputStream and - DataXceiver. (szetszwo) - - HDFS-1767. Namenode ignores non-initial block report from datanodes - when in safemode during startup. (Matt Foley via suresh) - - HDFS-1817. Move pipeline_Fi_[39-51] from TestFiDataTransferProtocol - to TestFiPipelineClose. (szetszwo) - - HDFS-1760. In FSDirectory.getFullPathName(..), it is better to return "/" - for root directory instead of an empty string. (Daryn Sharp via szetszwo) - - HDFS-1833. Reduce repeated string constructions and unnecessary fields, - and fix comments in BlockReceiver.PacketResponder. (szetszwo) - - HDFS-1486. Generalize CLITest structure and interfaces to faciliate - upstream adoption (e.g. for web testing). (cos) - - HDFS-1844. Move "fs -help" shell command tests from HDFS to COMMOM; see - also HADOOP-7230. (Daryn Sharp via szetszwo) - - HDFS-1840. In DFSClient, terminate the lease renewing thread when all files - being written are closed for a grace period, and start a new thread when - new files are opened for write. (szetszwo) - - HDFS-1854. make failure message more useful in - DFSTestUtil.waitReplication(). (Matt Foley via eli) - - HDFS-1562. Add rack policy tests. (eli) - - HDFS-1856. TestDatanodeBlockScanner waits forever, errs without giving - information. (Matt Foley via eli) - - HDFS-1295. Improve namenode restart times by short-circuiting the - first block reports from datanodes. (Matt Foley via suresh) - Corrected merge error in DataNode.java. (Matt Foley) - - HDFS-1843. Discover file not found early for file append. - (Bharath Mundlapudi via jitendra) - - HDFS-1862. Improve test reliability of HDFS-1594. (Aaron T. Myers via eli) - - HDFS-1846. Preallocate edit log with OP_INVALID instead of zero bytes - to ensure blocks are actually allocated. (Aaron T. Myers via todd) - - HDFS-1741. Provide a minimal pom file to allow integration of HDFS into Sonar - analysis (cos) - - HDFS-1870. Move and rename DFSClient.LeaseChecker to a seperated class - LeaseRenewer. (szetszwo) - - HDFS-1866. Document dfs.datanode.max.transfer.threads in hdfs-default.xml - (Harsh J Chouraria via todd) - - HDFS-1890. Improve the name, class and value type of the map - LeaseRenewer.pendingCreates. (szetszwo) - - HDFS-1865. Share LeaseRenewer among DFSClients so that there is only a - LeaseRenewer thread per namenode per user. (szetszwo) - - HDFS-1906. Remove logging exception stack trace in client logs when one of - the datanode targets to read from is not reachable. (suresh) - - HDFS-1378. Edit log replay should track and report file offsets in case of - errors. (Aaron T. Myers and Todd Lipcon via todd) - - HDFS-1917. Separate hdfs jars from common in ivy configuration. (Eric Yang - via szetszwo) - - HDFS-1899. GenericTestUtils.formatNamenode should be moved to DFSTestUtil - (Ted Yu via todd) - - HDFS-1117. Metrics 2.0 HDFS instrumentation. (Luke Lu via suresh) - - HDFS-1946. HDFS part of HADOOP-7291. (eli) - - HDFS-1945. Removed the deprecated fields in DataTransferProtocol. - (szetszwo) - - HDFS-1730. Use DaemonFactory from common and delete it from HDFS. - (Tanping via suresh) - - HDFS-1573. Add useful tracing information to Lease Renewer thread names - (todd) - - HDFS-1939. In ivy.xml, test conf should not extend common conf. - (Eric Yang via szetszwo) - - HDFS-1332 Include more information in exceptions and debug messages - when BlockPlacementPolicy cannot be satisfied. (Ted Yu via szetszwo) - - HDFS-1958. Confirmation should be more lenient of user input when - formatting the NameNode. (todd) - - HDFS-1905. Improve namenode -format command by not making -clusterId - parameter mandatory. (Bharath Mundlapudi via suresh) - - HDFS-1877. Add a new test for concurrent read and write. (CW Chung - via szetszwo) - - HDFS-1959. Better error message for missing namenode directory. (eli) - - HDFS-1996. ivy: hdfs test jar should be independent to common test jar. - (Eric Yang via szetszwo) - - HDFS-1812. TestHDFSCLI should clean up cluster in teardown method. - (Uma Maheswara Rao G via todd) - - HDFS-1884. Improve TestDFSStorageStateRecovery to properly throw in the - case of errors. (Aaron T. Myers via todd) - - HDFS-1727. fsck command should display command usage if user passes any - illegal argument. (Sravan Kumar via todd) - - HDFS-1636. If dfs.name.dir points to an empty dir, namenode format - shouldn't require confirmation. (Harsh J Chouraria via todd) - - HDFS-1966. Encapsulate individual DataTransferProtocol op headers. - (szetszwo) - - HDFS-2024. Format TestWriteRead source codes. (CW Chung via szetszwo) - - HDFS-1968. Enhance TestWriteRead to support position/sequential read, - append, truncate and verbose options. (CW Chung via szetszwo) - - HDFS-1986. Add option to get http/https address from - DFSUtil#getInfoServer(). (Tanping via suresh) - - HDFS-2029. In TestWriteRead, check visible length immediately after - openning the file and fix code style. (John George via szetszwo) - - HDFS-2040. Only build libhdfs if a flag is passed. (eli) - - HDFS-1586. Add InterfaceAudience and InterfaceStability annotations to - MiniDFSCluster. (suresh) - - HDFS-2003. Separate FSEditLog reading logic from edit log memory state - building logic. (Ivan Kelly via todd) - - HDFS-2066. Create a package and individual class files for - DataTransferProtocol. (szetszwo) - - HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. - (todd, nigel) - - HDFS-2046. Force entropy to come from non-true random for tests. (todd) - - HDFS-2073. Add @Override annotation to NameNode. (suresh) - - HDFS-420. Fuse-dfs should cache fs handles. (Brian Bockelman and eli) - - HDFS-1568. Improve the log messages in DataXceiver. (Joey Echeverria via - szetszwo) - - HDFS-2100. Improve TestStorageRestore. (atm) - - HDFS-2092. Remove some object references to Configuration in DFSClient. - (Bharath Mundlapudi via szetszwo) - - HDFS-2087. Declare methods in DataTransferProtocol interface, and change - Sender and Receiver to implement the interface. (szetszwo) - - HDFS-1723. quota errors messages should use the same scale. (Jim Plush via - atm) - - HDFS-2110. StreamFile and ByteRangeInputStream cleanup. (eli) - - HDFS-2107. Move block management code from o.a.h.h.s.namenode to a new - package o.a.h.h.s.blockmanagement. (szetszwo) - - HDFS-2109. Store uMask as member variable to DFSClient.Conf. (Bharath - Mundlapudi via szetszwo) - - HDFS-2111. Add tests for ensuring that the DN will start with a few bad - data directories. (Harsh J Chouraria via todd) - - HDFS-2134. Move DecommissionManager to the blockmanagement package. - (szetszwo) - - HDFS-1977. Stop using StringUtils.stringifyException(). - (Bharath Mundlapudi via jitendra) - - HDFS-2131. Add new tests for the -overwrite/-f option in put and - copyFromLocal by HADOOP-7361. (Uma Maheswara Rao G via szetszwo) - - HDFS-2140. Move Host2NodesMap to the blockmanagement package. (szetszwo) - - HDFS-2154. In TestDFSShell, use TEST_ROOT_DIR and fix some deprecated - warnings. (szetszwo) - - HDFS-2153. Move DFSClientAdapter to test and fix some javac warnings in - OfflineEditsViewerHelper. (szetszwo) - - HDFS-2159. Deprecate DistributedFileSystem.getClient() and fixed the - deprecated warnings in DFSAdmin. (szetszwo) - - HDFS-2157. Improve header comment in o.a.h.hdfs.server.namenode.NameNode. - (atm via eli) - - HDFS-2147. Move cluster network topology to block management and fix some - javac warnings. (szetszwo) - - HDFS-2141. Remove NameNode roles Active and Standby (they become - states of the namenode). (suresh) - - HDFS-2161. Move createNamenode(..), createClientDatanodeProtocolProxy(..) - and Random object creation to DFSUtil; move DFSClient.stringifyToken(..) - to DelegationTokenIdentifier. (szetszwo) - - HDFS-1774. Small optimization to FSDataset. (Uma Maheswara Rao G via eli) - - HDFS-2167. Move dnsToSwitchMapping and hostsReader from FSNamesystem to - DatanodeManager. (szetszwo) - - HDFS-2116. Use Mokito in TestStreamFile and TestByteRangeInputStream. - (Plamen Jeliazkov via shv) - - HDFS-2112. Move ReplicationMonitor to block management. (Uma Maheswara - Rao G via szetszwo) - - HDFS-1739. Add available volume size to the error message when datanode - throws DiskOutOfSpaceException. (Uma Maheswara Rao G via szetszwo) - - HDFS-2144. If SNN shuts down during initialization it does not log the - cause. (Ravi Prakash via atm) - - HDFS-2180. Refactor NameNode HTTP server into new class. (todd) - - HDFS-2198. Remove hardcoded configuration keys. (suresh) - - HDFS-2149. Move EditLogOp serialization formats into FsEditLogOp - implementations. (Ivan Kelly via todd) - - HDFS-2191. Move datanodeMap from FSNamesystem to DatanodeManager. - (szetszwo) - - HDFS-2200. Change FSNamesystem.LOG to package private. (szetszwo) - - HDFS-2195. Refactor StorageDirectory to not be an non-static inner class. - (todd via eli) - - HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams. - (todd via eli) - - HDFS-2199. Move blockTokenSecretManager from FSNamesystem to BlockManager. - (Uma Maheswara Rao G via szetszwo) - - HDFS-2187. Make EditLogInputStream act like an iterator over FSEditLogOps - (Ivan Kelly and todd via todd) - - HDFS-2225. Refactor edit log file management so it's not in classes - which should be generic to the type of edit log storage. (Ivan Kelly - via todd) - - HDFS-2108. Move datanode heartbeat handling from namenode package to - blockmanagement package. (szetszwo) - - HDFS-2226. Clean up counting of operations in FSEditLogLoader (todd) - - HDFS-2228. Move block and datanode code from FSNamesystem to - BlockManager and DatanodeManager. (szetszwo) - - HDFS-2238. In NamenodeFsck.toString(), uses StringBuilder.(..) instead of - string concatenation. (Uma Maheswara Rao G via szetszwo) - - HDFS-2230. ivy to resolve/retrieve latest common-tests jar published by - hadoop common maven build. (gkesavan) - - HDFS-2227. getRemoteEditLogManifest should pull its information from - FileJournalManager during checkpoint process (Ivan Kelly and Todd Lipcon - via todd) - - HDFS-2239. Reduce access levels of the fields and methods in FSNamesystem. - (szetszwo) - - HDFS-2241. Remove implementing FSConstants interface to just get the - constants from the interface. (suresh) - - HDFS-2237. Change UnderReplicatedBlocks from public to package private. - (szetszwo) - - HDFS-2233. Add WebUI tests with URI reserved chars. (eli) - - HDFS-2265. Remove unnecessary BlockTokenSecretManager fields/methods from - BlockManager. (szetszwo) - - HDFS-2260. Refactor BlockReader into an interface and implementation. - (todd) - - HDFS-2096. Mavenization of hadoop-hdfs (Alejandro Abdelnur via tomwhite) - - HDFS-2273. Refactor BlockManager.recentInvalidateSets to a new class. - (szetszwo) - - HDFS-2266. Add Namesystem and SafeMode interfaces to avoid directly - referring to FSNamesystem in BlockManager. (szetszwo) - - HDFS-1217. Change some NameNode methods from public to package private. - (Laxman via szetszwo) - - HDFS-2332. Add test for HADOOP-7629 (using an immutable FsPermission - object as an RPC parameter fails). (todd) - - HDFS-2363. Move datanodes size printing from FSNamesystem.metasave(..) - to BlockManager. (Uma Maheswara Rao G via szetszwo) - - HDFS-2209. Make MiniDFS easier to embed in other apps. (stevel) - - HDFS-2205. Log message for failed connection to datanode is not - followed by a success message. (Ravi Prakash via stevel) - - HDFS-2401. Running a set of methods in a Single Test Class. - (Jonathan Eagles via mahadev) - - HDFS-2471. Add federation documentation. (suresh) - - HDFS-2485. Improve code layout and constants in UnderReplicatedBlocks - (stevel) - - HDFS-2356. Support case insensitive query parameter names in WebHDFS. - (szetszwo) - - HDFS-2368. Move SPNEGO conf properties from hdfs-default.xml to - hdfs-site.xml. (szetszwo) - - HDFS-2395. Add a root element in the JSON responses of WebHDFS. - (szetszwo) - - HDFS-2427. Change the default permission in WebHDFS to 755 and add range - check/validation for all parameters. (szetszwo) - - HDFS-2501. Add version prefix and root methods to WebHDFS. (szetszwo) - - HDFS-1869. mkdirs should use the supplied permission for all of the created - directories. (Daryn Sharp via szetszwo) - - HDFS-2355. Federation: enable using the same configuration file across - all the nodes in the cluster. (suresh) - - HDFS-2371. Refactor BlockSender.java for better readability. (suresh) - - HDFS-2493. Remove reference to FSNamesystem in blockmanagement classes. - (szetszwo) - - HDFS-2294. Download of commons-daemon TAR should not be under target (tucu) - - HDFS-2322. the build fails in Windows because commons-daemon TAR cannot be - fetched. (tucu) - - HDFS-2436. Change FSNamesystem.setTimes(..) for allowing setting times on - directories. (Uma Maheswara Rao G via szetszwo) - - HDFS-2512. Add textual error message to data transfer protocol responses - (todd) - - HDFS-2521. Remove custom checksum headers from data transfer protocol - (todd) - - HDFS-2308. NamenodeProtocol.endCheckpoint is vestigial and can be removed. - (eli) - - OPTIMIZATIONS - - HDFS-1458. Improve checkpoint performance by avoiding unnecessary image - downloads and loading. (hairong) - - HDFS-1601. Pipeline ACKs are sent as lots of tiny TCP packets (todd) - - HDFS-1826. NameNode should save image to name directories in parallel - during upgrade. (Matt Foley via hairong) - - HDFS-2030. Improve usability of namenode -upgrade command. - (Bharath Mundlapudi via suresh) - - HDFS-2056. Update fetchdt usage. (Tanping Wang via jitendra) - - HDFS-2118. Couple dfs data dir improvements. (eli) - - HDFS-2500. Avoid file system operations in BPOfferService thread while - processing deletes. (todd) - - HDFS-2465. Add HDFS support for fadvise readahead and drop-behind. (todd) - - BUG FIXES - - HDFS-2347. Fix checkpointTxnCount's comment about editlog size. - (Uma Maheswara Rao G via mattf) - - HDFS-2011. Removal and restoration of storage directories on checkpointing - failure doesn't work properly. (Ravi Prakash via mattf) - - HDFS-1955. FSImage.doUpgrade() was made too fault-tolerant by HDFS-1826. - (mattf) - - HDFS-2061. Two minor bugs in BlockManager block report processing. (mattf) - - HDFS-1449. Fix test failures - ExtendedBlock must return - block file name in #getBlockName(). (suresh) - - HDFS-1680. Fix TestBalancer. (szetszwo) - - HDFS-1705. Balancer command throws NullPointerException. (suresh via - szetszwo) - - HDFS-1559. Add missing UGM overrides to TestRefreshUserMappings - (Todd Lipcon via eli) - - HDFS-1585. Fix build after HDFS-1547 (todd) - - HDFS-1684. Balancer cannot start with with multiple namenodes. (szetszwo) - - HDFS-1516. mvn-install is broken after 0.22 branch creation. (cos) - - HDFS-1360. TestBlockRecovery should bind ephemeral ports. - (Todd Lipcon via hairong) - - HDFS-1551. Fix pom templates dependency list (gkesavan) - - HDFS-1509. A savenamespace command writes the fsimage and edits into - all configured directories. (dhruba) - - HDFS-1540. Make Datanode handle errors from RPC calls to namenode - more elegantly. (dhruba) - - HDFS-1463. Accesstime of a file is not updated in safeMode. (dhruba) - - HDFS-863. Potential deadlock in TestOverReplicatedBlocks. - (Ken Goodhope via jghoman) - - HDFS-1607. Fix referenced to misspelled method name getProtocolSigature - (todd) - - HDFS-1610. Fix TestClientProtocolWithDelegationToken and TestBlockToken - on trunk after HADOOP-6904 (todd) - - HDFS-1600. Fix release audit warnings on trunk. (todd) - - HDFS-1691. Remove a duplicated static initializer for reading default - configurations in DFSck. (Alexey Diomin via szetszwo) - - HDFS-1748. Balancer utilization classification is incomplete. (szetszwo) - - HDFS-1738. change hdfs jmxget to return an empty string instead of - null when an attribute value is not available (tanping vi boryas) - - HDFS-1757. Don't compile fuse-dfs by default. (eli) - - HDFS-1770. TestFiRename fails due to invalid block size. (eli) - - HDFS-1797. Fix new findbugs warning introduced by HDFS-1120 (todd) - - HDFS-1611. Fix up some log messages in DFSClient and MBean registration - (Uma Maheswara Rao G via todd) - - HDFS-1543. Reduce dev. cycle time by moving system testing artifacts from - default build and push to maven for HDFS (Luke Lu via cos) - - HDFS-1818. TestHDFSCLI is failing on trunk after HADOOP-7202. - (Aaron T. Myers via todd) - - HDFS-1828. TestBlocksWithNotEnoughRacks intermittently fails assert. - (Matt Foley via eli) - - HDFS-1824. delay instantiation of file system object until it is - needed (linked to HADOOP-7207) (boryas) - - HDFS-1831. Fix append bug in FileContext and implement CreateFlag - check (related to HADOOP-7223). (suresh) - - HDFS-1594. When the disk becomes full Namenode is getting shutdown and - not able to recover. (Aaron T. Myers via eli) - - HDFS-1822. Handle editlog opcode conflict with 0.20.203 during upgrade, - by throwing an error to indicate the editlog needs to be empty. - (suresh) - - HDFS-1808. TestBalancer waits forever, errs without giving information. - (Matt Foley via eli) - - HDFS-1829. TestNodeCount waits forever, errs without giving information. - (Matt Foley via eli) - - HDFS-1860. when renewing/canceling DelegationToken over http we need to - pass exception information back to the caller.(boryas) - - HDFS-1871. Mapreduce build fails due to MiniDFSCluster change from - HDFS-1052. (suresh) - - HDFS-1876. One MiniDFSCluster constructor ignores numDataNodes parameter - (todd) - - HDFS-1773. Do not show decommissioned datanodes, which are not in both - include and exclude lists, on web and JMX interfaces. - (Tanping Wang via szetszwo) - - HDFS-1888. MiniDFSCluster#corruptBlockOnDatanodes() access must be - public. (suresh) - - HDFS-1889. incorrect path in start/stop dfs script. (John George via eli) - - HDFS-1891. Disable IPV6 for junit tests to fix TestBackupNode failure. - (suresh) - - HDFS-1898. Tests failing on trunk due to use of NameNode.format. - (todd via eli) - - HDFS-1902. Fix setrep path display for TestHDFSCLI. (Daryn Sharp - via szetszwo) - - HDFS-1827. Fix timeout problem in TestBlockReplacement. (Matt Foley - via szetszwo) - - HDFS-1908. Fix a NullPointerException in fi.DataTransferTestUtil. - (szetszwo) - - HDFS-1912. Update tests for FsShell standardized error messages. - (Daryn Sharp via szetszwo) - - HDFS-1903. Fix path display for rm/rmr in TestHDFSCLI and TestDFSShell. - (Daryn Sharp via szetszwo) - - HDFS-1627. Fix NullPointerException in Secondary NameNode. (hairong) - - HDFS-1928. Fix path display for touchz in TestHDFSCLI. - (Daryn Sharp via todd) - - HDFS-1938. Fix ivy-retrieve-hdfs dependence in build.xml and aop.xml. - (Eric Yang via szetszwo) - - HDFS-1929. TestEditLogFileOutputStream fails if running on same host as NN - (Aaron T. Myers via todd) - - HDFS-1933. Update TestDFSShell for improved "test" shell command. (Daryn - Sharp via todd) - - HDFS-1931. Update TestDFSShell for improved "du" shell command. (Daryn - Sharp via todd) - - HDFS-1439. HDFS Federation: Fix compilation error in TestFiHftp. (suresh) - - HDFS-1881. Federation: after taking snapshot the current directory - of datanode is empty. (Tanping Wang via suresh) - - HDFS-1927. Fix a bug which causes ip=null in NameNode audit log. - (John George via szetszwo) - - HDFS-1953. Federation: Change name node mxbean name in cluster web - console. (Tanping Wang via suresh) - - HDFS-1922. Fix recurring failure of TestJMXGet (Luke Lu via todd) - - HDFS-1371. One bad node can incorrectly flag many files as corrupt. - (Tanping Wang via jitendra) - - HDFS-1943. Fail to start datanode while start-dfs.sh is executed by - root user. (Wei Yongjun via jghoman) - - HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and - TestDFSShell. (Daryn Sharp via todd) - - HDFS-1999. Tests use deprecated configs. (Aaron T. Myers via eli) - - HDFS-1592. Datanode startup doesn't honor volumes.tolerated. - (Bharath Mundlapudi via jitendra) - - HDFS-1920. libhdfs does not build for ARM processors. - (Trevor Robinson via eli) - - HDFS-1936. Layout version change from HDFS-1822 causes upgrade failure. - (suresh) - - HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder. - (John George via szetszwo) - - HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode - as a singleton. (suresh via todd) - - HDFS-2022. ant binary should build libhdfs. (Eric Yang via eli) - - HDFS-2014. Change HDFS scripts to work in developer enviroment post - RPM packaging changes. (Eric Yang via suresh) - - HDFS-1995. Federation: Minor bug fixes and modification cluster web UI. - (Tanping Wang via suresh) - - HDFS-1907. Fix position read for reading still-being-written file in - DFSInputStream. (John George via szetszwo) - - HDFS-1923. In TestFiDataTransferProtocol2, reduce random sleep time period - and increase the number of datanodes. (szetszwo) - - HDFS-1149. Lease reassignment should be persisted to the edit log. - (Aaron T. Myers via todd) - - HDFS-1998. Federation: Make refresh-namenodes.sh refresh all the - namenode. (Tanping Wang via suresh) - - HDFS-2041. OP_CONCAT_DELETE doesn't properly restore modification time - of the concatenated file when edit logs are replayed. (todd) - - HDFS-2063. libhdfs test is broken. (Eric Yang via eli) - - HDFS-2067. Bump DATA_TRANSFER_VERSION constant in trunk after introduction - of protocol buffers in the protocol. (szetszwo via todd) - - HDFS-2069. Incorrect default trash interval value in the docs. - (Harsh J Chouraria via eli) - - HDFS-1942. Datanode must exist when all the block pool service threads - exit. (Bharath Mundlapudi via suresh) - - HDFS-1656. Fixes an issue to do with fetching of delegation tokens in - HftpFileSystem. Contributed by Kan Zhang. - - HDFS-1692. In secure mode, Datanode process doesn't exit when disks - fail. (Bharath Mundlapudi via suresh) - - HDFS-1734. 'Chunk size to view' option is not working in Name Node UI. - (Uma Maheswara Rao G via jitendra) - - HDFS-2086. If the include hosts list contains host names, after restarting - namenode, data nodes registration is denied. Contributed by Tanping Wang. - - HDFS-2082. SecondaryNameNode web interface doesn't show the right info. (atm) - - HDFS-1321. If service port and main port are the same, there is no clear - log message explaining the issue. (Jim Plush via atm) - - HDFS-1381. HDFS javadocs hard-code references to dfs.namenode.name.dir and - dfs.datanode.data.dir parameters (Jim Plush via atm) - - HDFS-2053. Bug in INodeDirectory#computeContentSummary warning. - (Michael Noll via eli) - - HDFS-1990. Fix resource leaks in BlockReceiver.close(). (Uma Maheswara - Rao G via szetszwo) - - HDFS-2034. Length in DFSInputStream.getBlockRange(..) becomes -ve when - reading only from a currently being written block. (John George via - szetszwo) - - HDFS-2132. Potential resource leak in EditLogFileOutputStream.close. (atm) - - HDFS-2120. on reconnect, DN can connect to NN even with different source - versions. (John George via atm) - - HDFS-2152. TestWriteConfigurationToDFS causing the random failures. (Uma - Maheswara Rao G via atm) - - HDFS-2114. re-commission of a decommissioned node does not delete - excess replicas. (John George via mattf) - - HDFS-1776. Bug in Concat code. (Bharath Mundlapudi via Dmytro Molkov) - - HDFS-2196. Make ant build system work with hadoop-common JAR generated - by Maven. (Alejandro Abdelnur via tomwhite) - - HDFS-2245. Fix a NullPointerException in BlockManager.chooseTarget(..). - (szetszwo) - - HDFS-2229. Fix a deadlock in namenode by enforcing lock acquisition - ordering. (szetszwo) - - HDFS-2235. Encode servlet paths. (eli) - - HDFS-2186. DN volume failures on startup are not counted. (eli) - - HDFS-2240. Fix a deadlock in LeaseRenewer by enforcing lock acquisition - ordering. (szetszwo) - - HDFS-73. DFSOutputStream does not close all the sockets. - (Uma Maheswara Rao G via eli) - - HDFS-1257. Fix a race condition on BlockManager.recentInvalidateSets. - (Eric Payne via szetszwo) - - HDFS-2267. DataXceiver thread name incorrect while waiting on op during - keepalive. (todd) - - HDFS-1480. All replicas of a block can end up on the same rack when - some datanodes are decommissioning. (todd) - - HDFS-2286. DataXceiverServer logs AsynchronousCloseException at shutdown - (todd) - - HDFS-2289. Ensure jsvc is bundled with the HDFS distribution artifact. - (Alejandro Abdelnur via acmurthy) - - HDFS-2323. start-dfs.sh script fails for tarball install (tomwhite) - - HDFS-2412. Add backwards-compatibility layer for renamed FSConstants - class (todd) - - HDFS-2414. Fix TestDFSRollback to avoid spurious failures. (todd) - - HDFS-2422. The NN should tolerate the same number of low-resource volumes - as failed volumes (atm) - - HDFS-2467. HftpFileSystem uses incorrect compare for finding delegation - tokens. (omalley) - - HDFS-2331. Fix WebHdfsFileSystem compilation problems for a bug in JDK - version < 1.6.0_26. (Abhijit Suresh Shingate via szetszwo) - - HDFS-2333. Change DFSOutputStream back to package private, otherwise, - there are two SC_START_IN_CTOR findbugs warnings. (szetszwo) - - HDFS-2366. Initialize WebHdfsFileSystem.ugi in object construction. - (szetszwo) - - HDFS-2361. hftp is broken, fixed username checks in JspHelper. (jitendra) - - HDFS-2403. NamenodeWebHdfsMethods.generateDelegationToken(..) does not use - the renewer parameter. (szetszwo) - - HDFS-2409. _HOST in dfs.web.authentication.kerberos.principal. (jitendra) - - HDFS-2404. WebHDFS liststatus json response is not correct. (suresh) - - HDFS-2441. Remove the Content-Type set by HttpServer.QuotingInputFilter in - WebHDFS responses. (szetszwo) - - HDFS-2428. Convert com.sun.jersey.api.ParamException$QueryParamException - to IllegalArgumentException and response it as http BAD_REQUEST in WebHDFS. - (szetszwo) - - HDFS-2424. Added a root element "HdfsFileStatuses" for the response - of WebHDFS listStatus. (szetszwo) - - MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra) - - HDFS-2439. Fix NullPointerException in WebHDFS when opening a non-existing - file or creating a file without specifying the replication parameter. - (szetszwo) - - HDFS-2453. Fix http response code for partial content in WebHDFS, added - getDefaultBlockSize() and getDefaultReplication() in WebHdfsFileSystem - and cleared content type in ExceptionHandler. (szetszwo) - - HDFS-2411. The the auth to local mappings are not being respected, with - WebHDFS enabled. (jitendra) - - HDFS-2494. Close the streams and DFSClient in DatanodeWebHdfsMethods. - (Uma Maheswara Rao G via szetszwo) - - HDFS-2298. Fix TestDfsOverAvroRpc by changing ClientProtocol to - not include multiple methods of the same name. (cutting) - - HDFS-2432. WebHDFS: response FORBIDDEN when setReplication on non-files; - clear umask before creating a flie; throw IllegalArgumentException if - setOwner with both owner and group empty; throw FileNotFoundException if - getFileStatus on non-existing files; fix bugs in getBlockLocations; and - changed getFileChecksum json response root to "FileChecksum". (szetszwo) - - HDFS-2065. Add null checks in DFSClient.getFileChecksum(..). (Uma - Maheswara Rao G via szetszwo) - - HDFS-2416. distcp with a WebHDFS uri on a secure cluster fails. (jitendra) - - HDFS-2527. WebHDFS: remove the use of "Range" header in Open; use ugi - username if renewer parameter is null in GetDelegationToken; response OK - when setting replication for non-files; rename GETFILEBLOCKLOCATIONS to - GET_BLOCK_LOCATIONS and state that it is a private unstable API; replace - isDirectory and isSymlink with enum {FILE, DIRECTORY, SYMLINK} in - HdfsFileStatus JSON object. (szetszwo) - - HDFS-2528. WebHDFS: set delegation kind to WEBHDFS and add a HDFS token - when http requests are redirected to datanode. (szetszwo) - - HDFS-2540. WebHDFS: change "Expect: 100-continue" to two-step write; change - "HdfsFileStatus" and "localName" respectively to "FileStatus" and - "pathSuffix" in JSON response. (szetszwo) - - BREAKDOWN OF HDFS-1073 SUBTASKS - - HDFS-1521. Persist transaction ID on disk between NN restarts. - (Ivan Kelly and Todd Lipcon via todd) - HDFS-1538. Refactor more startup and image loading code out of FSImage. - (todd) - HDFS-1729. Add code to detect valid length of an edits file. (todd) - HDFS-1793. Add code to inspect a storage directory with txid-based - filenames (todd) - HDFS-1794. Add code to list which edit logs are available on a remote NN - (todd) - HDFS-1858. Add state management variables to FSEditLog (Ivan Kelly and Todd - Lipcon via todd) - HDFS-1859. Add some convenience functions to iterate over edit log streams - (Ivan Kelly and Todd Lipcon via todd) - HDFS-1894. Add constants for LAYOUT_VERSIONs in edits log branch (todd) - HDFS-1892. Fix EditLogFileInputStream.getValidLength to be aware of - OP_INVALID filler (todd) - HDFS-1799. Refactor log rolling and filename management out of FSEditLog - (Ivan Kelly and Todd Lipcon via todd) - HDFS-1801. Remove use of timestamps to identify checkpoints and logs (todd) - HDFS-1930. TestDFSUpgrade failing in HDFS-1073 branch (todd) - HDFS-1800. Extend image checksumming to function with multiple fsimage - files per directory. (todd) - HDFS-1725. Set storage directories only at FSImage construction (Ivan Kelly - via todd) - HDFS-1926. Remove references to StorageDirectory from JournalManager - interface (Ivan Kelly via todd) - HDFS-1893. Change edit logs and images to be named based on txid (todd) - HDFS-1985. Clean up image transfer servlet (todd) - HDFS-1984. Enable multiple secondary namenodes to run simultaneously (todd) - HDFS-1987. Re-enable TestCheckpoint.testSecondaryImageDownload which was - not running previously. (todd) - HDFS-1993. TestCheckpoint needs to clean up between cases (todd) - HDFS-1992. Remove vestiges of NNStorageListener. (todd) - HDFS-1991. Some refactoring of Secondary NameNode to be able to share more - code with the BackupNode or CheckpointNode. (todd) - HDFS-1994. Fix race conditions when running two rapidly checkpointing - Secondary NameNodes. (todd) - HDFS-2001. Remove use of previous.checkpoint and lastcheckpoint.tmp - directories (todd) - HDFS-2015. Remove checkpointTxId from VERSION file. (todd) - HDFS-2016. Add infrastructure to remove or archive old and unneeded storage - files within the name directories. (todd) - HDFS-2047. Improve TestNamespace and TestEditLog in HDFS-1073 branch. - (todd) - HDFS-2048. Add upgrade tests and fix upgrade from 0.22 with corrupt image. - (todd) - HDFS-2027. Image inspector should return finalized logs before unfinalized - logs. (todd) - HDFS-2074. Determine edit log validity by truly reading and validating - transactions. (todd) - HDFS-2085. Finalize in-progress edit logs at startup. (todd) - HDFS-2026. SecondaryNameNode should properly handle the case where the - NameNode is reformatted. (todd) - HDFS-2077. Address checkpoint upload when one of the storage dirs is failed - (todd) - HDFS-2078. NameNode should not clear directory when restoring removed - storage. (todd) - HDFS-2088. Move edits log archiving logic into FSEditLog/JournalManager - (todd) - HDFS-2093. Handle case where an entirely empty log is left during NN crash - (todd) - HDFS-2102. Zero-pad edits filename to make them lexically sortable. (Ivan - Kelly via todd) - HDFS-2010. Fix NameNode to exit if all edit streams become inaccessible. - (atm via todd) - HDFS-2123. Checkpoint interval should be based on txn count, not size. - (todd) - HDFS-1979. Fix backupnode for new edits/image layout. (todd) - HDFS-2101. Fix remaining unit tests for new storage filenames. (todd) - HDFS-2133. Address remaining TODOs and pre-merge cleanup on HDFS-1073 - branch. (todd) - HDFS-1780. Reduce need to rewrite FSImage on startup. (todd) - HDFS-2104. Add a flag to the 2NN to format its checkpoint dirs on startup. - (todd) - HDFS-2135. Fix regression of HDFS-1955 in HDFS-1073 branch. (todd) - HDFS-2160. Fix CreateEditsLog test tool in HDFS-1073 branch. (todd) - HDFS-2168. Reenable TestEditLog.testFailedOpen and fix exposed bug. (todd) - HDFS-2169. Clean up TestCheckpoint and remove TODOs (todd) - HDFS-2170. Address remaining TODOs in HDFS-1073 branch. (todd) - HDFS-2172. Address findbugs and javadoc warnings in HDFS-1073 branch. - (todd) - - HDFS-2445. Ensure failed tests exit with proper error code. (Jonathan - Eagles via acmurthy) - -Release 0.22.1 - Unreleased - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - HDFS-2718. Optimize OP_ADD in edits loading. (shv) - - HDFS-2886. CreateEditLogs should generate a realistic edit log. (shv) - - BUG FIXES - - HDFS-2877. If locking of a storage dir fails, it will remove the other - NN's lock file on exit. (todd) - -Release 0.22.0 - 2011-11-29 - - INCOMPATIBLE CHANGES - - HDFS-1825. Remove thriftfs contrib. (nigel via eli) - - NEW FEATURES - - HDFS-992. Re-factor block access token implementation to conform to the - generic Token interface in Common (Kan Zhang and Jitendra Pandey via jghoman) - - HDFS-599. Allow NameNode to have a seprate port for service requests from - client requests. (Dmytro Molkov via hairong) - - HDFS-1004. Update NN to support Kerberized SSL from HADOOP-6584. - (jghoman and Kan Zhang via jghoman) - - HDFS-1005. Fsck security. (borya and Kan Zhang via jghoman) - - HDFS-1006. getImage/putImage http requests should be https for the case - of security enabled. (borya and jghoman via jghoman) - - HDFS-1033. In secure clusters, NN and SNN should verify that the remote - principal during image and edits transfer. (jghoman) - - HDFS-1023. Allow http server to start as regular principal if https - principal not defined. (jghoman) - - HDFS-1150. Verify datanodes' identities to clients in secure clusters. - (jghoman) - - HDFS-1330. Make RPCs to DataNodes timeout. (hairong) - Added additional unit tests per HADOOP-6889. (John George via mattf) - - HDFS-202. HDFS support of listLocatedStatus introduced in HADOOP-6870. - HDFS piggyback block locations to each file status when listing a - directory. (hairong) - - HDFS-1361. Add -fileStatus operation to NNThroughputBenchmark. (shv) - - HDFS-1435. Provide an option to store fsimage compressed. (hairong) - - HDFS-903. Support fsimage validation through MD5 checksum. (hairong) - - HDFS-1457. Provide an option to throttle image transmission between - pimary and secondary NameNodes. (Yifei Lu and hairong via hairong) - - HDFS-1164. TestHdfsProxy is failing. (Todd Lipcon via cos) - - HDFS-811. Add metrics, failure reporting and additional tests for HDFS-457. - (eli) - - HDFS-895. Allow hflush/sync to occur in parallel with new writes - to the file. (Todd Lipcon via hairong) - - HDFS-528. Add ability for safemode to wait for a minimum number of - live datanodes (Todd Lipcon via eli) - - HDFS-1753. Resource Leak in StreamFile. (Uma Maheswara Rao G via eli) - - IMPROVEMENTS - - HDFS-1304. Add a new unit test for HftpFileSystem.open(..). (szetszwo) - - HDFS-1096. fix for prev. commit. (boryas) - - HDFS-1096. allow dfsadmin/mradmin refresh of superuser proxy group - mappings (boryas) - - HDFS-1146. Javadoc for getDelegationTokenSecretManager in FSNamesystem (jnp via boryas) - - HDFS-1132. Refactor TestFileStatus (Eli Collins via cos) - - HDFS-1163. normalize property names for JT/NN kerberos principal - names in configuration (from HADOOP 6633) (boryas) - - HDFS-1003. authorization checks for inter-server protocol - (based on HADOOP-6600) (boryas) - - HDFS-1061. Memory footprint optimization for INodeFile object. - (Bharath Mundlapudi via jghoman) - - HDFS-1079. Throw exceptions as specified by the AbstractFileSystem - in HDFS implemenation and protocols. (suresh) - - HDFS-1112. Edit log buffer should not grow unfoundedly. (hairong) - - HDFS-1119. Introduce a GSet interface to BlocksMap. (szetszwo) - - HDFS-1184. Replace tabs in code with spaces. (Jeff Ames via jghoman) - - HDFS-1185. Remove duplicate now() functions in DataNode, FSNamesysetm. - (Jeff Ames via jghoman) - - HDFS-1183. Remove some duplicate code in NamenodeJspHelper.java. - (Jeff Ames via jghoman) - - HDFS-1190. Remove unused getNamenode() method from DataNode. - (Jeff Ames via jghoman) - - HDFS-1110. Reuses objects for commonly used file names in namenode to - reduce the heap usage. (suresh) - - HDFS-752. Add interfaces classification to to HDFS source code. (suresh) - - HDFS-947. An Hftp read request is redirected to a datanode that has - the most replicas of the blocks in the file. (Dmytro Molkov via dhruba) - - HDFS-1272. Fixes to take care of the changes in HADOOP-6845. - (Jitendra Pandey via ddas) - - HDFS-1298 - Add support in HDFS for new statistics added in FileSystem - to track the file system operations. (suresh) - - HDFS-1201. The HDFS component for HADOOP-6632. - (Kan Zhang & Jitendra Pandey via ddas) - - HDFS-1307 Add start time, end time and total time taken for FSCK to - FSCK report (suresh) - - HDFS-1302. The HDFS side of the changes corresponding to HADOOP-6861. - (Jitendra Pandey & Owen O'Malley via ddas) - - HDFS-1315. Add fsck event to audit log and remove other audit log events - corresponding to FSCK listStatus and open calls. (suresh) - - HDFS-1178. The NameNode servlets should not use RPC to connect to the - NameNode. (Kan Zhang via jghoman) - - HDFS-1130. Adds dfs.cluster.administrator ACL configuration that can - be used to control who can view the default hdfs servlets. (ddas) - - HDFS-1297. Fix some comments. (Jeff Ames via jghoman) - - HDFS-330. Datanode Web UIs should provide robots.txt. - (Allen Wittenauer via jghoman) - - HDFS-881. Refactor DataNode Packet header into DataTransferProtocol. - (Todd Lipcon via jghoman) - - HDFS-1036. docs for fetchdt - - HDFS-1318. Add JMX interface for read access to namenode and datanode - web UI information. (Tanping Wang via suresh). - - HDFS-1356. Provide information as to whether or not security is - enabled on web interface for NameNode (boryas) - - HDFS-1205. FSDatasetAsyncDiskService should name its threads. - (Todd Lipcon via eli) - - HDFS-1111. Introduce getCorruptFileBlocks() for fsck. (Sriram Rao via shv) - - HDFS-1395. Add @Override to FSDataset methods that implement - FSDatasetInterface methods. (suresh) - - HDFS-1383. Improve the error messages when using hftp://. (szetszwo) - - HDFS-1093. Change the FSNamesystem lock to a read/write lock. (dhruba) - - HDFS-1407. Change DataTransferProtocol methods to use Block instead - of individual elements of Block. (suresh) - - HDFS-1417. Add @Override to SimulatedFSDataset methods that implement - FSDatasetInterface methods. (suresh) - - HDFS-1426. Remove unused method BlockInfo#listCount. (hairong) - - HDFS-1472. Allow programmatic access to fsck output. - (Ramkumar Vadali via dhruba) - - HADOOP-7007. Update the hudson-test-patch ant target to work with the - latest test-patch.sh script (gkesavan) - - HDFS-1462. Refactor edit log loading to a separate class from edit log writing. - (Todd Lipcon via eli) - - HDFS-1485. Fix typo in BlockPlacementPolicy. (Jingguo Yao via shv) - - HDFS-1035. Generate Eclipse's .classpath file from Ivy config. (nigel) - - HDFS-1408. Herriot NN and DN clients should vend statistics. (cos) - - HDFS-1491 Update Hdfs to match the change of methods from protected to public - in AbstractFileSystem (Hadoop-6903) (sanjay) - - HDFS-1160. Improve some FSDataset warnings and comments. (eli) - - HDFS-556. Provide info on failed volumes in the web ui. (eli) - - HDFS-697. Enable asserts for tests by default. (eli) - - HDFS-1187. Modify fetchdt to allow renewing and canceling token. - (Owen O'Malley and Kan Zhang via jghoman) - - HDFS-1387. Update HDFS permissions guide for security. (Todd Lipcon via eli) - - HDFS-455. Make NN and DN handle in a intuitive way comma-separated - configuration strings. (Michele Catasta via eli) - - HDFS-1071. savenamespace should write the fsimage to all configured - fs.name.dir in parallel (Dmytro Molkov via jghoman) - - HDFS-1055. Improve thread naming for DataXceivers. - (Todd Lipcon and Ramkumar Vadali via eli). - - HDFS-718. Configuration parameter to prevent accidental formatting of - HDFS filesystem. (Andrew Ryan via jghoman) - - HDFS-1500. TestOfflineImageViewer failing on trunk. (Todd Lipcon - via hairong) - - HDFS-1483. DFSClient.getBlockLocations should indicate if corresponding - blocks are corrupt. (Patrick Kling via hairong) - - HDFS-259. Remove intentionally corrupt 0.13 directory layout creation. - (Todd Lipcon via eli) - - HDFS-1513. Fix a number of warnings. (eli) - - HDFS-1473. Refactor storage management into separate classes than fsimage - file reading/writing. (Todd Lipcon via eli) - - HDFS-1582. Remove auto-generated native build files. (rvs via eli) - - HDFS-1456. Provide builder for constructing instances of MiniDFSCluster. - (jghoman) - - HDFS-1861. Rename dfs.datanode.max.xcievers and bump its default value. - (eli) - - HDFS-1052. HDFS Federation - Merge of umbrella jira changes from - HDFS-1052 branch into trunk. - - HDFS-1835. DataNode should not depend on SHA1PRNG secure random generator - to generate a storage ID. (John Carrino via todd) - - HDFS-1947. DFSClient should use mapreduce.task.attempt.id. (eli) - - HDFS-1957. Add documentation for HFTP. (Ari Rabkin via todd) - - HDFS-1454. Update the documentation to reflect that clients don't write - blocks to local disk before copying to HDFS. (Harsh J Chouraria via todd) - - HDFS-1980. Move build/webapps deeper in the build directory heirarchy - to aid eclipse users. (todd) - - HDFS-1619. Remove AC_TYPE* from the libhdfs. (Roman Shaposhnik via eli) - - HDFS-1948 Forward port 'hdfs-1520 lightweight namenode operation to - trigger lease recovery' (stack) - - HDFS-1954. Improved corrupt files warning on NameNode web UI. - (Patrick Hunt via shv) - - HDFS-1409. BackupNode registration throwing - UnsupportedActionException("register") instead of "journal". - (Ching-Shen Chen via shv) - - HDFS-2054 BlockSender.sendChunk() prints ERROR for connection closures - encountered during transferToFully() (Kihwal Lee via stack) - - OPTIMIZATIONS - - HDFS-1140. Speedup INode.getPathComponents. (Dmytro Molkov via shv) - - HDFS-1081. Performance regression in - DistributedFileSystem::getFileBlockLocations in secure systems (jghoman) - - HDFS-1114. Implement LightWeightGSet for BlocksMap in order to reduce - NameNode memory footprint. (szetszwo) - - HDFS-1320. Add LOG.isDebugEnabled() guard for each LOG.debug(..). - (Erik Steffl via szetszwo) - - HDFS-1368. Add a block counter to DatanodeDescriptor. (hairong) - - HDFS-1434. Refactor Datanode#startDataNode method into smaller methods. - (suresh) - - HDFS-941. The DFS client should cache and reuse open sockets to datanodes - while performing reads. (bc Wong and Todd Lipcon via todd) - - BUG FIXES - - HDFS-1039. Adding test for JspHelper.getUGI(jnp via boryas) - - HDFS-1019. Incorrect default values for delegation tokens in - hdfs-default.xml (jnp via boryas) - - HDFS-1039. Service should be set in the token in JspHelper.getUGI(jnp via boryas) - - HDFS-1038. FIX. A test missed in a previous commit for this JIRA. (boryas) - - HDFS-1038. In nn_browsedfscontent.jsp fetch delegation token only - if security is enabled. (jnp via boryas) - - HDFS-1044. Cannot submit mapreduce job from secure client to - unsecure sever (boryas) - - HDFS-1021. specify correct server principal for RefreshAuthorizationPolicyProtocol - and RefreshUserToGroupMappingsProtocol protocols in DFSAdmin (for HADOOP-6612) (boryas) - - HDFS-970. fsync fsimage to disk before closing fsimage file. - (Todd Lipcon via dhruba) - - HDFS-1027. Update copyright year to 2010. (Ravi Phulari via jghoman) - - HDFS-1080. SecondaryNameNode image transfer should use the defined http - address rather than local ip address. (jghoman) - - HDFS-1198. Resolving cross-realm principals. (Jitendra Pandey via jghoman) - - HDFS-1118. Fix socketleak on DFSClient. (Zheng Shao via dhruba) - - HDFS-1192. refreshSuperUserGroupsConfiguration should use server side - configuration for the refresh (for HADOOP-6815) (boryas) - - HDFS-1036. in DelegationTokenFetch dfs.getURI returns no port (boryas) - - HDFS-1017. browsedfs jsp should call JspHelper.getUGI rather - than using createRemoteUser() (jnp via boryas) - - HDFS-1250. Namenode should reject block reports and block received - requests from dead datanodes (suresh) - - HDFS-1145. When NameNode is shutdown it does not try to exit - safemode anymore. (dhruba) - - HDFS-1202. DataBlockScanner throws NPE when updated before - initialized. (Todd Lipcon via dhruba) - - HDFS-882. Datanode logs the hostname and port its listening on. - (Steve Loughran via dhruba) - - HDFS-1238. ant eclipse-files has drifted again, (jghoman) - - HDFS-1045. In secure clusters, re-login is necessary for https - clients before opening connections. (jghoman) - - HDFS-1289. Datanode secure mode is broken. (Kan Zhang via jghoman) - - HDFS-1007. HFTP needs to be updated to use delegation tokens (boryas) - - HDFS-1085. HFTP read may fail silently on the client side if there is an - exception on the server side. (szetszwo) - - HDFS-1308. job conf key for the services name of DelegationToken for HFTP - url is constructed incorrectly in HFTPFileSystem (boryas) - - HDFS-1319. Fix location of re-login for secondary namenode from HDFS-999. - (jghoman) - - HDFS-1317. Remove the FILEPATH_PATTERN from hdfsproxy.AuthorizationFilter. - (Rohini Palaniswamy via szetszwo) - - HDFS-912. sed in build.xml on Solaris fails. (Allen Wittenauer via jghoman) - - HDFS-1296. using delegation token over hftp for long running - clients (boryas) - - HDFS-1334. open in HftpFileSystem does not add delegation tokens to the url. - (Jitendra Pandey via jghoman) - - HDFS-1301. TestHDFSProxy need to use server side conf for ProxyUser - stuff. (boryas) - - HDFS-1340. When security is turned off, there is a potential XSS attack. - This patch fixes it by removing delegationtoken string from the URL, - before returning a response to the client. (Jitendra Pandey via ddas) - - HDFS-1347. TestDelegationToken uses mortbay.log for logging (boryas) - - HDFS-1157. Modifications introduced by HDFS-1150 are breaking aspect's - bindings (cos) - - HDFS-1349. Remove empty java files. (Eli Collins) - - HDFS-1340. A null delegation token is appended to the url if security - is disabled when browsing filesystem. (boryas) - - HDFS-1352. Fix jsvc.location. (Eli Collins via jghoman) - - HDFS-1284. TestBlockToken fails. (Kan Zhang via jghoman) - - HDFS-1355. ant veryclean (clean-cache) doesn't clean enough. - (Luke Lu via jghoman) - - HDFS-1353. Remove most of getBlockLocation optimization. (jghoman) - - HDFS-1369. Invalid javadoc reference in FSDatasetMBean.java (Eli Collins) - - HDFS-829. hdfsJniHelper.c: #include is not portable. - (Allen Wittenauer via jghoman) - - HDFS-1310. The ClientDatanodeProtocol proxy should be stopped in - DFSInputStream.readBlockLength(..). (sam rash via szetszwo) - - HDFS-1357. HFTP traffic served by DataNode shouldn't use service port - on NameNode. (Kan Zhang via jghoman) - - HDFS-1419. HDFS Federation: Three test cases need minor modification after - the new block id change (Tanping Wang via suresh) - - HDFS-96. HDFS supports blocks larger than 2 GB. - (Patrick Kling via dhruba) - - HDFS-1433. Fix test failures - TestPread and TestFileLimit. (suresh) - - HDFS-1364. Makes long running HFTP-based applications do relogins - if necessary. (Jitendra Pandey via ddas) - - HDFS-1399. Distinct minicluster services (e.g. NN and JT) overwrite each - other's service policies. (Aaron T. Myers via tomwhite) - - HDFS-1440. Fix TestComputeInvalidateWork failure. (suresh) - - HDFS-1498. FSDirectory#unprotectedConcat calls setModificationTime - on a file. (eli) - - HDFS-1625. Ignore disk space values in TestDataNodeMXBean. (szetszwo) - - HDFS-1850. DN should transmit absolute failed volume count rather than - increments to the NN. (eli) - - HDFS-671. Documentation change for updated configuration keys. - (tomwhite via eli) - - HDFS-1544. Ivy resolve force mode should be turned off by default. - (Luke Lu via tomwhite) - - HDFS-1615. seek() on closed DFS input stream throws NullPointerException - (Scott Carey via todd) - - HDFS-1897. Documentation refers to removed option dfs.network.script - (Andrew Whang via todd) - - HDFS-1621. Fix references to hadoop-common-${version} in build.xml - (Jolly Chen via todd) - - HDFS-1505. saveNamespace appears to succeed even if all directories fail - to save. (Aaron T. Myers via todd) - - HDFS-1921. saveNamespace can cause NN to be unable to come up on restart - (Matt Foley via todd) - - HDFS-1925. SafeModeInfo should use the correct constant instead of a - hard-coded value for its default. (Joey Echeverria via todd) - - HDFS-1575. Viewing block from web UI is broken. (Aaron T. Myers via todd) - - HDFS-1932. Ensure that HDFS configuration deprecations are set up in every - spot that HDFS configurations are loaded. (Jolly Chen via todd) - - HDFS-1952. FSEditLog.open() appears to succeed even if all EDITS - directories fail. (Andrew Wang via todd) - - HDFS-1965. IPCs done using block token-based tickets can't reuse - connections (todd) - - HDFS-1978. All but first option in LIBHDFS_OPTS is ignored. (eli) - - HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper - (Aaron T. Myers via todd) - - HDFS-1997. Image transfer process misreports client side exceptions. - (todd via eli) - - HDFS-2000. Missing deprecation for io.bytes.per.checksum. - (Aaron T. Myers vie eli) - - HDFS-977. DataNode.createInterDataNodeProtocolProxy() guards a log - at the wrong level. (Harsh J Chouraria via todd) - - HDFS-1969. Running rollback on new-version namenode destroys the - namespace. (todd) - - HDFS-2039. TestNameNodeMetrics uses a bad test root path, preventing it - from running inside Eclipse. (todd) - - HDFS-988. saveNamespace race can corrupt the edits log. (eli) - - HDFS-2071. Use of isConnected() in DataXceiver is invalid. (Kihwal Lee - via todd) - - HDFS-1981. NameNode does not saveNamespace() when editsNew is empty. - (Uma Maheswara Rao G via shv) - - HDFS-2258. Reset lease limits to default values in TestLeaseRecovery2. (shv) - - HDFS-2232. Generalize regular expressions in TestHDFSCLI. - (Plamen Jeliazkov via shv) - - HDFS-2290. Block with corrupt replica is not getting replicated. - (Benoy Antony via shv) - - HDFS-2012. Balancer incorrectly treats nodes whose utilization equals - avgUtilization. (Uma Maheswara Rao G via shv) - - HDFS-2491. TestBalancer can fail when datanode utilization and - avgUtilization is exactly same. (Uma Maheswara Rao G via shv) - - HDFS-2452. OutOfMemoryError in DataXceiverServer takes down the DataNode - (Uma Maheswara Rao via cos) - - HDFS-2002. Incorrect computation of needed blocks in getTurnOffTip(). - (Plamen Jeliazkov via shv) - - HDFS-2573. TestFiDataXceiverServer is failing, not testing OOME (cos) - - HDFS-2514. Link resolution bug for intermediate symlinks with - relative targets. (eli) - - HDFS-1786. Some cli test cases expect a "null" message - (Uma Maheswara Rao G via todd) - - HDFS-1855. TestDatanodeBlockScanner.testBlockCorruptionRecoveryPolicy() - part 2 fails in two different ways. (Matt Foley via eli) - - HDFS-2346. TestHost2NodesMap & TestReplicasMap will fail depending upon - execution order of test methods (Laxman, Uma Maheswara Rao G via shv) - - HDFS-2287. TestParallelRead has a small off-by-one bug. (todd) - - HDFS-1466. TestFcHdfsSymlink relies on /tmp/test not existing. (eli) - - HDFS-874. TestHDFSFileContextMainOperations fails on weirdly - configured DNS hosts. (Todd Lipcon via eli) - - HDFS-1507. TestAbandonBlock should abandon a block. (eli) - - HDFS-1487. FSDirectory.removeBlock() should update diskspace count - of the block owner node (Zhong Wang via eli). - - HDFS-1467. Append pipeline never succeeds with more than one replica. - (Todd Lipcon via eli) - - HDFS-1167. New property for local conf directory in system-test-hdfs.xml - file. (Vinay Thota via cos) - - HDFS-1503. TestSaveNamespace fails. (Todd Lipcon via cos) - - HDFS-1524. Image loader should make sure to read every byte in image file. - (hairong) - - HDFS-1523. TestLargeBlock is failing on trunk. (cos) - - HDFS-1502. TestBlockRecovery triggers NPE in assert. (hairong via cos) - - HDFS-1532. Exclude Findbugs warning in FSImageFormat$Saver. (Todd Lipcon - via cos) - - HDFS-1527. SocketOutputStream.transferToFully fails for blocks >= 2GB on - 32 bit JVM. (Patrick Kling via cos) - - HDFS-1531. Clean up stack traces due to duplicate MXBean registration. - (Todd Lipcon via cos) - - HDFS-613. TestBalancer and TestBlockTokenWithDFS fail Balancer assert. - (Todd Lipcon via cos) - - HDFS-1511. 98 Release Audit warnings on trunk and branch-0.22. - (jghoman) - - HDFS-1560. dfs.data.dir permissions should default to 700. - (Todd Lipcon via eli) - - HDFS-1550. NPE when listing a file with no location. (hairong) - - HDFS-1542. Add test for HADOOP-7082, a deadlock writing Configuration to - HDFS. (todd) - - HDFS-1504. FSImageSaver should catch all exceptions, not just IOE. (todd) - - HDFS-884. DataNode throws IOException if all data directories are - unavailable. (Steve Loughran and shv) - - HDFS-1591. HDFS part of HADOOP-6642. (Chris Douglas, Po Cheung via shv) - - HDFS-900. Corrupt replicas are not processed correctly in block report (shv) - - HDFS-1529. Incorrect handling of interrupts in waitForAckedSeqno can cause - deadlock (todd) - - HDFS-1597. Batched edit log syncs can reset synctxid and throw assertions - (todd) - - HDFS-1602. Fix HADOOP-4885 for it is doesn't work as expected. (boryas) - - HDFS-1618. configure files that are generated as part of the released - tarball need to have executable bit set (Roman Shaposhnik via cos) - - HDFS-981. test-contrib fails due to test-cactus failure (cos) - - HDFS-1001. DataXceiver and BlockReader disagree on when to send/recv - CHECKSUM_OK. (bc Wong via eli) - - HDFS-1781. Fix the path for jsvc in bin/hdfs. (John George via szetszwo) - - HDFS-1782. Fix an NPE in FSNamesystem.startFileInternal(..). - (John George via szetszwo) - - HDFS-1821. Fix username resolution in NameNode.createSymlink(..) and - FSDirectory.addSymlink(..). (John George via szetszwo) - - HDFS-1806. TestBlockReport.blockReport_08() and _09() are timing-dependent - and likely to fail on fast servers. (Matt Foley via eli) - - HDFS-1845. Symlink comes up as directory after namenode restart. - (John George via eli) - - HDFS-1666. Disable failing hdfsproxy test TestAuthorizationFilter (todd) - - HDFS-1823. start-dfs.sh script fails if HADOOP_HOME is not set. - (tomwhite via eli) - -Release 0.21.1 - Unreleased - - HDFS-1411. Correct backup node startup command in hdfs user guide. - (Ching-Shen Chen via shv) - - BUG FIXES - - HDFS-1363. Eliminate second synchronized sections in appendFile(). (shv) - - HDFS-1413. Fix broken links to HDFS Wiki. (shv) - - HDFS-1420. Clover build doesn't generate per-test coverage (cos) - - HDFS-1444. Test related code of build.xml is error-prone and needs to be - re-aligned. (cos) - - HDFS-1343. Instrumented build should be concentrated in one build area (cos) - - HDFS-1452. ant compile-contrib is broken (cos) - - HDFS-1474. ant binary-system is broken (cos) - - HDFS-1292. Allow artifacts to be published to the staging Apache Nexus - Maven Repository. (Giridharan Kesavan via tomwhite) - - HDFS-1552. Remove java5 dependencies from build. (cos) - - HDFS-1189. Quota counts missed between clear quota and set quota. - (John George via szetszwo) - - HDFS-1665. Balancer misuses dfs.heartbeat.interval as milliseconds. - (szetszwo) - - HDFS-1728. SecondaryNameNode.checkpointSize is in bytes but not in MB. - (szetszwo) - - HDFS-1206. TestFiHFlush fails intermittently. (cos) - - HDFS-1548. Fault-injection tests are executed multiple times if invoked - with run-test-hdfs-fault-inject target (cos) - - HDFS-1552. Remove java5 dependencies from build. (cos) - - HDFS-996. JUnit tests should never depend on anything in conf (cos) - - HDFS-1612. Update HDFS design documentation for append, quota, symlink, - block placement and checkpoint/backup node features. (Joe Crobak - via szetszwo) - - HDFS-1596. Replace fs.checkpoint.* with dfs.namenode.checkpoint.* - in documentations. (Harsh J Chouraria via szetszwo) - -Release 0.21.0 - 2010-08-13 - - INCOMPATIBLE CHANGES - - HDFS-538. Per the contract elucidated in HADOOP-6201, throw - FileNotFoundException from FileSystem::listStatus rather than returning - null. (Jakob Homan via cdouglas) - - HDFS-602. DistributedFileSystem mkdirs throws FileAlreadyExistsException - instead of FileNotFoundException. (Boris Shkolnik via suresh) - - HDFS-544. Add a "rbw" subdir to DataNode data directory. (hairong) - - HDFS-576. Block report includes under-construction replicas. (shv) - - HDFS-636. SafeMode counts complete blocks only. (shv) - - HDFS-644. Lease recovery, concurrency support. (shv) - - HDFS-570. Get last block length from a data-node when opening a file - being written to. (Tsz Wo (Nicholas), SZE via shv) - - HDFS-657. Remove unused legacy data-node protocol methods. (shv) - - HDFS-658. Block recovery for primary data-node. (shv) - - HDFS-660. Remove deprecated methods from InterDatanodeProtocol. (shv) - - HDFS-512. Block.equals() and compareTo() compare blocks based - only on block Ids, ignoring generation stamps. (shv) - - HDFS-873. Configuration specifies data-node storage directories as URIs. - (shv) - - HDFS-905. Use the new UserGroupInformation from HDFS-6299. - (jghoman via omalley) - - HDFS-984. Persistent delegation tokens. (Jitendra Pandey via shv) - - HDFS-1016. HDFS side change for HADOOP-6569. This jira changes the - error message on the screen when cat a directory or a - non-existent file. (hairong) - - NEW FEATURES - - HDFS-1134. Large-scale Automated Framework. (cos) - - HDFS-436. Introduce AspectJ framework for HDFS code and tests. - (Konstantin Boudnik via szetszwo) - - HDFS-447. Add LDAP lookup to hdfsproxy. (Zhiyong Zhang via cdouglas) - - HDFS-459. Introduce Job History Log Analyzer. (shv) - - HDFS-461. Tool to analyze file size distribution in HDFS. (shv) - - HDFS-492. Add two JSON JSP pages to the Namenode for providing corrupt - blocks/replicas information. (Bill Zeller via szetszwo) - - HDFS-578. Add support for new FileSystem method for clients to get server - defaults. (Kan Zhang via suresh) - - HDFS-595. umask settings in configuration may now use octal or symbolic - instead of decimal. (Jakob Homan via suresh) - - HADOOP-6234. Updated hadoop-core and test jars to propagate new option - dfs.umaskmode in configuration. (Jakob Homan via suresh) - - HDFS-235. Add support for byte ranges in HftpFileSystem to serve - range of bytes from a file. (Bill Zeller via suresh) - - HDFS-385. Add support for an experimental API that allows a module external - to HDFS to specify how HDFS blocks should be placed. (dhruba) - - HADOOP-4952. Update hadoop-core and test jars to propagate new FileContext - file system application interface. (Sanjay Radia via suresh). - - HDFS-567. Add block forensics contrib tool to print history of corrupt and - missing blocks from the HDFS logs. - (Bill Zeller, Jitendra Nath Pandey via suresh). - - HDFS-610. Support o.a.h.fs.FileContext. (Sanjay Radia via szetszwo) - - HDFS-536. Support hflush at DFSClient. (hairong) - - HDFS-517. Introduce BlockInfoUnderConstruction to reflect block replica - states while writing. (shv) - - HDFS-565. Introduce block committing logic during new block allocation - and file close. (shv) - - HDFS-537. DataNode exposes a replica's meta info to BlockReceiver for the - support of dfs writes/hflush. It also updates a replica's bytes received, - bytes on disk, and bytes acked after receiving a packet. (hairong) - - HDFS-585. Datanode should serve up to visible length of a replica for read - requests. (szetszwo) - - HDFS-604. Block report processing for append. (shv) - - HDFS-619. Support replica recovery initialization in datanode for the new - append design. (szetszwo) - - HDFS-592. Allow clients to fetch a new generation stamp from NameNode for - pipeline recovery. (hairong) - - HDFS-624. Support a new algorithm for pipeline recovery and pipeline setup - for append. (hairong) - - HDFS-627. Support replica update in data-node. - (Tsz Wo (Nicholas), SZE and Hairong Kuang via shv) - - HDFS-642. Support pipeline close and close error recovery. (hairong) - - HDFS-631. Rename configuration keys towards API standardization and - backward compatibility. (Jitendra Nath Pandey via suresh) - - HDFS-669. Add unit tests framework (Mockito) (cos, Eli Collins) - - HDFS-731. Support new Syncable interface in HDFS. (hairong) - - HDFS-702. Add HDFS implementation of AbstractFileSystem. - (Sanjay Radio via suresh) - - HDFS-758. Add decommissioning status page to Namenode Web UI. - (Jitendra Nath Pandey via suresh) - - HDFS-814. Add an api to get the visible length of a DFSDataInputStream. - (szetszwo) - - HDFS-654. Add support new atomic rename functionality in HDFS for - supporting rename in FileContext. (suresh) - - HDFS-222. Support for concatenating of files into a single file - without copying. (Boris Shkolnik via hairong) - - HDFS-933. Adds Delegation token based authentication in the NameNode. - (Kan Zhang via ddas) - - HDFS-935. Adds a real user component in Delegation token. - (Jitendra Nath Pandey via ddas) - - HDFS-245. Adds a symlink implementation to HDFS. This complements the new - symlink feature added in HADOOP-6421 (Eli Collins via Sanjay Radia) - - HDFS-1009. Support Kerberos authorization in HDFSProxy. (Srikanth - Sundarrajan via szetszwo) - - HDFS-1091. Implement listStatus that returns an iterator of FileStatus. - (hairong) - - IMPROVEMENTS - - HDFS-381. Remove blocks from DataNode maps when corresponding file - is deleted. (Suresh Srinivas via rangadi) - - HDFS-377. Separate codes which implement DataTransferProtocol. - (szetszwo) - - HDFS-396. NameNode image and edits directories are specified as URIs. - (Luca Telloli via rangadi) - - HDFS-444. Allow to change probability levels dynamically in the fault - injection framework. (Konstantin Boudnik via szetszwo) - - HDFS-352. Documentation for saveNamespace command. (Ravi Phulari via shv) - - HADOOP-6106. Updated hadoop-core and test jars from hudson trunk - build #12. (Giridharan Kesavan) - - HDFS-204. Add a new metrics FilesInGetListingOps to the Namenode. - (Jitendra Nath Pandey via szetszwo) - - HDFS-278. HDFS Outputstream close does not hang forever. (dhruba) - - HDFS-443. Add a new metrics numExpiredHeartbeats to the Namenode. - (Jitendra Nath Pandey via szetszwo) - - HDFS-475. Add new ant targets for fault injection jars and tests. - (Konstantin Boudnik via szetszwo) - - HDFS-458. Create a new ant target, run-commit-test. (Jakob Homan - via szetszwo) - - HDFS-493. Change build.xml so that the fault-injected tests are executed - only by the run-test-*-fault-inject targets. (Konstantin Boudnik via - szetszwo) - - HDFS-446. Improvements to Offline Image Viewer. (Jakob Homan via shv) - - HADOOP-6160. Fix releaseaudit target to run on specific directories. - (gkesavan) - - HDFS-501. Use enum to define the constants in DataTransferProtocol. - (szetszwo) - - HDFS-508. Factor out BlockInfo from BlocksMap. (shv) - - HDFS-510. Rename DatanodeBlockInfo to be ReplicaInfo. - (Jakob Homan & Hairong Kuang via shv) - - HDFS-500. Deprecate NameNode methods deprecated in NameNodeProtocol. - (Jakob Homan via shv) - - HDFS-514. Change DFSClient.namenode from public to private. (Bill Zeller - via szetszwo) - - HDFS-496. Use PureJavaCrc32 in HDFS. (Todd Lipcon via szetszwo) - - HDFS-511. Remove redundant block searches in BlockManager. (shv) - - HDFS-504. Update the modification time of a file when the file - is closed. (Chun Zhang via dhruba) - - HDFS-498. Add development guide and documentation for the fault injection - framework. (Konstantin Boudnik via szetszwo) - - HDFS-524. Further DataTransferProtocol code refactoring. (szetszwo) - - HDFS-529. Use BlockInfo instead of Block to avoid redundant block searches - in BlockManager. (shv) - - HDFS-530. Refactor TestFileAppend* to remove code duplication. - (Konstantin Boudnik via szetszwo) - - HDFS-451. Add fault injection tests for DataTransferProtocol. (szetszwo) - - HDFS-409. Add more access token tests. (Kan Zhang via szetszwo) - - HDFS-546. DatanodeDescriptor iterates blocks as BlockInfo. (shv) - - HDFS-457. Do not shutdown datanode if some, but not all, volumes fail. - (Boris Shkolnik via szetszwo) - - HDFS-548. TestFsck takes nearly 10 minutes to run. (hairong) - - HDFS-539. Refactor fault injeciton pipeline test util for future reuse. - (Konstantin Boudnik via szetszwo) - - HDFS-552. Change TestFiDataTransferProtocol to junit 4 and add a few new - tests. (szetszwo) - - HDFS-563. Simplify the codes in FSNamesystem.getBlockLocations(..). - (szetszwo) - - HDFS-581. Introduce an iterator over blocks in the block report array.(shv) - - HDFS-549. Add a new target, run-with-fault-inject-testcaseonly, which - allows an execution of non-FI tests in FI-enable environment. (Konstantin - Boudnik via szetszwo) - - HDFS-173. Namenode will not block until a large directory deletion - completes. It allows other operations when the deletion is in progress. - (suresh) - - HDFS-551. Create new functional test for a block report. (Konstantin - Boudnik via hairong) - - HDFS-288. Redundant computation in hashCode() implementation. - (szetszwo via tomwhite) - - HDFS-412. Hadoop JMX usage makes Nagios monitoring impossible. - (Brian Bockelman via tomwhite) - - HDFS-472. Update hdfsproxy documentation. Adds a setup guide and design - document. (Zhiyong Zhang via cdouglas) - - HDFS-617. Support non-recursive create(). (Kan Zhang via szetszwo) - - HDFS-618. Support non-recursive mkdir(). (Kan Zhang via szetszwo) - - HDFS-574. Split the documentation between the subprojects. - (Corinne Chandel via omalley) - - HDFS-598. Eclipse launch task for HDFS. (Eli Collins via tomwhite) - - HDFS-641. Move all of the components that depend on map/reduce to - map/reduce. (omalley) - - HDFS-509. Redesign DataNode volumeMap to include all types of Replicas. - (hairong) - - HDFS-562. Add a test for NameNode.getBlockLocations(..) to check read from - un-closed file. (szetszwo) - - HDFS-543. Break FSDatasetInterface#writToBlock() into writeToRemporary, - writeToRBW, ad append. (hairong) - - HDFS-603. Add a new interface, Replica, which is going to replace the use - of Block in datanode. (szetszwo) - - HDFS-589. Change block write protocol to support pipeline recovery. - (hairong) - - HDFS-652. Replace BlockInfo.isUnderConstruction() with isComplete() (shv) - - HDFS-648. Change some methods in AppendTestUtil to public. (Konstantin - Boudnik via szetszwo) - - HDFS-662. Unnecessary info message from DFSClient. (hairong) - - HDFS-518. Create new tests for Append's hflush. (Konstantin Boudnik - via szetszwo) - - HDFS-688. Add configuration resources to DFSAdmin. (shv) - - HDFS-29. Validate the consistency of the lengths of replica and its file - in replica recovery. (szetszwo) - - HDFS-680. Add new access method to a copy of a block's replica. (shv) - - HDFS-704. Unify build property names to facilitate cross-projects - modifications (cos) - - HDFS-705. Create an adapter to access some of package-private methods of - DataNode from tests (cos) - - HDFS-710. Add actions with constraints to the pipeline fault injection - tests and change SleepAction to support uniform random sleeping over an - interval. (szetszwo) - - HDFS-713. Need to properly check the type of the test class from an aspect - (cos) - - HDFS-716. Define a pointcut for pipeline close and add a few fault - injection tests to simulate out of memory problem. (szetszwo) - - HDFS-719. Add 6 fault injection tests for pipeline close to simulate slow - datanodes and disk errors. (szetszwo) - - HDFS-616. Create functional tests for new design of the block report. (cos) - - HDFS-584. Fail the fault-inject build if any advices are mis-bound. (cos) - - HDFS-730. Add 4 fault injection tests to simulate non-responsive datanode - and out-of-memory problem for pipeline close ack. (szetszwo) - - HDFS-728. Create a comprehensive functional test for append. (hairong) - - HDFS-736. commitBlockSynchronization() updates block GS and length - in-place. (shv) - - HADOOP-5107. Use Maven ant tasks to publish the subproject jars. - (Giridharan Kesavan via omalley) - - HDFS-521. Create new tests for pipeline (cos) - - HDFS-764. Places the Block Access token implementation in hdfs project. - (Kan Zhang via ddas) - - HDFS-787. Upgrade some libraries to be consistent with common and - mapreduce. (omalley) - - HDFS-519. Create new tests for lease recovery (cos) - - HDFS-804. New unit tests for concurrent lease recovery (cos) - - HDFS-813. Enable the append test in TestReadWhileWriting. (szetszwo) - - HDFS-145. Cleanup inconsistent block length handling code in - FSNameSystem#addStoredBlock. (hairong) - - HDFS-127. Reset failure count in DFSClient for each block acquiring - operation. (Igor Bolotin via szetszwo) - - HDFS-520. Create new tests for block recovery. (hairong) - - HDFS-1067. Create block recovery tests that handle errors. (hairong) - - HDFS-1107. Turn on append by default. (shv) - - HDFS-968. Use StringBuilder instead of StringBuffer for better - performance. (Kay Kay via suresh) - - HDFS-703. Replace current fault injection implementation with one - from (cos) - - HDFS-754. Reduce ivy console output to observable level (cos) - - HDFS-832. HDFS side of HADOOP-6222. (cos) - - HDFS-840. Change tests to use FileContext test helper introduced in - HADOOP-6394. (Jitendra Nath Pandey via suresh) - - HDFS-685. Use the user-to-groups mapping service in the NameNode. - (boryas, acmurthy) - - HDFS-755. Read multiple checksum chunks at once in DFSInputStream. - (Todd Lipcon via tomwhite) - - HDFS-786. Implement getContentSummary in HftpFileSystem. - (Tsz Wo (Nicholas), SZE via cdouglas) - - HDFS-587. Add support for specifying queue name in mapreduce tests. - (Erik Steffl via suresh) - - HDFS-902 Move contrib/raid to MapReduce. (Eli Collins via omalley) - - HDFS-800. The last block of a file under construction may change to the - COMPLETE state in response to getAdditionalBlock or completeFileInternal. - (hairong) - - HDFS-899. Delegation Token Implementation - and corresponding changes in Namenode and DFS Api to issue, - renew and cancel delegation tokens. (jnp via boryas) - - HDFS-844. Log the filename when file locking fails. (tomwhite) - - HDFS-914. Refactor DFSOutputStream and DFSInputStream out of DFSClient. - (Todd Lipcon via tomwhite) - - HDFS-949. Move DelegationToken into Common so that it can be used by - MapReduce. (omalley) - - HDFS-930. Better error message for DATA_TRANSFER_VERSION mismatched. - (Kay Kay via szetszwo) - - HDFS-986. Delegation token renewing and cancelling should provide - meaningful exceptions when there are failures instead of returning - false. (omalley) - - HADOOP-6579. Upgrade the commons-codec library to 1.4. (omalley) - - HDFS-991. Allow authentication to the web ui via a delegation token. - (omalley) - - HDFS-994. Allow fetching of delegation token from NameNode for hftp. - (Jakob Homan via acmurthy) - - HDFS-998. Quote blocks streamed through jsps. (cdouglas) - - HDFS-729. NameNode API to list files that have missing blocks. - (Rodrigo Schmidt via dhruba) - - HDFS-850. The WebUI display more details about namenode memory usage. - (Dmytro Molkov via dhruba) - - HDFS-826. The DFSOutputStream has a API that returns the number of - active datanode(s) in the current pipeline. (dhruba) - - HDFS-985. HDFS should issue multiple RPCs for listing a large - directory. (hairong) - - HDFS-1043. NNThroughputBenchmark modifications to support benchmarking of - server-side user group resolution. (shv) - - HDFS-892. Optionally use Avro reflection for Namenode RPC. This - is not a complete implementation yet, but rather a starting point. - (cutting) - - HDFS-854. Datanode should scan devices in parallel to generate - block report. (Dmytro Molkov via jhoman) - - HDFS-1032. fsck has an option to list corrupt files. - (Andre Oriai via dhruba) - - HDFS-1024. SecondaryNameNode verifies size of fsimage and edits file. - (Dmytro Molkov via dhruba) - - HDFS-1011. hdfsproxy: Improve log messages by restoring the previous - thread name. (Srikanth Sundarrajan via szetszwo) - - HDFS-997. Allow datanode storage directory permissions to be configurable. - (Luke Lu via cdouglas) - - HDFS-1012. hdfsproxy: Support for fully qualified HDFS path in addition to - simple unqualified path. (Srikanth Sundarrajan via szetszwo) - - HDFS-993. Namenode should issue a delegation token only for kerberos - authenticated clients.(jnp via boryas) - - HDFS-1087. Modify audit log to use a StringBuilder rather than a Formatter. - (cdouglas) - - HDFS-1083. Update TestHDFSCLI not to expect exception class name - in error messages. (suresh) - - HDFS-1099. Add test for umask backward compatibility. (suresh) - - HDFS-1092. Use logging rather than System.err in MiniDFSCluster. - (Kay Kay via jghoman) - - HDFS-1047. Install/deploy source jars to Maven repo. - (Patrick Angeles via jghoman) - - HDFS-666. Unit test for FsShell -text. (cdouglas via jghoman) - - HDFS-1054. Remove unnecessary sleep after failure in nextBlockOutputStream. - (Todd Lipcon via jghoman) - - HDFS-921. Convert TestDFSClientRetries::testNotYetReplicatedErrors - to Mockito. (jghoman) - - HDFS-1100. Override unwrapException in TestFcHdfsSymlink to test - symlink API conformance. (Eli Collins via suresh). - - HDFS-1089. Remove uses of FileContext#isFile, isDirectory, and exists. - (Eli Collins via hairong) - - HDFS-1028. Efficient splitting of path components reduces the time - to load in fsimage by 20%. (Dmytro Molkov via dhruba) - - HDFS-1109. HFTP supports filenames that contains the character "+". - (Dmytro Molkov via dhruba) - - HDFS-853. The HDFS webUI displays the balanced-ness of the cluster. - (Dmytro Molkov via dhruba) - - HDFS-1126. Change HDFS to depend on Hadoop 'common' artifacts instead - of 'core'. (tomwhite) - - HDFS-995. Replace usage of FileStatus#isDir(). (Eli Collins via - tomwhite) - - HDFS-1161. Make DN minimum valid volumes configurable. - (Eli Collins via tomwhite) - - HDFS-1181. Move configuration and script files post split. (tomwhite) - - HDFS-1170. Add more assertions to TestLargeDirectoryDelete. - (Steve Loughran via tomwhite) - - HDFS-1199. Extract a subset of tests for smoke (DOA) validation. (cos) - - HDFS-1174. New properties for suspend and resume process. (Vinay Thota via - cos) - - HDFS-1277. [Herriot] New property for multi user list. (Vinay Thota via - cos) - - HDFS-806. Add new unit tests to the 10-mins 'run-commit-test' target (cos) - - OPTIMIZATIONS - - HDFS-946. NameNode should not return full path name when lisitng a - diretory or getting the status of a file. (hairong) - - BUG FIXES - - HDFS-76. Better error message to users when commands fail because of - lack of quota. Allow quota to be set even if the limit is lower than - current consumption. (Boris Shkolnik via rangadi) - - HADOOP-4687. HDFS is split from Hadoop Core. It is a subproject under - Hadoop (Owen O'Malley) - - HADOOP-6096. Fix Eclipse project and classpath files following project - split. (tomwhite) - - HDFS-195. Handle expired tokens when write pipeline is reestablished. - (Kan Zhang via rangadi) - - HDFS-181. Validate src path in FSNamesystem.getFileInfo(..). (Todd - Lipcon via szetszwo) - - HDFS-441. Remove TestFTPFileSystem. (szetszwo) - - HDFS-440. Fix javadoc broken links in DFSClient. (szetszwo) - - HDFS-480. Fix a typo in the jar name in build.xml. - (Konstantin Shvachko via gkesavan) - - HDFS-438. Check for NULL before invoking GenericArgumentParser in - DataNode. (Raghu Angadi) - - HDFS-415. BlockReceiver hangs in case of certain runtime exceptions. - (Konstantin Boudnik via rangadi) - - HDFS-462. loadFSImage should close edits file. (Jakob Homan via shv) - - HDFS-489. Update TestHDFSCLI for the -skipTrash option in rm. (Jakob Homan - via szetszwo) - - HDFS-445. pread() does not pick up changes to block locations. - (Kan Zhang via rangadi) - - HDFS-463. CreateEditLog utility broken after HDFS-396 (URI for - FSImage). (Suresh Srinivas via rangadi) - - HDFS-484. Fix bin-package and package target to package jar files. - (gkesavan) - - HDFS-490. Eliminate the deprecated warnings introduced by H-5438. - (He Yongqiang via szetszwo) - - HDFS-119. Fix a bug in logSync(), which causes NameNode block forever. - (Suresh Srinivas via shv) - - HDFS-534. Include avro in ivy. (szetszwo) - - HDFS-532. Allow applications to know that a read request failed - because block is missing. (dhruba) - - HDFS-561. Fix write pipeline READ_TIMEOUT in DataTransferProtocol. - (Kan Zhang via szetszwo) - - HDFS-553. BlockSender reports wrong failed position in ChecksumException. - (hairong) - - HDFS-568. Set mapred.job.tracker.retire.jobs to false in - src/test/mapred-site.xml for mapreduce tests to run. (Amareshwari - Sriramadasu via szetszwo) - - HDFS-15. All replicas end up on 1 rack. (Jitendra Nath Pandey via hairong) - - HDFS-586. TestBlocksWithNotEnoughRacks sometimes fails. - (Jitendra Nath Pandey via hairong) - - HADOOP-6243. Fixed a NullPointerException in handling deprecated keys. - (Sreekanth Ramakrishnan via yhemanth) - - HDFS-605. Do not run fault injection tests in the run-test-hdfs-with-mr - target. (Konstantin Boudnik via szetszwo) - - HDFS-606. Fix ConcurrentModificationException in invalidateCorruptReplicas() - (shv) - - HDFS-601. TestBlockReport obtains data directories directly from - MiniHDFSCluster. (Konstantin Boudnik via shv) - - HDFS-614. TestDatanodeBlockScanner obtains data directories directly from - MiniHDFSCluster. (shv) - - HDFS-612. Remove the use of org.mortbay.log.Log in FSDataset. (szetszwo) - - HDFS-622. checkMinReplication should count live nodes only. (shv) - - HDFS-629. Remove ReplicationTargetChooser.java along with fixing - import warnings generated by Eclipse. (dhruba) - - HDFS-637. DataNode sends a Success ack when block write fails. (hairong) - - HDFS-640. Fixed TestHDFSFileContextMainOperations.java build failure. (suresh) - - HDFS-547. TestHDFSFileSystemContract#testOutputStreamClosedTwice - sometimes fails with CloseByInterruptException. (hairong) - - HDFS-588. Fix TestFiDataTransferProtocol and TestAppend2 failures. (shv) - - HDFS-550. DataNode restarts may introduce corrupt/duplicated/lost replicas - when handling detached replicas. (hairong) - - HDFS-659. If the the last block is not complete, update its length with - one of its replica's length stored in datanode. (szetszwo) - - HDFS-649. Check null pointers for DataTransferTest. (Konstantin Boudnik - via szetszwo) - - HDFS-661. DataNode upgrade fails on non-existant current directory. - (hairong) - - HDFS-597. Mofication introduced by HDFS-537 breakes an advice binding in - FSDatasetAspects. (Konstantin Boudnik via szetszwo) - - HDFS-665. TestFileAppend2 sometimes hangs. (hairong) - - HDFS-676. Fix NPE in FSDataset.updateReplicaUnderRecovery() (shv) - - HDFS-673. BlockReceiver#PacketResponder should not remove a packet from - the ack queue before its ack is sent. (hairong) - - HDFS-682. Fix bugs in TestBlockUnderConstruction. (szetszwo) - - HDFS-668. TestFileAppend3#TC7 sometimes hangs. (hairong) - - HDFS-679. Appending to a partial chunk incorrectly assumes the - first packet fills up the partial chunk. (hairong) - - HDFS-722. Fix callCreateBlockWriteStream pointcut in FSDatasetAspects. - (szetszwo) - - HDFS-690. TestAppend2#testComplexAppend failed on "Too many open files". - (hairong) - - HDFS-725. Support the build error fix for HADOOP-6327. (Sanjay Radia via - szetszwo) - - HDFS-625. Fix NullPointerException thrown from ListPathServlet. (suresh) - - HDFS-735. TestReadWhileWriting has wrong line termination symbols (cos) - - HDFS-691. Fix an overflow error in DFSClient.DFSInputStream.available(). - (szetszwo) - - HDFS-733. TestBlockReport fails intermittently. (cos) - - HDFS-774. Intermittent race condition in TestFiPipelines (cos) - - HDFS-741. TestHFlush test doesn't seek() past previously written part of - the file (cos, szetszwo) - - HDFS-706. Intermittent failures in TestFiHFlush (cos) - - HDFS-646. Fix test-patch failure by adding test-contrib ant target. - (gkesavan) - - HDFS-791. Build is broken after HDFS-787 patch has been applied (cos) - - HDFS-792. TestHDFSCLI is failing. (Todd Lipcon via cos) - - HDFS-781. Namenode metrics PendingDeletionBlocks is not decremented. - (Suresh) - - HDFS-192. Fix TestBackupNode failures. (shv) - - HDFS-797. TestHDFSCLI much slower after HDFS-265 merge. (Todd Lipcon via cos) - - HDFS-824. Stop lease checker in TestReadWhileWriting. (szetszwo) - - HDFS-823. CheckPointer should use addInternalServlet for image-fetching - servlet (jghoman) - - HDFS-456. Fix URI generation for windows file paths. (shv) - - HDFS-812. FSNamesystem#internalReleaseLease throws NullPointerException on - a single-block file's lease recovery. (cos) - - HDFS-724. Pipeline hangs if one of the block receiver is not responsive. - (hairong) - - HDFS-564. Adding pipeline tests 17-35. (hairong) - - HDFS-849. TestFiDataTransferProtocol2#pipeline_Fi_18 sometimes fails. - (hairong) - - HDFS-762. Balancer causes Null Pointer Exception. - (Cristian Ivascu via dhruba) - - HDFS-868. Fix link to Hadoop Upgrade Wiki. (Chris A. Mattmann via shv) - - HDFS-880. TestNNLeaseRecovery fails on windows (cos, shv) - - HDFS-699. Primary datanode should compare replicas' on disk lengths. - (hairong) - - HDFS-897. Fix a bug related to generation stamp comparison in - ReplicasMap. (suresh) - - HDFS-793. Data node should receive the whole packet ack message before it - constructs and sends its own ack message for the packet. (hairong) - - HDFS-101. DFS write pipeline: DFSClient sometimes does not detect second - datanode failure. (hairong) - - HDFS-822. Appends to already-finalized blocks can rename across volumes. - (hairong) - - HDFS-1046. Fix Tomcat version in hdfsproxy/build.xml. (Srikanth - Sundarrajan via szetszwo) - - HDFS-1072. Fix TestReadWhileWriting failure. (Erik Steffl via shv) - - HDFS-913. Rename fault injection test TestRename.java to TestFiRename.java - to include it in tests run by ant target run-test-hdfs-fault-inject. - (suresh) - - HDFS-695. RaidNode should read in configuration from hdfs-site.xml. - (dhruba) - - HDFS-726. Eclipse .classpath template has outdated jar files and is - missing some new ones. (cos) - - HDFS-750. Fix build failure due to TestRename. (suresh) - - HDFS-712. Move libhdfs from mapreduce subproject to hdfs subproject. - (Eli Collins via dhruba) - - HDFS-757. Enable Unit test for HDFS Raid. (dhruba) - - HDFS-611. Prevent DataNode heartbeat times from increasing even when - the DataNode has many blocks to delete. (Zheng Shao via dhruba) - - HDFS-751. Fix TestCrcCorruption to pick up the correct datablocks to - corrupt. (dhruba) - - HDFS-763. Fix slightly misleading report from DataBlockScanner - about corrupted scans. (dhruba) - - HDFS-727. bug setting block size hdfsOpenFile (Eli Collins via cos) - - HDFS-756. libhdfs unit tests do not run. (Eli Collins via cos) - - HDFS-783. libhdfs tests brakes code coverage runs with Clover (cos) - - HDFS-785. Add Apache license to several namenode unit tests. - (Ravi Phulari via jghoman) - - HDFS-802. Update Eclipse configuration to match changes to Ivy - configuration (Edwin Chan via cos) - - HDFS-423. Unbreak FUSE build and fuse_dfs_wrapper.sh (Eli Collins via cos) - - HDFS-825. Build fails to pull latest hadoop-core-* artifacts (cos) - - HDFS-94. The Heap Size printed in the NameNode WebUI is accurate. - (Dmytro Molkov via dhruba) - - HDFS-767. An improved retry policy when the DFSClient is unable to fetch a - block from the datanode. (Ning Zhang via dhruba) - - HDFS-775. FSDataset calls getCapacity() twice. (stevel) - - HDFS-885. Datanode toString() NPEs on null dnRegistration. (stevel) - - HDFS-877. Client-driven block verification not functioning. (Todd - Lipcon via hairong) - - HDFS-630. In DFSOutputStream.nextBlockOutputStream(), the client can - exclude specific datanodes when locating the next block. - (Cosmin Lehene via Stack) - - HDFS-922. Remove unnecessary semicolon added by HDFS-877 that causes - problems for Eclipse compilation. (jghoman) - - HDFS-927 DFSInputStream retries too many times for new block locations - (Todd Lipcon via Stack) - - HDFS-938. Replace calls to UGI.getUserName() with UGI.getShortUserName() - (jghoman) - - HDFS-894. DatanodeID.ipcPort is not updated when existing node - re-registers. (Todd Lipcon via tomwhite) - - HDFS-965. Split TestDelegationToken in to two parts and fix configuration - to allow proxy users in the test. (Jitendra Pandey via omalley) - - HDFS-999. Secondary namenode should login using kerberos if security is - configured (boryas) - - HDFS-856. Hardcoded replication level for new files in fuse-dfs. - (Brian Bockelman via tomwhite) - - HDFS-857. Incorrect type for fuse-dfs capacity can cause "df" to return - negative values on 32-bit machines. (Brian Bockelman via tomwhite) - - HDFS-858. Incorrect return codes for fuse-dfs. (Brian Bockelman via - tomwhite) - - HDFS-859. fuse-dfs utime behavior causes issues with tar. - (Brian Bockelman via tomwhite) - - HDFS-861. fuse-dfs does not support O_RDWR. (Brian Bockelman via tomwhite) - - HDFS-961. dfs_readdir incorrectly parses paths. (Eli Collins via tomwhite) - - HDFS-1015. Fix intermittent failure in TestSecurityTokenEditLog. - (Jitendra Nath Pandey via suresh) - - HDFS-939. libhdfs test is broken. (Eli Collins via tomwhite) - - HDFS-1074. hdfsproxy: Fix bugs in TestProxyUtil. (Srikanth Sundarrajan - via szetszwo) - - HDFS-481. hdfsproxy: Bug Fixes + HdfsProxy to use proxy user to - impresonate the real user. (Srikanth Sundarrajan via szetszwo) - - HDFS-482. Move HsftpFileSystem's ssl.client.do.not.authenticate.server - configuration setting to ssl-client.xml. (Srikanth Sundarrajan via - szetszwo) - - HDFS-1010. hdfsproxy: Retrieve groups from UnixUserGroupInformation - instead of LdapEntry. (Srikanth Sundarrajan via szetszwo) - - HDFS-466. hdfs_write infinite loop when dfs fails and cannot write - files > 2 GB. (Pete Wyckoff via tomwhite) - - HDFS-651. HDFS Docs - fix listing of docs in the doc menu. - (Corinne Chandel via tomwhite) - - HDFS-1014. Error in reading delegation tokens from edit logs. - (Jitendra Nath Pandey via jhoman) - - HDFS-1088. Prevent renaming a symbolik link to its target. - (Eli Collins via suresh) - - HDFS-966. NameNode does not recovers lease when it is in safemode. - (dhruba) - - HDFS-833. Datanode shutdown should log problems with Storage.unlockAll() - (Steve Loughran via dhruba) - - HDFS-1101. TestDiskError.testLocalDirs() fails. (cdouglas via jghoman) - - HDFS-1031. Enhance the webUi to list a few of the corrupted files in HDFS. - (Andre Orian via dhruba) - - HDFS-1078. Create static and dynamic versions of libhdfs. - (Sam Rash via dhruba) - - HDFS-1104. Fsck triggers full GC on NameNode. (hairong) - - HDFS-1141. Closing a file is successful only if the client still has a - valid lease. (Todd Lipcon via dhruba) - - HDFS-1138. Prevent erroneous updation of modification time of a directory - when fsimage loads. (Dmytro Molkov via dhruba) - - HDFS-1000. Updates libhdfs to the new API for UGI (ddas) - - HDFS-609. Create a file with the append flag does not work in HDFS. - (tomwhite) - - HDFS-1255. Fix failing test-libhdfs.sh test. (tomwhite) - - HDFS-1256. libhdfs is missing from the tarball. (tomwhite) - - HDFS-1057. Concurrent readers hit ChecksumExceptions if following a - writer to very end of file. (sam rash via hairong) - - HDFS-1212. Harmonize HDFS JAR library versions with Common. (tomwhite) - - HDFS-1159. clean-cache target removes wrong ivy cache (cos) - - HDFS-1193. -mvn-system-deploy target is broken which inturn fails the - mvn-deploy task leading to unstable mapreduce build (Giridharan - Kesavan via cos) - - HDFS-1299. 'compile-fault-inject' never should be called directly. (cos) - - HDFS-1311. Running tests with 'testcase' cause triple execution of the - same test case (Cos) - - HDFS-1267. fuse-dfs does not compile. (Devaraj Das via tomwhite) - - HDFS-1598. Directory listing on hftp:// does not show .*.crc files. - (szetszwo) - - HDFS-1750. ListPathsServlet should not use HdfsFileStatus.getLocalName() - to get file name since it may return an empty string. (szetszwo) - -Release 0.20.3 - Unreleased - - IMPROVEMENTS - - BUG FIXES - - HDFS-1041. DFSClient.getFileChecksum(..) should retry if connection to - the first datanode fails. (szetszwo) - - HDFS-909. Wait until edits syncing is finishes before purging edits. - (Todd Lipcon via shv) - - HDFS-1258. Clearing namespace quota on "/" corrupts fs image. - (Aaron T. Myers via szetszwo) - - HDFS-1406. TestCLI fails on Ubuntu with default /etc/hosts. (cos) - -Release 0.20.203.0 - 2011-5-11 - - IMPROVEMENTS - - HADOOP-7259. Contrib modules should include the build.properties from - the enclosing hadoop directory. (omalley) - - BUG FIXES - - HDFS-132. Fix namenode to not report files deleted metrics for deletions - done while replaying edits during startup. (suresh & shv) - - HDFS-955. New implementation of saveNamespace() to avoid loss of edits - when name-node fails during saving. (shv) - -Release 0.20.2 - 2009-09-01 - - IMPROVEMENTS - - HDFS-737. Add full path name of the file to the block information and - summary of total number of files, blocks, live and deadnodes to - metasave output. (Jitendra Nath Pandey via suresh) - - HDFS-919. Create test to validate the BlocksVerified metric (Gary Murry - via cos) - - HDFS-907. Add tests for getBlockLocations and totalLoad metrics. - (Ravi Phulari via cos) - - BUG FIXES - - HDFS-686. NullPointerException is thrown while merging edit log and image. - (hairong) - - HDFS-677. Rename failure when both source and destination quota exceeds - results in deletion of source. (suresh) - - HDFS-709. Fix TestDFSShell failure due to rename bug introduced by - HDFS-677. (suresh) - - HDFS-579. Fix DfsTask to follow the semantics of 0.19, regarding non-zero - return values as failures. (Christian Kunz via cdouglas) - - HDFS-723. Fix deadlock in DFSClient#DFSOutputStream. (hairong) - - HDFS-596. Fix memory leak in hdfsFreeFileInfo() for libhdfs. - (Zhang Bingjun via dhruba) - - HDFS-185. Disallow chown, chgrp, chmod, setQuota, and setSpaceQuota when - name-node is in safemode. (Ravi Phulari via shv) - - HDFS-187. Initialize secondary namenode http address in TestStartup. - (Todd Lipcon via szetszwo) - - HDFS-464. Fix memory leaks in libhdfs. (Christian Kunz via suresh) - - HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli) - -Release 0.20.1 - 2009-09-01 - - IMPROVEMENTS - - HDFS-438. Improve help message for space quota command. (Raghu Angadi) - - BUG FIXES - - HDFS-167. Fix a bug in DFSClient that caused infinite retries on write. - (Bill Zeller via szetszwo) - - HDFS-527. Remove/deprecate unnecessary DFSClient constructors. (szetszwo) - - HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread - safe. (Suresh Srinivas and cdouglas) - - HDFS-761. Fix failure to process rename operation from edits log due to - quota verification. (suresh) diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt deleted file mode 100644 index 89390077396..00000000000 --- a/hadoop-mapreduce-project/CHANGES.txt +++ /dev/null @@ -1,8906 +0,0 @@ -Hadoop MapReduce Change Log - -Trunk (Unreleased) - - INCOMPATIBLE CHANGES - - MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically. - (Gera Shegalov and Karthik Kambatla via gera) - - MAPREDUCE-5653. DistCp does not honour config-overrides for - mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw) - - MAPREDUCE-4424. 'mapred job -list' command should show the job name - as well. (Avinash Kujur via aajisaka) - - MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. - (Siqi Li via gera) - - MAPREDUCE-6613. Change mapreduce.jobhistory.jhist.format default from - json to binary (rchiang via rkanter) - - NEW FEATURES - - MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk) - - MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation. - (Plamen Jeliazkov via shv) - - MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions - with poor implementations of Object#hashCode(). (Radim Kolar via cutting) - - MAPREDUCE-5232. Add a configuration to be able to log classpath and other - system properties on mapreduce JVMs startup. (Sangjin Lee via vinodkv) - - MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving - RM-restart. (Rohith via jianhe) - - MAPREDUCE-2841. Add a native implementation of MapOutputCollector. - (see section below for detailed breakdown) - - IMPROVEMENTS - - MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk) - - MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the - job-history related information. - - MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. - (Vinay Thota via amarrk) - - MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests. - (Vinay Thota via amarrk) - - MAPREDUCE-2836. Provide option to fail jobs when submitted to non-existent - fair scheduler pools. (Ahmed Radwan via todd) - - MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs - native. (tucu) - - MAPREDUCE-3149. Add a test to verify that TokenCache handles file system - uri with no authority. (John George via jitendra) - - MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides - client APIs cross MR1 and MR2 (Ahmed via tucu) - - MAPREDUCE-2944. Improve checking of input for JobClient.displayTasks() - (XieXianshan via harsh) - - MAPREDUCE-3956. Remove the use of the deprecated Syncable.sync() method from - TeraOutputFormat in the terasort example. (szetszwo) - - MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public. - (tomwhite) - - HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia) - - MAPREDUCE-3302. Remove the last dependency call from - org.apache.hadoop.record package in MR. (harsh) - - MAPREDUCE-2384. The job submitter should make sure to validate - jobs before creation of necessary files. (harsh) - - MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG - (madhukara phatak via bobby) - - MAPREDUCE-4686. hadoop-mapreduce-client-core fails compilation in Eclipse - due to missing Avro-generated classes (Chris Nauroth via harsh) - - MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive. - (Brandon Li via suresh) - - MAPREDUCE-5197. Add a service for checkpointing task state. - (Carlo Curino via cdouglas) - - MAPREDUCE-5189. Add policies and wiring to respond to preemption requests - from YARN. (Carlo Curino via cdouglas) - - MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state. - (Carlo Curino via cdouglas) - - MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after - MAPREDUCE-5196. (Remus Rusanu via cnauroth) - - MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple - interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong) - - MAPREDUCE-6013. [post-HADOOP-9902] mapred version is missing (Akira AJISAKA - via aw) - - MAPREDUCE-6250. deprecate sbin/mr-jobhistory-daemon.sh (aw) - - MAPREDUCE-6260. Convert site documentation to markdown (Masatake Iwasaki - via aw) - - MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml - (Ray Chiang via aw) - - MAPREDUCE-2632. Avoid calling the partitioner when the numReduceTasks is 1. - (Ravi Teja Ch N V and Sunil G via kasha) - - MAPREDUCE-6407. Migrate MAPREDUCE nativetask build to new CMake framework - (Alan Burlison via Colin P. McCabe) - - MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API - (Junping Du via jianhe) - - BUG FIXES - - MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob. - (Jinghui Wang via Eric Yang) - - MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2 - (Jason Lowe via bobby) - - MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. - (Ravi Prakash and Ravi Gummadi via amarrk) - - MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. - (Devaraj K and Amar Kamat via amarrk) - - MAPREDUCE-3412. Fix 'ant docs'. (amarrk) - - MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null. - (amarrk) - - MAPREDUCE-2950. [Gridmix] TestUserResolve fails in trunk. - (Ravi Gummadi via amarrk) - - MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and - ResourceUsageMatcher. (amarrk) - - MAPREDUCE-2978. Fixed test-patch to make Jenkins report correct number of - findBugs, correct links to findBugs artifacts and no links to the - artifacts when there are no warnings. (Tom White via vinodkv). - - MAPREDUCE-3664. Federation Documentation has incorrect configuration example. - (Brandon Li via jitendra) - - MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are - variable depth (ahmed via tucu) [IMPORTANT: this is dead code in trunk] - - MAPREDUCE-3990. MRBench allows Long-sized input-lines value - but parses CLI argument as an Integer. (harsh) - - MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen) - - MAPREDUCE-4685. DBCount should not use ACCESS. (Viji via harsh) - - MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml (tlipcon via harsh) - - MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it - (harsh) - - MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes - (Joe Mudd via aw) - - MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with - SerialUtils.hh (Simone Leo via aw) - - MAPREDUCE-4574. Fix TotalOrderParitioner to work with - non-WritableComparable key types. (harsh) - - MAPREDUCE-5012. Typo in javadoc for IdentityMapper class. (Adam Monsen - via suresh) - - MAPREDUCE-4987. TestMRJobs#testDistributedCache fails on Windows due to - classpath problems and unexpected behavior of symlinks (Chris Nauroth via - bikas) - - MAPREDUCE-5191. TestQueue#testQueue fails with timeout on Windows. (Ivan - Mitic via hitesh) - - MAPREDUCE-5717. Task pings are interpreted as task progress (jlowe) - - MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to - ProportionalCapacityPreemptionPolicy (Sunil G via devaraj) - - MAPREDUCE-5972. Fix typo 'programatically' in job.xml (and a few other - places) (Akira AJISAKA via aw) - - MAPREDUCE-6161. mapred hsadmin command missing from trunk (Allen Wittenauer - via jlowe) - - MAPREDUCE-4413. MR lib dir contains jdiff (which is gpl) (Nemon Lou via aw) - - MAPREDUCE-6234. TestHighRamJob fails due to the change in MAPREDUCE-5785. - (Masatake Iwasaki via kasha) - - MAPREDUCE-6343. JobConf.parseMaximumHeapSizeMB() fails to parse value - greater than 2GB expressed in bytes. (Hao Xia via kasha) - - MAPREDUCE-6396. TestPipeApplication fails by NullPointerException. - (Brahma Reddy Battula via aajisaka) - - MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT - to match mapred-default.xml. (Ray Chiang via devaraj) - - MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw) - - MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME - (Alan Burlison via aw) - - MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean - (Alan Burlison via aw) - - MAPREDUCE-6416. Not all platforms have d_type in struct dirent - (Alan Burlison via aw) - - MAPREDUCE-6435. MapReduce client assumes the world is x86 - (Alan Burlison via aw) - - MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee) - - MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa) - - BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS - - MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by - Binglin Chang - - MAPREDUCE-5994. Simplify ByteUtils and fix failing test. (todd) - - MAPREDUCE-5996. native-task: Rename system tests into standard directory - layout (todd) - - MAPREDUCE-5997. native-task: Use DirectBufferPool from Hadoop Common (todd) - - MAPREDUCE-6000. native-task: Simplify ByteBufferDataReader/Writer (todd) - - MAPREDUCE-5991. native-task should not run unit tests if native profile is - not enabled. (Binglin Chang) - - MAPREDUCE-5995. native-task: Revert changes to Text internals (todd) - - MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang) - - MAPREDUCE-5984. native-task: Reuse lz4 sources in hadoop-common (Binglin - Chang) - - MAPREDUCE-5976. native-task: should not fail to build if snappy is missing - (Manu Zhang) - - MAPREDUCE-5978. native-task: remove test case for not supported codec - Bzip2Codec and DefaultCodec (Manu Zhang) - - MAPREDUCE-6006. native-task: add native tests to maven and fix bug in - pom.xml (Binglin Chang via todd) - - MAPREDUCE-6026. native-task: fix logging (Manu Zhang via todd) - - MAPREDUCE-6035. native-task: sources/test-sources jar distribution (Manu - Zhang via todd) - - MAPREDUCE-5977. Fix or suppress native-task gcc warnings (Manu Zhang via - todd) - - MAPREDUCE-6054. native-task: Speed up tests (todd) - - MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is - sucessful (Binglin Chang) - - MAPREDUCE-6056. native-task: move system test working dir to target dir and - cleanup test config xml files (Manu Zhang via bchang) - - MAPREDUCE-6055. native-task: findbugs, interface annotations, and other misc - cleanup (todd) - - MAPREDUCE-6067. native-task: fix some counter issues (Binglin Chang) - - MAPREDUCE-6069. native-task: Lint/style fixes and removal of unused code - (todd) - - MAPREDUCE-6074. native-task: fix release audit, javadoc, javac warnings - (todd) - - MAPREDUCE-6077. native-task: Remove CustomModule examples in nativetask (seanzhong) - - MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang) - - MAPREDUCE-5801. Uber mode's log message is missing a vcore reason - (Steven Wong via aw) - - MAPREDUCE-6525. Fix test failure of TestMiniMRClientCluster.testRestart. - (Masatake Iwasaki via aajisaka) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-6431. JobClient should be an AutoClosable (haibochen via rkanter) - - MAPREDUCE-6634. Log uncaught exceptions/errors in various thread pools in - mapreduce. (Sidharta Seethana via vvasudev) - - MAPREDUCE-6627. Add machine-readable output to mapred job -history - command (rkanter) - - MAPREDUCE-6640. mapred job -history command should be able to take - Job ID (rkanter) - - MAPREDUCE-6622. Add capability to set JHS job cache to a - task-based limit (rchiang via rkanter) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6620. Jobs that did not start are shown as starting in 1969 in - the JHS web UI (haibochen via rkanter) - - MAPREDUCE-4785. TestMRApp occasionally fails (haibochen via rkanter) - -Release 2.8.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application - Master REST API. (Ryu Kobayashi via ozawa) - - MAPREDUCE-6364. Add a "Kill" link to Task Attempts page. (Ryu Kobayashi - via ozawa) - - MAPREDUCE-6304. Specifying node labels when submitting MR jobs. - (Naganarasimha G R via wangda) - - MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. - (Robert Kanter via kasha) - - IMPROVEMENTS - - MAPREDUCE-6471. Document distcp incremental copy - (Neelesh Srinivas Salian via harsh) - - MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused - (Neelesh Srinivas Salian via harsh) - - MAPREDUCE-6291. Correct mapred queue usage command. - (Brahma Reddu Battula via harsh) - - MAPREDUCE-579. Streaming "slowmatch" documentation. (harsh) - - MAPREDUCE-6287. Deprecated methods in org.apache.hadoop.examples.Sort - (Chao Zhang via harsh) - - MAPREDUCE-5190. Unnecessary condition test in RandomSampler. - (Jingguo Yao via harsh) - - MAPREDUCE-6239. Consolidate TestJobConf classes in - hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core - (Varun Saxena via harsh) - - MAPREDUCE-5807. Print usage by TeraSort job. (Rohith via harsh) - - MAPREDUCE-4653. TestRandomAlgorithm has an unused "import" statement. - (Amir Sanjar via harsh) - - MAPREDUCE-6100. replace "mapreduce.job.credentials.binary" with - MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY for better readability. - (Zhihai Xu via harsh) - - MAPREDUCE-6105. Inconsistent configuration in property - mapreduce.reduce.shuffle.merge.percent. (Ray Chiang via harsh) - - MAPREDUCE-4414. Add main methods to JobConf and YarnConfiguration, - for debug purposes. (Plamen Jeliazkov via harsh) - - MAPREDUCE-5755. MapTask.MapOutputBuffer#compare/swap should have - @Override annotation. (ozawa) - - MAPREDUCE-6282. Reuse historyFileAbsolute.getFileSystem in - CompletedJob#loadFullHistoryData for code optimization. - (zxu via rkanter) - - MAPREDUCE-6307. Remove property - mapreduce.tasktracker.taskmemorymanager.monitoringinterval. - (J.Andreina via aajisaka) - - MAPREDUCE-6192. Create unit test to automatically compare - MR related classes and mapred-default.xml (rchiang via rkanter) - - MAPREDUCE-6279. AM should explicity exit JVM after all services have - stopped (Eric Payne via jlowe) - - MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. - (Tsuyoshi Ozawa via aajisaka) - - MAPREDUCE-5981. Log levels of certain MR logs can be changed to DEBUG. - (Varun Saxena via devaraj) - - MAPREDUCE-5248. Let NNBenchWithoutMR specify the replication factor for - its test (Erik Paulson via jlowe) - - MAPREDUCE-6174. Combine common stream code into parent class for - InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) - - MAPREDUCE-6383. Pi job (QuasiMonteCarlo) should not try to read the - results file if its job fails. (Harsh J via devaraj) - - MAPREDUCE-6354. ShuffleHandler should be able to log shuffle connections - (Chang Li via jlowe) - - MAPREDUCE-6392. Document mapred class path options. - (Brahma Reddy Battula via cnauroth) - - MAPREDUCE-6395. Improve the commit failure messages in MRAppMaster recovery. - (Brahma Reddy Battula via gera) - - MAPREDUCE-6316. Task Attempt List entries should link to the task overview. - (Siqi Li via gera) - - MAPREDUCE-6305. AM/Task log page should be able to link back to the job. - (Siqi Li via gera) - - MAPREDUCE-6408. Queue name and user name should be printed on the job page. - (Siqi Li via gera) - - MAPREDUCE-6384. Add the last reporting reducer info for too many fetch - failure diagnostics (Chang Li via jlowe) - - MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. (Robert Kanter - via junping_du) - - MAPREDUCE-6477. Replace usage of deprecated NameNode.DEFAULT_PORT in - TestFileSystem. (Mingliang Liu via wheat9) - - MAPREDUCE-6483. Replace deprecated method NameNode.getUri() with - DFSUtilClient.getNNUri() in TestMRCredentials. (Mingliang Liu via wheat9) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - MAPREDUCE-6479. Add missing mapred job command options in mapreduce - document. (nijel via aajisaka) - - MAPREDUCE-6489. Fail fast rogue tasks that write too much to local disk - (Maysam Yabandeh via jlowe) - - MAPREDUCE-5763. Warn message about httpshuffle in NM logs. - (Akira AJISAKA via ozawa) - - MAPREDUCE-6499. Add elapsed time for retired job in JobHistoryServer WebUI. - (Lin Yiqun via aajisaka) - - MAPREDUCE-5870. Support for passing Job priority through Application - Submission Context in Mapreduce Side (Sunil G via jlowe) - - MAPREDUCE-6566. Add retry support to mapreduce CLI tool. - (Varun Vasudev via xgong) - - MAPREDUCE-6574. MR AM should print host of failed tasks. - (Mohammad Shahid Khan via wangda) - - MAPREDUCE-6584. Remove trailing whitespaces from mapred-default.xml. - (aajisaka) - - MAPREDUCE-6595. Fix findbugs warnings in OutputCommitter and - FileOutputCommitter. (Akira AJISAKA via junping_du) - - OPTIMIZATIONS - - MAPREDUCE-6376. Add avro binary support for jhist files (Ray Chiang via - jlowe) - - MAPREDUCE-6394. Speed up Task processing loop in HsTasksBlock#render() - (Ray Chiang via jlowe) - - MAPREDUCE-6473. Job submission can take a long time during Cluster - initialization (Kuhu Shukla via jlowe) - - MAPREDUCE-6626. Reuse ObjectMapper instance in MapReduce. - (Lin Yiqun via aajisaka) - - BUG FIXES - - MAPREDUCE-6314. TestPipeApplication fails on trunk. - (Varun Vasudev via harsh) - - MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with hidden - files/folders. (Maysam Yabandeh via harsh) - - MAPREDUCE-6213. NullPointerException caused by job history server addr not - resolvable. (Peng Zhang via harsh) - - MAPREDUCE-6281. Fix javadoc in Terasort. (Albert Chu via ozawa) - - MAPREDUCE-6242. Progress report log is incredibly excessive in - application master. (Varun Saxena via devaraj) - - MAPREDUCE-6294. Remove an extra parameter described in Javadoc of - TockenCache. (Brahma Reddy Battula via ozawa) - - MAPREDUCE-5875. Make Counter limits consistent across JobClient, - MRAppMaster, and YarnChild. (Gera Shegalov via kasha) - - MAPREDUCE-6199. AbstractCounters are not reset completely on - deserialization (adhoot via rkanter) - - MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which - causes counter limits are not reset correctly. - (Zhihai Xu via harsh) - - MAPREDUCE-4844. Counters / AbstractCounters have constant references not - declared final. (Brahma Reddy Battula via gera) - - MAPREDUCE-6266. Job#getTrackingURL should consistently return a proper URL - (rchiang via rkanter) - - MAPREDUCE-6293. Set job classloader on uber-job's LocalContainerLauncher - event thread. (Sangjin Lee via gera) - - MAPREDUCE-6297. Task Id of the failed task in diagnostics should link to - the task page. (Siqi Li via gera) - - MAPREDUCE-6330. Fix typo in Task Attempt API's URL in documentations. - (Ryu Kobayashi via ozawa) - - MAPREDUCE-6333. TestEvents,TestAMWebServicesTasks,TestAppController are - broken due to MAPREDUCE-6297. (Siqi Li via gera) - - MAPREDUCE-6341. Fix typo in mapreduce tutorial. (John Michael Luy - via ozawa) - - MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster - logs. (Rohit Agarwal via gera) - - MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs - unnecessary "null" strings. (Akira AJISAKA via ozawa) - - MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce. - lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. (Ray Chiang via ozawa) - - MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. - (Akira AJISAKA via ozawa) - - MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - MAPREDUCE-6342. Make POM project names consistent. (Rohith via devaraj) - - MAPREDUCE-3383. Duplicate job.getOutputValueGroupingComparator() in - ReduceTask (Binglin Chang via jlowe) - - MAPREDUCE-4750. Enable NNBenchWithoutMR in MapredTestDriver (Liang Xie - and Jason Lowe via raviprak) - - MAPREDUCE-6359. In RM HA setup, "Cluster" tab links populated with AM - hostname instead of RM. (zhaoyunjiong via junping_du) - - MAPREDUCE-6353. Divide by zero error in MR AM when calculating available - containers. (Anubhav Dhoot via kasha) - - MAPREDUCE-5465. Tasks are often killed before they exit on their own - (Ming Ma via jlowe) - - MAPREDUCE-6360. TestMapreduceConfigFields is placed in wrong dir, - introducing compile error (Arshad Mohammad via vinayakumarb) - - MAPREDUCE-6366. mapreduce.terasort.final.sync configuration in TeraSort - doesn't work. (Takuya Fukudome via ozawa) - - MAPREDUCE-5708. Duplicate String.format in YarnOutputFiles.getSpillFileForWrite. - (Konstantin Weitz via devaraj) - - MAPREDUCE-6204. TestJobCounters should use new properties instead of - JobConf.MAPRED_TASK_JAVA_OPTS. (Sam Liu via ozawa) - - MAPREDUCE-6374. Distributed Cache File visibility should check permission - of full path (Chang Li via jlowe) - - MAPREDUCE-5965. Hadoop streaming throws error if list of input files is - high. Error is: "error=7, Argument list too long at if number of input - file is high" (wilfreds via rkanter) - - MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview. - (Siqi Li via gera) - - MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes - (Ray Chiang via ozawa). - - MAPREDUCE-6350. JobHistory doesn't support fully-functional search. - (Siqi Li via devaraj) - - MAPREDUCE-6389. Fix BaileyBorweinPlouffe CLI usage message. (Brahma Reddy Battula - via devaraj) - - MAPREDUCE-6373. The logger reports total input paths but it is referring - to input files. (Bibin A Chundatt via devaraj) - - MAPREDUCE-6405. NullPointerException in App Attempts page. - (Siqi Li and Gera Shegalov via devaraj) - - MAPREDUCE-6403. Fix typo in the usage of NNBench. - (Jagadesh Kiran N via aajisaka) - - MAPREDUCE-6400. Multiple shuffle transfer fails because input is closed - too early (Brahma Reddy Battula, Akira AJISAKA, and Gera Shegalov via - jlowe) - - MAPREDUCE-6413. TestLocalJobSubmission is failing with unknown host - (zhihai xu via jlowe) - - MAPREDUCE-6121. JobResourceUpdater#compareFs() doesn't handle HA namespaces - (rchiang via rkanter) - - MAPREDUCE-6420. Interrupted Exception in LocalContainerLauncher should be - logged in warn/info level (Chang Li via jlowe) - - MAPREDUCE-6418. MRApp should not shutdown LogManager during shutdown - (Chang Li via jlowe) - - MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in - MapReduce Tutorial. (Tsuyoshi Ozawa via cdouglas) - - MAPREDUCE-6421. Fix findbugs warning in - RMContainerAllocator.reduceNodeLabelExpression. - (Brahma Reddy Battula via aajisaka) - - MAPREDUCE-6427. Fix typo in JobHistoryEventHandler. (Ray Chiang via cdouglas) - - MAPREDUCE-6433. launchTime may be negative. (Zhihai Xu) - - MAPREDUCE-5817. Mappers get rescheduled on node transition even after all - reducers are completed. (Sangjin Lee via kasha) - - MAPREDUCE-6357. MultipleOutputs.write() API should document that output - committing is not utilized when input path is absolute. - (Dustin Cote via aajisaka) - - MAPREDUCE-6357. MultipleOutputs.write() API should document that output - committing is not utilized when input path is absolute. - (Dustin Cote via aajisaka) - - MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. - (Zhihai Xu) - - MAPREDUCE-5002. AM could potentially allocate a reduce container to a map - attempt (Chang Li via jlowe) - - MAPREDUCE-6460. TestRMContainerAllocator. - testAttemptNotFoundCausesRMCommunicatorException fails. (Zhihai Xu) - - MAPREDUCE-6484. Yarn Client uses local address instead of RM address as - token renewer in a secure cluster when RM HA is enabled. (Zhihai Xu) - - MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) - - MAPREDUCE-6494. Permission issue when running archive-logs tool as - different users (rkanter) - - MAPREDUCE-6485. Create a new task attempt with failed map task priority - if in-progress attempts are unassigned. (Xianyin Xin via rohithsharmaks) - - MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead - of HADOOP_HOME (rkanter) - - MAPREDUCE-6302. Preempt reducers after a configurable timeout irrespective - of headroom. (kasha) - - MAPREDUCE-6495. Docs for archive-logs tool (rkanter) - - MAPREDUCE-6508. TestNetworkedJob fails consistently due to delegation - token changes on RM. (Akira AJISAKA via junping_du) - - MAPREDUCE-6515. Update Application priority in AM side from AM-RM heartbeat - (Sunil G via jlowe) - - MAPREDUCE-6533. testDetermineCacheVisibilities of - TestClientDistributedCacheManager is broken (Chang Li via jlowe) - - MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration. - (Gabor Liptak via aajisaka) - - MAPREDUCE-6557. Tests in mapreduce-client-app are writing outside of - target. (Akira AJISAKA via junping_du) - - MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn - user when using DefaultContainerExecutor (rkanter) - - MAPREDUCE-6419. JobHistoryServer doesn't sort properly based on Job ID when Job id's - exceed 9999. (Mohammad Shahid Khan via rohithsharmaks) - - MAPREDUCE-6589. TestTaskLog outputs a log under directory other than - target/test-dir. (aajisaka) - - MAPREDUCE-6593. TestJobHistoryEventHandler.testTimelineEventHandling fails - on trunk because of NPE. (Naganarasimha G R via aajisaka) - - MAPREDUCE-6068. Illegal progress value warnings in map tasks. (Binglin Chang - via junping_du) - - MAPREDUCE-6601. Fix typo in Job#setUseNewAPI. (Kai Sasaki via aajisaka) - - MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and - mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. - (Kai Sasaki via aajisaka) - - MAPREDUCE-6614. Remove unnecessary code in TestMapreduceConfigFields. - (Kai Sasaki via aajisaka) - - MAPREDUCE-6610. JobHistoryEventHandler should not swallow timeline response - (Li Lu via jianhe) - - MAPREDUCE-6563. Streaming documentation contains a stray '%' character. - (cnauroth) - - MAPREDUCE-6616. Fail to create jobhistory file if there are some multibyte - characters in the job name. (Kousuke Saruta via aajisaka) - - MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader - and IndexOutOfBoundsException. (Junping Du via vvasudev) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-6637. Testcase Failure : TestFileInputFormat.testSplitLocationInfo. - (Brahma Reddy Battula via wang) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6377. JHS sorting on state column not working in webUi. - (zhihai xu via devaraj) - - MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly - misleading (Nathan Roberts via jlowe) - - MAPREDUCE-6583. Clarify confusing sentence in MapReduce tutorial document. - (Kai Sasaki via aajisaka) - - MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in - MRAppMaster#parsePreviousJobHistory (Bibin A Chundatt via jlowe) - - MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient thread. - (Xuan Gong via jlowe) - - MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal() (Xuan Gong - via jlowe) - - MAPREDUCE-6413. TestLocalJobSubmission is failing with unknown host - (zhihai xu via jlowe) - - MAPREDUCE-6191. Improve clearing stale state of Java serialization - testcase. (Sam Liu via Eric Yang) - - MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader - and IndexOutOfBoundsException. (Junping Du via vvasudev) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6425. ShuffleHandler passes wrong "base" parameter to getMapOutputInfo - if mapId is not in the cache. (zhihai xu via devaraj) - - MAPREDUCE-6426. TestShuffleHandler#testGetMapOutputInfo is failing. - (zhihai xu via devaraj) - - MAPREDUCE-6439. AM may fail instead of retrying if RM shuts down during the - allocate call. (Anubhav Dhoot via kasha) - - MAPREDUCE-6442. Stack trace is missing when error occurs in client protocol - provider's constructor (Chang Li via ozawa) - - MAPREDUCE-6474. ShuffleHandler can possibly exhaust nodemanager file - descriptors (Kuhu Shukla via jlowe) - - MAPREDUCE-6472. MapReduce AM should have java.io.tmpdir=./tmp to be - consistent with tasks (Naganarasimha G R via jlowe) - - MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle - multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira - AJISAKA via jlowe) - - MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong - position/key information for uncompressed input sometimes. (Zhihai Xu via - jlowe) - - MAPREDUCE-5982. Task attempts that fail from the ASSIGNED state can - disappear (Chang Li via jlowe) - - MAPREDUCE-6492. AsyncDispatcher exit with NPE on - TaskAttemptImpl#sendJHStartEventForAssignedFailTask (Bibin A Chundatt via - jlowe) - - MAPREDUCE-6518. Set SO_KEEPALIVE on shuffle connections (Chang Li via - jlowe) - - MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to - avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. - (zhihai xu via devaraj) - - MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary() - (Junping Du via jlowe) - - MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when - strategy is dynamic (Kuhu Shukla via kihwal) - -Release 2.7.1 - 2015-07-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6252. JobHistoryServer should not fail when encountering a - missing directory. (Craig Welch via devaraj) - - MAPREDUCE-6334. Fetcher#copyMapOutput is leaking usedMemory upon - IOException during InMemoryMapOutput shuffle handler (Eric Payne via jlowe) - - MAPREDUCE-6339. Job history file is not flushed correctly because isTimerActive - flag is not set true when flushTimerTask is scheduled. (zhihai xu via devaraj) - - MAPREDUCE-6259. IllegalArgumentException due to missing job submit time - (zhihai xu via jlowe) - - MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so - that they don't fail on history-server backed by DFSes with not so strong - guarantees. (Craig Welch via vinodkv) - - MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at - the end. (Arun Suresh via kasha) - - MAPREDUCE-6410. Fixed MapReduce JobHistory server to use the right (login) - UGI to refresh log and cleaner settings. (Varun Saxena via vinodkv) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-6227. DFSIO for truncate. (shv via yliu) - - MAPREDUCE-6228. Add truncate operation to SLive. (Plamen Jeliazkov via shv) - - MAPREDUCE-5583. Ability to limit running map and reduce tasks. - (Jason Lowe via junping_du) - - IMPROVEMENTS - - MAPREDUCE-6149. Document override log4j.properties in MR job. - (Junping Du via harsh) - - MAPREDUCE-6194. Bubble up final exception in failures during creation - of output collectors (Varun Saxena via harsh) - - MAPREDUCE-5420. Remove mapreduce.task.tmp.dir from mapred-default.xml - (James Carman via harsh) - - MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle - log (Gera Shegalov via jlowe) - - MAPREDUCE-6046. Change the class name for logs in RMCommunicator - (Sahil Takiar via devaraj) - - HADOOP-11032. Replace use of Guava's Stopwatch with Hadoop's StopWatch - (ozawa) - - MAPREDUCE-6173. Document the configuration of deploying MR over distributed - cache with enabling wired encryption at the same time. - (Junping Du via xgong) - - MAPREDUCE-6141. History server leveldb recovery store (jlowe) - - MAPREDUCE-6150. Update document of Rumen (Masatake Iwasaki via aw) - - MAPREDUCE-6151. Update document of GridMix (Masatake Iwasaki via aw) - - MAPREDUCE-6143. add configuration for mapreduce speculative execution in - MR2 (zxu via rkanter) - - MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors - (aajisaka) - - MAPREDUCE-6253. Update use of Iterator to Iterable. (Ray Chiang via devaraj) - - MAPREDUCE-5335. Rename Job Tracker terminology in ShuffleSchedulerImpl. - (devaraj via ozawa) - - MAPREDUCE-4431. mapred command should print the reason on killing already - completed jobs. (devaraj via ozawa) - - MAPREDUCE-6255. Fix JobCounter's format to use grouping separator. - (Ryu Kobayashi via ozawa) - - MAPREDUCE-6256. Removed unused private methods in o.a.h.mapreduce.Job.java. - (Naganarasimha G R via ozawa) - - MAPREDUCE-6225. Fix new findbug warnings in hadoop-mapreduce-client-core. - (Varun Saxena via junping_du) - - MAPREDUCE-6264. Remove httpclient dependency from hadoop-mapreduce-client. - (Brahma Reddy Battula via aajisaka) - - MAPREDUCE-5612. Add javadoc for TaskCompletionEvent.Status. - (Chris Palmer via aajisaka) - - MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public - API in DistCp. (Jing Zhao via vinodkv) - - MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the - application and forcefully kill. (Eric Payne via junping_du) - - MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable - to better control to launch/kill containers. (Zhihai Xu via ozawa) - - MAPREDUCE-6292. Use org.junit package instead of junit.framework in - TestCombineFileInputFormat. (aajisaka) - - OPTIMIZATIONS - - MAPREDUCE-6169. MergeQueue should release reference to the current item - from key and value at the end of the iteration to save memory. - (Zhihai Xu via kasha) - - MAPREDUCE-6059. Speed up history server startup time (Siqi Li via aw) - - MAPREDUCE-4815. Speed up FileOutputCommitter#commitJob for many output - files. (Siqi Li via gera) - - BUG FIXES - - MAPREDUCE-6210. Use getApplicationAttemptId() instead of getApplicationId() - for logging AttemptId in RMContainerAllocator.java (Leitao Guo via aajisaka) - - MAPREDUCE-6177. Minor typo in the EncryptedShuffle document about - ssl-client.xml (Yangping Wu via harsh) - - MAPREDUCE-5918. LineRecordReader can return the same decompressor to - CodecPool multiple times (Sergey Murylev via raviprak) - - MAPREDUCE-6162. mapred hsadmin fails on a secure cluster (jlowe) - - MAPREDUCE-6049. AM JVM does not exit if MRClientService graceful shutdown - fails (Rohith via devaraj) - - MAPREDUCE-5568. Fixed CompletedJob in JHS to show progress percentage - correctly in case the number of mappers or reducers is zero. (MinJi Kim via - jianhe) - - MAPREDUCE-6172. TestDbClasses timeouts are too aggressive (Varun Saxena - via jlowe) - - MAPREDUCE-6160. Potential NullPointerException in MRClientProtocol - interface implementation. (Rohith via jlowe) - - MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output - directory. (gera) - - MAPREDUCE-6045. need close the DataInputStream after open it in - TestMapReduce.java (zxu via rkanter) - - MAPREDUCE-6206. TestAggregatedTransferRate fails on non-US systems (Jens - Rabe via jlowe) - - MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath - (Varun Saxena via cnauroth) - - MAPREDUCE-6231. Grep example job is not working on a fully-distributed - cluster. (aajisaka) - - MAPREDUCE-6243. Fix findbugs warnings in hadoop-rumen. (Masatake Iwasaki - via aajisaka) - - MAPREDUCE-5988. Fix dead links to the javadocs in mapreduce project. - (aajisaka) - - MAPREDUCE-6186. Redundant call to requireJob() while displaying the conf - page (Rohit Agarwal via jlowe) - - MAPREDUCE-6233. org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort - failed in trunk (zxu via rkanter) - - MAPREDUCE-6221. Stringifier is left unclosed in Chain#getChainElementConf(). - (Ted Yu via ozawa) - - MAPREDUCE-4286. TestClientProtocolProviderImpls passes on failure - conditions. (Devaraj K via ozawa) - - MAPREDUCE-6261. NullPointerException if MapOutputBuffer.flush invoked - twice (Tsuyoshi OZAWA via jlowe) - - MAPREDUCE-2815. JavaDoc does not generate correctly for - MultithreadedMapRunner. (Chris Palmer via aajisaka) - - MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. - (Varun Saxena via kasha) - - MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi - via ozawa) - - MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma - Reddy Battula via ozawa) - - MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc - comments. (Akira AJISAKA and Andrew Purtell via ozawa) - - MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa) - - MAPREDUCE-6277. Job can post multiple history files if attempt loses - connection to the RM (Chang Li via jlowe) - - MAPREDUCE-6275. Race condition in FileOutputCommitter v2 for - user-specified task output subdirs (Gera Shegalov and Siqi Li via jlowe) - - MAPREDUCE-6285. ClientServiceDelegate should not retry upon - AuthenticationException. (Jonathan Eagles via ozawa) - -Release 2.6.5 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6413. TestLocalJobSubmission is failing with unknown host - (zhihai xu via jlowe) - - MAPREDUCE-6191. Improve clearing stale state of Java serialization - testcase. (Sam Liu via Eric Yang) - - MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader - and IndexOutOfBoundsException. (Junping Du via vvasudev) - -Release 2.6.4 - 2016-02-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-6436. JobHistory cache issue. (Kai Sasaki via zxu) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6577. MR AM unable to load native library without - MR_AM_ADMIN_USER_ENV set (sjlee) - - MAPREDUCE-5982. Task attempts that fail from the ASSIGNED state can - disappear (Chang Li via jlowe) - - MAPREDUCE-6492. AsyncDispatcher exit with NPE on - TaskAttemptImpl#sendJHStartEventForAssignedFailTask (Bibin A Chundatt via - jlowe) - - MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple - mappers. (Vlad Sharanhovich and Bibin A Chundatt via aajisaka) - - MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in - MRAppMaster#parsePreviousJobHistory (Bibin A Chundatt via jlowe) - - MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient thread. - (Xuan Gong via jlowe) - - MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal() (Xuan Gong - via jlowe) - -Release 2.6.3 - 2015-12-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6273. HistoryFileManager should check whether summaryFile exists to - avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state. - (zhihai xu via devaraj) - - MAPREDUCE-6528. Memory leak for HistoryFileManager.getJobSummary() - (Junping Du via jlowe) - - MAPREDUCE-6377. JHS sorting on state column not working in webUi. - (zhihai xu via devaraj) - - MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle - multibyte record delimiters well (Vinayakumar B, Rushabh Shah, and Akira - AJISAKA via jlowe) - - MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong - position/key information for uncompressed input sometimes. (Zhihai Xu via - jlowe) - - MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly - misleading (Nathan Roberts via jlowe) - - MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause - duplicate records (wilfreds via rkanter) - -Release 2.6.2 - 2015-10-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6454. Fixed MapReduce to modify HADOOP_CLASSPATH to have distributed - cache files so that child processes running hadoop scripts can access these - files. (Junping Du via vinodkv) - - MAPREDUCE-6334. Fetcher#copyMapOutput is leaking usedMemory upon - IOException during InMemoryMapOutput shuffle handler (Eric Payne via jlowe) - - MAPREDUCE-6497. Fix wrong value of JOB_FINISHED event in - JobHistoryEventHandler. (Shinichi Yamashita via aajisaka) - -Release 2.6.1 - 2015-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of - reusing conections. (Kannan Rajah via ozawa) - - MAPREDUCE-6166. Reducers do not validate checksum of map outputs when - fetching directly to disk. (Eric Payne via gera) - - MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken - service name properly. (Jason Lowe via jianhe) - - MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal - to a reducer. (Jason Lowe via junping_du) - - MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) - - MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own - class. (Chris Trezzo via kasha) - - MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options - which is a regression from MR1 (zxu via rkanter) - - MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM - tokens when they roll-over. (Jason Lowe via vinodkv) - - MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge - (Gera Shegalov via jlowe) - - MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between - copySucceeded() in one thread and copyFailed() in another thread on the - same host. (Junping Du via ozawa) - -Release 2.6.0 - 2014-11-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. - (Robert Kanter via zjshen) - - IMPROVEMENTS - - MAPREDUCE-5971. Move the default options for distcp -p to - DistCpOptionSwitch. (clamb via wang) - - MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with - compatible/incompatible changes (Junping Du via jlowe) - - MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and - Koji Noguchi via aw) - - MAPREDUCE-4791. Javadoc for KeyValueTextInputFormat should include default - separator and how to change it (Akira AJISAKA via aw) - - MAPREDUCE-5906. Inconsistent configuration in property - "mapreduce.reduce.shuffle.input.buffer.percent" (Akira AJISAKA via aw) - - MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with - fallback. (Todd Lipcon via kasha) - - MAPREDUCE-5130. Add missing job config options to mapred-default.xml - (Ray Chiang via Sandy Ryza) - - MAPREDUCE-5891. Improved shuffle error handling across NM restarts - (Junping Du via jlowe) - - MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN - scheduler resource type is memory plus cpu. (Peng Zhang and Varun Vasudev - via zjshen) - - MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw) - - MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling (Gera - Shegalov via jlowe) - - MAPREDUCE-6018. Added an MR specific config to enable emitting job history - data to the timeline server. (Robert Kanter via zjshen) - - MAPREDUCE-6052. Supported overriding the default container-log4j.properties - file per job. (Junping Du via zjshen) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5866. TestFixedLengthInputFormat fails in windows. - (Varun Vasudev via cnauroth) - - MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current - attempt is the last retry. (Wangda Tan via zjshen) - - MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader - enabled if custom output format/committer is used (Sangjin Lee via jlowe) - - MAPREDUCE-5756. CombineFileInputFormat.getSplits() including directories - in its results (Jason Dere via jlowe) - - MAPREDUCE-6014. New task status field in task attempts table can lead to - an empty web page (Mit Desai via jlowe) - - MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH - (jlowe) - - MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens - (jlowe) - - MAPREDUCE-5878. some standard JDK APIs are not part of system classes - defaults (Sangjin Lee via jlowe) - - MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm - (Akira AJISAKA via aw) - - MAPREDUCE-5943. Separate mapred commands from CommandManual.apt.vm - (Akira AJISAKA via aw) - - MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus - and getStatus (Akira AJISAKA via aw) - - MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw) - - MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors - in mapreduce.Job (Akira AJISAKA via aw) - - MAPREDUCE-5950. incorrect description in distcp2 document (Akira AJISAKA - via aw) - - MAPREDUCE-5998. CompositeInputFormat javadoc is broken (Akira AJISAKA via - aw) - - MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw) - - MAPREDUCE-6032. Made MR jobs write job history files on the default FS when - the current context's FS is different. (Benjamin Zhitomirsky via zjshen) - - MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before - concluding the failure by configuration. (Yunjiong Zhao via zjshen) - - MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2 (chang - li via jlowe) - - MAPREDUCE-6012. DBInputSplit creates invalid ranges on Oracle. - (Wei Yan via kasha) - - MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir - creation on Windows. (zjshen) - - MAPREDUCE-5885. build/test/test.mapred.spill causes release audit warnings - (Chen He via jlowe) - - BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS - - MAPREDUCE-5890. Support for encrypting Intermediate - data and spills in local filesystem. (asuresh via tucu) - - MAPREDUCE-6007. Add support to distcp to preserve raw.* namespace - extended attributes. (clamb) - - MAPREDUCE-6041. Fix TestOptionsParser. (clamb) - -- - - MAPREDUCE-6051. Fix typos in log messages. (Ray Chiang via cdouglas) - - MAPREDUCE-5931. Validate SleepJob command line parameters (Gera Shegalov - via jlowe) - - MAPREDUCE-6063. Correct spill size calculation for spills wrapping the - circular buffer. (zhihai xu via cdouglas) - - MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is - disabled because of too much CPUs (Tsuyoshi OZAWA via jlowe) - - MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length - files (jlowe) - - MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but - is not documented (Tsuyoshi OZAWA via jlowe) - - MAPREDUCE-6090. mapred hsadmin getGroups fails to connect in some cases - (Robert Kanter via jlowe) - - MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs. - (Zhihai Xu via kasha) - - MAPREDUCE-6091. YARNRunner.getJobStatus() fails with - ApplicationNotFoundException if the job rolled off the RM view (Sangjin - Lee via jlowe) - - MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs (Gera Shegalov - via jlowe) - - MAPREDUCE-6104. TestJobHistoryParsing.testPartialJob fails in branch-2 - (Mit Desai via jlowe) - - MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb - via aw) - - MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) - - MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. - (Junping Du via zjshen) - - MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold - in mapred-default should be moved into description tags (Tsuyoshi OZAWA - via aw) - - MAPREDUCE-5796. Use current version of the archive name in - DistributedCacheDeploy document (Akira AJISAKA via aw) - - MAPREDUCE-5945. Update the description of GenericOptionsParser -jt - option (Akira AJISAKA via aw) - - MAPREDUCE-6087. Fixed wrong config name of - MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS. Contributed by - Akira AJISAKA. (Akira AJISAKA via jianhe) - - MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk - (Akira AJISAKA via jlowe) - - MAPREDUCE-6029. TestCommitterEventHandler fails in trunk (Mit Desai via - jlowe) - - MAPREDUCE-6122. TestLineRecordReader may fail due to test data files checked - out of git with incorrect line endings. (cnauroth) - - MAPREDUCE-6123. TestCombineFileInputFormat incorrectly starts 2 - MiniDFSCluster instances. (cnauroth) - - MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails (Mit Desai via - jlowe) - - MAPREDUCE-6115. TestPipeApplication#testSubmitter fails in trunk (Binglin - Chang via jlowe) - - MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting - for maps (Siqi Li via jlowe) - - MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in - client (Rohith via jlowe) - - MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. - (Junping Du via jianhe) - - MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and - TestMRTimelineEventHandling. (Zhijie Shen via vinodkv) - - MAPREDUCE-6022. map_input_file is missing from streaming job environment. - (jlowe via kihwal) - - MAPREDUCE-6048. Fixed TestJavaSerialization failure. (Varun Vasudev via - jianhe) - - MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect - with no authority in job jar path. (Gera Shegalov via jlowe) - - MAPREDUCE-5958. Wrong reduce task progress if map output is compressed - (Emilio Coppa and jlowe via kihwal) - - MAPREDUCE-6156. Fetcher - connect() doesn't handle connection refused - correctly (Junping Du via jlowe) - -Release 2.5.2 - 2014-11-19 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - -Release 2.5.1 - 2014-09-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-6033. Updated access check for displaying job information - (Yu Gao via Eric Yang) - -Release 2.5.0 - 2014-08-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5671. NaN can be created by client and assign to Progress (Chen - He via jeagles) - - MAPREDUCE-5665. Add audience annotations to MiniMRYarnCluster and - MiniMRCluster. (Anubhav Dhoot via kasha) - - MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles) - - MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles) - - MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason - Lowe via jeagles) - - MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout (Mit Desai - via kihwal) - - MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down - (Gera Shegalov via jlowe) - - MAPREDUCE-5836. Fix typo in RandomTextWriter (Akira AJISAKA via jeagles) - - MAPREDUCE-5852. Prepare MapReduce codebase for JUnit 4.11. (cnauroth) - - MAPREDUCE-5639. Port DistCp2 document to trunk (Akira AJISAKA via jeagles) - - MAPREDUCE-5812. Make job context available to - OutputCommitter.isRecoverySupported() (Mohammad Kamrul Islam via jlowe) - - MAPREDUCE-5638. Port Hadoop Archives document to trunk (Akira AJISAKA via - jeagles) - - MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE, - MAX_CHUNKS_IDEAL, MIN_RECORDS_PER_CHUNK and SPLIT_RATIO to be configurable. - (Tsuyoshi OZAWA via szetszwo) - - MAPREDUCE-5637. Convert Hadoop Streaming document to APT (Akira AJISAKA via - jeagles) - - MAPREDUCE-5636. Convert MapReduce Tutorial document to APT (Akira AJISAKA - via jeagles) - - MAPREDUCE-5774. Job overview in History UI should list reducer phases in - chronological order. (Gera Shegalov via kasha) - - MAPREDUCE-5652. NM Recovery. ShuffleHandler should handle NM restarts. - (Jason Lowe via kasha) - - MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not - need to be volatile. (Tsuyoshi OZAWA via junping_du) - - MAPREDUCE-5809. Enhance distcp to support preserving HDFS ACLs. (cnauroth) - - MAPREDUCE-5899. Support incremental data copy in DistCp. (jing9) - - MAPREDUCE-5886. Allow wordcount example job to accept multiple input paths. - (cnauroth) - - MAPREDUCE-5834. Increased test-timeouts in TestGridMixClasses to avoid - occassional failures. (Mit Desai via vinodkv) - - MAPREDUCE-5896. InputSplits should indicate which locations have the block - cached in memory. (Sandy Ryza via kasha) - - MAPREDUCE-5844. Add a configurable delay to reducer-preemption. - (Maysam Yabandeh via kasha) - - MAPREDUCE-5790. Made it easier to enable hprof profile options by default. - (Gera Shegalov via vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5759. Remove unnecessary conf load in Limits (Sandy Ryza) - - MAPREDUCE-5014. Extend Distcp to accept a custom CopyListing. - (Srikanth Sundarrajan via amareshwari) - - MAPREDUCE-5775. Remove unnecessary job.setNumReduceTasks in SleepJob.createJob - (jhanver chand sharma via devaraj) - - MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly - (Eric Payne via jlowe) - - MAPREDUCE-5642. TestMiniMRChildTask fails on Windows. - (Chuan Liu via cnauroth) - - MAPREDUCE-5846. Rumen doesn't understand JobQueueChangedEvent (Nathan Roberts via raviprak) - - MAPREDUCE-5837. MRAppMaster fails when checking on uber mode. (wheat9) - - MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed - (jlowe) - - MAPREDUCE-5884. History server uses short user name when canceling tokens - (Mohammad Kamrul Islam via jlowe) - - MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe - via jeagles) - - MAPREDUCE-5814. fat jar with *-default.xml may fail when - mapreduce.job.classloader=true. (Gera Shegalov via jlowe) - - MAPREDUCE-5309. 2.0.4 JobHistoryParser can't parse certain failed job - history files generated by 2.0.3 history server (Rushabh S Shah via jlowe) - - MAPREDUCE-5862. Line records longer than 2x split size aren't handled - correctly (bc Wong via jlowe) - - MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog. - (Kousuke Saruta via devaraj) - - MAPREDUCE-5777. Support utf-8 text with Byte Order Marker. - (Zhihai Xu via kasha) - - MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs) - (Yi Liu via umamahesh) - - MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth) - - MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event - at COMMIT_PENDING state. (Zhijie Shen via jianhe) - - MAPREDUCE-5939. StartTime showing up as the epoch time in JHS UI after - upgrade (Chen He via jlowe) - - MAPREDUCE-5900. Changed to the interpret container preemption exit code as a - task attempt killing event. (Mayank Bansal via zjshen) - - MAPREDUCE-5868. Fixed an issue with TestPipeApplication that was causing the - nightly builds to fail. (Akira Ajisaka via vinodkv) - - MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side - resource configuration for deciding uber-mode on map-only jobs. (Siqi Li via - vinodkv) - - MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly - assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) - - MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process - is shutting down. (Wangda Tan via zjshen) - -Release 2.4.1 - 2014-06-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5830. Added back the private API HostUtil.getTaskLogUrl(..) for - binary compatibility with older clients like Hive 0.13. (Akira Ajisaka via - vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen) - - MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in - Windows. (Xuan Gong via vinodkv) - - MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster - gracefully handle empty-queue names. (Akira Ajisaka via vinodkv) - - MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right permissions - on Windows for temporary files and thus also fix the test-issue with - TestHistoryServerFileSystemStateStoreService. (Varun Vasudev via vinodkv) - - MAPREDUCE-5828. Fixed a test issue with TestMapReduceJobControl that was - causing it to fail on Windows. (vinodkv) - - MAPREDUCE-5827. TestSpeculativeExecutionWithMRApp fails. - (Zhijie Shen via cnauroth) - - MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally. - (Zhijie Shen via cnauroth) - - MAPREDUCE-5832. Fixed TestJobClient to not fail on JDK7 or on Windows. (Jian - He and Vinod Kumar Vavilapalli via vinodkv) - - MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill - (Sangjin Lee via jlowe) - - MAPREDUCE-5843. Fixed TestMRKeyValueTextInputFormat to not leak files and - thus avoid failing on Windows. (Varun Vasudev via vinodkv) - - MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state - (Ming Ma via jlowe) - - MAPREDUCE-5821. Avoid unintentional reallocation of byte arrays in segments - during merge. (Todd Lipcon via cdouglas) - -Release 2.4.0 - 2014-04-07 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-5787. Added the ability to keep alive shuffle connections in the - MapReduce shuffle-handler. (Rajesh Balamohan via vinodkv) - - IMPROVEMENTS - - MAPREDUCE-5464. Add analogs of the SLOTS_MILLIS counters that jive with the - YARN resource model (Sandy Ryza) - - MAPREDUCE-5732. Report proper queue when job has been automatically placed - (Sandy Ryza) - - MAPREDUCE-5699. Allow setting tags on MR jobs (kasha) - - MAPREDUCE-5761. Added a simple log message to denote when encrypted shuffle - is on in the shuffle-handler. (Jian He via vinodkv) - - MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via - jlowe) - - MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level - inside the ApplicationMaster log. (Jian He via vinodkv) - - MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera - Shegalov via jlowe) - - MAPREDUCE-5553. Allow users to easily access - completed/pending/successful/failed tasks on MR AM web-ui. (Paul Han via - acmurthy) - - MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to - handle cross platform application submissions. (Jian He via vinodkv) - - MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block - location calls in parallel. (Siddharth Seth via vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5746. Job diagnostics can implicate wrong task for a failed job. - (Jason Lowe via kasha) - - MAPREDUCE-5670. CombineFileRecordReader should report progress when moving - to the next file (Chen He via jlowe) - - MAPREDUCE-5757. ConcurrentModificationException in JobControl.toList - (jlowe) - - MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect - to the YARN's web-app proxy with the correct scheme prefix. (Jian He via - vinodkv) - - MAPREDUCE-5768. TestMRJobs.testContainerRollingLog fails on trunk (Gera - Shegalov via jlowe) - - MAPREDUCE-5780. SliveTest should use the specified path to get the - particular FileSystem instead of using the default FileSystem. (szetszwo) - - MAPREDUCE-5028. Fixed a bug in MapTask that was causing mappers to fail - when a large value of io.sort.mb is set. (Karthik Kambatla via vinodkv) - - MAPREDUCE-5778. JobSummary does not escape newlines in the job name (Akira - AJISAKA via jlowe) - - MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page - (Rushabh S Shah via jlowe) - - MAPREDUCE-5794. SliveMapper always uses default FileSystem. (szetszwo via - Arpit Agarwal) - - MAPREDUCE-5751. MR app master fails to start in some cases if - mapreduce.job.classloader is true (Sangjin Lee via jlowe) - - MAPREDUCE-5688. TestStagingCleanup fails intermittently with JDK7 (Mit - Desai via jeagles) - - MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed - before registering with RM (Rohith via jlowe) - - MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt - finish time (Rushabh S Shah via jlowe) - - MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly - override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. (Varun Vasudev via - vinodkv) - - MAPREDUCE-5791. Shuffle phase is slow in Windows - - FadviseFileRegion::transferTo does not read disks efficiently. - (Nikola Vujic via cnauroth) - - MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it - recovers from a commit during a previous attempt. (Xuan Gong via vinodkv) - - MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape - hyphens and thus avoid parsing errors. (Akira AJISAKA via vinodkv) - - MAPREDUCE-5810. Removed the faulty and failing streaming test - TestStreamingTaskLog. (Akira Ajisaka via vinodkv) - - MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the - local-filesystem, rather than rely on the classpath. (Gera Shegalov via - acmurthy) - -Release 2.3.1 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.3.0 - 2014-02-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-5265. History server admin service to refresh user and superuser - group mappings (Ashwin Shankar via jlowe) - - MAPREDUCE-5356. Ability to refresh aggregated log retention period and - check interval (Ashwin Shankar via jlowe) - - MAPREDUCE-5386. Ability to refresh history server job retention and job - cleaner settings (Ashwin Shankar via jlowe) - - MAPREDUCE-5411. Refresh size of loaded job cache on history server (Ashwin - Shankar via jlowe) - - MAPREDUCE-5332. Support token-preserving restart of history server (jlowe) - - IMPROVEMENTS - - MAPREDUCE-5329. Allow MR applications to use additional AuxServices, - which are compatible with the default MapReduce shuffle. - (Avner BenHanoch via sseth) - - MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters (Tzuyoshi Ozawa via Sandy - Ryza) - - MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write - out text files without separators (Sandy Ryza) - - MAPREDUCE-5596. Allow configuring the number of threads used to serve - shuffle connections (Sandy Ryza via jlowe) - - MAPREDUCE-434. LocalJobRunner limited to single reducer (Sandy Ryza and - Aaron Kimball via Sandy Ryza) - - MAPREDUCE-4421. Run MapReduce framework via the distributed cache (jlowe) - - MAPREDUCE-1176. FixedLengthInputFormat and FixedLengthRecordReader - (Mariappan Asokan and BitsOfInfo via Sandy Ryza) - - MAPREDUCE-5613. DefaultSpeculator holds and checks hashmap that is always - empty (Gera Shegalov via Sandy Ryza) - - MAPREDUCE-5431. Missing pom dependency in MR-client (Timothy St. Clair - via stevel) - - MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope - (Ted Yu via stevel) - - MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) - - MAPREDUCE-5052. Job History UI and web services confusing job start time and - job submit time (Chen He via jeagles) - - MAPREDUCE-5692. Add explicit diagnostics when a task attempt is killed due - to speculative execution (Gera Shegalov via Sandy Ryza) - - MAPREDUCE-5550. Task Status message (reporter.setStatus) not shown in UI - with Hadoop 2.0 (Gera Shegalov via Sandy Ryza) - - MAPREDUCE-3310. Custom grouping comparator cannot be set for Combiners (tucu) - - MAPREDUCE-5672. Provide optional RollingFileAppender for container log4j - (syslog) (Gera Shegalov via jlowe) - - MAPREDUCE-5725. Make explicit that TestNetworkedJob relies on the Capacity - Scheduler (Sandy Ryza) - - MAPREDUCE-5744. Job hangs because - RMContainerAllocator$AssignedRequests.preemptReduce() violates the - comparator contract (Gera Shegalov via kasha) - - OPTIMIZATIONS - - MAPREDUCE-4680. Job history cleaner should only check timestamps of files in - old enough directories (Robert Kanter via Sandy Ryza) - - MAPREDUCE-5484. YarnChild unnecessarily loads job conf twice (Sandy Ryza) - - MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again - in Limits (Sandy Ryza) - - MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when - fetch fails (Sandy Ryza) - - BUG FIXES - - MAPREDUCE-5569. FloatSplitter is not generating correct splits (Nathan - Roberts via jlowe) - - MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly (Chuan Liu - via cnauroth) - - MAPREDUCE-5518. Fixed typo "can't read paritions file". (Albert Chu - via devaraj) - - MAPREDUCE-5561. org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl - testcase failing on trunk (Karthik Kambatla via jlowe) - - MAPREDUCE-5598. TestUserDefinedCounters.testMapReduceJob is flakey - (Robert Kanter via jlowe) - - MAPREDUCE-5604. TestMRAMWithNonNormalizedCapabilities fails on Windows due to - exceeding max path length. (cnauroth) - - MAPREDUCE-5451. MR uses LD_LIBRARY_PATH which doesn't mean anything in - Windows. (Yingda Chen via cnauroth) - - MAPREDUCE-5409. MRAppMaster throws InvalidStateTransitonException: Invalid - event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl (Gera - Shegalov via jlowe) - - MAPREDUCE-5674. Missing start and finish time in mapred.JobStatus. - (Chuan Liu via cnauroth) - - MAPREDUCE-5650. Job fails when hprof mapreduce.task.profile.map/reduce.params - is specified (Gera Shegalov via Sandy Ryza) - - MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal - task-state (Ashwin Shankar via jlowe) - - MAPREDUCE-5380. Invalid mapred command should return non-zero exit code - (Stephen Chu via jlowe) - - MAPREDUCE-5404. HSAdminServer does not use ephemeral ports in minicluster - mode (Ted Yu via jlowe) - - MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang - via bobby) - - MAPREDUCE-5514. Fix TestRMContainerAllocator. (Zhijie Shen via acmurthy) - - MAPREDUCE-5102. fix coverage org.apache.hadoop.mapreduce.lib.db and - org.apache.hadoop.mapred.lib.db (Aleksey Gorshkov, Andrey Klochkov, and - Nathan Roberts via jlowe) - - MAPREDUCE-5585. TestCopyCommitter#testNoCommitAction Fails on JDK7 - (jeagles) - - MAPREDUCE-5186. mapreduce.job.max.split.locations causes some splits - created by CombineFileInputFormat to fail (Robert Parker and Jason Lowe - via jlowe) - - MAPREDUCE-5610. TestSleepJob fails in jdk7 (Jonathan Eagles via jlowe) - - MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too - high. (cnauroth) - - MAPREDUCE-5625. TestFixedLengthInputFormat fails in jdk7 environment - (Mariappan Asokan via jeagles) - - MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should - have taken more than 5 seconds in jdk7 (Jonathan Eagles via jlowe) - - MAPREDUCE-5645. TestFixedLengthInputFormat fails with native libs (Mit - Desai via jeagles) - - MAPREDUCE-5632. TestRMContainerAllocator#testUpdatedNodes fails (jeagles) - - MAPREDUCE-5656. bzip2 codec can drop records when reading data in splits - (jlowe) - - MAPREDUCE-5623. TestJobCleanup fails because of RejectedExecutionException - and NPE. (jlowe) - - MAPREDUCE-5679. TestJobHistoryParsing has race condition (Liyin Liang via - jlowe) - - MAPREDUCE-5687. Fixed failure in TestYARNRunner caused by YARN-1446. (Jian He - via vinodkv) - - MAPREDUCE-5694. Fixed MR AppMaster to shutdown the LogManager so as to avoid - losing syslog in some conditions. (Mohammad Kamrul Islam via vinodkv) - - MAPREDUCE-5685. Fixed a bug with JobContext getCacheFiles API inside the - WrappedReducer class. (Yi Song via vinodkv) - - MAPREDUCE-5689. MRAppMaster does not preempt reducers when scheduled maps - cannot be fulfilled. (lohit via kasha) - - MAPREDUCE-5724. JobHistoryServer does not start if HDFS is not running. - (tucu) - - MAPREDUCE-5729. mapred job -list throws NPE (kasha) - - MAPREDUCE-5693. Restore MRv1 behavior for log flush (Gera Shegalov via - jlowe) - - MAPREDUCE-5723. MR AM container log can be truncated or empty. - (Mohammad Kamrul Islam via kasha) - - MAPREDUCE-5743. Fixed the test failure in TestRMContainerAllocator. - (Ted Yu and Vinod Kumar Vavilapalli via zjshen) - -Release 2.2.0 - 2013-10-13 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta - via tgraves) - - MAPREDUCE-5488. Changed MR client to keep trying to reach the application - when it sees that on attempt's AM is down. (Jian He via vinodkv) - - MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config - mapreduce.ssl.enabled to enable https and disabling it by default as MR AM - needs to set up its own certificates etc and not depend on clusters'. - (Omkar Vinit Joshi via vinodkv) - - MAPREDUCE-5505. Clients should be notified job finished only after job - successfully unregistered (Zhijie Shen via bikas) - - MAPREDUCE-5503. Fixed a test issue in TestMRJobClient. (Jian He via vinodkv) - - MAPREDUCE-5170. Fixed a wrong log message in CombineFileInputFormat class. - (Sangjin Lee via vinodkv) - - MAPREDUCE-5525. Increase timeout of TestDFSIO.testAppend and - TestMRJobsWithHistoryService.testJobHistoryData. (Chuan Liu via cnauroth) - - MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert - Parker via jlowe) - - MAPREDUCE-5531. Fix compat with hadoop-1 in mapreduce.(TaskID, - TaskAttemptID) by re-introducing missing constructors. (Robert Kanter via - acmurthy) - - MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow - times out (Robert Kanter via jlowe) - - MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner - by re-introducing (get,set)PartitionFile which takes in JobConf. (Robert - Kanter via acmurthy) - - MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after - the job is really done - a bug caused by MAPREDUCE-5505. (Zhijie Shen via - vinodkv) - - MAPREDUCE-5551. Fix compat with hadoop-1 in - SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing - missing constructors. (Zhijie Shen via acmurthy) - - MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza) - - MAPREDUCE-5536. Fixed MR AM and JHS to respect - mapreduce.jobhistory.webapp.https.address. (Omkar Vinit Joshi via vinodkv) - - MAPREDUCE-5530. Fix compat with hadoop-1 in - mapred.lib.CombinFileInputFormat by re-introducing - isSplittable(FileSystem, Path) api and ensuring semantic compatibility. - (Robert Kanter via acmurthy) - - MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN. - (Zhijie Shen via acmurthy) - - MAPREDUCE-5554. hdfs-site.xml included in hadoop-mapreduce-client-jobclient - tests jar is breaking tests for downstream components (Robert Kanter via - Sandy Ryza) - - MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting - feature in RM requests (Zhijie Shen via bikas) - - MAPREDUCE-5442. $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on - Windows. (Yingda Chen via cnauroth) - - MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that - aren't heart-beating for a while, so that we can aggressively speculate - instead of waiting for task-timeout (Xuan Gong via vinodkv) - - MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir - cleanup, sending job-end notification correctly when unregister with RM - fails. (Zhijie Shen via vinodkv) - -Release 2.1.1-beta - 2013-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5478. TeraInputFormat unnecessarily defines its own FileSplit - subclass (Sandy Ryza) - - MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else - but just before ClientService to avoid race conditions during RM restart. - (Jian He via vinodkv) - - MAPREDUCE-5379. Include token tracking ids in jobconf. (kkambatl via tucu) - - MAPREDUCE-5523. Added separate configuration properties for https for JHS - without which even when https is enabled, it starts on http port itself. - (Omkar Vinit Joshi via vinodkv) - - OPTIMIZATIONS - - MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race - conditions (jlowe via kihwal) - - MAPREDUCE-5462. In map-side sort, swap entire meta entries instead of - indexes for better cache performance. (Sandy Ryza) - - MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus - (Hairong Kuang and Jason Lowe via jlowe) - - BUG FIXES - - MAPREDUCE-5385. Fixed a bug with JobContext getCacheFiles API. (Omkar Vinit - Joshi via vinodkv) - - MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is - stopped (Karthik Kambatla via jlowe) - - MAPREDUCE-5251. Reducer should not implicate map attempt if it has - insufficient space to fetch map output (Ashwin Shankar via jlowe) - - MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via - jlowe) - - MAPREDUCE-5358. MRAppMaster throws invalid transitions for JobImpl - (Devaraj K via jlowe) - - MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the - input path dir (Devaraj K via jlowe) - - MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe) - - MAPREDUCE-5367. Local jobs all use same local working directory - (Sandy Ryza) - - MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7 (Robert - Parker via jlowe) - - MAPREDUCE-5454. TestDFSIO fails intermittently on JDK7 (Karthik Kambatla - via Sandy Ryza) - - MAPREDUCE-5001. LocalJobRunner has race condition resulting in job - failures (Sandy Ryza via jlowe) - - MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate - AMs in case they are exiting because of errors and thus help history-server - pick up the right history file for the last successful AM. (Jian He via - vinodkv) - - MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. (vinodkv via - acmurthy) - - MAPREDUCE-5470. LocalJobRunner does not work on Windows. (Sandy Ryza via - cnauroth) - - MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory - only after unregistering from the RM. (Jian He via vinodkv) - - MAPREDUCE-5483. revert MAPREDUCE-5357. (rkanter via tucu) - - MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM - commands to reboot, so that client can continue to track the overall job. - (Jian He via vinodkv) - - MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe) - - MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE (Nemon Lou via - devaraj) - - MAPREDUCE-5020. Compile failure with JDK8 (Trevor Robinson via tgraves) - - MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS - (Nemon Lou via devaraj) - - MAPREDUCE-5493. Cleanup in-memory & on-disk segments to prevent leak on - shuffle completion. (jlowe via acmurthy) - -Release 2.1.0-beta - 2013-08-22 - - INCOMPATIBLE CHANGES - - MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan - Gong via vinodkv) - - MAPREDUCE-5234. Change mapred.TaskReport and mapreduce.TaskReport for binary - compatibility with mapred in 1.x but incompatible with 0.23.x. (Mayank Bansal - via vinodkv) - - MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run - 1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of - ProgramDriver. (Zhijie Shen via vinodkv) - - MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job - when compared to 1.x. This breaks 0.23.x users of one API in Job. (Mayank - Bansal via vinodkv) - - MAPREDUCE-5237. Add methods that were removed from ClusterStatus back into - 2.x so as to be compatible with 1.x. Incompatible as - ClusterStatus.UNINITIALIZED_MEMORY_VALUE is a long now and so breaks 0.23.x - but it shouldn't be a big deal in reality. (Zhijie Shen via vinodkv) - - NEW FEATURES - - HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows - Azure environments. (See breakdown of tasks below for subtasks and - contributors) - - IMPROVEMENTS - - MAPREDUCE-3008. Improvements to cumulative CPU emulation for short running - tasks in Gridmix. (amarrk via tgraves) - - MAPREDUCE-5033. mapred shell script should respect usage flags - (--help -help -h). (Andrew Wang via atm) - - MAPREDUCE-4892. Modify CombineFileInputFormat to not skew input slits' - allocation on small clusters. (Bikas Saha via vinodkv) - - MAPREDUCE-4990. Construct debug strings conditionally in - ShuffleHandler.Shuffle#sendMapOutput(). (kkambatl via tucu) - - MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred - (Aleksey Gorshkov via bobby) - - MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via - acmurthy) - - MAPREDUCE-3787. [Gridmix] Optimize job monitoring and STRESS mode for - faster job submission. (amarrk via tgraves) - - MAPREDUCE-5079. Changes job recovery to restore state directly from job - history, instaed of simulating state machine events. - (Jason Lowe and Robert Parker via sseth) - - MAPREDUCE-4981. Add WordMean, WordMedian, WordStandardDeviation - to ExamplesDriver. (Plamen Jeliazkov via shv) - - MAPREUDUCE-5059. Change average merge time on Job overview page to be the - time delta between the end of the shuffle and the start of the reduce. - (Omkar Vinit Joshi via vinodkv) - - MAPREDUCE-4985. Add compression option to TestDFSIO usage. - (Plamen Jeliazkov via shv) - - MAPREDUCE-5152. Make MR App to simply pass through the container from RM - instead of extracting and populating information itself to start any - container. (vinodkv) - - MAPREDUCE-5175. Updated MR App to not set envs that will be set by NMs - anyways after YARN-561. (Xuan Gong via vinodkv) - - MAPREDUCE-5069. add concrete common implementations of - CombineFileInputFormat (Sangjin Lee via bobby) - - MAPREDUCE-5145. Changed default max-attempts to be more than one for MR jobs - inline with YARN. (Zhijie Shen via vinodkv) - - MAPREDUCE-5036. Default shuffle handler port should not be 8080. - (Sandy Ryza via tomwhite) - - MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support - binary compatibility with hadoop-1 examples. (Zhijie Shen via vinodkv) - - MAPREDUCE-5157. Bring back old sampler related code so that we can support - binary compatibility with hadoop-1 sorter example. (Zhijie Shen via vinodkv) - - MAPREDUCE-5222. Bring back some methods and constants in Jobclient for - binary compatibility with mapred in 1.x. (Karthik Kambatla via vinodkv) - - MAPREDUCE-5235. Bring back old fields and exceptions in Counters for - binary compatibility with mapred in 1.x. (Mayank Bansal via vinodkv) - - MAPREDUCE-5246. Specify application-type at the time of job submission after - YARN-563. (Mayank Bansal via vinodkv) - - MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary - compatibility with mapred in 1.x (Mayank Bansal via vinodkv) - - MAPREDUCE-5270. Migrated MR app from using BuilderUtil factory methods to - individual record factory methods. (Jian He via vinodkv) - - MAPREDUCE-5263. Bring back old methods and fields in - filecache.DistributedCache for binary compatibility with mapred in 1.x. - (Zhijie Shen via vinodkv) - - MAPREDUCE-5228. Bring back FileInputFormat.Counter and - FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. - (Mayank Bansal via vinodkv) - - MAPREDUCE-5176. Add annotation for tagging tasks as responsive to - preemption. (Carlo Curino, cdouglas) - - MAPREDUCE-5275. Bring back a couple of APIs in mapreduce.security.TokenCache - for binary compatibility with 1.x mapreduce APIs. (Mayank Bansal via vinodkv) - - MAPREDUCE-5231. Bring back a constructor in mapred's - DBInputFormat.DBRecordReader for binary compatibility with 1.x mapred APIs. - (Zhijie Shen via vinodkv) - - MAPREDUCE-5273. Bring back a couple of protected variables in mapred and - mapreduce CombineFileRecordReader for binary compatibility with 1.x APIs. - (Mayank Bansal via vinodkv) - - MAPREDUCE-5280. Bring back removed constructor and a method in mapreduce - ClusterMetrics for binary compatibility with 1.x APIs. (Mayank Bansal via - vinodkv) - - MAPREDUCE-5289. Updated MR App to use Token directly after YARN-717. (Jian He - via vinodkv) - - MAPREDUCE-5229. Put back FileOutputCommitter.TEMP_DIR_NAME in mapreduce for - binary compatibility with 1.x APIs. (Zhijie Shen via vinodkv) - - MAPREDUCE-5274. Bring back SecureShuffleUtils.toHex in mapreduce for binary - compatibility with 1.x APIs. (Mayank Bansal via vinodkv) - - MAPREDUCE-5300. Fix backward incompatibility for - o.a.h.mapreduce.filecache.DistributedCache. (Zhijie Shen via acmurthy) - - MAPREDUCE-5283. Over 10 different tests have near identical - implementations of AppContext (Sandy Ryza via jlowe) - - MAPREDUCE-5199. Removing ApplicationTokens file as it is no longer needed. - (Daryn Sharp via vinodkv) - - MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents. - (cdouglas via acmurthy) - - MAPREDUCE-5184. Document compatibility for MapReduce applications in - hadoop-2 vis-a-vis hadoop-1. (Zhijie Shen via acmurthy) - - MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (cdouglas) - - MAPREDUCE-5326. Added version to shuffle header. (Zhijie Shen via - acmurthy) - - MAPREDUCE-5333. Add test that verifies MRAM works correctly when sending - requests with non-normalized capabilities. (ywskycn via tucu) - - MAPREDUCE-5398. MR changes for YARN-513 (Jian He via bikas) - - OPTIMIZATIONS - - MAPREDUCE-4974. Optimising the LineRecordReader initialize() method - (Gelesh via bobby) - - MAPREDUCE-5268. Improve history server startup performance (Karthik - Kambatla via jlowe) - - MAPREDUCE-5352. Optimize node local splits generated by - CombineFileInputFormat. (sseth) - - BUG FIXES - - MAPREDUCE-4671. AM does not tell the RM about container requests which are - no longer needed. (Bikas Saha via sseth) - - MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu) - - MAPREDUCE-5000. Fixes getCounters when speculating by fixing the selection - of the best attempt for a task. (Jason Lowe via sseth) - - MAPREDUCE-4994. Addendum fixing testcases failures. (sandyr via tucu) - - MAPREDUCE-4846. Some JobQueueInfo methods are public in MR1 but protected - in MR2. (Sandy Ryza via tomwhite) - - MAPREDUCE-5013. mapred.JobStatus compatibility: MR2 missing constructors - from MR1. (Sandy Ryza via tomwhite) - - MAPREDUCE-4951. Container preemption interpreted as task failure. - (Sandy Ryza via tomwhite) - - MAPREDUCE-5008. Merger progress miscounts with respect to EOF_MARKER. - (Sandy Ryza via tomwhite) - - MAPREDUCE-4693. History server should include counters for failed tasks. - (Xuan Gong via sseth) - - MAPREDUCE-4896. mapred queue -info spits out ugly exception when queue does - not exist. (sandyr via tucu) - - MAPREDUCE-3685. Fix bugs in MergeManager to ensure compression codec is - appropriately used and that on-disk segments are correctly sorted on - file-size. (Anty Rao and Ravi Prakash via acmurthy) - - MAPREDUCE-4571. TestHsWebServicesJobs fails on jdk7. (tgraves via tucu) - - MAPREDUCE-4716. TestHsWebServicesJobsQuery.testJobsQueryStateInvalid - fails with jdk7. (tgraves via tucu) - - MAPREDUCE-5075. DistCp leaks input file handles since ThrottledInputStream - does not close the wrapped InputStream. (Chris Nauroth via szetszwo) - - MAPREDUCE-3872. Fix an event handling races in ContainerLauncherImpl. - (Robert Kanter via sseth) - - MAPREDUCE-5062. Fix MR AM to read max-retries from the RM. (Zhijie Shen via - vinodkv) - - MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when - input data directory already exists and -generate opton is - given.(ravigummadi via tgraves) - - MAPREDUCE-2722. [Gridmix] Gridmix simulated job's map's hdfsBytesRead - counter is wrong when compressed input is used.(ravigummadi via tgraves) - - MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a - job if the trace contains null taskStatus for a task. (ravigummadi via - tgraves) - - MAPREDUCE-4087. [Gridmix] GenerateDistCacheData job of Gridmix can - become slow in some cases (ravigummadi via tgraves). - - MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related - code. (Karthik Kambatla via sseth) - - MAPREDUCE-4083. [Gridmix] NPE in cpu emulation. (amarrk via tgraves) - - MAPREDUCE-4100. [Gridmix] Bug fixed in compression emulation feature for - map only jobs. (amarrk via tgraves) - - MAPREDUCE-4356. [Rumen] Provide access to the method - ParsedTask.obtainTaskAttempts(). (ravigummadi via tgraves) - - MAPREDUCE-4149. [Rumen] Rumen fails to parse certain counter - strings. (ravigummadi via tgraves) - - MAPREDUCE-3757. [Rumen] Fixed Rumen Folder to adjust shuffleFinished and - sortFinished times when needed. (Ravi Gummadi via tgraves) - - MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. (Omkar Vinit - Joshi via vinodkv) - - MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot - command from the RM. (Jian He via jlowe) - - MAPREDUCE-5113. Streaming input/output types are ignored with java - mapper/reducer. (sandyr via tucu) - - MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu) - - MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error - (Thomas Graves via jlowe) - - MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA (Amir - Sanjar via jlowe) - - MAPREDUCE-5139. Update MR AM to use the modified startContainer API after - YARN-486. (Xuan Gong via vinodkv) - - MAPREDUCE-5151. Update MR AM to use standard exit codes from the API after - YARN-444. (Sandy Ryza via vinodkv) - - MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) - - MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server - configs. (sandyr via tucu) - - MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and - FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu) - - MAPREDUCE-5078. TestMRAppMaster fails on Windows due to mismatched path - separators. (Chris Nauroth via sseth) - - MAPREDUCE-4932. mapreduce.job#getTaskCompletionEvents incompatible with - Hadoop 1. (rkanter via tucu) - - MAPREDUCE-5163. Update MR App to not use API utility methods for collections - after YARN-441. (Xuan Gong via vinodkv) - - MAPREDUCE-5066. Added a timeout for the job.end.notification.url. (Ivan - Mitic via acmurthy) - - MAPREDUCE-5146. application classloader may be used too early to load - classes. (Sangjin Lee via tomwhite) - - MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent - with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that - cleanup is now called even if there is an error. The old mapred API - already ensures that Mapper.close and Reducer.close are invoked during - error handling. Note that it is an incompatible change, however end-users - can override Mapper.run and Reducer.run to get the old (inconsistent) - behaviour. (acmurthy) - - MAPREDUCE-5166. Fix ConcurrentModificationException due to insufficient - synchronization on updates to task Counters. (Sandy Ryza via acmurthy) - - MAPREDUCE-5181. RMCommunicator should not use AMToken from the env. - (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-5178. Update MR App to set progress in ApplicationReport after - YARN-577. (Hitesh Shah via vinodkv) - - MAPREDUCE-5167. Update MR App after YARN-562 to use the new builder API - for the container. (Jian He via vinodkv) - - MAPREDUCE-5179. Fix unit test in TestHSWebServices which fails when - versionInfo has parantheses like when running on a git checkout. (Hitesh Shah - via vinodkv) - - MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the - default minimum block size. (Andrew Wang via atm) - - MAPREDUCE-5205. Fixed MR App to load tokens correctly. (vinodkv) - - MAPREDUCE-5204. Handling YarnRemoteException separately from IOException in - MR app after YARN-629. (Xuan Gong via vinodkv) - - MAPREDUCE-5209. Fix units in a ShuffleScheduler log message. - (Tsuyoshi OZAWA via cdouglas) - - MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in - MR App's use of ClientRMProtocol after YARN-631. (Xuan Gong via vinodkv) - - MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in - MR App's use of AMRMProtocol after YARN-630. (Xuan Gong via vinodkv) - - MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw - InterruptedException so its incompatible with MR1. (rkanter via tucu) - - MAPREDUCE-5239. Updated MR App to reflect YarnRemoteException changes after - YARN-634. (Siddharth Seth via vinodkv) - - MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading - local files. (Omkar Vinit Joshi via vinodkv) - - MAPREDUCE-5220. Setter methods in TaskCompletionEvent are public in MR1 and - protected in MR2. (sandyr via tucu) - - MAPREDUCE-5240. Fix a bug in MRAppMaster because of which OutputCommitter - could not access credentials set by the user. (vinodkv) - - MAPREDUCE-5244. Two functions changed their visibility in JobStatus. - (zjshen via tucu) - - MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific - counters page for failed job. (Ashwin Shankar via jlowe) - - MAPREDUCE-5257. Fix issues in TestContainerLauncherImpl after YARN-617. - (Omkar Vinit Joshi via vinodkv) - - MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after - YARN-716. (Siddharth Seth via vinodkv) - - MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead - of the entire Container in the startContainer call - YARN-684. - (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-5299. Fix backward incompatibility for TaskCompletionEvent by - adding back setTaskID. (Zhijie Shen via acmurthy) - - MAPREDUCE-5296. Fix backward incompatibility for JobControl by adding the - omitted addJob. (Zhijie Shen via acmurthy) - - MAPREDUCE-5245. Added back constants to JobConf to fix incompatibilities. - (Zhijie Shen via acmurthy) - - MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public - after YARN-748. (Jian He via vinodkv) - - MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming - YarnRemoteException to YarnException. (Siddharth Seth via vinodkv) - - MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching - multiple compressed map outputs (Nathan Roberts via jlowe) - - MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe - via daryn) - - MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators - missmatch. (Ivan Mitic via cnauroth) - - MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar, - Devaraj K, and B Anil Kumar via jlowe) - - MAPREDUCE-5334. Fix failing unit tests - TestContainerLauncher, - TestContainerLauncherImpl. (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-5325. MR changes related to YARN-727. ClientRMProtocol.getAllApplications - should accept ApplicationType as a parameter. (Xuan Gong via hitesh) - - MAPREDUCE-5291. Change MR App to use updated property names in - container-log4j.properties. (Zhijie Shen via sseth) - - MAPREDUCE-5303. Changed MR app after moving ProtoBase to package impl.pb via - YARN-724. (Jian He via vinodkv) - - MAPREDUCE-5312. TestRMNMInfo is failing. (sandyr via tucu) - - MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents - methods have incompatible signature changes. (kkambatl via tucu) - - MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle. - (Steve Loughran via vinodkv) - - MAPREDUCE-5319. Set user.name in job.xml. (Xuan Gong via acmurthy) - - MAPREDUCE-5310. MRAM should not normalize allocation request capabilities. - (tucu) - - MAPREDUCE-5213. Re-assess TokenCache methods marked @Private. - (kkambatl via tucu) - - MAPREDUCE-5412. Update MR app to use multiple containers API of - ContainerManager after YARN-926. (Jian He via vinodkv) - - MAPREDUCE-5421. Fixed TestNonExistentJob failure after YARN-873. (Junping Du - via vinodkv) - - MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker - via jlowe) - - MAPREDUCE-5399. Unnecessary Configuration instantiation in IFileInputStream - slows down merge. (Stanislav Barton via Sandy Ryza) - - BREAKDOWN OF HADOOP-8562 SUBTASKS - - MAPREDUCE-4739. Some MapReduce tests fail to find winutils. - (Chris Nauroth via suresh) - - MAPREDUCE-4780. MapReduce distribution build fails on Windows. - (Chris Nauroth via suresh) - - MAPREDUCE-4790. MapReduce build script would be more readable using abspath. - (Chris Nauroth via suresh) - - MAPREDUCE-4869. Fix TestMapReduceChildJVM. (Chris Nauroth via acmurthy) - - MAPREDUCE-4870. Fix TestMRJobsWithHistoryService. (Chris Nauroth via acmurthy) - - MAPREDUCE-4983. Fixed various platform specific assumptions in various tests, - so that they can pass on Windows too. (Chris Nauroth via vinodkv) - - HADOOP-9372. Fix bad timeout annotations on tests. - (Arpit Agarwal via suresh) - - MAPREDUCE-4885. Streaming tests have multiple failures on Windows. (Chris - Nauroth via bikas) - - MAPREDUCE-5177. Use common utils FileUtil#setReadable/Writable/Executable & - FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) - - MAPREDUCE-5349. TestClusterMapReduceTestCase and TestJobName fail on Windows - in branch-2. (Chuan Liu via cnauroth) - - MAPREDUCE-5355. MiniMRYarnCluster with localFs does not work on Windows. - (Chuan Liu via cnauroth) - - MAPREDUCE-5359. JobHistory should not use File.separator to match timestamp - in path. (Chuan Liu via cnauroth) - - MAPREDUCE-5357. Job staging directory owner checking could fail on Windows. - (Chuan Liu via cnauroth) - - MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format. - (Chuan Liu via cnauroth) - - MAPREDUCE-5366. TestMRAsyncDiskService fails on Windows. (Chuan Liu via - cnauroth) - - MAPREDUCE-5187. Create mapreduce command scripts on Windows. (Chuan Liu via - cnauroth) - - MAPREDUCE-4374. Fix child task environment variable config and add support - for Windows. (Chuan Liu via cnauroth) - -Release 2.0.6-alpha - 08/22/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - -Release 2.0.5-alpha - 06/06/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5240 inside of FileOutputCommitter the initialized Credentials cache - appears to be empty. (vinodkv) - -Release 2.0.4-alpha - 2013-04-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. - (Sandy Ryza via tomwhite) - - MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is - submitting a job (Daryn Sharp via cos) - - MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus - fix failures in renewal of HistoryServer's delegations tokens. (Siddharth - Seth via vinodkv) - - MAPREDUCE-5083. MiniMRCluster should use a random component when creating an - actual cluster (Siddharth Seth via hitesh) - - MAPREDUCE-5094. Disabled memory monitoring by default in MiniMRYarnCluster - to avoid some downstream tests failing. (Siddharth Seth via vinodkv) - -Release 2.0.3-alpha - 2013-02-06 - - INCOMPATIBLE CHANGES - - MAPREDUCE-4123. Remove the 'mapred groups' command, which is no longer - supported. (Devaraj K via sseth) - - MAPREDUCE-4938. Use token request messages defined in hadoop common. - (suresh) - - NEW FEATURES - - MAPREDUCE-4520. Added support for MapReduce applications to request for - CPU cores along-with memory post YARN-2. (acmurthy) - - MAPREDUCE-4810. Added new admin command options for MR AM. (Jerry Chen via - vinodkv) - - MAPREDUCE-4049. Experimental api to allow for alternate shuffle plugins. - (Avner BenHanoch via acmurthy) - - MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu) - - MAPREDUCE-4808. Refactor MapOutput and MergeManager to facilitate reuse - by Shuffle implementations. (masokan via tucu) - - IMPROVEMENTS - - MAPREDUCE-3678. The Map tasks logs should have the value of input - split it processed. (harsh) - - MAPREDUCE-4616. Improve javadoc for MultipleOutputs. (Tony Burton via - acmurthy) - - HADOOP-8911. CRLF characters in source and text files. - (Raja Aluri via suresh) - - MAPREDUCE-4723. Fix warnings found by findbugs 2. (Sandy Ryza via eli) - - MAPREDUCE-4703. Add the ability to start the MiniMRClientCluster using - the configurations used before it is being stopped. (ahmed.radwan via tucu) - - MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in - MR1 but not MR2. (Sandy Ryza via tomwhite) - - MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished - applications that YARN's ResourceManager doesn't keep track off anymore - (Derek Dagit via vinodkv) - - MAPREDUCE-4920. Use security token protobuf definition from hadoop common. - (Suresh Srinivas via vinodkv) - - MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus - calls. (sandyr via tucu) - - MAPREDUCE-4949. Enable multiple pi jobs to run in parallel. (sandyr via tucu) - - MAPREDUCE-4809. Change visibility of classes for pluggable sort changes. - (masokan via tucu) - - MAPREDUCE-4838. Add additional fields like Locality, Avataar to the - JobHistory logs. (Zhijie Shen via sseth) - - MAPREDUCE-4971. Minor extensibility enhancements to Counters & - FileOutputFormat. (Arun C Murthy via sseth) - - MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort. - (tucu) - - OPTIMIZATIONS - - MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of - containers to get maximum locality. (Bikas Saha via vinodkv) - - BUG FIXES - - MAPREDUCE-4272. SortedRanges.Range#compareTo is not spec compliant. - (Yu Gao via llu) - - MAPREDUCE-4607. Race condition in ReduceTask completion can result in Task - being incorrectly failed. (Bikas Saha via tomwhite) - - MAPREDUCE-4646. Fixed MR framework to send diagnostic information correctly - to clients in case of failed jobs also. (Jason Lowe via vinodkv) - - MAPREDUCE-4674. Hadoop examples secondarysort has a typo - "secondarysrot" in the usage. (Robert Justice via eli) - - MAPREDUCE-4681. Fix unit tests broken by HDFS-3910. (acmurthy) - - MAPREDUCE-4712. mr-jobhistory-daemon.sh doesn't accept --config - (Vinod Kumar Vavilapalli via tgraves) - - MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite) - - MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO. - (Brandon Li via suresh) - - MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and - UNASSIGNED states. (Mayank Bansal via sseth) - - MAPREDUCE-1806. CombineFileInputFormat does not work with paths not on default FS. (Gera Shegalov via tucu) - - MAPREDUCE-4777. In TestIFile, testIFileReaderWithCodec relies on - testIFileWriterWithCodec. (Sandy Ryza via tomwhite) - - MAPREDUCE-4800. Cleanup o.a.h.mapred.MapTaskStatus - remove unused - code. (kkambatl via tucu) - - MAPREDUCE-4861. Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal. - (kkambatl via tucu) - - MAPREDUCE-4856. TestJobOutputCommitter uses same directory as - TestJobCleanup. (Sandy Ryza via tomwhite) - - MAPREDUCE-4278. Cannot run two local jobs in parallel from the same - gateway. (Sandy Ryza via tomwhite) - - MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce - system JARs. (tomwhite) - - MAPREDUCE-4936. JobImpl uber checks for cpu are wrong (Arun C Murthy via - jlowe) - - MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR. - (rkanter via tucu) - - MAPREDUCE-4923. Add toString method to TaggedInputSplit. (sandyr via tucu) - - MAPREDUCE-4948. Fix a failing unit test TestYARNRunner.testHistoryServerToken. - (Junping Du via sseth) - - MAPREDUCE-4803. Remove duplicate copy of TestIndexCache. (Mariappan Asokan - via sseth) - - MAPREDUCE-2264. Job status exceeds 100% in some cases. - (devaraj.k and sandyr via tucu) - - MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7. - (Arpit Agarwal via suresh) - - MAPREDUCE-4884. Streaming tests fail to start MiniMRCluster due to missing - queue configuration. (Chris Nauroth via suresh) - - MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via atm) - -Release 2.0.2-alpha - 2012-09-07 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process - command-line. (ahmed via tucu) - - MAPREDUCE-4417. add support for encrypted shuffle (tucu) - - MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu) - - MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) - - MAPREDUCE-4438. Add a simple, generic client to run 'easy' AMs in YARN. - (Bikas Saha via acmurthy) - - IMPROVEMENTS - - MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved - (Jason Lowe via bobby) - - MAPREDUCE-4511. Add IFile readahead (ahmed via tucu) - - MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached - (rkanter via tucu) - - MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal - interface to allow schedulers to maintain their own. (acmurthy) - - MAPREDUCE-4146. Support limits on task status string length and number of - block locations in branch-2. (Ahmed Radwan via tomwhite) - - MAPREDUCE-3871. Allow symlinking in LocalJobRunner DistributedCache. - (tomwhite) - - MAPREDUCE-3921. MR AM should act on node health status changes. - (Bikas Saha via sseth) - - MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in - mapred-default.xml (Rui Kubo via harsh) - - MAPREDUCE-3907. Document entries mapred-default.xml for the - jobhistory server. (Eugene Koontz via harsh) - - MAPREDUCE-3906. Fix inconsistency in documentation regarding - mapreduce.jobhistory.principal. (Eugene Koontz via harsh) - - MAPREDUCE-4432. Confusing warning message when GenericOptionsParser - is not used. (Gabriel Reid via harsh) - - MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease - development of new applications. (Bikas Saha via acmurthy) - - MAPREDUCE-4447. Remove aop from cruft from the ant build. (eli) - - MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler. - (Todd Lipcon and Siddharth Seth via sseth) - - MAPREDUCE-4580. Change MapReduce to use the yarn-client module. - (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on - jdk7. (Thomas Graves via vinodkv) - - MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than - rely on YARN. (acmurthy) - - BUG FIXES - - MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in - YarnConfiguration. (ahmed via tucu) - - MAPREDUCE-4406. Users should be able to specify the MiniCluster ResourceManager - and JobHistoryServer ports. (ahmed via tucu) - - MAPREDUCE-4407. Add hadoop-yarn-server-tests--tests.jar to hadoop dist - package. (ahmed via tucu) - - MAPREDUCE-4465. Update description of yarn.nodemanager.address property. - (bowang via tucu) - - MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files - get deleted from tasktracker. (mayank_bansal via tucu) - - MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) - - MAPREDUCE-4494. TestFifoScheduler failing with Metrics source QueueMetrics,q0=default - already exists!. (ahmed.radwan via tucu) - - MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration. - (ahmed.radwan via tucu) - - MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name - for compatibility reasons is creating incorrect counter name. - (Jarek Jarcec Cecho via tomwhite) - - MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not added to the - classpath (rkanter via tucu) - - MAPREDUCE-4577. HDFS-3672 broke - TestCombineFileInputFormat.testMissingBlocks() test. (atm) - - MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu) - - MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies. - (tucu via tomwhite) - - MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in - MR2. (tomwhite) - - MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby) - - MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu) - - MAPREDUCE-4148. MapReduce should not have a compile-time dependency on - HDFS. (tomwhite) - - MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn - jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu) - - MAPREDUCE-4002. MultiFileWordCount job fails if the input path is not - from default file system. (Bhallamudi Venkata Siva Kamesh via todd) - - MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta. - (todd via bobby) - - MAPREDUCE-4262. NM gives wrong log message saying "Connected to - ResourceManager" before trying to connect. (Devaraj K via tgraves) - - MAPREDUCE-4276. Allow setting yarn.nodemanager.delete.debug-delay-sec - property to "-1" for easier container debugging. (ahmed via tucu) - - MAPREDUCE-4224. TestFifoScheduler throws - org.apache.hadoop.metrics2.MetricsException (Devaraj K via tgraves) - - MAPREDUCE-3493. Add the default mapreduce.shuffle.port property - to mapred-default.xml (Madhukara Phatak via harsh) - - MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize() - without a Path - Failure when using ViewFileSystem. (Ahmed Radwan via eli) - - MAPREDUCE-4313. TestTokenCache doesn't compile due - TokenCache.getDelegationToken compilation error (bobby) - - MAPREDUCE-3873. Fixed NodeManagers' decommissioning at RM to accept IP - addresses also. (xieguiming via vinodkv) - - MAPREDUCE-4306. Fix distributed shell to work with users other than the one - running the daemons. (Ahmed Radwan via sseth) - - MAPREDUCE-4031. Prevent a Node Manager hang during shutdown. - (Devaraj K via sseth) - - MAPREDUCE-4336. Distributed Shell fails when used with the CapacityScheduler - (ahmed via tucu) - - MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion. - (Devaraj K via sseth) - - MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem - (ahmed via tucu) - - MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) - - MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby) - - MAPREDUCE-4383. HadoopPipes.cc needs to include unistd.h. - (Andy Isaacson via eli) - - MAPREDUCE-2739. Update installation docs (remove YarnClientFactory) (bowang via tucu) - - MAPREDUCE-3993. Graceful handling of codec errors during decompression - (kkambatl via tucu) - - MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) - - MAPREDUCE-4441. Fix build issue caused by MR-3451 (kkambatl via tucu) - - HADOOP-8499. Lower min.user.id to 500 for the tests. - (Colin Patrick McCabe via eli) - - MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager - #determineTimestamps (Bhallamudi via bobby) - - MAPREDUCE-4380. Empty Userlogs directory is getting created under logs - directory (Devaraj K via bobby) - - MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume - HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. (vinodkv via - acmurthy) - -Release 2.0.0-alpha - 05-23-2012 - - INCOMPATIBLE CHANGES - - MAPREDUCE-3545. Remove Avro RPC. (suresh) - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-2887. Due to HADOOP-7524, change RPC to allow multiple protocols - including multuple versions of the same protocol (Sanjay Radia) - - MAPREDUCE-2934. MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup - process (atm) - - HADOOP-7862. MR changes to work with HADOOP 7862: Move the support for - multiple protocols to lower layer so that Writable, PB and Avro can all - use it (Sanjay Radia) - - MAPREDUCE-3909 Javadoc the Service interfaces (stevel) - - MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in - MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth) - - MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files - splitting. (harsh) - - MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley - via acmurthy) - - MAPREDUCE-3970. Add ServiceOperations class to aid working with Services - (stevel) - - MAPREDUCE-3353. Add a channel between RM and AM to get information on - nodes. (Bikas Saha via acmurthy) - - MAPREDUCE-3955. Change MR to use ProtobufRpcEngine from hadoop-common - instead of ProtoOverHadoopRpcEngine. (Jitendra Nath Pandey via sseth) - - MAPREDUCE-4103. Fix HA docs for changes to shell command fencer args (todd) - - MAPREDUCE-4093. Improve RM WebApp start up when proxy address is not set - (Devaraj K vai bobby) - - MAPREDUCE-4138. Reduce memory usage of counters due to non-static nested - classes. (tomwhite) - - MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec - configuration property (Eugene Koontz via tgraves) - - MAPREDUCE-4219. make default container-executor.conf.dir be a path - relative to the container-executor binary. (rvs via tucu) - - MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager - (tucu) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-3740. Fixed broken mapreduce compilation after the patch for - HADOOP-7965. (Devaraj K via vinodkv) - - MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch - for HDFS-2895. (Suresh Srinivas via vinodkv) - - MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas - Graves via mahadev) - - MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu) - - MAPREDUCE-3728. ShuffleHandler can't access results when configured in a - secure mode (ahmed via tucu) - - MAPREDUCE-3952. In MR2, when Total input paths to process == 1, - CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu) - - MAPREDUCE-3578. Starting nodemanager as root gives "Unknown -jvm option" - (tomwhite) - - MAPREDUCE-3348. Fixed a bug in MR client to redirect to JobHistoryServer - correctly when RM forgets the app. (Devaraj K via vinodkv) - - MAPREDUCE-3974. TestSubmitJob in MR1 tests doesn't compile after HDFS-162 - merge. (atm) - - MAPREDUCE-4007. JobClient getJob(JobID) should return NULL if the job - does not exist (for backwards compatibility) (tucu) - - MAPREDUCE-3431 NPE in Resource Manager shutdown. (stevel) - - MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) - - MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response - (todd) - - MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR - (xieguiming via harsh) - - MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs - is called prior to copying job.xml. (Jane Chen via acmurthy) - - MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via - bobby) - - MAPREDUCE-4082. hadoop-mapreduce-client-app's mrapp-generated-classpath - file should not be in the module JAR (tucu) - - MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu) - - MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu) - - MAPREDUCE-4095. TestJobInProgress#testLocality uses a bogus topology. - (Colin Patrick McCabe via eli) - - MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu) - - MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath - file in classpath (rvs via tucu) - - MAPREDUCE-4113. Fix tests org.apache.hadoop.mapred.TestClusterMRNotification - (Devaraj K via bobby) - - MAPREDUCE-4112. Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase - (Devaraj K via bobby) - - MAPREDUCE-4111. Fix tests in org.apache.hadoop.mapred.TestJobName (Devaraj - K via bobby) - - MAPREDUCE-4110. Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath & - org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers (Devaraj K via - bobby) - - MAPREDUCE-4105. Yarn RackResolver ignores rack configurations. - (Ahmed Radwan via tomwhite) - - MAPREDUCE-3869. Fix classpath for DistributedShell application. (Devaraj K - via sseth) - - MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. (Devaraj K - via szetszwo) - - MAPREDUCE-4076. Stream job fails with ZipException when use yarn jar - command (Devaraj K via bobby) - - MAPREDUCE-4108. Fix tests in org.apache.hadoop.util.TestRunJar - (Devaraj K via tgraves) - - MAPREDUCE-4107. Fix tests in org.apache.hadoop.ipc.TestSocketFactory - (Devaraj K via tgraves) - - MAPREDUCE-4147. YARN should not have a compile-time dependency on HDFS. - (tomwhite) - - MAPREDUCE-4008. ResourceManager throws MetricsException on start up - saying QueueMetrics MBean already exists (Devaraj K via tgraves) - - MAPREDUCE-3867. MiniMRYarn/MiniYarn uses fixed ports (tucu) - - MAPREDUCE-4141. clover integration broken, also mapreduce poms are - pulling in clover as a dependency. (phunt via tucu) - - MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. - (phunt via tomwhite) - - MAPREDUCE-4202. TestYarnClientProtocolProvider is broken (Daryn Sharp via - bobby) - - MAPREDUCE-3173. MRV2 UI doesn't work properly without internet (Devaraj K - via bobby) - - MAPREDUCE-3958. RM: Remove RMNodeState and replace it with NodeState - (Bikas Saha via bobby) - - MAPREDUCE-4231. Update RAID to use the new BlockCollection interface. - (szetszwo) - - MAPREDUCE-4483. 2.0 build does not work (John George via bobby) - - MAPREDUCE-4444. nodemanager fails to start when one of the local-dirs is - bad (Jason Lowe via bobby) - -Release 0.23.10 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5640. Rename TestLineRecordReader in jobclient module (Jason Lowe - via jeagles) - - MAPREDUCE-3191. docs for map output compression incorrectly reference - SequenceFile (Chen He via jeagles) - - OPTIMIZATIONS - - MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus - (Hairong Kuang and Jason Lowe via jlowe) - - BUG FIXES - - MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the - input path dir (Devaraj K via jlowe) - - MAPREDUCE-5380. Invalid mapred command should return non-zero exit code - (Stephen Chu via jlowe) - - MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via - jlowe) - - MAPREDUCE-5251. Reducer should not implicate map attempt if it has - insufficient space to fetch map output (Ashwin Shankar via jlowe) - - MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker - via jlowe) - - MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe) - - MAPREDUCE-5001. LocalJobRunner has race condition resulting in job - failures (Sandy Ryza via jlowe) - - MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe) - - MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta - via tgraves) - - MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert - Parker via jlowe) - - MAPREDUCE-5586. TestCopyMapper#testCopyFailOnBlockSizeDifference fails when - run from hadoop-tools/hadoop-distcp directory (jeagles) - - MAPREDUCE-5587. TestTextOutputFormat fails on JDK7 (jeagles) - - MAPREDUCE-5373. TestFetchFailure.testFetchFailureMultipleReduces could fail - intermittently (jeagles) - -Release 0.23.9 - 2013-07-08 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5084. fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and - org.apache.hadoop.mapreduce.v2.hs.webapp (Aleksey Gorshkov via jeagles) - - OPTIMIZATIONS - - MAPREDUCE-5268. Improve history server startup performance (Karthik - Kambatla via jlowe) - - BUG FIXES - - MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching - multiple compressed map outputs (Nathan Roberts via jlowe) - - MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe - via daryn) - - MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar, - Devaraj K, and B Anil Kumar via jlowe) - - MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal - task-state (Ashwin Shankar via jlowe) - -Release 0.23.8 - 2013-06-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes - are different on source/target (Mithun Radhakrishnan via kihwal) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI - (Aleksey Gorshkov via tgraves) - - MAPREDUCE-5147. Maven build should create - hadoop-mapreduce-client-app-VERSION.jar directly (Robert Parker via tgraves) - - MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific - counters page for failed job. (Ashwin Shankar via jlowe) - -Release 0.23.7 - 2013-04-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes - (Aleksey Gorshkov via bobby) - - MAPREDUCE-4989. JSONify DataTables input data for Attempts page (Ravi - Prakash via jlowe) - - MAPREDUCE-5027. Shuffle does not limit number of outstanding connections - (Robert Parker via jeagles) - - MAPREDUCE-4972. Coverage fixing for org.apache.hadoop.mapreduce.jobhistory - (Aleksey Gorshkov via bobby) - - OPTIMIZATIONS - - MAPREDUCE-4946. Fix a performance problem for large jobs by reducing the - number of map completion event type conversions. (Jason Lowe via sseth) - - MAPREDUCE-4822. Unnecessary conversions in History Events. (Chu Tong via - jlowe) - - BUG FIXES - - MAPREDUCE-4458. Warn if java.library.path is used for AM or Task - (Robert Parker via jeagles) - - MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with - speculative attempts (Robert Parker via jlowe) - - MAPREDUCE-5009. Killing the Task Attempt slated for commit does not clear - the value from the Task commitAttempt member (Robert Parker via jeagles) - - MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but - mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via - jeagles) - - MAPREDUCE-4794. DefaultSpeculator generates error messages on normal - shutdown (Jason Lowe via jeagles) - - MAPREDUCE-5043. Fetch failure processing can cause AM event queue to - backup and eventually OOM (Jason Lowe via bobby) - - MAPREDUCE-5023. History Server Web Services missing Job Counters (Ravi - Prakash via tgraves) - - MAPREDUCE-5060. Fetch failures that time out only count against the first - map task (Robert Joseph Evans via jlowe) - - MAPREDUCE-5042. Reducer unable to fetch for a map task that was recovered - (Jason Lowe via bobby) - - MAPREDUCE-5053. java.lang.InternalError from decompression codec cause - reducer to fail (Robert Parker via jeagles) - - MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves) - - MAPREDUCE-5007. fix coverage org.apache.hadoop.mapreduce.v2.hs (Aleksey - Gorshkov via tgraves) - - MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error - (Thomas Graves via jlowe) - -Release 0.23.6 - 2013-02-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-4811. JobHistoryServer should show when it was started in WebUI - About page (Ravi Prakash via jlowe) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-4802. Takes a long time to load the task list on the AM for - large jobs (Ravi Prakash via bobby) - - MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby) - - MAPREDUCE-4825. JobImpl.finished doesn't expect ERROR as a final job state - (jlowe via bobby) - - MAPREDUCE-4817. Hardcoded task ping timeout kills tasks localizing large - amounts of data (tgraves) - - MAPREDUCE-4836. Elapsed time for running tasks on AM web UI tasks page is 0 - (Ravi Prakash via jeagles) - - MAPREDUCE-4842. Shuffle race can hang reducer (Mariappan Asokan via jlowe) - - MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP (Robert - Parker via jlowe) - - MAPREDUCE-4793. Problem with adding resources when using both -files and - -file to hadoop streaming (jlowe) - - MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while - speculating (jlowe) - - MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output - (Albert Chu via jlowe) - - MAPREDUCE-4813. AM timing out during job commit (jlowe via bobby) - - MAPREDUCE-4279. getClusterStatus() fails with null pointer exception when - running jobs in local mode (Devaraj K via bobby) - - MAPREDUCE-4832. MR AM can get in a split brain situation (jlowe) - - MAPREDUCE-4894. Renewal / cancellation of JobHistory tokens (Siddharth - Seth via tgraves) - - MAPREDUCE-4819. AM can rerun job after reporting final job status to the - client (bobby and Bikas Saha via bobby) - - MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally - exits (Jason Lowe via tgraves) - - MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery (Jerry - Chen via jlowe) - - MAPREDUCE-4921. JobClient should acquire HS token with RM principal - (daryn via bobby) - - MAPREDUCE-4934. Maven RAT plugin is not checking all source files (tgraves) - - MAPREDUCE-4678. Running the Pentomino example with defaults throws - java.lang.NegativeArraySizeException (Chris McConnell via harsh) - - MAPREDUCE-4925. The pentomino option parser may be buggy. - (Karthik Kambatla via harsh) - -Release 0.23.5 - 2012-11-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-4596. Split StateMachine state from states seen by MRClientProtocol - for Job, Task and TaskAttempt. (Siddarth Seth via vinodkv) - - MAPREDUCE-4752. Reduce MR AM memory usage through String Interning (Robert - Evans via tgraves) - - MAPREDUCE-4266. remove Ant remnants from MR (tgraves via bobby) - - MAPREDUCE-4666. JVM metrics for history server (jlowe via jeagles) - - OPTIMIZATIONS - - MAPREDUCE-4720. Browser thinks History Server main page JS is taking too - long (Ravi Prakash via bobby) - - BUG FIXES - - MAPREDUCE-4554. Job Credentials are not transmitted if security is turned - off (Benoy Antony via bobby) - - MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs - inaccessible despite the presence of a valid history file. (Jason Lowe - via sseth) - - MAPREDUCE-4521. mapreduce.user.classpath.first incompatibility with 0.20/1.x - (Ravi Prakash via bobby) - - MAPREDUCE-4721. Task startup time in JHS is same as job startup time. - (Ravi Prakash via bobby) - - MAPREDUCE-4479. Fix parameter order in assertEquals() in - TestCombineInputFileFormat.java (Mariappan Asokan via bobby) - - MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many - reducers complete consecutively. (Jason Lowe via vinodkv) - - MAPREDUCE-4740. only .jars can be added to the Distributed Cache - classpath. (Robert Joseph Evans via jlowe) - - MAPREDUCE-4229. Intern counter names in the JT (Miomir Boljanovic and bobby via daryn) - - MAPREDUCE-4741. WARN and ERROR messages logged during normal AM shutdown. - (Vinod Kumar Vavilapalli via jlowe) - - MAPREDUCE-4730. Fix Reducer's EventFetcher to scale the map-completion - requests slowly to avoid HADOOP-8942. (Jason Lowe via vinodkv) - - MAPREDUCE-4748. Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED. (jlowe) - - MAPREDUCE-4724. job history web ui applications page should be sorted to - display last app first (tgraves via bobby) - - MAPREDUCE-4746. The MR Application Master does not have a config to set - environment variables (Rob Parker via bobby) - - MAPREDUCE-4729. job history UI not showing all job attempts. (Vinod - Kumar Vavilapalli via jlowe) - - MAPREDUCE-4763 repair test TestUmbilicalProtocolWithJobToken (Ivan A. - Veselovsky via bobby) - - MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when - configured (jlowe via bobby) - - MAPREDUCE-4772. Fetch failures can take way too long for a map to be - restarted (bobby) - - MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit - (Mark Fuhs via bobby) - - MAPREDUCE-4774. JobImpl does not handle asynchronous task events in FAILED - state (jlowe via bobby) - - MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby) - - MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby) - - MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe - via bobby) - - MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by - default (Ravi Prakash via bobby) - - MAPREDUCE-4517. Too many INFO messages written out during AM to RM heartbeat - (Jason Lowe via tgraves) - - MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact - the RM (jlowe via bobby) - - MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely - closed channels (jlowe via bobby) - -Release 0.23.4 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-2786. Add compression option for TestDFSIO. - (Plamen Jeliazkov via shv) - - MAPREDUCE-4645. Provide a random seed to Slive to make the sequence - of file names deterministic. (Ravi Prakash via shv) - - MAPREDUCE-4651. Benchmarking random reads with DFSIO. (shv) - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-4647. We should only unjar jobjar if there is a lib directory - in it. (Robert Evans via tgraves) - - MAPREDUCE-4691. Historyserver can report "Unknown job" after RM says job - has completed (Robert Joseph Evans via jlowe) - - MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE - (jlowe via bobby) - -Release 0.23.3 - - INCOMPATIBLE CHANGES - - MAPREDUCE-4072. User set java.library.path seems to overwrite default - creating problems native lib loading (Anupam Seth via bobby) - - MAPREDUCE-3812. Lower default allocation sizes, fix allocation - configurations and document them (Harsh J via bobby) - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-4059. The history server should have a separate pluggable - storage/query interface. (Robert Evans via tgraves) - - MAPREDUCE-3942. Randomize master key generation for - ApplicationTokenSecretManager and roll it every so often. (Vinod Kumar - Vavilapalli via sseth) - - MAPREDUCE-4151. RM scheduler web page should filter apps to those that - are relevant to scheduling (Jason Lowe via tgraves) - - MAPREDUCE-4134. Remove references of mapred.child.ulimit etc. since they - are not being used any more (Ravi Prakash via bobby) - - MAPREDUCE-3972. Fix locking and exception issues in JobHistory server. - (Robert Joseph Evans via sseth) - - MAPREDUCE-4161. create sockets consistently (Daryn Sharp via bobby) - - MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range. - (bobby via tgraves) - - MAPREDUCE-4210. Expose listener address for WebApp (Daryn Sharp via bobby) - - MAPREDUCE-4162. Correctly set token service (Daryn Sharp via bobby) - - MAPREDUCE-4301. Dedupe some strings in MRAM for memory savings - (bobby via tgraves) - - MAPREDUCE-4267. mavenize pipes (tgraves via bobby) - - MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby - via tgraves) - - MAPREDUCE-4569. Fixed TestHsWebServicesJobsQuery to pass on JDK7 by not - depending on test order. (Thomas Graves via vinodkv) - - OPTIMIZATIONS - - MAPREDUCE-3850. Avoid redundant calls for tokens in TokenCache (Daryn - Sharp via bobby) - - BUG FIXES - - MAPREDUCE-4092. commitJob Exception does not fail job (Jon Eagles via - bobby) - - MAPREDUCE-4089. Hung Tasks never time out. (Robert Evans via tgraves) - - MAPREDUCE-4024. RM webservices can't query on finalStatus (Tom Graves - via bobby) - - MAPREDUCE-4060. Multiple SLF4J binding warning (Jason Lowe via bobby) - - MAPREDUCE-3983. TestTTResourceReporting can fail, and should just be - deleted (Ravi Prakash via bobby) - - MAPREDUCE-4012 Hadoop Job setup error leaves no useful info to users - (when LinuxTaskController is used). (tgraves) - - MAPREDUCE-4062. AM Launcher thread can hang forever (tgraves via bobby) - - MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory - on a node. (Eric Payne via bobby) - - MAPREDUCE-3999. Tracking link gives an error if the AppMaster hasn't - started yet (Ravi Prakash via bobby) - - MAPREDUCE-4020. Web services returns incorrect JSON for deep queue tree - (Anupam Seth via tgraves) - - MAPREDUCE-3672. Killed maps shouldn't be counted towards - JobCounter.NUM_FAILED_MAPS. (Anupam Seth via tgraves) - - MAPREDUCE-3682 Tracker URL says AM tasks run on localhost. - (Ravi Prakash via tgraves) - - MAPREDUCE-3082. Archive command take wrong path for input file with current - directory (John George via bobby) - - MAPREDUCE-3650. testGetTokensForHftpFS() fails (Ravi Prakash via bobby) - - MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail - (Ravi Prakash via tgraves) - - MAPREDUCE-4073. CS assigns multiple off-switch containers when using - multi-level-queues (Siddharth Seth via bobby) - - MAPREDUCE-4051. Remove the empty hadoop-mapreduce-project/assembly/all.xml - file (Ravi Prakash via bobby) - - MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K - via bobby) - - MAPREDUCE-4099. ApplicationMaster may fail to remove staging directory - (Jason Lowe via bobby) - - MAPREDUCE-4017. Add jobname to jobsummary log (tgraves and Koji Noguchi - via bobby) - - MAPREDUCE-4040. History links should use hostname rather than IP address. - (Bhallamudi Venkata Siva Kamesh via sseth) - - MAPREDUCE-4099 amendment. ApplicationMaster will remove staging directory - after the history service is stopped. (Jason Lowe via sseth) - - MAPREDUCE-3932. Fix the TaskAttempt state machine to handle - CONTIANER_LAUNCHED and CONTIANER_LAUNCH_FAILED events in additional - states. (Robert Joseph Evans via sseth) - - MAPREDUCE-4140. mapreduce classes incorrectly importing - "clover.org.apache.*" classes. (Patrick Hunt via tomwhite) - - MAPREDUCE-4050. For tasks without assigned containers, changes the node - text on the UI to N/A instead of a link to null. (Bhallamudi Venkata Siva - Kamesh via sseth) - - MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to - also be completed. (Bikas Saha via bobby) - - MAPREDUCE-4144. Fix a NPE in the ResourceManager when handling node - updates. (Jason Lowe via sseth) - - MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves) - - MAPREDUCE-4160. some mrv1 ant tests fail with timeout - due to 4156 - (tgraves) - - MAPREDUCE-4074. Client continuously retries to RM When RM goes down - before launching Application Master (xieguiming via tgraves) - - MAPREDUCE-4159. Job is running in Uber mode after setting - "mapreduce.job.ubertask.maxreduces" to zero (Devaraj K via bobby) - - MAPREDUCE-4165. Committing is misspelled as commiting in task logs - (John Eagles via bobby) - - MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via - bobby) - - MAPREDUCE-3947. yarn.app.mapreduce.am.resource.mb not documented - (Devaraj K via bobby) - - MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves - via bobby) - - MAPREDUCE-4133. MR over viewfs is broken (John George via bobby) - - MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection - (Jonathan Eagles via bobby) - - MAPREDUCE-3613. web service calls header contains 2 content types - (tgraves) - - MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles - via bobby) - - MAPREDUCE-4189. TestContainerManagerSecurity is failing (Devaraj K via - bobby) - - MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing - scope test (Radim Kolar via bobby) - - MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page sorts - does not work correctly (Jonathon Eagles via tgraves) - - MAPREDUCE-4212. TestJobClientGetJob sometimes fails - (Daryn Sharp via tgraves) - - MAPREDUCE-4211. Error conditions (missing appid, appid not found) are - masked in the RM app page (Jonathan Eagles via bobby) - - MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via bobby) - - MAPREDUCE-4048. NullPointerException exception while accessing the - Application Master UI (Devaraj K via bobby) - - MAPREDUCE-4220. RM apps page starttime/endtime sorts are incorrect - (Jonathan Eagles via bobby) - - MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup. - (tomwhite) - - MAPREDUCE-4215. RM app page shows 500 error on appid parse error - (Jonathon Eagles via tgraves) - - MAPREDUCE-4237. TestNodeStatusUpdater can fail if localhost has a domain - associated with it (bobby) - - MAPREDUCE-4233. NPE can happen in RMNMNodeInfo. (bobby) - - MAPREDUCE-4238. mavenize data_join. (tgraves) - - MAPREDUCE-4102. job counters not available in Jobhistory webui for - killed jobs (Bhallamudi Venkata Siva Kamesh via tgraves) - - MAPREDUCE-3543. Mavenize Gridmix. (tgraves) - - MAPREDUCE-4197. Include the hsqldb jar in the hadoop-mapreduce tar - file (Ravi Prakash via tgraves) - - MAPREDUCE-4269. documentation: Gridmix has javadoc warnings in - StressJobFactory (Jonathon Eagles via tgraves). - - MAPREDUCE-3870. Invalid App Metrics - (Bhallamudi Venkata Siva Kamesh via tgraves). - - MAPREDUCE-4152. map task left hanging after AM dies trying to connect to RM - (Tom Graves via bobby) - - MAPREDUCE-4297. Usersmap file in gridmix should not fail on empty lines - (Ravi Prakash via bobby) - - MAPREDUCE-4302. NM goes down if error encountered during log aggregation - (Daryn Sharp via bobby) - - MAPREDUCE-3350. Per-app RM page should have the list of application-attempts - like on the app JHS page (Jonathon Eagles via tgraves) - - MAPREDUCE-3842. Stop webpages from automatic refreshing (tgraves) - - MAPREDUCE-3927. Shuffle hang when set map.failures.percent - (Bhallamudi Venkata Siva Kamesh via tgraves) - - MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for - capacity and maximum-capacity settings (Karthik Kambatla via tgraves) - - MAPREDUCE-4341. add types to capacity scheduler properties documentation - (Karthik Kambatla via tgraves) - - MAPREDUCE-4270. Move the data_join test classes to the correct path. - (Thomas Graves via sseth) - - MAPREDUCE-3889. job client tries to use /tasklog interface, but that - doesn't exist anymore (Devaraj K via bobby) - - MAPREDUCE-4320. gridmix mainClass wrong in pom.xml (tgraves) - - MAPREDUCE-4295. RM crashes due to DNS issue (tgraves) - - MAPREDUCE-4228. mapreduce.job.reduce.slowstart.completedmaps is not working - properly (Jason Lowe via bobby) - - MAPREDUCE-4392. Counters.makeCompactString() changed behavior from 0.20 - (Jason Lowe via bobby) - - MAPREDUCE-4384. Race conditions in IndexCache (Kihwal Lee via tgraves) - - MAPREDUCE-4387. RM gets fatal error and exits during TestRM - (Kihwal Lee via tgraves) - - MAPREDUCE-4379. Node Manager throws java.lang.OutOfMemoryError: Java heap - space due to org.apache.hadoop.fs.LocalDirAllocator.contexts (Devaraj K - via bobby) - - MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via - bobby) - - MAPREDUCE-4300. OOM in AM can turn it into a zombie. (Robert Evans via - tgraves) - - MAPREDUCE-4252. MR2 job never completes with 1 pending task (Tom White via - bobby) - - MAPREDUCE-3940. ContainerTokens should have an expiry interval. (Siddharth - Seth and Vinod Kumar Vavilapalli via vinodkv) - - MAPREDUCE-4419. ./mapred queue -info -showJobs displays all - the jobs irrespective of (Devaraj K via bobby) - - MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via - bobby) - - MAPREDUCE-4437. Race in MR ApplicationMaster can cause reducers to never be - scheduled (Jason Lowe via bobby) - - MAPREDUCE-4449. Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig - (Ahmed Radwan via bobby) - - MAPREDUCE-4283. Display tail of aggregated logs by default (Jason Lowe via - bobby) - - MAPREDUCE-4448. Fix NM crash during app cleanup if aggregation didn't - init. (Jason Lowe via daryn) - - MAPREDUCE-3893. allow capacity scheduler configs maximum-applications and - maximum-am-resource-percent configurable on a per queue basis (tgraves via - bobby) - - MAPREDUCE-4467. IndexCache failures due to missing synchronization - (Kihwal Lee via tgraves) - - MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans - via tgraves) - - MAPREDUCE-4456. LocalDistributedCacheManager can get an - ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves) - - MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby) - - MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans - via tgraves) - - MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at - perticular level is sucessfull (Mayank Bansal via bobby) - - MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at - FAILED (Robert Evans via tgraves) - - MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or - parallel use (due to a /tmp file with static name) (Robert Evans via - jeagles) - - MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans - via tgraves) - - MAPREDUCE-4503. Should throw InvalidJobConfException if duplicates found in - cacheArchives or cacheFiles (Robert Evans via jeagles) - - MAPREDUCE-3782. teragen terasort jobs fail when using webhdfs:// (Jason - Lowe via bobby) - - MAPREDUCE-4053. Counters group names deprecation is wrong, iterating over - group names deprecated names don't show up (Robert Evans via tgraves) - - MAPREDUCE-3506. Calling getPriority on JobInfo after parsing a history log - with JobHistoryParser throws a NullPointerException (Jason Lowe via bobby) - - MAPREDUCE-4570. ProcfsBasedProcessTree#constructProcessInfo() prints a - warning if procfsDir//stat is not found. (Ahmed Radwan via bobby) - - MAPREDUCE-4600. TestTokenCache.java from MRV1 no longer compiles (daryn - via bobby) - - MAPREDUCE-4612. job summary file permissions not set when its created - (tgraves via bobby) - - MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby) - - MAPREDUCE-4611. MR AM dies badly when Node is decommissioned (Robert - Evans via tgraves) - - MAPREDUCE-4604. In mapred-default, mapreduce.map.maxattempts & - mapreduce.reduce.maxattempts defaults are set to 4 as well as - mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles) - - MAPREDUCE-4633. history server doesn't set permissions on all subdirs - (tgraves via bobby) - - MAPREDUCE-4641. Exception in commitJob marks job as successful in job - history (Jason Lowe via bobby) - -Release 0.23.2 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - MAPREDUCE-3849. Change TokenCache's reading of the binary token file - (Daryn Sharp via bobby) - - MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's - environmental variables in TestMiniMRChildTask. (Tom White via vinodkv) - - MAPREDUCE-3877 Add a test to formalise the current state transitions - of the yarn lifecycle. (stevel) - - MAPREDUCE-3866. Fixed the bin/yarn script to not print the command line - unnecessarily. (vinodkv) - - MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the - cluster without waiting for expiry. (Jason Lowe via vinodkv) - - MAPREDUCE-2793. Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct - format on the web pages. (Bikas Saha via vinodkv) - - MAPREDUCE-3614. Fixed MR AM to close history file quickly and send a correct - final state to the RM when it is killed. (Ravi Prakash via vinodkv) - - MAPREDUCE-3497. Added docs for YARN CLI. (tgraves via acmurthy) - - MAPREDUCE-3954. Added new envs to separate heap size for different daemons - started via bin scripts. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value - (Jason Lowe via bobby) - - MAPREDUCE-4034. Unable to view task logs on history server with - mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby) - - MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks - (Jason Lowe via bobby) - - MAPREDUCE-3989. Cap space usage of default log4j rolling policy. - (Patrick Hunt via eli) - - OPTIMIZATIONS - - MAPREDUCE-3901. Modified JobHistory records in YARN to lazily load job and - task reports so as to improve UI response times. (Siddarth Seth via vinodkv) - - MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to - minimize counter names lookup time. (Siddarth Seth via vinodkv) - - MAPREDUCE-3944. Change the history jobs/ webservice to return partial job - info for a significant performance improvement. (Robert Joseph Evans via - sseth) - - BUG FIXES - MAPREDUCE-3918 proc_historyserver no longer in command line arguments for - HistoryServer (Jon Eagles via bobby) - - MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering - DeletionService threads (Jason Lowe via bobby) - - MAPREDUCE-3680. FifoScheduler web service rest API can print out invalid - JSON. (B Anil Kumar via tgraves) - - MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas - Graves via mahadev) - - MAPREDUCE-3736. Variable substitution depth too large for fs.default.name - causes jobs to fail (ahmed via tucu). - - MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode - HTTPS parameters. (todd) - - MAPREDUCE-3583. Change pid to String and stime to BigInteger in order to - avoid NumberFormatException caused by overflow. (Zhihong Yu via szetszwo) - - MAPREDUCE-3634. Fixed all daemons to crash instead of hanging around when - their EventHandlers get exceptions. (vinodkv) - - MAPREDUCE-3798. Fixed failing TestJobCleanup.testCusomCleanup() and moved it - to the maven build. (Ravi Prakash via vinodkv) - - MAPREDUCE-3884. PWD should be first in the classpath of MR tasks (tucu) - - MAPREDUCE-3878. Null user on filtered jobhistory job page (Jonathon Eagles - via tgraves) - - MAPREDUCE-3738. MM can hang during shutdown if AppLogAggregatorImpl thread - dies unexpectedly (Jason Lowe via sseth) - - MAPREDUCE-3904 Job history produced with mapreduce.cluster.acls.enabled - false can not be viewed with mapreduce.cluster.acls.enabled true - (Jonathon Eagles via tgraves) - - MAPREDUCE-3910. Fixed a bug in CapacityScheduler LeafQueue which was causing - app-submission to fail. (John George via vinodkv) - - MAPREDUCE-3686. Fixed two bugs in Counters because of which web app displays - zero counter values for framework counters. (Bhallamudi Venkata Siva Kamesh - via vinodkv) - - MAPREDUCE-3913. RM application webpage is unresponsive after 2000 jobs - (Jason Lowe via tgraves) - - MAPREDUCE-3922. Fixed build to not compile 32bit container-executor binary - by default on all platforms. (Hitesh Shah via vinodkv) - - MAPREDUCE-3790. Broken pipe on streaming job can lead to truncated output for - a successful job (Jason Lowe via bobby) - - MAPREDUCE-3816. capacity scheduler web ui bar graphs for used capacity wrong - (tgraves via bobby) - - MAPREDUCE-3930. Fixed an NPE while accessing the AM page/webservice for a - task attempt without an assigned container. (Robert Joseph Evans via - sseth) - - MAPREDUCE-3931. Changed PB implementation of LocalResource to take locks - so that race conditions don't fail tasks by inadvertantly changing the - timestamps. (Siddarth Seth via vinodkv) - - MAPREDUCE-3687. If AM dies before it returns new tracking URL, proxy - redirects to http://N/A/ and doesn't return error code (Ravi Prakash via - bobby) - - MAPREDUCE-3920. Revise yarn default port number selection - (Dave Thompson via tgraves) - - MAPREDUCE-3903. Add support for mapreduce admin users. (Thomas Graves via - sseth) - - MAPREDUCE-3706. Fix circular redirect error in job-attempts page. (bobby - via acmurthy) - - MAPREDUCE-3896. Add user information to the delegation token issued by the - history server. (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-3792. Fix "bin/mapred job -list" to display all jobs instead of - only the jobs owned by the user. (Jason Lowe via vinodkv) - - MAPREDUCE-3929. Fixed output of 'bin/mapred queue -showacl' command to - clarify ACLs for users. (John George via acmurthy) - - MAPREDUCE-3960. Fix web-proxy to forward request to AM with configured - hostname or IP. (tgraves via acmurthy) - - MAPREDUCE-3897. Fixed computation of maxActiveAppsPerUser for queues by - using capacity and not max-capacity since we are already scaling it by - userLimitFactor. (Eric Payne via acmurthy) - - MAPREDUCE-3009. Fixed node link on JobHistory webapp. (chackaravarthy via - vinodkv) - - MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via - bobby) - - MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from - ResourceManager. (Devaraj K & Eric Payne via acmurthy) - - MAPREDUCE-3976. TestRMContainerAllocator failing (Jason Lowe via bobby) - - MAPREDUCE-3961. Map/ReduceSlotMillis computation incorrect (Siddharth Seth - via bobby) - - MAPREDUCE-3977. LogAggregationService leaks log aggregator objects - (Jason Lowe via bobby) - - MAPREDUCE-3975. Default value not set for Configuration parameter - mapreduce.job.local.dir (Eric Payne via bobby) - - MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job' - whose tasks don't write any outputs. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager - Application Info page. (Jason Lowe via sseth) - - MAPREDUCE-4006. History server container log web UI sometimes combines - stderr/stdout/syslog contents together (Siddharth Seth via tgraves) - - MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) - -Release 0.23.1 - 2012-02-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev) - - MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas - Graves via vinodkv) - - MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster - (Anupam Seth via mahadev) - - IMPROVEMENTS - - MAPREDUCE-3297. Moved log related components into yarn-common so that - HistoryServer and clients can use them without depending on the - yarn-server-nodemanager module. (Siddharth Seth via vinodkv) - - MAPREDUCE-3336. Replaced guice internal.Preconditions api usage with the - public Preconditions API. (Thomas Graves via vinodkv) - - MAPREDUCE-3280. Removed the unnecessary job user-name configuration in - mapred-site.xml. (vinodkv) - - MAPREDUCE-3370. Fixed MiniMRYarnCluster and related tests to not use - a hard-coded path for the mr-app jar. (Ahmed Radwan via vinodkv) - - MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves - via mahadev) - - MAPREDUCE-3341. Enhance logging of initalized queue limit values. - (Anupam Seth via mahadev) - - MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles - via mahadev) - - MAPREDUCE-3331. Improvement to single node cluster setup documentation for - 0.23 (Anupam Seth via mahadev) - - MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor - has wrong configuration or permissions. (Hitesh Shah via vinodkv) - - MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution. - (tucu) - - MAPREDUCE-3373. Hadoop scripts unconditionally source - "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite) - - MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden. - (Bruno Mahé via tomwhite) - - MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via - mahadev) - - MAPREDUCE-3371. Review and improve the yarn-api javadocs. (Ravi Prakash - via mahadev) - - MAPREDUCE-3238. Small cleanup in SchedulerApp. (Todd Lipcon via mahadev) - - MAPREDUCE-3413. RM web ui applications not sorted in any order by default. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3045. Fixed UI filters to not filter on hidden title-numeric - sort fields. (Jonathan Eagles via sseth) - - MAPREDUCE-3448. TestCombineOutputCollector javac unchecked warning on mocked - generics (Jonathan Eagles via mahadev) - - MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. (Ahmed Radwan via - sseth) - - MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces - introduced in MAPREDUCE-3169. (Ahmed Radwan via tomwhite) - - MAPREDUCE-3518. mapred queue -info -showJobs throws NPE. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3391. Making a trivial change to correct a log message in - DistributedShell app's AM. (Subroto Sanyal via vinodkv) - - MAPREDUCE-3547. Added a bunch of unit tests for the the RM/NM webservices. - (Thomas Graves via acmurthy) - - MAPREDUCE-3610. Remove use of the 'dfs.block.size' config for default block - size fetching. Use FS#getDefaultBlocksize instead. (Sho Shimauchi via harsh) - - MAPREDUCE-3478. Cannot build against ZooKeeper 3.4.0. (Tom White via mahadev) - - MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration - for the thread loop interval separate from task-timeout configuration - property. (Siddharth Seth via vinodkv) - - MAPREDUCE-3312. Modified MR AM to not send a stop-container request for - a container that isn't launched at all. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end - notification URL. (Ravi Prakash via vinodkv) - - MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the - job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv) - - MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR AM. - Improved the earlier patch to not to JobHistoryServer repeatedly. - (Anupam Seth via vinodkv) - - MAPREDUCE-3553. Add support for data returned when exceptions thrown from web - service apis to be in either xml or in JSON. (Thomas Graves via mahadev) - - MAPREDUCE-3641. Making CapacityScheduler more conservative so as to - assign only one off-switch container in a single scheduling - iteration. (Arun C Murthy via vinodkv) - - MAPREDUCE-3692. yarn-resourcemanager out and log files can get big. (eli) - - MAPREDUCE-3710. Improved FileInputFormat to return better locality for the - last split. (Siddarth Seth via vinodkv) - - MAPREDUCE-2765. DistCp Rewrite. (Mithun Radhakrishnan via mahadev) - - MAPREDUCE-3737. The Web Application Proxy's is not documented very well. - (Robert Evans via mahadev) - - MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable - values for working at scale. (Hitesh Shah via vinodkv) - - MAPREDUCE-3693. Added mapreduce.admin.user.env to mapred-default.xml. - (Roman Shapshonik via acmurthy) - - MAPREDUCE-3732. Modified CapacityScheduler to use only users with pending - requests for computing user-limits. (Arun C Murthy via vinodkv) - - MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1 - second. (Vinod KV via mahadev) - - MAPREDUCE-3754. Modified RM UI to filter applications based on state of the - applications. (vinodkv) - - MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from - hadoop-server-common. (Mahadev Konar via vinodkv) - - MAPREDUCE-3771. Un-deprecated the old mapred apis, port of MAPREDUCE-1735. - (acmurthy) - - MAPREDUCE-3784. Fixed CapacityScheduler so that maxActiveApplications and - maxActiveApplicationsPerUser per queue are not too low for small - clusters. (Arun C Murthy via vinodkv) - - OPTIMIZATIONS - - MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar - Vavilapalli via sseth) - - MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM - immediately after downloading a resource instead of always waiting for a - second. (Siddarth Seth via vinodkv) - - MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv) - - MAPREDUCE-3569. TaskAttemptListener holds a global lock for all - task-updates. (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-3511. Removed a multitude of cloned/duplicate counters in the AM - thereby reducing the AM heap size and preventing full GCs. (vinodkv) - - MAPREDUCE-3618. Fixed TaskHeartbeatHandler to not hold a global lock for all - task-updates. (Siddarth Seth via vinodkv) - - MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush - for every event slowing down AM. (Siddarth Seth via vinodkv) - - MAPREDUCE-3718. Change default AM heartbeat interval to 1 second. (Hitesh - Shah via sseth) - - MAPREDUCE-3360. Added information about lost/rebooted/decommissioned nodes - on the webapps. (Bhallamudi Venkata Siva Kamesh and Jason Lowe via vinodkv) - - MAPREDUCE-3756. Made single shuffle limit configurable. (Hitesh Shah via - acmurthy) - - MAPREDUCE-3811. Made jobclient-to-AM retries configurable. (sseth via - acmurthy) - - BUG FIXES - - MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob - and fixed bugs in it. (Devaraj K via vinodkv) - - MAPREDUCE-3215. Reenabled and fixed bugs in the failing test - TestNoJobSetupCleanup. (Hitesh Shah via vinodkv) - - MAPREDUCE-3219. Reenabled and fixed bugs in the failing test - TestDelegationToken. (Hitesh Shah via vinodkv) - - MAPREDUCE-3217. Reenabled and fixed bugs in the failing ant test - TestAuditLogger. (Devaraj K via vinodkv) - - MAPREDUCE-3291. App fail to launch due to delegation token not - found in cache (Robert Evans via mahadev) - - MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to - ReduceContext.ValueIterator. (Brock Noland via tomwhite) - - MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue - name. (Jonathan Eagles via vinodkv) - - MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing - TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv) - - MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to - which per-container connections to NodeManager were lingering long enough - to hit the ulimits on number of processes. (vinodkv) - - MAPREDUCE-3392. Fixed Cluster's getDelegationToken's API to return null - when there isn't a supported token. (John George via vinodkv) - - MAPREDUCE-3379. Fixed LocalResourceTracker in NodeManager to remove deleted - cache entries correctly. (Siddharth Seth via vinodkv) - - MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are - accessible through all UI servers (Jonathan Eagles via mahadev) - - MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command - timeouts correctly. (vinodkv) - - MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed - by the integration tests. (Hitesh Shah via vinodkv) - - MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev) - - MAPREDUCE-3447. mapreduce examples not working (mahadev) - - MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev) - - MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah - via mahadev) - - MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger - (Bruno Mahe via mahadev) - - MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot - be lesser than capacity for any queue. (acmurthy) - - MAPREDUCE-3464. mapreduce jsp pages missing DOCTYPE. (Dave Vronay via mattf) - - MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to - cut down noise. (acmurthy) - - MAPREDUCE-3433. Finding counters by legacy group name returns empty - counters. (tomwhite) - - MAPREDUCE-3450. NM port info no longer available in JobHistory. - (Siddharth Seth via mahadev) - - MAPREDUCE-3488. Streaming jobs are failing because the main class - isnt set in the pom files. (mahadev) - - MAPREDUCE-3463. Second AM fails to recover properly when first AM is killed with - java.lang.IllegalArgumentException causing lost job. (Siddharth Seth via mahadev) - - MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3443. JobClient and Job should function in the context of the - UGI which created them. (Mahadev Konar via sseth) - - MAPREDUCE-3460. MR AM can hang if containers are allocated on a node - blacklisted by the AM. (Hitesh Shah and Robert Joseph Evans via sseth) - - MAPREDUCE-3453. RM web ui application details page shows RM cluster about - information. (Jonathan Eagles via sseth) - - MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite) - - MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for - $HADOOP_*_HOME (Eric Payne via mahadev) - - MAPREDUCE-3458. Fix findbugs warnings in hadoop-examples. (Devaraj K - via mahadev) - - MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as - ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev) - - MAPREDUCE-3496. Fixed client to print queue acls in consistent order. - (Jonathan Eagles via acmurthy) - - MAPREDUCE-3147. Handle leaf queues with the same name properly. (Ravi Prakash via - mahadev) - - MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value - for queues (Anupam Seth via mahadev) - - MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory. - (chackaravarthy via mahadev) - - MAPREDUCE-3519. Fixed a deadlock in NodeManager LocalDirectories's handling - service. (Ravi Gummadi via vinodkv) - - MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23. - (tomwhite) - - MAPREDUCE-3328. mapred queue -list output inconsistent and missing child - queues. (Ravi Prakash via mahadev) - - MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue - -showacls (Jonathan Eagles via mahadev) - - MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led - to container localization occuring in wrong directories. (acmurthy) - - MAPREDUCE-3542. Support "FileSystemCounter" legacy counter group name for - compatibility. (tomwhite) - - MAPREDUCE-3426. Fixed MR AM in uber mode to write map intermediate outputs - in the correct directory to work properly in secure mode. (Hitesh Shah via - vinodkv) - - MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via - mahadev) - - MAPREDUCE-3398. Fixed log aggregation to work correctly in secure mode. - (Siddharth Seth via vinodkv) - - MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the - ResourceManager. (Arun C Murthy via vinodkv) - - MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing - all its retries. (Ravi Prakash via vinodkv) - - MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Robert Joseph - Evans via sseth) - - MAPREDUCE-3560. TestRMNodeTransitions is failing on trunk. - (Siddharth Seth via mahadev) - - MAPREDUCE-3487. Fixed JobHistory web-UI to display links to single task's - counters' page. (Jason Lowe via vinodkv) - - MAPREDUCE-3564. Fixed failures in TestStagingCleanup and TestJobEndNotifier - tests. (Siddharth Seth via vinodkv) - - MAPREDUCE-3422. Counter display names are not being picked up. (Jonathan - Eagles via sseth) - - MAPREDUCE-3366. Mapreduce component should use consistent directory structure - layout as HDFS/common (Eric Yang via mahadev) - - MAPREDUCE-3387. Fixed AM's tracking URL to always go through the proxy, even - before the job started, so that it works properly with oozie throughout - the job execution. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a url - without a port. (atm via harsh) - - MAPREDUCE-3563. Fixed LocalJobRunner to work correctly with new mapreduce - apis. (acmurthy) - - MAPREDUCE-3376. Fixed Task to ensure it passes reporter to combiners using - old MR api. (Subroto Sanyal via acmurthy) - - MAPREDUCE-3339. Fixed MR AM to stop considering node blacklisting after the - number of nodes blacklisted crosses a threshold. (Siddharth Seth via vinodkv) - - MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that - yarn daemons can start. (Arun C Murthy via vinodkv) - - MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations - thereby solving race conditions in MR AM shutdown. (vinodkv) - - MAPREDUCE-3604. Fixed streaming to use new mapreduce.framework.name to - check for local mode. (acmurthy) - - MAPREDUCE-3521. Fixed streaming to ensure it doesn't silently ignore - unknown arguments. (Robert Evans via acmurthy) - - MAPREDUCE-3522. Ensure queues inherit ACLs from parent if they aren't - explicitly specified. (Jonathan Eagles via acmurthy) - - MAPREDUCE-3608. Fixed compile issue with MAPREDUCE-3522. (mahadev via - acmurthy) - - MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce - ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv) - - MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when - adding a file/archive to the path. (Dick King via tucu) - - MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly - (sseth) - - MAPREDUCE-3595. Add missing TestCounters#testCounterValue test from branch - 1 to 0.23 (Tom White via sseth) - - MAPREDUCE-3566. Fixed MR AM to construct CLC only once across all tasks. - (vinodkv via acmurthy) - - MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for - performance reasons. (vinodkv via acmurthy) - - MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth) - - MAPREDUCE-3326. Added detailed information about queue's to the - CapacityScheduler web-ui. (Jason Lowe via acmurthy) - - MAPREDUCE-3548. Added more unit tests for MR AM & JHS web-services. - (Thomas Graves via acmurthy) - - MAPREDUCE-3617. Removed wrong default value for - yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan - Eagles via acmurthy) - - MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml - missing license header. (Hitesh Shah via tucu). - - MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu) - - MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu) - - MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off - by default. (tucu) - - MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. (jeagles via tucu) - - MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu) - - MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with - classpath from the build machine. (tucu) - - MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be added as - ivy dependency. (tucu) - - MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives dependency. - (tucu) - - MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev - via acmurthy) - - MAPREDUCE-3616. Thread pool for launching containers in MR AM not - expanding as expected. (vinodkv via sseth) - - MAPREDUCE-3639. Fixed TokenCache to work with absent FileSystem canonical - service-names. (Siddharth Seth via vinodkv) - - MAPREDUCE-3380. Token infrastructure for running clients which are not kerberos - authenticated. (mahadev) - - MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev) - - MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev) - - MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev) - - MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization - fails. (Thomas Graves via mahadev) - - MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken. - (Jason Lowe via mahadev) - - MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs - may subsequently report as running. (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort - benchmark consistently. (Siddarth Seth via vinodkv) - - MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral - web port is configured. (Bhallamudi Venkata Siva Kamesh via vinodkv) - - MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable - speculating either maps or reduces. (Eric Payne via vinodkv) - - MAPREDUCE-3649. Job End notification gives an error on calling back. - (Ravi Prakash via mahadev) - - MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe - via mahadev) - - MAPREDUCE-2450. Fixed a corner case with interrupted communication threads - leading to a long timeout in Task. (Rajesh Balamohan via acmurthy) - - MAPREDUCE-3669. Allow clients to talk to MR HistoryServer using both - delegation tokens and kerberos. (mahadev via acmurthy) - - MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread - pool (tomwhite) - - MAPREDUCE-3582. Move successfully passing MR1 tests to MR2 maven tree. - (ahmed via tucu) - - MAPREDUCE-3698. Client cannot talk to the history server in secure mode. - (mahadev) - - MAPREDUCE-3689. RM web UI doesn't handle newline in job name. - (Thomas Graves via mahadev) - - MAPREDUCE-3549. write api documentation for web service apis for RM, NM, - mapreduce app master, and job history server (Thomas Graves via mahadev) - - MAPREDUCE-3705. ant build fails on 0.23 branch. (Thomas Graves via - mahadev) - - MAPREDUCE-3691. webservices add support to compress response. - (Thomas Graves via mahadev) - - MAPREDUCE-3702. internal server error trying access application master - via proxy with filter enabled (Thomas Graves via mahadev) - - MAPREDUCE-3646. Remove redundant URL info from "mapred job" output. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3681. Fixed computation of queue's usedCapacity. (acmurthy) - - MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable. - (ahmed via tucu) - - MAPREDUCE-3714. Fixed EventFetcher and Fetcher threads to shut-down properly - so that reducers don't hang in corner cases. (vinodkv) - - MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client- - jobclient-tests.jar. (mahadev) - - MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs. - (mahadev) - - MAPREDUCE-3630. Fixes a NullPointer exception while running TeraGen - if a - map is asked to generate 0 records. (Mahadev Konar via sseth) - - MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent - maxCapacities. (acmurthy) - - MAPREDUCE-3713. Fixed the way head-room is allocated to applications by - CapacityScheduler so that it deducts current-usage per user and not - per-application. (Arun C Murthy via vinodkv) - - MAPREDUCE-3721. Fixed a race in shuffle which caused reduces to hang. - (sseth via acmurthy) - - MAPREDUCE-3733. Add Apache License Header to hadoop-distcp/pom.xml. - (mahadev) - - MAPREDUCE-3735. Add distcp jar to the distribution (tar). - (mahadev) - - MAPREDUCE-3720. Changed bin/mapred job -list to not print job-specific - information not available at RM. (vinodkv via acmurthy) - - MAPREDUCE-3742. "yarn logs" command fails with ClassNotFoundException. - (Jason Lowe via mahadev) - - MAPREDUCE-3703. ResourceManager should provide node lists in JMX output. - (Eric Payne via mahadev) - - MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use - java.io.File.createTempFile to create temporary files as part of their - tasks. (Jonathan Eagles via vinodkv) - - MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug. - (ramya via acmurthy) - - MAPREDUCE-3764. Fixed resource usage metrics for queues and users. - (acmurthy) - - MAPREDUCE-3749. ConcurrentModificationException in counter groups. - (tomwhite) - - MAPREDUCE-3762. Fixed default CapacityScheduler configs. (mahadev via - acmurthy) - - MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration - correctly, thus tests using doAs do not work. (johnvijoe via tucu) - - MAPREDUCE-3696. MR job via oozie does not work on hadoop 23. - (John George via mahadev) - - MAPREDUCE-3427. Fix streaming unit tests broken after mavenization. - (Hitesh Shah via acmurthy) - - MAPREDUCE-3640. Allow AMRecovery to work with partial JobHistory files. - (Arun C Murthy via sseth) - - MAPREDUCE-3752. Modified application limits to include queue max-capacities - besides the usual user limits. (Arun C Murthy via vinodkv) - - MAPREDUCE-3744. Fix the yarn logs command line. Improve error messages for - mapred job -logs. (Jason Lowe via sseth) - - MAPREDUCE-3780. Fixed a bug where applications killed before getting - activated were not getting cleaned up properly. (Hitesh Shah via acmurthy) - - MAPREDUCE-3708. Metrics: Incorrect Apps Submitted Count (Bhallamudi via - mahadev) - - MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong - jobtoken file (tucu) - - MAPREDUCE-3711. Fixed MR AM recovery so that only single selected task - output is recovered and thus reduce the unnecessarily bloated recovery - time. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3760. Changed active nodes list to not contain unhealthy nodes - on the webUI and metrics. (vinodkv) - - MAPREDUCE-3417. Fixed job-access-controls to work with MR AM and - JobHistoryServer web-apps. (Jonathan Eagles via vinodkv) - - MAPREDUCE-3803. Fix broken build of raid contrib due to HDFS-2864. - (Ravi Prakash via suresh) - - MAPREDUCE-3791. can't build site in hadoop-yarn-server-common. - (mahadev) - - MAPREDUCE-3723. TestAMWebServicesJobs & TestHSWebServicesJobs - incorrectly asserting tests (Bhallamudi Venkata Siva Kamesh - via mahadev) - - MAPREDUCE-3795. "job -status" command line output is malformed. - (vinodkv via mahadev) - - MAPREDUCE-3759. ClassCastException thrown in -list-active-trackers when - there are a few unhealthy nodes (vinodkv via mahadev) - - MAPREDUCE-3775. Change MiniYarnCluster to escape special chars in testname. - (Hitesh Shah via mahadev) - - MAPREDUCE-3765. FifoScheduler does not respect yarn.scheduler.fifo.minimum- - allocation-mb setting (Hitesh Shah via mahadev) - - MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish - time to RM Web-UI. (acmurthy) - - MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) - - MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined - hascode/equals for ResourceRequest & Priority. (vinodkv via acmurthy) - - MAPREDUCE-3813. Added a cache for resolved racks. (vinodkv via acmurthy) - - MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps - but no reduces. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks - (Dave Thompson via bobby) - - MAPREDUCE-3354. Changed scripts so that jobhistory server is started by - bin/mapred instead of bin/yarn. (Jonathan Eagles via acmurthy) - - MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end - of the task. (sseth via acmurthy) - - MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter - enums for compatibility (Tom White via mahadev) - - MAPREDUCE-3697. Support binary compatibility for Counters after - MAPREDUCE-901. (mahadev via acmurthy) - - MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via - mahadev) - - MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive - jobs. (Arpit Gupta via acmurthy) - - MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via - mahadev) - - MAPREDUCE-3436. JobHistory webapp address should use the host configured - in the jobhistory address. (Ahmed Radwan via sseth) - - MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when - requesting containers so that scheduler can give off data local containers - correctly. (Siddarth Seth via vinodkv) - - MAPREDUCE-3833. Fixed a bug in reinitiaziling of queues. (Jason Lowe via - acmurthy) - - MAPREDUCE-3826. Fixed a bug in RM web-ui which broke sorting. (Jonathan - Eagles via acmurthy) - - MAPREDUCE-3823. Ensure counters are calculated only once after a job - finishes. (Vinod Kumar Vavilapalli via sseth) - - MAPREDUCE-3827. Changed Counters to use ConcurrentSkipListMap for - performance. (vinodkv via acmurthy) - - MAPREDUCE-3822. Changed FS counter computation to use all occurences of - the same FS scheme, instead of randomly using one. (Mahadev Konar via - sseth) - - MAPREDUCE-3834. Changed MR AM to not add the same rack entry multiple times - into the container request table when multiple hosts for a split happen to - be on the same rack. (Siddarth Seth via vinodkv) - - MAPREDUCE-3828. Ensure that urls in single-node mode are correct. (sseth - via acmurthy) - - MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk) - - MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting - (Ravi Prakash via bobby) - - MAPREDUCE-3843. Job summary log file found missing on the RM host - (Anupam Seth via tgraves) - - MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then - the recovery. (vinodkv) - - MAPREDUCE-3802. Added test to validate that AM can crash multiple times and - still can recover successfully after MAPREDUCE-3846. (vinodkv) - - MAPREDUCE-3858. Task attempt failure during commit results in task never completing. - (Tom White via mahadev) - - MAPREDUCE-3856. Instances of RunningJob class givs incorrect job tracking - urls when mutiple jobs are submitted from same client jvm. (Eric Payne via - sseth) - - MAPREDUCE-3880. Changed LCE binary to be 32-bit. (acmurthy) - -Release 0.23.0 - 2011-11-01 - - INCOMPATIBLE CHANGES - - MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of - JobTrackerStatus. (tomwhite) - - MAPREDUCE-2430. Remove mrunit contrib. (nigel via eli) - - MAPREDUCE-2606. Remove IsolationRunner. (Alejandro Abdelnur via eli) - - NEW FEATURES - - MAPREDUCE-2682. Add "mapred classpath" command to print classpath - for MR applications. (vinodkv via acmurthy) - - MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix. - (Amar Kamat and Ravi Gummadi via amarrk) - - MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix. - (amarrk) - - MAPREDUCE-2543. [Gridmix] High-Ram feature emulation in Gridmix. (amarrk) - - MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk) - - MAPREDUCE-2473. Add "mapred groups" command to query the server-side groups - resolved for a user. (Aaron T. Myers via todd) - - MAPREDUCE-461. Enable ServicePlugins for the JobTracker. - (Fredrik Hedberg via tomwhite) - - MAPREDUCE-2521. Create RPM and Debian packages for MapReduce. Changes - deployment layout to be consistent across the binary tgz, rpm, and deb. - (Eric Yang via omalley) - - MAPREDUCE-2323. Add metrics to the fair scheduler. (todd) - - MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for - tasks. (Dick King via acmurthy) - - MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by - Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey - Naisbitt, Josh Wills, Jonathan Eagles, Krishna Ramachandran, Luke Lu, Mahadev - Konar, Robert Evans, Sharad Agarwal, Siddharth Seth, Thomas Graves, and Vinod - Kumar Vavilapalli. - - MAPREDUCE-2930. Added the ability to be able to generate graphs from the - state-machine definitions. (Binglin Chang via vinodkv) - - MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate - alternate frameworks on YARN. (Hitesh Shah via acmurthy) - - MAPREDUCE-3104. Implemented Application-acls. (vinodkv) - - MAPREDUCE-2708. Designed and implemented MR Application Master recovery to - make MR AMs resume their progress after restart. (Sharad Agarwal via vinodkv) - - MAPREDUCE-2858. Added a WebApp Proxy for applications. (Robert Evans via - acmurthy) - - IMPROVEMENTS - - MAPREDUCE-2187. Reporter sends progress during sort/merge. (Anupam Seth via - acmurthy) - - MAPREDUCE-2365. Add counters to track bytes (read,written) via - File(Input,Output)Format. (Siddharth Seth via acmurthy) - - MAPREDUCE-2680. Display queue name in job client CLI. (acmurthy) - - MAPREDUCE-2679. Minor changes to sync trunk with MR-279 branch. (acmurthy) - - MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a - ServiceProvider for the actual implementation. (tomwhite via acmurthy) - - MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk) - - MAPREDUCE-2563. [Gridmix] Add High-Ram emulation system tests to - Gridmix. (Vinay Kumar Thota via amarrk) - - MAPREDUCE-2104. [Rumen] Add Cpu, Memory and Heap usages to - TraceBuilder's output. (amarrk) - - MAPREDUCE-2554. [Gridmix] Add distributed cache emulation system tests - to Gridmix. (Vinay Kumar Thota via amarrk) - - MAPREDUCE-2543. [Gridmix] High-Ram feature emulation testcase. (amarrk) - - MAPREDUCE-2469. Task counters should also report the total heap usage of - the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk) - - MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to - Gridmix. (Vinay Kumar Thota via amarrk) - - MAPREDUCE-2517. [Gridmix] Add system tests to Gridmix. - (Vinay Kumar Thota via amarrk) - - MAPREDUCE-2492. The new MapReduce API should make available task's - progress to the task. (amarrk) - - MAPREDUCE-2153. Bring in more job configuration properties in to the trace - file. (Rajesh Balamohan via amarrk) - - MAPREDUCE-1461. Feature to instruct rumen-folder utility to skip jobs worth - of specific duration. (Rajesh Balamohan via amarrk) - - MAPREDUCE-2172. Added test-patch.properties required by test-patch.sh - (nigel) - - MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba) - - MAPREDUCE-2215. A more elegant FileSystem#listCorruptFileBlocks API - (RAID changes) (Patrick Kling via hairong) - - MAPREDUCE-1831. BlockPlacement policy for HDFS-RAID. - (Scott Chen via dhruba) - - MAPREDUCE-1906. Lower minimum heartbeat interval for TaskTracker - (Scott Carey and Todd Lipcon via todd) - - MAPREDUCE-1382. MRAsyncDiscService should tolerate missing local.dir. - (Zheng Shao and tomwhite via tomwhite) - - MAPREDUCE-2263. MapReduce side of HADOOP-6904: RPC compatibility. - (hairong) - - MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen) - - MAPREDUCE-2334. Update BlockPlacementPolicyRaid for the new method - in BlockPlacementPolicy. (szetszwo) - - MAPREDUCE-2254. Allow setting of end-of-record delimiter for - TextInputFormat (Ahmed Radwan via todd) - - MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support). - (Greg Roelofs via tomwhite) - - MAPREDUCE-2206. The task-cleanup tasks should be optional. (schen) - - MAPREDUCE-2225. MultipleOutputs should not require the use of 'Writable'. - (Harsh J Chouraria via tomwhite) - - MAPREDUCE-1811. Job.monitorAndPrintJob() should print status of the job - at completion. (Harsh J Chouraria via tomwhite) - - MAPREDUCE-993. bin/hadoop job -events <#-of-events> - help message is confusing. (Harsh J Chouraria via tomwhite) - - MAPREDUCE-2302. Add static factory methods in GaloisField. (schen) - - MAPREDUCE-2351. mapred.job.tracker.history.completed.location should - support an arbitrary filesystem URI. (tomwhite) - - MAPREDUCE-2239. BlockPlacementPolicyRaid should call getBlockLocations - only when necessary. (schen) - - MAPREDUCE-2331. Add coverage of task graph servlet to fair scheduler system - test. (todd) - - MAPREDUCE-2367. Allow using a file to exclude certain tests from build. - (todd) - - MAPREDUCE-2202. Generalize CLITest structure and interfaces to faciliate - upstream adoption (e.g. for web or system testing). (cos) - - MAPREDUCE-2420. JobTracker should be able to renew delegation token over - HTTP (Boris Shkolnik via jitendra) - - MAPREDUCE-2474. Add docs to the new API Partitioner on how to access the - Job Configuration. (Harsh J Chouraria via todd) - - MAPREDUCE-2475. Disable IPV6 for junit tests. (suresh srinivas via mahadev) - - MAPREDUCE-2422. Removed unused internal methods from DistributedCache. - (tomwhite) - - MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with - failed fetch notifications in the JobTracker log. - (Jeffrey Naisbitt via cdouglas) - - MAPREDUCE-869. Documentation for config to set map/reduce task environment - (Alejandro Abdelnur via todd) - - MAPREDUCE-2410. Add entry to streaming FAQ about how streaming reducers - receive keys. (Harsh J Chouraria via todd) - - MAPREDUCE-2499. MR part of HADOOP-7291. (eli) - - MAPREDUCE-2497. Missing spaces in error messages. (eli) - - MAPREDUCE-2502. JobSubmitter should use mapreduce.job.maps instead of - its deprecated equivalent. (eli via todd) - - MAPREDUCE-2381. JobTracker instrumentation not consistent about error - handling. (Philip Zeyliger via tomwhite) - - MAPREDUCE-2449. Allow for command line arguments when performing - "Run on Hadoop" action in Eclipse plugin. (Jeff Zemerick via todd) - - MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects - from build artifacts. (Eric Yang via omalley) - - MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in - log4j.properties (todd) - - MAPREDUCE-2516. Rename webinterface.private.actions to - mapreduce.jobtracker.webinterface.trusted (Ari Rabkin via todd) - - MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev) - - HADOOP-7259. Contrib modules should include the build.properties from - the enclosing hadoop directory. (omalley) - - MAPREDUCE-2494. Order distributed cache deletions by LRU. (Robert Joseph - Evans via cdouglas) - - MAPREDUCE-2452. Makes the cancellation of delegation tokens happen in a - separate thread. (ddas) - - HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects. - (todd, nigel) - - MAPREDUCE-2249. Check the reflexive property of Counters objects when - comparing equality. (Devaraj K via todd) - - MAPREDUCE-2623. Update ClusterMapReduceTestCase to use - MiniDFSCluster.Builder (Harsh J Chouraria via eli) - - MAPREDUCE-2602. Allow setting of end-of-record delimiter for - TextInputFormat for the old API. (Ahmed Radwan via todd) - - MAPREDUCE-2705. Permits parallel multiple task launches. - (Thomas Graves via ddas) - - MAPREDUCE-2489. Jobsplits with random hostnames can make the queue - unusable (jeffrey naisbit via mahadev) - - MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn. - (thomas graves via mahadev) - - MAPREDUCE-2701. app/Job.java needs UGI for the user that launched it. - (Robert Evans via mahadev) - - MAPREDUCE-2652. Enabled multiple NMs to be runnable on a single node by - making shuffle service port to be truely configurable. (Robert Evans via - vinodkv) - - MAPREDUCE-2735. Add an applications summary log to ResourceManager. - (Thomas Graves via acmurthy) - - MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running - applications per-queue & per-user. (acmurthy) - Configuration changes: - add yarn.capacity-scheduler.maximum-am-resource-percent - - MAPREDUCE-2774. Add startup message to ResourceManager & NodeManager on - startup. (Venu Gopala Rao via acmurthy) - - MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. (Thomas - Graves via acmurthy) - - MAPREDUCE-2864. Normalize configuration variable names for YARN. (Robert - Evans via acmurthy) - - MAPREDUCE-2690. Web-page for FifoScheduler. (Eric Payne via acmurthy) - - MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem - and block management APIs. (szetszwo) - - MAPREDUCE-2933. Change allocate call to return ContainerStatus for - completed containers rather than Container. (acmurthy) - - MAPREDUCE-2675. Reformat JobHistory Server main page to be more - useful. (Robert Joseph Evans via vinodkv). - - MAPREDUCE-2896. Simplify all apis to in - org.apache.hadoop.yarn.api.records.* to be get/set only. Added javadocs to - all public records. (acmurthy) - - MAPREDUCE-2676. MR-279: JobHistory Job page needs reformatted. (Robert Evans via - mahadev) - - MAPREDUCE-2899. Replace major parts of ApplicationSubmissionContext with a - ContainerLaunchContext (Arun Murthy via mahadev) - - MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can - gracefully exit. (Abhijit Suresh Shingate via vinodkv) - - MAPREDUCE-2672. MR-279: JobHistory Server needs Analysis this job. - (Robert Evans via mahadev) - - MAPREDUCE-2965. Streamlined the methods hashCode(), equals(), compareTo() - and toString() for all IDs. (Siddharth Seth via vinodkv) - - MAPREDUCE-2726. Added job-file to the AM and JobHistoryServer web - interfaces. (Jeffrey Naisbitt via vinodkv) - - MAPREDUCE-2880. Improve classpath-construction for mapreduce AM and - containers. (Arun C Murthy via vinodkv) - - MAPREDUCE-3055. Simplified ApplicationAttemptId passing to - ApplicationMaster via environment variable. (vinodkv) - - MAPREDUCE-3092. Removed a special comparator for JobIDs in JobHistory as - JobIDs are already comparable. (Devaraj K via vinodkv) - - MAPREDUCE-3099. Add docs for setting up a single node MRv2 cluster. - (mahadev) - - MAPREDUCE-3001. Added task-specific counters to AppMaster and JobHistory - web-UIs. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3098. Fixed RM and MR AM to report YarnApplicationState and - application's FinalStatus separately. (Hitesh Shah via vinodkv) - - MAPREDUCE-2889. Added documentation for writing new YARN applications. - (Hitesh Shah via acmurthy) - - MAPREDUCE-3134. Added documentation the CapacityScheduler. (acmurthy) - - MAPREDUCE-3013. Removed YarnConfiguration.YARN_SECURITY_INFO and its usage - as it doesn't affect security any more. (vinodkv) - - MAPREDUCE-2907. Changed log level for various messages in ResourceManager - from INFO to DEBUG. (Ravi Prakash via vinodkv) - - MAPREDUCE-2702. Added a new API in OutputCommitter for recovering - the outputs of tasks from a crashed job so as to support MR Application - Master recovery. (Sharad Agarwal and Arun C Murthy via vinodkv) - - MAPREDUCE-2738. Added the missing cluster level statistics on the RM web - UI. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the - current NodeManager code. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3161. Improved some javadocs and fixed some typos in - YARN. (Todd Lipcon via vinodkv) - - MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task - recovery. (acmurthy) - - MAPREDUCE-3133. Running a set of methods in a Single Test Class. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3059. QueueMetrics do not have metrics for aggregate - containers-allocated and aggregate containers-released. - (Devaraj K via mahadev) - - MAPREDUCE-3187. Add names for various unnamed threads in MR2. - (Todd Lipcon and Siddharth Seth via mahadev) - - MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both - non-secure and secure mode for both HDFS & YARN. (acmurthy) - - MAPREDUCE-3068. Added a whitelist of environment variables for containers - from the NodeManager and set MALLOC_ARENA_MAX for all daemons and - containers. (Chris Riccomini via acmurthy) - - MAPREDUCE-3144. Augmented JobHistory with the information needed for - serving aggregated logs. (Siddharth Seth via vinodkv) - - MAPREDUCE-3163. JobClient spews errors when killing MR2 job. - (mahadev) - - MAPREDUCE-3239. Use new createSocketAddr API in MRv2 to give better - error messages on misconfig (Todd Lipcon via mahadev) - - MAPREDUCE-2747. Cleaned up LinuxContainerExecutor binary sources and changed - the configuration to use yarn names. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3205. Fix memory specifications to be physical rather than - virtual, allowing for a ratio between the two to be configurable. (todd - via acmurthy) - - MAPREDUCE-2986. Fixed MiniYARNCluster to support multiple NodeManagers. - (Anupam Seth via vinodkv) - - MAPREDUCE-2736. Remove unused contrib components dependent on MR1. (eli) - - MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the - JobHistoryServer. (Siddharth Seth via vinodkv) - - MAPREDUCE-3146. Added a MR specific command line to dump logs for a - given TaskAttemptID. (Siddharth Seth via vinodkv) - - MAPREDUCE-3275. Added documentation for AM WebApp Proxy. (Robert Evans via - acmurthy) - - MAPREDUCE-3322. Added a better index.html and an brief overview of YARN - architecture. (acmurthy) - - OPTIMIZATIONS - - MAPREDUCE-2026. Make JobTracker.getJobCounters() and - JobInProgress.getCounters() aquire locks in a shorter time period. - (Joydeep Sen Sarma via schen) - - MAPREDUCE-2740. MultipleOutputs in new API creates needless - TaskAttemptContexts. (todd) - - MAPREDUCE-901. Efficient framework counters. (llu via acmurthy) - - MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents - JIT compilation of inner classes methods in ReduceContextImpl. - - BUG FIXES - - MAPREDUCE-2603. Disable High-Ram emulation in system tests. - (Vinay Kumar Thota via amarrk) - - MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via - acmurthy) - - MAPREDUCE-1978. Rumen TraceBuilder should provide recursive - input folder scanning. - - MAPREDUCE-2416. Remove the restriction of specifying group names in - users-list file for Gridmix in RoundRobinUserResolver mode. - - MAPREDUCE-2417. Fix Gridmix in RoundRobinUserResolver mode to - map testing/proxy users to unique users in a trace. - - MAPREDUCE-2307. Exception thrown in Jobtracker logs, when the Scheduler - configured is FairScheduler. (Devaraj K via matei) - - MAPREDUCE-2199. build is broken 0.22 branch creation. (cos) - - MAPREDUCE-1752. Implement getFileBlockLocations in HarFilesystem. - (Patrick Kling via dhruba) - - MAPREDUCE-2155. RaidNode should optionally use the mapreduce jobs to - fix missing blocks. (Patrick Kling via dhruba) - - MAPREDUCE-1334. Fix TestIndexUpdater by ignoring _SUCCESS file in HDFS. - (Kay Kay via yhemanth) - - MAPREDUCE-2232. Add missing methods to TestMapredGroupMappingServiceRefresh. - (Todd Lipcon via eli) - - MAPREDUCE-2271. Fix TestSetupTaskScheduling failure on trunk. - (Liyin Liang via todd) - - MAPREDUCE-2290. Fix compilation error in TestTaskCommit. (eli) - - MAPREDUCE-2294. Fix compilation error in mumak. (todd) - - MAPREDUCE-2300. Fix TestUmbilicalProtocolWithJobToken on trunk after - HADOOP-6904. (todd) - - MAPREDUCE-2296. Fix references to misspelled method name - getProtocolSigature (todd) - - MAPREDUCE-2311. Fix TestFairScheduler failure (schen) - - MAPREDUCE-1996. API: Reducer.reduce() method detail misstatement. - (Harsh J Chouraria via tomwhite) - - MAPREDUCE-2203. Wrong javadoc for TaskRunner's appendJobJarClasspaths - method. (Jingguo Yao via tomwhite) - - MAPREDUCE-2074. Task should fail when symlink creation fails. - (Priyo Mustafi via tomwhite) - - MAPREDUCE-1242. Chain APIs error misleading. - (Harsh J Chouraria via tomwhite) - - MAPREDUCE-2379. Adds missing DistributedCache configurations in - mapred-default.xml (Todd Lipcon via amareshwari) - - MAPREDUCE-2348. Disable mumak tests on trunk since they currently time out - (todd) - - MAPREDUCE-2395. TestBlockFixer timing out on trunk. (Ramkumar Vadali via - todd) - - MAPREDUCE-2426. Make TestFairSchedulerSystem fail with more verbose output - (todd) - - MAPREDUCE-2448. NoSuchMethodError: - org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..) (eli) - - MAPREDUCE-2460. Fix flaky test TestFairSchedulerSystem. (todd) - - MAPREDUCE-2451. Log the details from health check script at the - JobTracker. (Thomas Graves via cdouglas) - - MAPREDUCE-2467. HDFS-1052 changes break the raid contrib module in - MapReduce. (suresh srinivas via mahadev) - - MAPREDUCE-2258. IFile reader closes stream and compressor in wrong order. - (todd via tomwhite) - - MAPREDUCE-2518. The t flag is missing in distcp help message. (Wei Yongjun - via szetszwo) - - MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message. - (Jonathan Eagles via cdouglas) - - MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid - diagnosis of related issues. (Jonathan Eagles via cdouglas) - - MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup - thread dies. (Robert Joseph Evans via cdouglas) - - MAPREDUCE-2470. Fix NPE in RunningJobs::getCounters. (Robert Joseph Evans - via cdouglas) - - MAPREDUCE-2536. Update FsShell -mv command usage in TestMRCLI. (Daryn - Sharp via szetszwo) - - MAPREDUCE-2529. Add support for regex-based shuffle metric counting - exceptions. (Thomas Graves via cdouglas) - - MAPREDUCE-2559. ant binary fails due to missing c++ lib dir. (eli) - - MAPREDUCE-2573. Fix new findbugs warning introduced by MAPREDUCE-2494. - (Robert Joseph Evans via todd) - - MAPREDUCE-2581. Spelling errors in log messages. (Tim Sell via eli) - - MAPREDUCE-2588. Change raid to the new DataTransferProtocol API. (szetszwo) - - MAPREDUCE-2576. Typo in comment in SimulatorLaunchTaskAction.java. - (Tim Sell via jghoman) - - MAPREDUCE-2550. Fix bin/mapred to work properly from within a source - checkout (Eric Yang via todd) - - MAPREDUCE-2620. Update RAID for HDFS-2087. (szetszwo) - - MAPREDUCE-2624. Update RAID for HDFS-2107. (szetszwo) - - MAPREDUCE-2670. Fixing spelling mistake in FairSchedulerServlet.java. (eli) - - MAPREDUCE-2710. Update JobSubmitter.printTokens(..) for HDFS-2161. - (szetszwo) - - MAPREDUCE-2409. DistributedCache maps files and archives to the same path, - despite semantic incompatibility. (Siddharth Seth via cdouglas) - - MAPREDUCE-2575. TestMiniMRDFSCaching fails if test.build.dir is set - to something other than build/test (Thomas Graves via mahadev) - - MAPREDUCE-2622. Remove the last remaining reference to the deprecated - configuration "io.sort.mb". (Harsh J Chouraria via todd) - - MAPREDUCE-2732. Remove directly accessing FSNamesystem.LOG from - TestCopyFiles and TestDistCh. (szetszwo) - - MAPREDUCE-2463. Job history files are not moved to done folder when job - history location is hdfs. (Devaraj K via szetszwo) - - MAPREDUCE-2243. Close streams propely in a finally-block to avoid leakage - in CompletedJobStatusStore, TaskLog, EventWriter and TotalOrderPartitioner. - (Devaraj K via szetszwo) - - MAPREDUCE-2741. Make ant build system work with hadoop-common JAR - generated by Maven. (Alejandro Abdelnur via tomwhite) - - MAPREDUCE-2760. mapreduce.jobtracker.split.metainfo.maxsize typoed - in mapred-default.xml. (todd via eli) - - MAPREDUCE-2797. Update mapreduce tests and RAID for HDFS-2239. (szetszwo) - - MAPREDUCE-2805. Update RAID for HDFS-2241. (szetszwo) - - MAPREDUCE-2837. Ported bug fixes from y-merge to prepare for MAPREDUCE-279 - merge. (acmurthy) - - MAPREDUCE-2541. Fixed a race condition in IndexCache.removeMap. (Binglin - Chang via acmurthy) - - MAPREDUCE-2458. Rename sanitized pom.xml in build directory to work around IDE - bug (Luke Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Clear application notification if sent once - to NodeManager (mahadev) - - MAPREDUCE-2433. YARNApplicationConstants hard code app master jar version (Luke - Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Implement restart for resource manager - phase 1 - Helper classes to store and restore the data structures. (mahadev) - - MAPREDUCE-2414. Change MRv2 to use generic interfaces. (Siddharth Seth via - acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Implement health-checks for the node - - server side(ResourceManager) changes. (vinodkv) - - MAPREDUCE-2405: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2) - (Greg Roelofs via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Implementing Containers' memory monitoring. - (vinodkv) - - MAPREDUCE-2440. Name clashes in TypeConverter (luke via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add fail count to the command line of the - application master. (mahadev) - - MAPREDUCE-2424. Polish uber-AppMaster: add uber-AM counters and GUI indicators. - (Greg Roelofs via mahadev) - - MAPREDUCE-2405. Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2). - (Greg Roelofs and Sharad Agarwal via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix YarnRemoteException to give more - details. (Siddharth Seth via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. WebApp for Job History (Krishna - Ramachandran via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Install sanitized poms for downstream - sanity (Luke Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add HistoryCleanerService to Job History - server. (Krishna Ramachandran via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred job -list' and - 'bin/mapred job -list-active-trackers'. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred queue [-info - [-showJobs]] [-list] and enhanced 'bin/mapred job -list' to show queue and - ApplicationMaster information. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed computation of user-limits at - runtime. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Added functionality to refresh queues at - runtime via the 'bin/yarn rmadmin' command. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Added functionality to stop/start queues. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Introducing web-UI for NodeManager and - linking it from RM UI. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix race condition in TestJobHistoryEvents - and TestJobHistoryParsing. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Add Containers' logs' view to NM UI and - link it from AM UI. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Add ACLs for queues and command-line - utilities for viewing them. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Recovery of MR Application Master from - failures. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Added support High-RAM applications in - CapacityScheduler. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Completing the ZooKeeper Store for - ResourceManager state. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Reorient container localization to be - per-container rather than per-application. (cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix file creation in - JobHistoryEventHandler. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Disable ContainerMonitoring for non-linux - systems. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix container launch w/ inconsistent - credential file naming. (cdouglas) - - MAPREDUCE-2434. Metrics for ResourceManager. (Luke Lu via acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. RM Restart Phase 2 - Completed the recovery - of components in the RM (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix to send finish application event only - when the application is finished (mahadev) - - MAPREDUCE-2462. Write job conf along with JobHistory, other minor improvements. - (Siddharth Seth via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Implement 'delay scheduling' for better - locality in CapacityScheduler and improved high-ram applications. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Implement Job Acls in MR Application - Master. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Moving userlogs out of container work-dir - into a separate directory structure. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Completing RM Restart. Completed Phase 3 of - making sure events are logged and restored (mahadev) - - MAPREDUCE-2468. Add metrics for NM Shuffle. (Luke Lu via cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Adding user log handling for YARN. Making - NM put the user-logs on DFS and providing log-dump tools. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing three tight-loops in RM that are - causing high cpu-usage. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Replacing FileContext usage with FileSystem - to work around security authentication issues with FileContext against a secure - DFS. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Client reconnect to restarted AM. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix refreshProxy in ClientServiceDelegate. - (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix Null Pointer in TestUberAM. (sharad) - - MAPREDUCE-2478. Improve history server. (Siddharth Seth via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Improved TestJobHistoryEvents and - TestJobHistoryParsing. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix NM to use multiple disks for local - files and the userlogs. (vinodkv) - - MAPREDUCE-2480: MR App should not depend on hard coded version of shuffle (luke - lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Propagate error back to client in case of a - job submission failure (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix assembly to add mapreduce shell scripts - to the assembly package. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix TestQueueMetrics. (Luke Lu via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Removal of stale application-log dirs from - NM local disks. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Add license header and minor cleanup in - history server. (Siddharth Seth via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Minor fix for install instructions. - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix a race in MR task that was causing MR - containers to overwrite each other's job.xml. Also fix leaking attempt-dirs in - app-local-dir. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Adding valid state to ASM on a finish when - its already completed and also disble UberAM. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed CS user limits. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed reservation's bad interaction with - delay scheduling in CS. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Support mapreduce old (0.20) APIs. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Support fail-fast for MR jobs. (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for clearing container requests on an - AM failure and add tostring methods to taskids and taskattemptids for better - grep support. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Speed up communication between MR AM and RM - by relying on a new config rather than AM_EXPIRY_INTERVAL which is too large. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity to use - parent's absolute-capacity rather than the leaf queue's absolute-capacity. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in JobIDPbImpl that's causing - AM to crash randomly. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity in - ParentQueue to use its parent's absolute-capacity rather than its own - absolute-capacity. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Launching bin/yarn and bin/mapred only - *once* in AM for constructing classpaths to avoid multiple forks and huge vmem - usage by AM. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler to release unused - reservations on application completion. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler (LeafQueue) to not - allocate DATA_LOCAL containers when they are not required on the rack. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Makes uber-task disabled by default (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Make logging and memory for AM configurable - for the user via command line (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in previous patch (r1103657). - Now bin/yarn truly shouldn't be launched multiple times in a single AM. - (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug to do with setting the staging - dir. (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed Composite Service to shutdown - services if an error occurs on starting any one of those (mahadev & chris) - - MAPREDUCE-279. Fix in MR-279 branch. Fix the tests to use jvm fork mode to avoid - errors in shutting down services (sidharth seth) - - MAPREDUCE-2500. PB factories are not thread safe (Siddharth Seth via mahadev) - - MAPREDUCE-2504. race in JobHistoryEventHandler stop (Siddharth Seth via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix job hang if the AM launch fails. - (mahadev) - - MAPREDUCE-2509. Fix NPE in UI for pending attempts. (Luke Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add junit jar to lib in assembly (mahadev - and luke) - - MAPREDUCE-279. Fix in MR-279 branch. Distributed cache bug fix to pass Terasort. - (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix null pointer exception in kill task - attempt (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Refactored RMContainerAllocator to release - unused containers. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Changed Scheduler to return available limit - to AM in the allocate api. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix nodemanager expiry to not throw OOM. - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Use DefaultContainerExecutor for - integration tests. (cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in test case (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for regression on the scheduling of - reduces before maps are done (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix distributed-cache related bugs. - (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Ensure unused containers released by AM are - correctly counted for queue-capacity. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix TestRuntimeEstimators (Siddharth Seth - via ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix queue refresh to correctly record newly - added queues in CapacityScheduler. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Added metrics for tracking reservations in - CapacityScheduler. (Luke Lu via acmurthy) - - MAPREDUCE-2522. Security for JobHistory service. (Siddharth Seth via mahadev) - - MAPREDUCE-2534. Fix CI breaking hard coded version in jobclient pom. (Luke Lu - via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add public cache. (cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Made number of RPC server threads - configurable. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Added acl check for RMAdmin. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Adding job kill for any state that the job - is in with access control. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add debug statements for AM not launching - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing race condition leader to hung jobs - in scheduler negotiator (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Add debug config for delaying delete of - local files. (cdouglas) - - MAPREDUCE-2527. Metrics for MRAppMaster (Luke lu via mahadev) - - MAPREDUCE-2532. Metrics for NodeManager (Luke Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed an NPE during handling of unnecessary - reservations in CS. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for public dist cache to work with non - default hdfs (mahadev &ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Making streaming -file option work. Also - minor fixes for successful compilation of contrib tests. (vinodkv) - - MAPREDUCE-2536. Backporting changes to MR-279. - - MAPREDUCE-279. Fix in MR-279 branch. Bugfix for using user staging directory for - history files (Siddharth Seth via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. First fix for making basic speculative - execution work (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fixes for TestFail/Kill (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Set correct version of avro-maven-plugin - that is available in apache maven repositories. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing and reneabling - TestContainerTokenSecretManager. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Cleaning up configuration constants in - mapreduce modules. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing NPE on speculator in MRAppMaster and - making job-history optional in tests to make test goal succeed. (vinodk and - sharadag). - - MAPREDUCE-279. Fix in MR-279 branch. Fixed NPE in CS by checking Application - state before scheduling and fixing synchronization in CS. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Making pipes work with YARN. Changed pipes - to get log-locations from an environmental variable. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Ensure 'lost' NodeManagers are dealt - appropriately, the containers are released correctly. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Adding some more logging for AM expiry logs - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Reduce ramp up and zero maps support. - (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Allowing hdfs calls from streaming/pipes - tasks. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Added ability to decommission nodes and - completed RM administration tools to achieve parity with JobTracker. (acmurthy) - - MAPREDUCE-2551. Added JobSummaryLog. (Siddharth Seth via acmurthy) - - MAPREDUCE-2552. Fixed NPE in CompletedJob in JobHistoryServer. (Siddharth Seth - via acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix reduce slow start. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed TestFifoScheduler. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix build issue for using yarn.version - instead of hadoop-mapred.version (mahadev and giri) - - MAPREDUCE-279. Fix in MR-279 branch. Fixes in the handling of KILL events in the - SUCCEEDED state for tasks in the application master (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestRMNMRPCResponseId. - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix a deadlock in the resourcemanager. - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. NodeStatus.getNodeHealthStatus().setBlah - broken (Siddharth Seth) - - MAPREDUCE-279. Fix in MR-279 branch. Fix another NPE in TestRMNMRPCResponseId. - (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestNMExpiry (siddharth - seth) - - MAPREDUCE-279. Fix in MR-279 branch. Making each node aggregate all its - user-logs to a separate hdfs file. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of max-capacity for a - queue, also fixed a bug in registration of NodeManagers. (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. More cleaning up constants, removing stale - code, and making conspicuous the envs that apps depend on to be provided by - YARN. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix container size rounding in AM and - headroom in RM. (acmurthy and sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Disable Job acls until fixed (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix to report job status if the application - is KILLED/FAILED. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix a corner case in headroom computation - - now reservations are taken into account and headroom is computed much later to - account for allocations/reservations. (acmurthy) - - MAPREDUCE-2537. The RM writes its log to - yarn-mapred-resourcemanager-.out (Robert Evans via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix logging for showing the state of job - (FAILED/KILLED/SUCCEEDED) when it completes (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Re-enabled TestCapacityScheduler. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Support for min and max container capacity. - (acmurthy and sharad) - - MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via acmurthy) - - MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via - acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing the wrong config key used in - JobHistory that prevented configuring move-thread interval. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed inconsistency in QueueACL enums. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Fix various issues with Web UI's. (Luke Lu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in Task abort for - old mapreduce apis. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Add deletion of distributed cache - resources. (cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Disable aggregation of logs onto DFS till - JobHistoryServer starts serving logs. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Cleanup redundant code in TaskAttemptImpl. - (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Work around broken signaling in public - cache. (cdouglas) - - MAPREDUCE-2566. YarnConfiguration should reloadConfiguration if instantiated - with a non YarnConfiguration object. (Siddharth Seth) - - MAPREDUCE-279. Fix in MR-279 branch. Fully resolve paths when launching - containers. (Siddharth Seth) - - MAPREDUCE-279. Fix in MR-279 branch. Re-enabling Uber-AM feature. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed deadlock during expiring NMs. - (acmurthy) - - MAPREDUCE-279. Fix in MR-279 branch. Solving NPEs during - ContainerManager#StopContainer. Also removing the unused - ContainerManager#CleanupContainer api. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Remove retries in dist cache so that NM's - do not shutdown (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix classpath construction for Task. - (vinodkv via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Sending Job diagnostics from AM to RM and - redirect to history-url on job completion. (vinodkv and sharadag) - - MAPREDUCE-279. Fix in MR-279 branch. Added clover in pom dependency. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster - Linux desktop. (Luke lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Including source files in release - distribution (Luke Lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster - Linux desktop (part 2) (Luke lu via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Disable Uber AM. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Added few job diagnostic messages. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Add ability to includes src files in - assembly target for maven (Luke Lu via mahadev) - - MAPREDUCE-2582. Cleanup JobHistory event generation.(Siddharth Seth via sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix rounding off problem in reduce ramp up. - (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix more rounding off problems in reduce - ramp up. Also fix a bug preventing the application of the cap on reduce ramp-up. - (Sharad Agarwal via vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fix to exclude images dir into the tar - distribution (Luke Lu via gkesavan) - - MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext - to FileSystem in YarnRunner to handle distributed cache path resolutions on - non-default filesystems. (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Display failed/killed attempts of the task - on MR AM UI separately. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Miscellaneous UI fixes + source code - formatting for MR JobHistoryEventHandler. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing broken link to logs for container on - NM web UI. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing the bug which was causing FAILED - jobs to be displayed as COMPLETED on the RM UI. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Job level node blacklisting. (sharad) - - MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in history event handling - (Siddharth Seth via mahadev) - - MAPREDUCE-2569. Ensure root queue allocated 100% capacity. (Jonathan Eagles via - cdouglas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix ClassCastException in JobHistoryServer - for certain jobs. (Siddharth Seth via llu) - - MAPREDUCE-279. Fix in MR-279 branch. Changes for invoking rack resolution in the - RM and in the AM (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix concurrent modification exception in - the Capacity Scheduler (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix stopContainer for setsid challenged - platforms. (llu) - - MAPREDUCE-2587. Generate yarn version for UI. (Thomas Graves via lluts page to - the history server UI. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Bug fix to set correct state on containers - so as to avoid duplicate containers from RM to AM. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Hack until MAPREDUCE-2365 is fixed to make - PIG work with MRV2. (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext - to FileSystem in TaskAttemptImpl to handle distributed cache path resolutions on - non-default filesystems. (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix NPE when requesting attempts for - completed jobs. (Siddharth Seth via llu) - - MAPREDUCE-279. Fix in MR-279 branch. Improve logging for AM when requesting - containers to show the right ask and release fields (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix race condition between multiple - localizers on a single node. (cdouglas via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix RM app start/finish time and - diagnostics. (llu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix to schedule reduces irrespective of the - headroom when all maps are done so as to avoid stall in reduce-scheduling when - slow-start is disabled. (Sharad Agarwal via vinodkv). - - MAPREDUCE-279. Fix in MR-279 branch. Disabling locality-wait in - CapacityScheduler for now to prevent uber-slow scheduling for apps with no - data-locality constraints (sleep-job like). (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Fixing scheduling deadlock in AM because of - incorrect headRoom values from RM. The bug happens when AM releases containers - and RM decrements current memory usage twice for all those containers. (vinodkv) - - MAPREDUCE-2611. Fix counters, finish times etc. in job history. (Siddharth Seth - via llu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for ConcurrentModification exception - while iterating through tokens in a UGI in ContainerLauncherImpl. (ddas) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in YarnChild that was causing - lots of tasks to fail. (vinodkv) - - MAPREDUCE-2615. Make killJob go through AM and fix JobSummaryLog. (Siddharth - Seth via llu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in release - reserved containers in capacity scheduler (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix diagnostics display for more than 100 - apps in RM. (llu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix some invalid transitions in the RM. - (vinodkv via ddas) - - MAPREDUCE-2618. Fix NPE in 0 map 0 reduce jobs. (Jeffrey Naisbitt via llu) - - MAPREDUCE-2625. Add version info to nodemanager info page. (Jonathan Eagles via - llu) - - MAPREDUCE-279. Fix in MR-279 branch. (1) Faster retries from AM to HistoryServer - (2) Correct diagnostics for containers. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Major ASM cleanup. Streamlining classes, - interface and events. (vinodkv) - - MAPREDUCE-279. Fix in MR-279 branch. Reinstate resolve path fixes for viewfs. - (Siddharth Seth via llu) - - MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record. (Josh - Wills via sharad) - - MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. (Josh Wills - via vinodkv) - - MAPREDUCE-2628. Add compiled on date to NM and RM info/about page. - - MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a ServiceProvider - for the actual implementation. (tomwhite via acmurthy) - - MAPREDUCE-2663. Refactoring StateMachineFactory inner classes. (ahmed radwan via - mahadev) - - MAPREDUCE-2678. minimum-user-limit-percent no longer honored. (naisbitt via - mahadev) - - MAPREDUCE-2630. refreshQueues leads to NPEs when used w/FifoScheduler. (Josh - Wills via mahadev) - - MAPREDUCE-2644. NodeManager fails to create containers when NM_LOG_DIR is not - explicitly set in the Configuration. (Josh Wills via vinodkv) - - MAPREDUCE-2661. Fix TaskImpl to not access MapTaskImpl. (Ahmed Radwan via - sharad) - - HADOOP-6929. Backport changes to MR-279 (mahadev and owen) - - HADOOP-6929. Making Security Info abstract and not an interface (mahadev) - - MAPREDUCE-2667. mapred job -kill leaves application in RUNNING state (thomas - graves via mahadev) - - MAPREDUCE-2664. Implement JobCounters for Mtions as asynchronous. (vinodkv, - sharad and acmurthy) - - MAPREDUCE-2773. server.api.records.NodeHealthStatus renamed but not updated in - client NodeHealthStatus.java (Thomas Graves via mahadev) - - MAPREDUCE-2772. Fix MR-279 build after common mavenization. (Robert Joseph Evans - via llu) - - MAPREDUCE-2772. Fix MR-279 build after common mavenization, part 2. (Thomas - Graves via llu) - - MAPREDUCE-279. Fix in MR-279 branch. Harmonize slf4j versions. (llu) - - MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in FifoScheduler. (mahadev) - - MAPREDUCE-2776. Fix some of the yarn findbug warnings. (Siddharth Seth via - mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules, - part 1 (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules - part 2 (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules - part 3 (mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix the poms to enable 0.23 snapshots for - hdfs/common from apache nightly builds (gkesavan) - - MAPREDUCE-279. Fix in MR-279 branch. Fix ivy conf to work with the hadoop common - trunk maven build changes. (Giridharan Kesavan) - - MAPREDUCE-279. Fix in MR-279 branch. Patch for findbugs warnings in Resource - Manager (Siddharth Seth via mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fix for running ant targets to use the - right set of common/test jars (gkesavan via mahadev) - - MAPREDUCE-2782. Unit tests for CapacityScheduler. (acmurthy) - - MAPREDUCE-2706. Log job submission failures. (Jeffrey Naisbitt via acmurthy) - - MAPREDUCE-2781. mr279 RM application finishtime not set (Thomas Graves via - mahadev) - - MAPREDUCE-279. Fix in MR-279 branch. Fixed CS locality wait factor. (acmurthy) - - MAPREDUCE-2808. pull MAPREDUCE-2797 into mr279 branch (Thomas Graves via - mahadev) - - MAPREDUCE-2639. Bug fixes in speculate.DataStatistics. (Josh Wills via acmurthy) - - MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old - apis. (Siddharth Seth via acmurthy) - - MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals 0. - (Jeffrey Naisbitt via acmurthy) - - MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new - and old apis. (Siddharth Seth via acmurthy) - - MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals - 0. (Jeffrey Naisbitt via acmurthy) - - MAPREDUCE-2860. Fix log4j logging in the maven test cases. (mahadev) - - MAPREDUCE-2867. Remove Unused TestApplicaitonCleanup in resourcemanager/applicationsmanager. - (mahadev) - - MAPREDUCE-2868. ant build broken in hadoop-mapreduce dir (mahadev, giri and arun via mahadev) - - MAPREDUCE-2649. Handling of finished applications in RM. (Thomas Graves - via acmurthy) - - MAPREDUCE-2838. Fix MapReduce builds to use new hadoop-common test jars. - (gkesavan via acmurthy) - - MAPREDUCE-2859. Fix eclipse plugin contrib module compilation (gkesavan) - - MAPREDUCE-2846. Fix missing synchronization in the task log management. - (omalley) - - MAPREDUCE-2807. Fix AM restart and client redirection. (sharad) - - MAPREDUCE-2877. Add missing Apache license header in some files in MR - and also add the rat plugin to the poms. (mahadev) - - MAPREDUCE-2796. Set start times for MR applications for clients to see. - (Devaraj K via acmurthy) - - MAPREDUCE-2879. Fix version for MR-279 to 0.23.0. (acmurthy) - - MAPREDUCE-2881. Fix to include log4j 1.2.16 depenency (gkesavan) - - MAPREDUCE-2885. Fix mapred-config.sh to look for hadoop-config.sh in - HADOOP_COMMON_HOME/libexec. (acmurthy) - - MAPREDUCE-2893. Remove duplicate entry of YarnClientProtocolProvider in - ClientProtocolProvider services file. (Liang-Chi Hsieh via acmurthy) - - MAPREDUCE-2891. Javadoc for AMRMProtocol and related records. (acmurthy) - - MAPREDUCE-2898. Javadoc for ContainerManager protocol and related records. - (acmurthy) - - MAPREDUCE-2904. Fixed bin/yarn to correctly include HDFS jars and - clean up of stale refs to pre-mavenized Hadoop Common and HDFS. - (Sharad Agarwal and Arun C. Murthy via acmurthy) - - MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth - and Mahadev Konar via mahadev) - - MAPREDUCE-2886. Fix Javadoc warnings in MapReduce. (mahadev) - - MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records. - (acmurthy) - - MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for - common daemon. (mahadev) - - MAPREDUCE-2917. Fixed corner case in container reservation which led to - starvation and hung jobs. (acmurthy) - - MAPREDUCE-2756. Better error handling in JobControl for failed jobs. - (Robert Evans via acmurthy) - - MAPREDUCE-2716. MRReliabilityTest job fails because of missing - job-file. (Jeffrey Naisbitt via vinodkv) - - MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd) - - MAPREDUCE-2687. Fix NodeManager to use the right version of - LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy) - - MAPREDUCE-2800. Set final progress for tasks to ensure all task information - is correctly logged to JobHistory. (Siddharth Seth via acmurthy) - - MAPREDUCE-2938. Log application submission failure in CapacityScheduler. - (acmurthy) - - MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev) - - MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy) - - MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via - acmurthy) - - MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via - acmurthy) - - MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans - via acmurthy) - - MAPREDUCE-2937. Ensure reason for application failure is displayed to the - user. (mahadev via acmurthy) - - MAPREDUCE-2953. Fix a race condition on submission which caused client to - incorrectly assume application was gone by making submission synchronous - for RMAppManager. (Thomas Graves via acmurthy) - - MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy) - - MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization - in protocol buffer records. (Siddharth Seth via vinodkv) - - MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than - Configuration. (mahadev via acmurthy) - - MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList - (jobs) (Thomas Graves via mahadev) - - MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources - and related tests. (Siddharth Seth via vinodkv) - - MAPREDUCE-2749. Ensure NM registers with RM after starting all its services - correctly. (Thomas Graves via acmurthy) - - MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration - from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv) - - MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService. - (Thomas Graves via acmurthy) - - MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via - acmurthy) - - MAPREDUCE-2995. Better handling of expired containers in MapReduce - ApplicationMaster. (vinodkv via acmurthy) - - MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via - acmurthy) - - MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service - startup fails. (Ravi Teja via vinodkv) - - MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly - enforce locality constraints. (acmurthy) - - MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to - JobHistoryServer in secure mode. (vinodkv) - - MAPREDUCE-2987. Fixed display of logged user on RM Web-UI. (Thomas Graves - via acmurthy) - - MAPREDUCE-3006. Fixed MapReduce AM to exit only after properly writing out - history file. (vinodkv) - - MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed - jobs. (Devaraj K via vinodkv) - - MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to - reject all NMs. (Devaraj K via vinodkv) - - MAPREDUCE-3042. Fixed default ResourceTracker address. (Chris Riccomini - via acmurthy) - - MAPREDUCE-3038. job history server not starting because conf() missing - HsController (Jeffrey Naisbitt via mahadev) - - MAPREDUCE-3004. Fix ReduceTask to not assume 'local' mode in YARN. (Hitesh - Shah via acmurthy) - - MAPREDUCE-3017. The Web UI shows FINISHED for killed/successful/failed jobs. - (mahadev) - - MAPREDUCE-3040. Fixed extra copy of Configuration in - YarnClientProtocolProvider and ensured MiniMRYarnCluster sets JobHistory - configuration for tests. (acmurthy) - - MAPREDUCE-3018. Fixed -file option for streaming. (mahadev via acmurthy) - - MAPREDUCE-3036. Fixed metrics for reserved resources in CS. (Robert Evans - via acmurthy) - - MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork - bin/mapred too many times. (vinodkv via acmurthy) - - MAPREDUCE-3023. Fixed clients to display queue state correctly. (Ravi - Prakash via acmurthy) - - MAPREDUCE-2970. Fixed NPEs in corner cases with different configurations - for mapreduce.framework.name. (Venu Gopala Rao via vinodkv) - - MAPREDUCE-3062. Fixed default RMAdmin address. (Chris Riccomini - via acmurthy) - - MAPREDUCE-3066. Fixed default ResourceTracker address for the NodeManager. - (Chris Riccomini via acmurthy) - - MAPREDUCE-3044. Pipes jobs stuck without making progress. (mahadev) - - MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to - correct log-files. (Ravi Teja Ch N V via vinodkv) - - MAPREDUCE-3073. Fixed build issues in MR1. (mahadev via acmurthy) - - MAPREDUCE-2691. Increase threadpool size for launching containers in - MapReduce ApplicationMaster. (vinodkv via acmurthy) - - - MAPREDUCE-2990. Fixed display of NodeHealthStatus. (Subroto Sanyal via - acmurthy) - - MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf - RPCs. (vinodkv via acmurthy) - - MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics - for AM failures in a couple of corner cases. (Arun C Murthy via vinodkv) - - MAPREDUCE-3064. 27 unit test failures with Invalid - "mapreduce.jobtracker.address" configuration value for - JobTracker: "local" (Venu Gopala Rao via mahadev) - - MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than - (ApplicationId, startCount) consistently. (acmurthy) - - MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on - priority. (Sharad Agarwal and Arun C Murthy via vinodkv) - - MAPREDUCE-3031. Proper handling of killed containers to prevent stuck - containers/AMs on an external kill signal. (Siddharth Seth via vinodkv) - - MAPREDUCE-2984. Better error message for displaying completed containers. - (Devaraj K via acmurthy) - - MAPREDUCE-3071. app master configuration web UI link under the Job menu - opens up application menu. (thomas graves via mahadev) - - MAPREDUCE-3067. Ensure exit-code is set correctly for containers. (Hitesh - Shah via acmurthy) - - MAPREDUCE-2999. Fix YARN webapp framework to properly filter servlet - paths. (Thomas Graves via vinodkv) - - MAPREDUCE-3095. fairscheduler ivy including wrong version for hdfs. - (John George via mahadev) - - MAPREDUCE-3054. Unable to kill submitted jobs. (mahadev) - - MAPREDUCE-3021. Change base urls for RM web-ui. (Thomas Graves via - acmurthy) - - MAPREDUCE-3041. Fixed ClientRMProtocol to provide min/max resource - capabilities along-with new ApplicationId for application submission. - (Hitesh Shah via acmurthy) - - MAPREDUCE-2843. Fixed the node-table to be completely displayed and making - node entries on RM UI to be sortable. (Abhijit Suresh Shingate via vinodkv) - - MAPREDUCE-3110. Fixed TestRPC failure. (vinodkv) - - MAPREDUCE-3078. Ensure MapReduce AM reports progress correctly for - displaying on the RM Web-UI. (vinodkv via acmurthy) - - MAPREDUCE-3114. Fixed invalid ApplicationURL on RM WebUI. (Subroto Sanyal - via vinodkv) - - MAPREDUCE-2791. Added missing info on 'job -status' output. (Devaraj K via - acmurthy) - - MAPREDUCE-2996. Add uber-ness information to JobHistory. (Jonathan Eagles - via acmurthy) - - MAPREDUCE-3050. Add ability to get resource usage information for - applications and nodes. (Robert Evans via acmurthy) - - MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root - of the install properly. (Xie Xianshan via acmurthy) - - MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames. - (vinodkv via acmurthy) - - MAPREDUCE-3112. Fixed recursive sourcing of HADOOP_OPTS environment - variable. (Eric Yang) - - MAPREDUCE-3056. Changed the default staging directory to not include - user.name to prevent issues with non-secure mode. (Devaraj K via vinodkv) - - MAPREDUCE-2913. Fixed TestMRJobs.testFailingMapper to assert the correct - TaskCompletionEventStatus. (Jonathan Eagles via vinodkv) - - MAPREDUCE-2794. [MR-279] Incorrect metrics value for AvailableGB per - queue per user. (John George via mahadev) - - MAPREDUCE-2783. Fixing RM web-UI to show no tracking-URL when AM - crashes. (Eric Payne via vinodkv) - - MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security - mode.(vinodkv) - - MAPREDUCE-2751. Modified NodeManager to stop leaving around local files - after application finishes. (Siddharth Seth via vinodkv) - - MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn - frameworks. (Hitesh Shah via acmurthy) - - MAPREDUCE-2802. Ensure JobHistory filenames have jobId. (Jonathan Eagles - via acmurthy) - - MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer. - (Anupam Seth via acmurthy) - - MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by - MAPREDUCE-2702. (mahadev via acmurthy) - - MAPREDUCE-3123. Fix NM to quote symlink names to escape special - characters. (Hitesh Shah via acmurthy) - - MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying - job submission files to fail fast. (Abhijit Suresh Shingate via acmurthy) - - MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being - set to yarn. (Hitesh Shah via acmurthy) - - MAPREDUCE-3167. container-executor is not being packaged with the assembly - target. (mahadev) - - MAPREDUCE-3020. Fixed TaskAttemptImpl to log the correct node-address for - a finished Reduce task. (Chackaravarthy via vinodkv) - - MAPREDUCE-2668. Fixed AuxServices to send a signal on application-finish - to all the services. (Thomas Graves via vinodkv) - - MAPREDUCE-3126. Fixed a corner case in CapacityScheduler where headroom - wasn't updated on changes to cluster size. (acmurthy) - - MAPREDUCE-3140. Fixed the invalid JobHistory URL for failed - applications. (Subroto Sanyal via vinodkv) - - MAPREDUCE-3125. Modified TaskImpl to consider only non-failed, non-killed - task-attempts for obtaining task's progress. (Hitesh Shah via vinodkv) - - MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM - restart. (Jonathan Eagles via acmurthy) - - MAPREDUCE-2789. Complete schedulingInfo on CLI. (Eric Payne via acmurthy) - - MAPREDUCE-3170. Fixed job output commit for deep hierarchies. (Hitesh Shah - via acmurthy) - - MAPREDUCE-3124. Fixed location of native libs i.e. libhadoop.so for - containers. (John George via acmurthy) - - MAPREDUCE-3057. Job History Server goes of OutOfMemory with 1200 Jobs - and Heap Size set to 10 GB. (Eric Payne via mahadev) - - MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles - via mahadev) - - MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message - when HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. (todd & acmurthy - via acmurthy) - - MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via - mahadev) - - MAPREDUCE-3127. Changed default value of yarn.resourcemanager.acl.enable - to true and added some more documentation. (acmurthy) - - MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error - information about failed tasks. (Devaraj K via vinodkv) - - MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX. - (Arun Murthy via mahadev) - - MAPREDUCE-3197. TestMRClientService failing on building clean checkout of - branch 0.23 (mahadev) - - MAPREDUCE-2762. Cleanup MR staging directory on completion. (mahadev via - acmurthy) - - MAPREDUCE-3165. Ensure logging options are set correctly for MR AM and - tasks. (todd via acmurthy) - - MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev) - - MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for - tests. (vinodkv) - - MAPREDUCE-3162. Separated application-init and container-init event types - in NodeManager's Application state machine. (Todd Lipcon via vinodkv) - - MAPREDUCE-3176. Fixed ant mapreduce tests that are timing out because - of wrong framework name. (Hitesh Shah via vinodkv) - - MAPREDUCE-3181. Fixed MapReduce runtime to load yarn-default.xml and - yarn-site.xml. (acmurthy) - - MAPREDUCE-2788. Normalize resource requests in FifoScheduler - appropriately. (Ahmed Radwan via acmurthy) - - MAPREDUCE-2693. Fix NPE in job-blacklisting. (Hitesh Shah via acmurthy) - - MAPREDUCE-3208. Fix NPE task/container log appenders. (liangzhwa via - acmurthy) - - MAPREDUCE-3212. Fix usage/help message for bin/yarn. (Bhallamudi Venkata - Siva Kamesh via acmurthy) - - MAPREDUCE-3179. Ensure failed tests exit with right error code. (Jonathan - Eagles via acmurthy) - - MAPREDUCE-3188. Ensure correct shutdown in services. (todd via acmurthy) - - MAPREDUCE-3226. Fix shutdown of fetcher threads. (vinodkv via acmurthy) - - MAPREDUCE-3070. Fix NodeManager to use ephemeral ports by default. - (Devaraj K via acmurthy) - - MAPREDUCE-3242. Trunk compilation broken with bad interaction from - MAPREDUCE-3070 and MAPREDUCE-3239. (mahadev) - - MAPREDUCE-3058. Fixed MR YarnChild to report failure when task throws an - error and thus prevent a hanging task and job. (vinodkv) - - MAPREDUCE-3087. Fixed the mapreduce classpath to correctly include the - generated-classpath file needed for tests. (Ravi Prakash via vinodkv) - - MAPREDUCE-3233. Fixed a bug in MR Job so as to be able to restart the - application on AM crash. (Mahadev Konar via vinodkv) - - MAPREDUCE-3028. Added job-end notification support. (Ravi Prakash via - acmurthy) - - MAPREDUCE-3249. Ensure shuffle-port is correctly used duringMR AM recovery. - (vinodkv via acmurthy) - - MAPREDUCE-3252. Fix map tasks to not rewrite data an extra time when - map output fits in spill buffer. (todd) - - MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application - directories during app-init. (todd via acmurthy) - - MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy) - - MAPREDUCE-2746. Yarn servers can't communicate with each other with - hadoop.security.authorization set to true (acmurthy via mahadev) - - MAPREDUCE-2821. Added missing fields (resourcePerMap & resourcePerReduce) - to JobSummary logs. (mahadev via acmurthy) - - MAPREDUCE-3253. Fixed ContextFactory to clone JobContext correctly. - (acmurthy) - - MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. (Hitesh - Shah via acmurthy) - - MAPREDUCE-3269. Fixed log4j properties to correctly set logging options - for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy) - - MAPREDUCE-2977. Fix ResourceManager to renew HDFS delegation tokens for - applications. (acmurthy) - - MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM - and prints a lots of logs. (vinodkv via mahadev) - - MAPREDUCE-3254. Fixed streaming to set the job.jar by using the right - JobConf ctor. (acmurthy) - - MAPREDUCE-3264. mapreduce.job.user.name needs to be set automatically. - (acmurthy via mahadev) - - MAPREDUCE-3175. Add authorization to admin web-pages such as /stacks, /jmx - etc. (Jonathan Eagles via acmurthy) - - MAPREDUCE-3257. Added authorization checks for the protocol between - ResourceManager and ApplicationMaster. (vinodkv via acmurthy) - - MAPREDUCE-3259. Added java.library.path of NodeManager to - ContainerLocalizer in LinuxContainerExecutor. (Kihwal Lee via acmurthy) - - MAPREDUCE-3279. Fixed TestJobHistoryParsing which assumed user name to be - mapred all the time. (Siddharth Seth via acmurthy) - - MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its - containers (process-trees) irrespective of whether the container succeeded, - or killed. (Hitesh Shah via vinodkv) - - MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv) - - MAPREDUCE-3228. Fixed MR AM to timeout RPCs to bad NodeManagers. (vinodkv - via acmurthy) - - MAPREDUCE-3284. Moved JobQueueClient to hadoop-mapreduce-client-core. - (acmurthy) - - MAPREDUCE-3282. bin/mapred job -list throws exception. (acmurthy via - mahadev) - - MAPREDUCE-3186. User jobs are getting hanged if the Resource manager - process goes down and comes up while job is getting executed. - (Eric Payne via mahadev) - - MAPREDUCE-3209. Jenkins reports 160 FindBugs warnings (mahadev) - - MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly. - (Siddharth Seth via acmurthy) - - MAPREDUCE-3290. Fixed a NPE in ClientRMService. (acmurthy) - - MAPREDUCE-3185. RM Web UI does not sort the columns in some cases. - (Jonathan Eagles via mahadev) - - MAPREDUCE-3292. In secure mode job submission fails with Provider - org.apache.hadoop.mapreduce.security.token.JobTokenIndentifier$Renewer - not found. (mahadev) - - MAPREDUCE-3296. Fixed the remaining nine FindBugs warnings. (vinodkv) - - MAPREDUCE-2775. Fixed ResourceManager and NodeManager to force a - decommissioned node to shutdown. (Devaraj K via vinodkv) - - MAPREDUCE-3304. Fixed intermittent test failure due to a race in - TestRMContainerAllocator#testBlackListedNodes. (Ravi Prakash via acmurthy) - - MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing - NodeManager to crash. (vinodkv) - - MAPREDUCE-3256. Added authorization checks for the protocol between - NodeManager and ApplicationMaster. (vinodkv via acmurthy) - - MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a - task-scheduling deadlock. (Robert Joseph Evans via vinodkv) - - MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing - TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv) - - MAPREDUCE-2766. Fixed NM to set secure permissions for files and directories - in distributed-cache. (Hitesh Shah via vinodkv) - - MAPREDUCE-2696. Fixed NodeManager to cleanup logs in a thread when logs' - aggregation is not enabled. (Siddharth Seth via vinodkv) - - MAPREDUCE-3262. Fixed Container's state-machine in NodeManager to handle - a couple of events in failure states correctly. (Hitesh Shah and Siddharth - Seth via vinodkv) - - MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present. - (chakravarthy via acmurthy) - - MAPREDUCE-3321. Disabled a few MR tests for 0.23. (Hitesh Shah via - acmurthy) - - MAPREDUCE-3220. Fixed TestCombineOutputCollector. (Devaraj K via acmurthy) - - MAPREDUCE-3103. Implement Job ACLs for MRAppMaster. - (mahadev) - - MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk) - - MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying - on current history file name format. (Ravi Gummadi) - - MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file - names also. (Ravi Gummadi) - - MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas). - - MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra) - - MAPREDUCE-3192. Fix Javadoc warning in JobClient.java and Cluster.java. - (jitendra) - - MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core. - (tomwhite via acmurthy) - - MAPREDUCE-3316. Rebooted link is not working properly. - (Bhallamudi Venkata Siva Kamesh via mahadev) - - MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname. - (Ravi Gummadi via mahadev) - - MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/ - Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev) - - MAPREDUCE-3337. Added missing license headers. (acmurthy) - -Release 0.22.1 - Unreleased - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - MAPREDUCE-3837. Job tracker is not able to recover jobs after crash. - (Mayank Bansal via shv) - -Release 0.22.0 - 2011-11-29 - - INCOMPATIBLE CHANGES - - MAPREDUCE-1866. Removes deprecated class - org.apache.hadoop.streaming.UTF8ByteArrayUtils. (amareshwari) - - MAPREDUCE-1664. Changes the behaviour of the combination of job-acls - when they function together with queue-acls. (Ravi Gummadi via vinodkv) - - MAPREDUCE-2994. Fixed a bug in ApplicationID parsing that affects RM - UI. (Devaraj K via vinodkv) - - MAPREDUCE-1788. o.a.h.mapreduce.Job shouldn't make a copy of the JobConf. - (Arun Murthy via mahadev) - - NEW FEATURES - - MAPREDUCE-1804. Stress-test tool for HDFS introduced in HDFS-708. - (Joshua Harlow via shv) - - MAPREDUCE-220. Collect cpu and memory statistics per task. (Scott Chen via - acmurthy) - - MAPREDUCE-1970. Reed-Solomon code implementation for HDFS RAID. - (Scott Chen via dhruba) - - MAPREDUCE-2169. Integrated Reed-Solomon code with RaidNode. (Ramkumar - Vadali via schen) - - MAPREDUCE-2936. Contrib Raid compilation broken after HDFS-1620. (vinodkv) - - IMPROVEMENTS - - MAPREDUCE-2141. Add an "extra data" field to Task for use by Mesos. (matei) - - MAPREDUCE-2140. Regenerate fair scheduler design doc PDF. (matei) - - MAPREDUCE-1546. Redirect all job pages to corresponding history page - if job is not in JT memory. (Scott Chen via sharad) - - MAPREDUCE-1092. Enable assertions for unit tests. (Eli Collins via - cdouglas) - - MAPREDUCE-1680. Add a metric recording JobTracker heartbeats processed. - (Dick King via cdouglas) - - MAPREDUCE-1761. FairScheduler allows separate configuration of node - and rack locality wait time (Scott Chen via dhruba) - - MAPREDUCE-1539. authorization checks for inter-server protocol - (based on HADOOP-6600) (Boris Shkolnik via shv) - - MAPREDUCE-1798. Names the configuration keys for the Kerberos - principals better. (Boris Shkolnik via ddas) - - MAPREDUCE-1773. streaming doesn't support jobclient.output.filter. - (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1785. Add streaming config option for not emitting the key. - (Eli Collins via sharad) - - MAPREDUCE-572. If #link is missing from uri format of -cacheArchive - then streaming does not throw error. (Amareshwari Sriramadasu via - vinodkv) - - MAPREDUCE-1545. Add timestamps for first task type launched in job summary. - (Luke Lu via cdouglas) - - MAPREDUCE-1543. Add an audit log for authentication events. (Amar Kamat and - Luke Lu via cdouglas) - - MAPREDUCE-1762. Add ability to set values of task counters. (Scott Chen via - cdouglas) - - MAPREDUCE-1533. Reduce overhead of logging and string manipulation during - heartbeat processing. (Amar Kamat and Dick King via cdouglas) - - MAPREDUCE-1516. JobTracker issues delegation tokens only if the user's - authentication is Kerberos. (Jitendra Pandey via ddas) - - MAPREDUCE-647. Update distcp forrest documentation to reflect the changes - of HADOOP-5472, MAPREDUCE-642 and HADOOP-5620. (Rodrigo Schmidt via - szetszwo) - - MAPREDUCE-1851. Documents configuration parameters in streaming. - (amareshwari) - - MAPREDUCE-1868. Add a read and connection timeout to JobClient while - pulling tasklogs. (Krishna Ramachandran via acmurthy) - - MAPREDUCE-1778. Ensure failure to setup CompletedJobStatusStore is not - silently ignored by the JobTracker. (Krishna Ramachandran via acmurthy) - - MAPREDUCE-1850. Includes job submit host information (name and ip) in - jobconf and jobdetails display (Krishna Ramachandran via amareshwari) - - MAPREDUCE-1893. Slive with multiple reducers. (shv) - - MAPREDUCE-1248. Fixes redudant memory copying in StreamKeyValUtil. - (Ruibang He via amareshwari) - - MAPREDUCE-1840. Enhancements to Gridmix benchmark simulating user - diversity, queue replay, and task duration for JobTracker load testing. - Also includes compatibility with security enhancements, and scalability - improvements. (Amar Kamat, Rahul Singh, Hong Tang, and cdouglas) - - MAPREDUCE-1848. Put number of speculative, data local, rack local - tasks in JobTracker metrics. (Scott Chen via dhruba) - - MAPREDUCE-1935. Makes the Distcp to work in a secure environment. - (Boris Shkolnik via ddas) - - MAPREDUCE-1945. The MapReduce component for HADOOP-6632. - (Kan Zhang & Jitendra Pandey via ddas) - - MAPREDUCE-1936. Modify Gridmix3 to support more tunable parameters for - stress submission and sleep jobs. (Hong Tang via cdouglas) - - MAPREDUCE-1733. Makes pipes applications secure. (Jitendra Pandey via ddas) - - MAPREDUCE-1566. Adds a configuration attribute using which job clients can - specify a credentials file. The tokens from there will be passed to the job. - (Jitendra Pandey and Owen O'Malley via ddas) - - MAPREDUCE-1624. Documents the job credentials and associated details to do - with delegation tokens (on the client side). - (Jitendra Pandey and Devaraj Das via ddas) - - MAPREDUCE-1834. TestSimulatorDeterministicReplay timesout on trunk. - (Hong Tang via mahadev) - - MAPREDUCE-1993. Fixes test failure - TestTrackerDistributedCacheManagerWithLinuxTaskController. (Devaraj Das - via amareshwari) - - MAPREDUCE-1523. Making Mumak work with Capacity-Scheduler (Anirban Das - via mahadev) - - MAPREDUCE-1920. Enables completed jobstatus store by default. (Tom White - via amareshwari) - - MAPREDUCE-1881. Improve TaskTrackerInstrumentation to enable collection of - advanced metrics. (Matei Zaharia via acmurthy) - - MAPREDUCE-1548. Hadoop archives preserve times and other properties from - original files. (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1517. Supports streaming job to run in the background. (Bochun Bai - via amareshwari) - - MAPREDUCE-1819. RaidNode is now smarter in submitting Raid jobs. (Ramkumar - Vadali via schen) - - MAPREDUCE-2132. A command line option in RaidShell to fix blocks using raid - - MAPREDUCE-2147. Remove redundant lines in JobInProgress ctor. - (Harsh J Chouraria via cutting) - - HADOOP-7007. Update the hudson-test-patch ant target to work with the - latest test-patch.sh script (gkesavan) - - MAPREDUCE-1818. RaidNode can specify scheduling parameters. (Ramkumar - Vadali via schen) - - MAPREDUCE-2051. Contribute a fair scheduler preemption system test. - (Todd Lipcon via tomwhite) - - MAPREDUCE-1892. RaidNode can allow layered policies more efficiently. - (Ramkumar Vadali via schen) - - MAPREDUCE-1592. Generate Eclipse's .classpath file from Ivy config. - (tomwhite via nigel) - - MAPREDUCE-2073. TestTrackerDistributedCacheManager should be up-front - about requirements on build environment. (Todd Lipcon via tomwhite) - - MAPREDUCE-2093. Herriot JT and TT clients should vend statistics. (cos) - - MAPREDUCE-2167. Faster directory traversal for raid node. (Ramkumar Vadali - via schen) - - MAPREDUCE-1931. Gridmix forrest documentation . (Ranjit Mathew via vinodkv). - - MAPREDUCE-2184. Port DistRaid.java to new mapreduce API. (Ramkumar Vadali - via schen) - - MAPREDUCE-1878. Add MRUnit documentation. (Aaron Kimball via tomwhite) - - MAPREDUCE-2180. Add coverage of fair scheduler servlet to system test (todd) - - MAPREDUCE-2250. Fix logging in raid code. (Ramkumar Vadali via schen) - - MAPREDUCE-2260. Remove auto-generated native build files. (rvs via eli) - - MAPREDUCE-2314. configure files that are generated as part of the released - tarball need to have executable bit set (rvs via cos) - - MAPREDUCE-1159. Limit Job name in web UI to be 80 char long. (Harsh J - Chouraria via szetszwo) - - MAPREDUCE-2337. Remove dependence of public MapReduce API on classes in - server package. (tomwhite) - - MAPREDUCE-2383. Improve documentation of DistributedCache methods (Harsh J - Chouraria via todd) - - MAPREDUCE-2222. Ivy resolve force mode should be turned off by default. - (Luke Lu via tomwhite) - - MAPREDUCE-2103. task-controller shouldn't require o-r permissions. - (todd via eli) - - MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler. - (matei via eli) - - MAPREDUCE-3138. Add a utility to help applications bridge changes in - Context Objects APIs due to MAPREDUCE-954. (omalley via acmurthy) - - OPTIMIZATIONS - - MAPREDUCE-1354. Enhancements to JobTracker for better performance and - scalability. (Arun C. Murthy & Richard King via acmurthy) - - MAPREDUCE-1829. JobInProgress.findSpeculativeTask should use min() to - find the candidate instead of sort(). (Scott Chen via vinodkv) - - BUG FIXES - - MAPREDUCE-1845. FairScheduler.tasksToPreempt() can return negative number. - (Scott Chen via matei) - - MAPREDUCE-1707. TaskRunner can get NPE in getting ugi from TaskTracker. - (Vinod Kumar Vavilapalli) - - MAPREDUCE-1532. Ensures that delegation tokens is obtained as the - actual user when the proxy-user is used for submitting jobs. Also - refactors the DelegationTokenToRenew class. (ddas) - - MAPREDUCE-1558. Fixes MRAdmin to look up the principal of the - JobTracker and use that in the RefreshUserToGroupsMapping protocol and - RefreshAuthorizationPolicyProtocol. (Boris Shkolnik via ddas) - - MAPREDUCE-1662. Remove unused methods from TaskRunner. (Amareshwari - Sriramadasu via cdouglas) - - MAPREDUCE-1617. Use IPv4 stack for unit tests. (Amar Kamat and Luke Lu via - cdouglas) - - MAPREDUCE-1599. Fixes MRBench so that it reuses tokens across jobs - correctly. (Jitendra Nath Pandey via ddas) - - MAPREDUCE-1836. Refresh for proxy superuser config (mr part for HDFS-1096). - (Boris Shkolnik via shv) - - MAPREDUCE-1505. Create RPC client on job submission, not in cstr of Job - instance. (Dick King via cdouglas) - - MAPREDUCE-1813. NPE in PipeMapred.MRErrorThread. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1225. Fixes DistributedCache to check if the file is fresh or not, - for the first localization also. (Zhong Wang via amareshwari) - - MAPREDUCE-1559. Fixes the token renewer to use the JobTracker's - credentials for talking to the NameNode. (ddas) - - MAPREDUCE-1492. Delete obsolete har files used on the parity files - of hdfs raid. (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1857. Removes unused configuration parameters in streaming. - (amareshwari) - - MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root - paths. (Aaron Kimball via zshao) - - MAPREDUCE-1863. Fix NPE in Rumen when processing null CDF for failed task - attempts. (Amar Kamat via cdouglas) - - MAPREDUCE-1864. Removes uninitialized/unused variables in - org.apache.hadoop.streaming.PipeMapRed. (amareshwari) - - MAPREDUCE-1888. Fixes Streaming to override output key and value types, - only if mapper/reducer is a command. (Ravi Gummadi via amareshwari) - - MAPREDUCE-577. Fixes duplicate records in StreamXmlRecordReader. - (Ravi Gummadi via amareshwari) - - MAPREDUCE-1894. Fixed a bug in DistributedRaidFileSystem.readFully() - that was causing it to loop infinitely. (Ramkumar Vadali via dhruba) - - MAPREDUCE-1838. Reduce the time needed for raiding a bunch of files - by randomly assigning files to map tasks. (Ramkumar Vadali via dhruba) - - MAPREDUCE-1820. Fix InputSampler to clone sampled keys. (Alex Kozlov via - cdouglas) - - MAPREDUCE-1528. Incorporates the changes to the credentials API done in - HADOOP-6845. Also, introduces Credentials in JobConf, and in JobContext. - (Jitendra Pandey and Arun Murthy via ddas) - - MAPREDUCE-1865. Rumen should also support jobhistory files generated using - trunk. (Amar Kamat via amareshwari) - - MAPREDUCE-1621. Fixes NPE in TextOutputReader.getLastOutput if it has never - read any output. (amareshwari) - - MAPREDUCE-1911. Fixes errors in -info message in streaming. (amareshwari) - - MAPREDUCE-1772. Corrects errors in streaming documentation in forrest. - (amareshwari) - - MAPREDUCE-1925. Fix failing TestRumenJobTraces. (Ravi Gummadi via cdouglas) - - MAPREDUCE-1718. Fixes a bug in the construction of jobconf key for the - mapping that the tasks use at runtime for looking up delegation tokens. - (Boris Shkolnik via ddas) - - MAPREDUCE-1701. Fixes a problem to do with exception handling in - delegation-token renewals. (Boris Shkolnik via ddas) - - MAPREDUCE-1686. Fixes StreamUtil.goodClassOrNull to find classes without - package names. (Paul Burkhardt via amareshwari) - - MAPREDUCE-1288. Fixes TrackerDistributedCacheManager to take into account - the owner of the localized file in the mapping from cache URIs to - CacheStatus objects. (ddas) - - MAPREDUCE-1982. Fixes Rumen's TraceBuilder to extract job name from either - of configuration properties "mapreduce.job.name" and "mapred.job.name". - (Ravi Gummadi via amareshwari) - - MAPREDUCE-1958. The MapReduce part corresponding to the HADOOP-6873. - (Boris Shkolnik & Owen O'Malley via ddas) - - MAPREDUCE-1900. TaskTracker and JobTracker closes FileSystems, opened on - behalf of users that it no longer requires. (Kan Zhang and ddas via ddas) - - MAPREDUCE-1992. Fixes a problem to do with bringing up the JobTracker in - unsecure mode. (Kan Zhang via ddas) - - MAPREDUCE-1999. Fixes ClientProtocol to use the correct - DelegationTokenSelector. (Jitendra Pandey via ddas) - - MAPREDUCE-1780. AccessControlList.toString() is used for serialization of - ACL in JobStatus.java. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1961. Fix ConcurrentModificationException in Gridmix during - shutdown. (Hong Tang via cdouglas) - - MAPREDUCE-2000. Fix parsing of JobHistory lines in Rumen when quotes are - escaped. (Hong Tang via cdouglas) - - MAPREDUCE-2022. Fixes compilation errors in TestSubmitJob. (amareshwari) - - MAPREDUCE-1670. RAID policies should not scan their own destination path. - (Ramkumar Vadali via dhruba) - - MAPREDUCE-1668. RaidNode Hars a directory only if all its parity files - have been created. (Ramkumar Vadali via dhruba) - - MAPREDUCE-2021. Fixes duplicate hostnames in CombineFileInputFormat's - split locations. (amareshwari) - - MAPREDUCE-1375. Fixes flaky test TestFileArgs. (Todd Lipcon via - amareshwari) - - MAPREDUCE-2023. TestDFSIO should not stop reading if curSize != bufferSize. - (Hong Tang via szetszwo) - - MAPREDUCE-2031. Fixes test failures TestTaskLauncher and - TestTaskTrackerLocalization. (Ravi Gummadi via amareshwari) - - MAPREDUCE-2046. Fixes CombineFileInputFormat to allow splits with size - less than DFS block size. (dhruba borthakur via amareshwari) - - MAPREDUCE-1975. Fixes unnecessary InterruptedException log in gridmix. - (Ravi Gummadi via amareshwari) - - MAPREDUCE-1597. Fixes CombineFileInputFormat to work with non-splittable - files. (amareshwari) - - MAPREDUCE-2032. Fixes TestJobCleanup to cleanup test directory in - tearDown. (Dick King via amareshwari) - - MAPREDUCE-1979. Fixes "Output directory already exists" error in gridmix - when gridmix.output.directory is not defined. (Ravi Gummadi via - amareshwari) - - MAPREDUCE-1918. Adds documentation to Rumen. (Amar Kamat via amareshwari) - - MAPREDUCE-2078. Fixes TraceBuilder to generate traces when a globbed job - history path is given. (Amar Kamat via amareshwari) - - MAPREDUCE-1989. Fixes error message in gridmix when user resolver is set - and no user list is given. (Ravi Gummadi via amareshwari) - - MAPREDUCE-2067. Distinct minicluster services (e.g. NN and JT) overwrite - each other's service policies. (Aaron T. Myers via tomwhite) - - MAPREDUCE-2029. DistributedRaidFileSystem removes itself from FileSystem - cache when it is closed. (Ramkumar Vadali via dhruba) - - MAPREDUCE-1816. HAR files used for RAID parity-bite have configurable - partfile size. (Ramkumar Vadali via dhruba) - - MAPREDUCE-2082. Fixes Pipes to create the jobtoken file in the right - place. (Jitendra Pandey via ddas) - - MAPREDUCE-2095. Fixes Gridmix to run from compressed traces. (Ranjit - Mathew via amareshwari) - - MAPREDUCE-1908. DistributedRaidFileSystem now handles ChecksumException - correctly. (Ramkumar Vadali via schen) - - MAPREDUCE-2126. JobQueueJobInProgressListener's javadoc is inconsistent - with source code. (Jingguo Yao via tomwhite) - - MAPREDUCE-2143. HarFileSystem is able to handle spaces in pathnames. - (Ramkumar Vadali via dhruba) - - MAPREDUCE-1867. Remove unused methods in - org.apache.hadoop.streaming.StreamUtil. (amareshwari via tomwhite) - - MAPREDUCE-2146. Raid does not affect access time of a source file. - (Ramkumar Vadali via dhruba) - - MAPREDUCE-2150. RaidNode periodically fixes corrupt blocks. (Ramkumar Vadali via - schen) - - MAPREDUCE-2099. RaidNode recreates outdated parity HARs. (Ramkumar Vadali - via schen) - - MAPREDUCE-2173. Fix race condition in TestBlockFixer that was - causing intermittent failure (Patrick Kling via dhruba) - - MAPREDUCE-2142. Refactor RaidNode so that the map-reduce component is - clearly separated out. (Patrick Kling via dhruba) - - MAPREDUCE-2179. Fix RaidBlockSender compilation failure. (Ramkumar Vadali - via schen) - - MAPREDUCE-2034. TestSubmitJob triggers NPE instead of permissions error. - (Todd Lipcon via tomwhite) - - MAPREDUCE-2195. New property for local conf directory in - system-test-mapreduce.xml file. (cos) - - MAPREDUCE-1783. FairScheduler initializes tasks only when the job can be - run. (Ramkumar Vadali via schen) - - MAPREDUCE-2224. Fix synchronization bugs in JvmManager. (todd) - - MAPREDUCE-714. JobConf.findContainingJar unescapes unnecessarily on linux (todd) - - MAPREDUCE-2096. Secure local filesystem IO from symlink vulnerabilities (todd) - - MAPREDUCE-2234. If Localizer can't create task log directory, it should fail - on the spot. (todd) - - MAPREDUCE-2219. JobTracker should not try to remove mapred.system.dir - during startup. (todd) - - MAPREDUCE-2207. Task-cleanup task should not be scheduled on the node that - the task just failed. (Liyin Liang via schen) - - MAPREDUCE-2084. Remove deprecate annotation for package file. The package - classes themselves are already deprecated. This removes an Eclipse error. - (tomwhite via nigel) - - MAPREDUCE-2248. DistributedRaidFileSystem should unraid only the corrupt - block (Ramkumar Vadali via schen) - - MAPREDUCE-1085. For tasks, "ulimit -v -1" is being run when user doesn't - specify a ulimit (todd) - - MAPREDUCE-2282. Fix TestMRServerPorts for the changes in - TestHDFSServerPorts. (shv via szetszwo) - - MAPREDUCE-2238. Fix permissions handling to avoid leaving undeletable - directories in local dirs. (todd) - - MAPREDUCE-2277. TestCapacitySchedulerWithJobTracker needs to wait for jobs - to complete before testing status. (todd) - - MAPREDUCE-2253. Servlets should specify content type (todd) - - MAPREDUCE-2283. Add timeout for Raid Tests (Ramkumar Vadali via schen) - - MAPREDUCE-1754. Replace mapred.persmissions.supergroup with an - acl : mapreduce.cluster.administrators (Amareshwari Sriramadasu via shv) - - MAPREDUCE-2256. FairScheduler fairshare preemption from multiple pools may - preempt all tasks from one pool causing that pool to go below fairshare. - (Priyo Mustafi via shv) - - MAPREDUCE-2281. MR part of HADOOP-6642. (Chris Douglas, Po Cheung via shv) - - MAPREDUCE-2200. TestUmbilicalProtocolWithJobToken is failing without Krb - evironment: needs to be conditional. (cos) - - MAPREDUCE-2077. Resolve name clash in the deprecated - o.a.h.util.MemoryCalculatorPlugin (Luke Lu via shv) - - MAPREDUCE-2188. The new API MultithreadedMapper doesn't initialize - RecordReader. (Owen O'Malley via shv) - - MAPREDUCE-1915. Fix IndexOutOfBoundsException in IndexCache. - (Priyo Mustafi via shv) - - MAPREDUCE-1974. Fix multiple preemtions of the same task in FairScheduler. - (Scott Chen via shv) - - MAPREDUCE-2304. Fix TestMRCLI to allow hostname with a hyphen (-). - (Priyo Mustafi via shv) - - MAPREDUCE-1825. jobqueue_details.jsp and FairSchedulerServelet should not - call finishedMaps and finishedReduces when job is not initialized. - (Scott Chen via shv) - - MAPREDUCE-2285. MiniMRCluster does not start after ant test-patch - (todd) - - MAPREDUCE-2315. javadoc is failing in nightly build (todd) - - MAPREDUCE-2054. Hierarchical queue implementation broke dynamic queue - addition in Dynamic Scheduler. (Thomas Sandholm via tomwhite) - - MAPREDUCE-2272. Job ACL file should not be executable - (Harsh J Chouraria via todd) - - MAPREDUCE-2241. ClusterWithLinuxTaskController should accept relative path - on the command line. (todd) - - MAPREDUCE-2251. Remove unused mapreduce.job.userhistorylocation config. - (Harsh J Chouraria via todd) - - MAPREDUCE-2284. TestLocalRunner.testMultiMaps times out (todd) - - MAPREDUCE-2336. Tool-related packages should be in the Tool javadoc group. - (tomwhite) - - MAPREDUCE-2394. JUnit output format doesn't propagate into raid contrib - build. (todd) - - MAPREDUCE-2392. TaskTracker shutdown in the tests sometimes take 60s. - (tomwhite) - - MAPREDUCE-2437. SLive uses only part* files to generating the final report. - (shv) - - MAPREDUCE-2428. start-mapred.sh script fails if HADOOP_HOME is not set. - (tomwhite via eli) - - MAPREDUCE-2445. Fix TestMiniMRWithDFSWithDistinctUsers to be a valid test. - (todd) - - MAPREDUCE-2457. Job submission should inject group.name on the JobTracker - (Alejandro Abdelnur via todd) - - MAPREDUCE-2472. Extra whitespace in mapred.child.java.opts breaks JVM - initialization. (Aaron T. Myers via todd) - - MAPREDUCE-2222. Ivy resolve force mode should be turned off by default. - (Luke Lu via tomwhite) - - MAPREDUCE-2486. Incorrect snapshot dependency published in .pom files - (todd) - - MAPREDUCE-2327. MapTask doesn't need to put username information in - SpillRecord. (todd via tomwhite) - - MAPREDUCE-2515. MapReduce code references some deprecated options - (Ari Rabkin via todd) - - MAPREDUCE-2487. ChainReducer uses MAPPER_BY_VALUE instead of - REDUCER_BY_VALUE. (Devaraj K via todd) - - MAPREDUCE-2185. Fix infinite loop at creating splits using - CombineFileInputFormat. (Ramkumar Vadali via schen) - - MAPREDUCE-2571. CombineFileInputFormat.getSplits throws a - java.lang.ArrayStoreException. (Bochun Bai via todd) - - MAPREDUCE-2767. Remove Linux task-controller. (Milind Bhandarkar via shv) - - MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity - scheduler if queue names contain special symbols. (Priyo Mustafi via shv) - - MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file. - (Ming Ma via shv) - - MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via - acmurthy) - - MAPREDUCE-3139. SlivePartitioner generates negative partitions. (jghoman) - -Release 0.21.1 - Unreleased - - NEW FEATURES - - MAPREDUCE-2040. Forrest Documentation for Dynamic Priority Scheduler. - (Thomas Sandholm via tomwhite) - - BUG FIXES - - MAPREDUCE-1897. trunk build broken on compile-mapred-test (cos) - - MAPREDUCE-1280. Update Eclipse plugin to the new eclipse.jdt API. - (Alex Kozlov via szetszwo) - - MAPREDUCE-1984. herriot TestCluster fails because exclusion is not there - (Balaji Rajagopalan via cos) - - MAPREDUCE-2090. Clover build doesn't generate per-test coverage. (cos) - - MAPREDUCE-2134. ant binary-system is broken in mapreduce project. (cos) - - MAPREDUCE-1905. Fixes Context.setStatus() and progress() apis. - (amareshwari) - - MAPREDUCE-1809. Ant build changes for Streaming system tests in contrib - projects. (Vinay Kumar Thota via amareshwari) - - MAPREDUCE-2223. TestMRCLI might fail on Ubuntu with default /etc/hosts - (cos) - - MAPREDUCE-2228. Remove java5 dependencies from build. (cos) - - MAPREDUCE-1929. Allow artifacts to be published to the staging Apache Nexus - Maven Repository. (tomwhite) - - MAPREDUCE-2317. Fix a NPE in HadoopArchives. (Devaraj K via szetszwo) - - MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. - (Bruno Mahé via eli) - -Release 0.21.0 - 2010-08-13 - - INCOMPATIBLE CHANGES - - MAPREDUCE-516. Fix the starvation problem in the Capacity Scheduler - when running High RAM Jobs. (Arun Murthy via yhemanth) - - MAPREDUCE-358. Change org.apache.hadoop.examples. AggregateWordCount - and org.apache.hadoop.examples.AggregateWordHistogram to use new - mapreduce api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-245. Change Job and jobcontrol classes to use the List interface - rather than ArrayList in APIs. (Tom White via cdouglas) - - MAPREDUCE-766. Enhanced list-blacklisted-trackers to display reasons - for blacklisting a node. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-817. Add a cache for retired jobs with minimal job info and - provide a way to access history file url. (sharad) - - MAPREDUCE-711. Moved Distributed Cache from Common to Map/Reduce - project. (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-895. Per the contract elucidated in HADOOP-6201, throw - FileNotFoundException from FileSystem::listStatus rather than returning - null. (Jakob Homan via cdouglas) - - MAPREDUCE-479. Provide full task id to map output servlet rather than the - reduce id, only. (Jiaqi Tan via cdouglas) - - MAPREDUCE-873. Simplify job recovery. Incomplete jobs are resubmitted on - jobtracker restart. Removes a public constructor in JobInProgress. (sharad) - - HADOOP-6230. Moved process tree and memory calculator related classes from - Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-157. Refactor job history APIs and change the history format to - JSON. (Jothi Padmanabhan via sharad) - - MAPREDUCE-849. Rename configuration properties. (Amareshwari Sriramadasu - via sharad) - - MAPREDUCE-1287. Only call the partitioner with more than one reducer. - (cdouglas) - - MAPREDUCE-1385. Use the new UserGroupInformation from HADOOP-6299. - (ddas via omalley) - - MAPREDUCE-1493. Authorization for job-history pages. (vinodkv) - - MAPREDUCE-1607. Task controller may not set permissions for a - task cleanup attempt's log directory (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1683. Remove JNI calls from ClusterStatus cstr. (Arun Murthy and - Luke Lu via cdouglas) - - MAPREDUCE-1855. Makes the refresh methods (for groups and proxy users) - independent of the client side configuration. (Boris Shkolnik via ddas) - - NEW FEATURES - - MAPREDUCE-1774. Large-scale Automated Framework (Sharad Agarwal, Sreekanth - Ramakrishnan, Konstantin Boudnik, at all via cos) - - MAPREDUCE-706. Support for FIFO pools in the fair scheduler. - (Matei Zaharia) - - MAPREDUCE-546. Provide sample fair scheduler config file in conf/ and use - it by default if no other config file is specified. (Matei Zaharia) - - MAPREDUCE-551. Preemption support in the Fair Scheduler. (Matei Zaharia) - - MAPREDUCE-567. Add a new example MR that always fails. (Philip Zeyliger - via tomwhite) - - MAPREDUCE-211. Provides ability to run a health check script on the - tasktracker nodes and blacklist nodes if they are unhealthy. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-637. Add an example, distbbp, which able to compute the n th bit - of Pi for some large n. (szetszwo) - - MAPREDUCE-532. Provide a way to limit the number of used slots - per queue in the capacity scheduler. - (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-467. Provide ability to collect statistics about total tasks - and succeeded tasks in different time windows. (sharad) - - MAPREDUCE-740. Log a job-summary at the end of a job, while allowing it - to be configured to use a custom appender if desired. (acmurthy) - - MAPREDUCE-814. Provide a way to configure completed job history files - to be on HDFS. (sharad) - - MAPREDUCE-800. MRUnit should support the new API. (Aaron Kimball via - tomwhite) - - MAPREDUCE-798. MRUnit should be able to test a succession of MapReduce - passes. (Aaron Kimball via tomwhite) - - MAPREDUCE-768. Provide an option to dump jobtracker configuration in JSON - format to standard output. (V.V.Chaitanya Krishna via yhemanth) - - MAPREDUCE-824. Add support for a hierarchy of queues in the capacity - scheduler. (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-751. Add Rumen, a tool for extracting statistics from job tracker - logs and generating job traces for simulation and analysis. (Dick King via - cdouglas) - - MAPREDUCE-830. Add support for splittable compression to TextInputFormats. - (Abdul Qadeer via cdouglas) - - MAPREDUCE-861. Add support for hierarchical queues in the Map/Reduce - framework. (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-776. Add Gridmix, a benchmark processing Rumen traces to simulate - a measured mix of jobs on a cluster. (cdouglas) - - MAPREDUCE-862. Enhance JobTracker UI to display hierarchical queues. - (V.V.Chaitanya Krishna via yhemanth) - - MAPREDUCE-777. Brand new apis to track and query jobs as a - replacement for JobClient. (Amareshwari Sriramadasu via acmurthy) - - MAPREDUCE-775. Add native and streaming support for Vertica as an input - or output format taking advantage of parallel read and write properties of - the DBMS. (Omer Trajman via ddas) - - MAPREDUCE-679. XML-based metrics as JSP servlet for JobTracker. - (Aaron Kimball via tomwhite) - - MAPREDUCE-980. Modify JobHistory to use Avro for serialization. (cutting) - - MAPREDUCE-728. Add Mumak, a Hadoop map/reduce simulator. (Arun C Murthy, - Tamas Sarlos, Anirban Dasgupta, Guanying Wang, and Hong Tang via cdouglas) - - MAPREDUCE-1383. Automates fetching of delegation tokens in File*Formats - Distributed Cache and Distcp. Also, provides a config - mapreduce.job.hdfs-servers that the jobs can populate with a comma - separated list of namenodes. The job client automatically fetches - delegation tokens from those namenodes. (Boris Shkolnik via ddas) - - MAPREDUCE-698. Per-pool task limits for the fair scheduler. - (Kevin Peterson via matei) - - MAPREDUCE-1026. Does mutual authentication of the shuffle - transfers using a shared JobTracker generated key. - (Boris Shkolnik via ddas) - - MAPREDUCE-744. Introduces the notion of a public distributed cache. - (Devaraj Das) - - MAPREDUCE-1338. Introduces the notion of token cache using which - tokens and secrets can be sent by the Job client to the JobTracker. - (Boris Shkolnik via ddas) - - HDFS-503. This patch implements an optional layer over HDFS that - implements offline erasure-coding. It can be used to reduce the - total storage requirements of HDFS. (dhruba) - - MAPREDUCE-1432. Adds hooks in the jobtracker and tasktracker - for loading the tokens in the user's ugi. This is required - for the copying of files from the hdfs. (ddas) - - MAPREDUCE-1335. Adds SASL Kerberos/Digest authentication in MapReduce. - (Kan Zhang via ddas) - - MAPREDUCE-1464. Makes a compatible change in JobTokenIdentifier to - account for HADOOP-6510. (Jitendra Nath Pandey via ddas) - - MAPREDUCE-1433. Add a delegation token for MapReduce. (omalley) - - MAPREDUCE-1307. Introduces the Job level ACLs feature. - (Vinod Kumar Vavilapalli via ddas) - - MAPREDUCE-1430. JobTracker automatically renews delegation tokens for jobs. - (Boris Shkolnik via ddas) - - MAPREDUCE-1455. Introduces job-level authorization for mapreduce servlets. - (Ravi Gummadi via vinodkv) - - IMPROVEMENTS - - MAPREDUCE-463. Makes job setup and cleanup tasks as optional. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-502. Allow jobtracker to be configured with zero completed jobs - in memory. (Amar Kamat via sharad) - - MAPREDUCE-416. Moves the history file to a "done" folder whenever a job - completes. (Amar Kamat via ddas) - - MAPREDUCE-646. Increase srcfilelist replication number in dictcp job. - (Ravi Gummadi via szetszwo) - - HADOOP-6106. Updated hadoop-core and test jars from hudson trunk - build #12. (Giridharan Kesavan) - - MAPREDUCE-642. A option to distcp that allows preserving the full - source path of a file in the specified destination directory. - (Rodrigo Schmidt via dhruba) - - MAPREDUCE-686. Move TestSpeculativeExecution.Fake* into a separate class - so that it can be used by other tests. (Jothi Padmanabhan via sharad) - - MAPREDUCE-625. Modify TestTaskLimits to improve execution time. - (Jothi Padmanabhan via sharad) - - MAPREDUCE-465. Deprecate o.a.h.mapred.lib.MultithreadedMapRunner and add - test for o.a.h.mapreduce.lib.MultithreadedMapper. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-701. Improves the runtime of the TestRackAwareTaskPlacement - by making it a unit test. (Jothi Padmanabhan via ddas) - - MAPREDUCE-371. Change KeyFieldBasedComparator and KeyFieldBasedPartitioner - to use new api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-623. Resolve javac warnings in mapreduce. (Jothi Padmanabhan - via sharad) - - MAPREDUCE-655. Change KeyValueLineRecordReader and KeyValueTextInputFormat - to use new mapreduce api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-632. Merge TestCustomOutputCommitter with - TestCommandLineJobSubmission. (Jothi Padmanabhan via sharad) - - MAPREDUCE-627. Improves execution time of TestTrackerBlacklistAcrossJobs. - (Jothi Padmanabhan via ddas) - - MAPREDUCE-630. Improves execution time of TestKillCompletedJob. - (Jothi Padmanabhan via ddas) - - MAPREDUCE-626. Improves the execution time of TestLostTracker. - (Jothi Padmanabhan via ddas) - - MAPREDUCE-353. Makes the shuffle read and connection timeouts - configurable. (Ravi Gummadi via ddas) - - MAPREDUCE-739. Allow relative paths to be created in archives. (Mahadev - Konar via cdouglas) - - MAPREDUCE-772. Merge HADOOP-4010 changes to LineRecordReader into mapreduce - package. (Abdul Qadeer via cdouglas) - - MAPREDUCE-785. Separate sub-test of TestReduceFetch to be included in - MR-670. (Jothi Padmanabhan via cdouglas) - - MAPREDUCE-784. Modify TestUserDefinedCounters to use LocalJobRunner - instead of MiniMR. (Jothi Padmanabhan via sharad) - - HADOOP-6160. Fix releaseaudit target to run on specific directories. - (gkesavan) - - MAPREDUCE-782. Use PureJavaCrc32 in SpillRecord. (Todd Lipcon via - szetszwo) - - MAPREDUCE-369. Change org.apache.hadoop.mapred.lib.MultipleInputs to - use new api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-373. Change org.apache.hadoop.mapred.lib.FieldSelectionMapReduce - to use new api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-628. Improves the execution time of TestJobInProgress. - (Jothi Padmanabhan via ddas) - - MAPREDUCE-793. Creates a new test that consolidates a few tests to - include in the commit-test list. (Jothi Padmanabhan via ddas) - - MAPREDUCE-797. Adds combiner support to MRUnit MapReduceDriver. - (Aaron Kimball via johan) - - MAPREDUCE-656. Change org.apache.hadoop.mapred.SequenceFile* classes - to use new mapreduce api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-670. Creates ant target for 10 mins patch test build. - (Jothi Padmanabhan via gkesavan) - - MAPREDUCE-375. Change org.apache.hadoop.mapred.lib.NLineInputFormat - and org.apache.hadoop.mapred.MapFileOutputFormat to use new api. - (Amareshwari Sriramadasu via ddas) - - MAPREDUCE-779. Added node health failure counts into - JobTrackerStatistics. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-842. Setup secure permissions for localized job files, - intermediate outputs and log files on tasktrackers. - (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-478. Allow map and reduce jvm parameters, environment variables - and ulimit to be set separately. - Configuration changes: - add mapred.map.child.java.opts - add mapred.reduce.child.java.opts - add mapred.map.child.env - add mapred.reduce.child.ulimit - add mapred.map.child.env - add mapred.reduce.child.ulimit - deprecated mapred.child.java.opts - deprecated mapred.child.env - deprecated mapred.child.ulimit - (acmurthy) - - MAPREDUCE-767. Remove the dependence on the CLI 2.0 snapshot. - (Amar Kamat via omalley) - - MAPREDUCE-712. Minor efficiency tweaks to RandomTextWriter. (cdouglas) - - MAPREDUCE-870. Remove the job retire thread and the associated - config parameters. (sharad) - - MAPREDUCE-874. Rename the PiEstimator example to QuasiMonteCarlo. - (szetszwo) - - MAPREDUCE-336. Allow logging level of map/reduce tasks to be configurable. - Configuration changes: - add mapred.map.child.log.level - add mapred.reduce.child.log.level - (acmurthy) - - MAPREDUCE-355. Update mapred.join package to use the new API. (Amareshwari - Sriramadasu via cdouglas) - - HADOOP-6184. Updated hadoop common and test jars to get the new API - in Configuration for dumping in JSON format from Hudson trunk build #68. - (yhemanth) - - MAPREDUCE-476. Extend DistributedCache to work locally (LocalJobRunner). - (Philip Zeyliger via tomwhite) - - MAPREDUCE-825. JobClient completion poll interval of 5s causes slow tests - in local mode. (Aaron Kimball via tomwhite) - - MAPREDUCE-910. Support counters in MRUnit. (Aaron Kimball via cdouglas) - - MAPREDUCE-788. Update gridmix2 to use the new API (Amareshwari Sriramadasu - via cdouglas) - - MAPREDUCE-875. Make DBRecordReader execute queries lazily. (Aaron Kimball - via enis) - - MAPREDUCE-318. Modularizes the shuffle code. (Jothi Padmanabhan and - Arun Murthy via ddas) - - MAPREDUCE-936. Allow a load difference for fairshare scheduler. - (Zheng Shao via dhruba) - - MAPREDUCE-370. Update MultipleOutputs to use the API, merge funcitonality - of MultipleOutputFormat. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-898. Changes DistributedCache to use the new API. - (Amareshwari Sriramadasu via ddas) - - MAPREDUCE-144. Includes dump of the process tree in task diagnostics when - a task is killed due to exceeding memory limits. - (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-945. Modifies MRBench and TestMapRed to use ToolRunner so that - options such as queue name can be passed via command line. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-963. Deprecate o.a.h.mapred.FileAlreadyExistsException and - replace it with o.a.h.fs.FileAlreadyExistsException. (Boris Shkolnik - via szetszwo) - - MAPREDUCE-960. Remove an unnecessary intermediate copy and obsolete API - from KeyValueLineRecordReader. (cdouglas) - - MAPREDUCE-930. Modify Rumen to resolve paths in the canonical way, rather - than defaulting to the local filesystem. (cdouglas) - - MAPREDUCE-944. Extend the LoadManager API of the fair-share scheduler - to support regulating tasks for a job based on resources currently in use - by that job. (dhruba) - - MAPREDUCE-973. Move FailJob and SleepJob from examples to test. (cdouglas - via omalley) - - MAPREDUCE-966. Modify Rumen to clean up interfaces and simplify integration - with other tools. (Hong Tang via cdouglas) - - MAPREDUCE-856. Setup secure permissions for distributed cache files. - (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-885. More efficient SQL queries for DBInputFormat. (Aaron Kimball - via enis) - - MAPREDUCE-284. Enables ipc.client.tcpnodelay in Tasktracker's Child. - (Ravi Gummadi via sharad) - - MAPREDUCE-916. Split the documentation to match the project split. - (Corinne Chandel via omalley) - - MAPREDUCE-649. Validate a copy by comparing the source and destination - checksums in distcp. Also adds an intra-task retry mechanism for errors - detected during the copy. (Ravi Gummadi via cdouglas) - - MAPREDUCE-654. Add a -dryrun option to distcp printing a summary of the - file data to be copied, without actually performing the copy. (Ravi Gummadi - via cdouglas) - - MAPREDUCE-664. Display the number of files deleted by distcp when the - -delete option is specified. (Ravi Gummadi via cdouglas) - - MAPREDUCE-781. Let the name of distcp jobs be configurable. (Venkatesh S - via cdouglas) - - MAPREDUCE-975. Add an API in job client to get the history file url for - a given job id. (sharad) - - MAPREDUCE-905. Add Eclipse launch tasks for MapReduce. (Philip Zeyliger - via tomwhite) - - MAPREDUCE-277. Makes job history counters available on the job history - viewers. (Jothi Padmanabhan via ddas) - - MAPREDUCE-893. Provides an ability to refresh queue configuration - without restarting the JobTracker. - (Vinod Kumar Vavilapalli and Rahul Kumar Singh via yhemanth) - - MAPREDUCE-1011. Add build.properties to svn and git ignore. (omalley) - - MAPREDUCE-954. Change Map-Reduce context objects to be interfaces. - (acmurthy) - - MAPREDUCE-639. Change Terasort example to reflect the 2009 updates. - (omalley) - - MAPREDUCE-1063. Document gridmix benchmark. (cdouglas) - - MAPREDUCE-931. Use built-in interpolation classes for making up task - runtimes in Rumen. (Dick King via cdouglas) - - MAPREDUCE-1012. Mark Context interfaces as public evolving. (Tom White via - cdouglas) - - MAPREDUCE-971. Document use of distcp when copying to s3, managing timeouts - in particular. (Aaron Kimball via cdouglas) - - HDFS-663. DFSIO for append. (shv) - - HDFS-641. Move all of the components that depend on map/reduce to - map/reduce. (omalley) - - HADOOP-5107. Use Maven ant tasks to publish artifacts. (Giridharan Kesavan - via omalley) - - MAPREDUCE-1229. Allow customization of job submission policy in Mumak. - (Hong Tang via cdouglas) - - MAPREDUCE-1317. Reduce the memory footprint of Rumen objects by interning - host Strings. (Hong Tang via cdouglas) - - MAPREDUCE-1097. Add support for Vertica 3.5 to its contrib module. (Omer - Trajman via cdouglas) - - MAPREDUCE-1627. HadoopArchives should not uses a method in DistCp. - (szetszwo) - - MAPREDUCE-1198. Alternatively schedule different types of tasks in - fair share scheduler. (Scott Chen via matei) - - MAPREDUCE-707. Provide a jobconf property for explicitly assigning a job to - a pool in the Fair Scheduler. (Alan Heirich via matei) - - MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter. - Enhanced FileOutputCommitter to create a _SUCCESS file for successful - jobs. (Amar Kamat & Jothi Padmanabhan via acmurthy) - - MAPREDUCE-1103. Added more metrics to Jobtracker. (sharad) - - MAPREDUCE-1048. Add occupied/reserved slot usage summary on jobtracker UI. - (Amareshwari Sriramadasu and Hemanth Yamijala via sharad) - - MAPREDUCE-1090. Modified log statement in TaskMemoryManagerThread to - include task attempt id. (yhemanth) - - MAPREDUCE-1189. Reduce ivy console output to ovservable level (cos) - - MAPREDUCE-1167. ProcfsBasedProcessTree collects rss memory information. - (Scott Chen via dhruba) - - MAPREDUCE-1231. Added a new DistCp option, -skipcrccheck, so that the CRC - check during setup can be skipped. (Jothi Padmanabhan via szetszwo) - - MAPREDUCE-1190. Add package documentation for BBP example. - (Tsz Wo (Nicholas) Sze via cdouglas) - - MAPREDUCE-1119. When tasks fail to report status, show tasks's stack dump - before killing. (Aaron Kimball via tomwhite) - - MAPREDUCE-1185. Redirect running job url to history url if job is already - retired. (Amareshwari Sriramadasu and Sharad Agarwal via sharad) - - MAPREDUCE-1050. Introduce a mock object testing framework. (tomwhite) - - MAPREDUCE-1084. Implementing aspects development and fault injeciton - framework for MapReduce. (Sreekanth Ramakrishnan via cos) - - MAPREDUCE-1209. Move common specific part of the test TestReflectionUtils - out of mapred into common. (Todd Lipcon via tomwhite) - - MAPREDUCE-967. TaskTracker does not need to fully unjar job jars. - (Todd Lipcon via tomwhite) - - MAPREDUCE-1083. Changes in MapReduce so that group information of users - can be refreshed in the JobTracker via command line. - (Boris Shkolnik via ddas) - - MAPREDUCE-181. Changes the job submission process to be secure. - (Devaraj Das) - - MAPREDUCE-1250. Refactors the JobToken to use Common's Token interface. - (Kan Zhang via ddas) - - MAPREDUCE-896. Enhance tasktracker to cleanup files that might have - been created by user tasks with non-writable permissions. - (Ravi Gummadi via yhemanth) - - MAPREDUCE-372. Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer - to use new mapreduce api. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-1295. Add a tool in Rumen for folding and manipulating job - traces. (Dick King via cdouglas) - - MAPREDUCE-1302. TrackerDistributedCacheManager deletes file - asynchronously, thus reducing task initialization delays. - (Zheng Shao via dhruba) - - MAPREDUCE-1218. TaskTrackers send cpu and memory usage of - node to JobTracker. (Scott Chen via dhruba) - - MAPREDUCE-847. Fix Releaseaudit warning count to zero - (Giridharan Kesavan) - - MAPREDUCE-1337. Use generics in StreamJob to improve readability of that - class. (Kay Kay via cdouglas) - - MAPREDUCE-361. Port terasort example to the new mapreduce API. (Amareshwari - Sriramadasu via cdouglas) - - MAPREDUCE-1367. LocalJobRunner should support parallel mapper execution. - (Aaron Kimball via tomwhite) - - MAPREDUCE-64. Eliminate io.sort.record.percent from MapTask configuration. - (cdouglas) - - MAPREDUCE-1440. Replace the long user name in MapReduce with the local - name. (omalley) - - MAPREDUCE-1470. Move delegation tokens from HDFS to Common so that - MapReduce can use them too. (omalley) - - MAPREDUCE-1425. Reduce memory usage by archive. (mahadev via szetszwo) - - MAPREDUCE-1441. Trim whitespace from directory lists pulled from the - configuration. (Todd Lipcon via cdouglas) - - MAPREDUCE-1309. Refactor Rumen trace generator to improve code structure - and add extensible support for log formats. (Dick King via cdouglas) - - MAPREDUCE-1503. Delegation token renewing and cancelling should provide - meaningful exceptions when there are failures instead of returning - false. (omalley) - - HADOOP-6579. Upgrade commons-codec library to 1.4. (omalley) - - MAPREDUCE-1423. Improve performance of CombineFileInputFormat when multiple - pools are configured. (Dhruba Borthakur via zshao) - - MAPREDUCE-1454. Quote user supplied strings in Tracker servlets. (cdouglas) - - MAPREDUCE-1408. Add customizable job submission policies to Gridmix. (Rahul - Singh via cdouglas) - - MAPREDUCE-1527. Better warning logged when mapred.queue.names is - overshadowed by mapred-queues.xml. (Hong Tang via acmurthy) - - MAPREDUCE-1403. Save the size and number of distributed cache artifacts in - the configuration. (Arun Murthy via cdouglas) - - MAPREDUCE-1482. Truncate state string and diagnostic information in - TaskStatus. (Amar Kamat via szetszwo) - - MAPREDUCE-1593. [Rumen] Improvements to random seed generation (tamas via - mahadev) - - MAPREDUCE-1460. Oracle support in DataDrivenDBInputFormat. - (Aaron Kimball via tomwhite) - - MAPREDUCE-1569. Pass configuration through mocked contexts in MRUnit. - (Chris White via cdouglas) - - MAPREDUCE-1590. Move HarFileSystem from Hadoop Common to Mapreduce tools. - (mahadev) - - MAPREDUCE-1629. Get rid of fakeBlockLocations() on HarFileSystem, since - it's not used (mahadev) - - MAPREDUCE-1489. DataDrivenDBInputFormat should not query the database - when generating only one split. (Aaron Kimball via tomwhite) - - MAPREDUCE-1514. Add documentation on replication, permissions, new options, - limitations and internals of har. (mahadev via szetszwo) - - MAPREDUCE-1428. Make block size and the size of archive created files - configurable. (mahadev via szetszwo) - - MAPREDUCE-1656. JobStory should provide queue info. (hong via mahadev) - - MAPREDUCE-1466. Record number of files processed in FileInputFormat in the - Configuration for offline analysis. (Luke Lu and Arun Murthy via cdouglas) - - MAPREDUCE-1538. TrackerDistributedCacheManager manages the - number of files. (Scott Chen via dhruba) - - MAPREDUCE-1673. Scripts to start and stop RaidNode. - (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1659. RaidNode writes temp files on configured tmp directory and - add random numbers to their names to avoid conflicts - (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1221. Allow admins to control physical memory limits per-task - and per-node. (Scott Chen via acmurthy) - - MAPREDUCE-1065. Update mapred tutorial to use the new API. (Aaron Kimball - via cdouglas) - - MAPREDUCE-1304. Add a task counter tracking time spent in GC. (Aaron - Kimball via cdouglas) - - MAPREDUCE-1570. Add grouping comparators to MRUnit. (Chris White via - cdouglas) - - MAPREDUCE-1650. Exclude Private elements from generated MapReduce - Javadoc. (tomwhite) - - MAPREDUCE-1625. Improve grouping of packages in Javadoc. (tomwhite) - - MAPREDUCE-1417. Forrest documentation should be updated to reflect - the changes in MAPREDUCE-744. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache - in a background thread. (Scott Chen via zshao) - - MAPREDUCE-1749. Move configuration strings out of JobContext so that it - can be made public stable. (omalley) - - MAPREDUCE-1623. Apply audience and stability notations to Hadoop - Map-Reduce. (tomwhite via acmurthy) - - MAPREDUCE-1751. Change MapReduce to depend on Hadoop 'common' artifacts - instead of 'core'. (tomwhite) - - MAPREDUCE-1535. Replace usage of FileStatus#isDir(). (Eli Collins via - tomwhite) - - MAPREDUCE-1832. Allow file sizes less than 1MB in DFSIO benchmark. (shv) - - MAPREDUCE-1404. Move Cluster-Setup and Single-Node-Setup Docs from - MapReduce to Common. (tomwhite) - - MAPREDUCE-1697. Document the behavior of -file option and deprecate it - in favour of -files option in streaming. (Amareshwari Sriramadasu - via vinodkv) - - MAPREDUCE-1033. Resolve location of scripts and configuration files after - project split. (tomwhite) - - MAPREDUCE-1018. Document changes to the memory management and scheduling - model. (Hemanth Yamijala via vinodkv) - - MAPREDUCE-1896. [Herriot] New property for multi user list. (Vinay Thota - via cos) - - MAPREDUCE-1812. New properties for suspend and resume process. (Vinay - Thota via cos) - - OPTIMIZATIONS - - MAPREDUCE-270. Fix the tasktracker to optionally send an out-of-band - heartbeat on task-completion for better job-latency. (acmurthy) - Configuration changes: - add mapreduce.tasktracker.outofband.heartbeat - - MAPREDUCE-1186. Modified code in distributed cache to set permissions - only on required set of localized paths. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1501. FileInputFormat supports multi-level, recursive - directory listing. (Zheng Shao via dhruba) - - MAPREDUCE-1556. upgrade to Avro 1.3.0. (cutting via tomwhite) - - MAPREDUCE-1613. Install/deploy source jars to Maven repo - (Patrick Angeles via ddas) - - MAPREDUCE-1610. Forrest documentation should be updated to reflect - the changes in MAPREDUCE-856. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1853. Adds caching for TaskAttemptContext in MultipleOutputs. - (Torsten Curdt via amareshwari) - - BUG FIXES - - MAPREDUCE-878. Rename fair scheduler design doc to - fair-scheduler-design-doc.tex and add Apache license header (matei) - - HADOOP-4687. MapReduce is split from Hadoop Core. It is a subproject under - Hadoop (Owen O'Malley) - - HADOOP-6096. Fix Eclipse project and classpath files following project - split. (tomwhite) - - MAPREDUCE-419. Reconcile mapred.userlog.limit.kb defaults in configuration - and code. (Philip Zeyliger via cdouglas) - - MAPREDUCE-2. Fixes a bug in KeyFieldBasedPartitioner in handling empty - keys. (Amar Kamat via sharad) - - MAPREDUCE-130. Delete the jobconf copy from the log directory of the - JobTracker when the job is retired. (Amar Kamat via sharad) - - MAPREDUCE-657. Fix hardcoded filesystem problem in CompletedJobStatusStore. - (Amar Kamat via sharad) - - MAPREDUCE-179. Update progress in new RecordReaders. (cdouglas) - - MAPREDUCE-658. Replace NPE in distcp with a meaningful error message when - the source path does not exist. (Ravi Gummadi via cdouglas) - - MAPREDUCE-671. Update ignore list to include untracked, generated - build artifacts and config files. (cdouglas) - - MAPREDUCE-433. Use more reliable counters in TestReduceFetch. (cdouglas) - - MAPREDUCE-124. Fix a bug in failure handling of abort task of - OutputCommiter. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-694. Fix to add jsp-api jars to capacity-scheduler classpath. - (Giridharan Kesavan) - - MAPREDUCE-702. Fix eclipse-plugin jar target (Giridharan Kesavan) - - MAPREDUCE-522. Replace TestQueueCapacities with simpler test case to - test integration between capacity scheduler and MR framework. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-683. Fixes an initialization problem in the JobHistory. - The initialization of JobHistoryFilesManager is now done in the - JobHistory.init call. (Amar Kamat via ddas) - - MAPREDUCE-708. Fixes a bug to allow updating the reason for - blacklisting a node on the JobTracker UI. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-709. Fixes message displayed for a blacklisted node where - the reason for blacklisting is due to the health check script - timing out. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-676. Existing diagnostic rules fail for MAP ONLY jobs. - (Suhas Gogate via tomwhite) - - MAPREDUCE-722. Fixes a bug with tasktracker reservations for - high memory jobs in capacity scheduler. - (Vinod Kumar Vavilapalli via yhemanth) - - HADOOP-6090. Updates gridmix script to use new mapreduce api output - format. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-732. Removed spurious log statements in the node - blacklisting logic. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-734. Fix a ConcurrentModificationException in unreserving - unused reservations for a job when it completes. - (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-733. Fix a RuntimeException while unreserving trackers - that are blacklisted for a job. - (Arun Murthy and Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-677. Fix timeout in TestNodeRefresh. (Amar Kamat via - sharad) - - MAPREDUCE-153. Fix timeout in TestJobInProgressListener. (Amar - Kamat via sharad) - - MAPREDUCE-742. Fix output messages and java comments in the Pi related - examples. (szetszwo) - - MAPREDUCE-565. Fix partitioner to work with new API. (Owen O'Malley via - cdouglas) - - MAPREDUCE-680. Fix so MRUnit can handle reuse of Writable objects. - (Aaron Kimball via johan) - - MAPREDUCE-18. Puts some checks for cross checking whether a reduce - task gets the correct shuffle data. (Ravi Gummadi via ddas) - - MAPREDUCE-771. Fix scheduling of setup and cleanup tasks to use - free slots instead of tasks for scheduling. (yhemanth) - - MAPREDUCE-717. Fixes some corner case issues in speculative - execution heuristics. (Devaraj Das) - - MAPREDUCE-716. Make DBInputFormat work with Oracle. (Aaron Kimball - via tomwhite) - - MAPREDUCE-735. Fixes a problem in the KeyFieldHelper to do with - the end index for some inputs (Amar Kamat via ddas) - - MAPREDUCE-682. Removes reservations on tasktrackers which are - blacklisted. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-743. Fixes a problem to do with progress reporting - in the map phase. (Ravi Gummadi via ddas) - - MAPREDUCE-765. Eliminate the deprecated warnings introduced by H-5438. - (He Yongqiang via szetszwo) - - MAPREDUCE-383. Fix a bug in Pipes combiner due to bytes count not - getting reset after the spill. (Christian Kunz via sharad) - - MAPREDUCE-809. Fix job-summary logs to correctly record status of FAILED - and KILLED jobs. (acmurthy) - - MAPREDUCE-792. Fix unchecked warnings in DBInputFormat. (Aaron Kimball - via szetszwo) - - MAPREDUCE-760. Fix a timing issue in TestNodeRefresh. (Amar Kamat via - sharad) - - MAPREDUCE-40. Keep memory management backwards compatible for job - configuration parameters and limits. (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-587. Fixes a OOM issue in TestStreamingExitStatus. - (Amar Kamat via ddas) - - MAPREDUCE-408. Fixes an assertion problem in TestKillSubProcesses - (Ravi Gummadi via ddas) - - MAPREDUCE-659. Fix gridmix2 compilation. (Giridharan Kesavan) - - MAPREDUCE-796. Fixes a ClassCastException in an exception log in - MultiThreadedMapRunner. (Amar Kamat via ddas) - - MAPREDUCE-808. Fixes a serialization problem in TypedBytes. - (Klaas Bosteels via ddas) - - MAPREDUCE-845. Fix a findbugs heap size problem in build.xml and add - a new property findbugs.heap.size. (Lee Tucker via szetszwo) - - MAPREDUCE-838. Fixes a problem in the way commit of task outputs - happens. The bug was that even if commit failed, the task would - be declared as successful. (Amareshwari Sriramadasu via ddas) - - MAPREDUCE-813. Updates Streaming and M/R tutorial documents. - (Corinne Chandel via ddas) - - MAPREDUCE-805. Fixes some deadlocks in the JobTracker due to the fact - the JobTracker lock hierarchy wasn't maintained in some JobInProgress - method calls. (Amar Kamat via ddas) - - MAPREDUCE-799. Fixes so all of the MRUnit self-tests run. - (Aaron Kimball via johan) - - MAPREDUCE-848. Fixes a problem to do with TestCapacityScheduler - failing (Amar Kamat via ddas) - - MAPREDUCE-840. DBInputFormat leaves open transaction. - (Aaron Kimball via tomwhite) - - MAPREDUCE-859. Adds Avro and its dependencies required by Hadoop - common. (Ravi Gummadi via sharad) - - MAPREDUCE-867. Fix ivy conf to look for avro jar from maven repo. - (Giridharan Kesavan) - - MAPREDUCE-877. Added avro as a dependency to contrib ivy settings. - (Tsz Wo (Nicholas) Sze via yhemanth) - - MAPREDUCE-852. In build.xml, remove the Main-Class, which is incorrectly - set in tools, and rename the target "tools-jar" to "tools". (szetszwo) - - MAPREDUCE-773. Sends progress reports for compressed gzip inputs in maps. - Fixes a native direct buffer leak in LineRecordReader classes. - (Hong Tang and ddas) - - MAPREDUCE-832. Reduce number of warning messages printed when - deprecated memory variables are used. (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-745. Fixes a testcase problem to do with generation of JobTracker - IDs. (Amar Kamat via ddas) - - MAPREDUCE-834. Enables memory management on tasktrackers when old - memory management parameters are used in configuration. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-818. Fixes Counters#getGroup API. (Amareshwari Sriramadasu - via sharad) - - MAPREDUCE-807. Handles the AccessControlException during the deletion of - mapred.system.dir in the JobTracker. The JobTracker will bail out if it - encounters such an exception. (Amar Kamat via ddas) - - MAPREDUCE-430. Fix a bug related to task getting stuck in case of - OOM error. (Amar Kamat via ddas) - - MAPREDUCE-871. Fix ownership of Job/Task local files to have correct - group ownership according to the egid of the tasktracker. - (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative - execution. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-924. Fixes the TestPipes testcase to use Tool. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-903. Add Avro jar to eclipse classpath. - (Philip Zeyliger via tomwhite) - - MAPREDUCE-943. Removes a testcase in TestNodeRefresh that doesn't make - sense in the new Job recovery model. (Amar Kamat via ddas) - - MAPREDUCE-764. TypedBytesInput's readRaw() does not preserve custom type - codes. (Klaas Bosteels via tomwhite) - - HADOOP-6243. Fixes a NullPointerException in handling deprecated keys. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-968. NPE in distcp encountered when placing _logs directory on - S3FileSystem. (Aaron Kimball via tomwhite) - - MAPREDUCE-826. harchive doesn't use ToolRunner / harchive returns 0 even - if the job fails with exception (koji Noguchi via mahadev) - - MAPREDUCE-839. unit test TestMiniMRChildTask fails on mac os-x (hong tang - via mahadev) - - MAPREDUCE-112. Add counters for reduce input, output records to the new API. - (Jothi Padmanabhan via cdouglas) - - MAPREDUCE-648. Fix two distcp bugs: (1) it should not launch a job if all - src paths are directories, and (2) it does not skip copying when updating - a single file. (Ravi Gummadi via szetszwo) - - MAPREDUCE-946. Fix a regression in LineRecordReader where the - maxBytesToConsume parameter is not set correctly. (cdouglas) - - MAPREDUCE-977. Missing jackson jars from Eclipse template. (tomwhite) - - MAPREDUCE-988. Fix a packaging issue in the contrib modules. (Hong Tang via - cdouglas) - - MAPREDUCE-971. distcp does not always remove distcp.tmp.dir. (Aaron Kimball - via tomwhite) - - MAPREDUCE-995. Fix a bug in JobHistory where tasks completing after the job - is closed cause a NPE. (Jothi Padmanabhan via cdouglas) - - MAPREDUCE-953. Fix QueueManager to dump queue configuration in JSON format. - (V.V. Chaitanya Krishna via yhemanth) - - MAPREDUCE-645. Prevent distcp from running a job when the destination is a - file, but the source is not. (Ravi Gummadi via cdouglas) - - MAPREDUCE-1002. Flushed writer in JobQueueClient so queue information is - printed correctly. (V.V. Chaitanya Krishna via yhemanth) - - MAPREDUCE-1003. Fix compilation problem in eclipse plugin when - eclipse.home is set. (Ravi Gummadi via yhemanth) - - MAPREDUCE-941. Vaidya script fails on Solaris. (Chad Metcalf - via tomwhite) - - MAPREDUCE-912. Add and standardize Apache license headers. (Chad Metcalf - via cdouglas) - - MAPREDUCE-1022. Fix compilation of vertica testcases. (Vinod Kumar - Vavilapalli via acmurthy) - - MAPREDUCE-1000. Handle corrupt history files in JobHistory.initDone(). - (Jothi Padmanabhan via sharad) - - MAPREDUCE-1028. Fixed number of slots occupied by cleanup tasks to one - irrespective of slot size for the job. - (Ravi Gummadi via yhemanth) - - MAPREDUCE-964. Fixed start and finish times of TaskStatus to be - consistent, thereby fixing inconsistencies in metering tasks. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-1076. Deprecate ClusterStatus and add javadoc in ClusterMetrics. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-979. Fixed JobConf APIs related to memory parameters to return - values of new configuration variables when deprecated variables are - disabled. (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-1030. Modified scheduling algorithm to return a map and reduce - task per heartbeat in the capacity scheduler. - (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-1071. Use DataInputStream rather than FSDataInputStream in the - JobHistory EventReader. (Hong Tang via cdouglas) - - MAPREDUCE-986. Fix Rumen to work with truncated task lines. (Dick King via - cdouglas) - - MAPREDUCE-1029. Fix failing TestCopyFiles by restoring the unzipping of - HDFS webapps from the hdfs jar. (Aaron Kimball and Jothi Padmanabhan via - cdouglas) - - MAPREDUCE-769. Make findbugs and javac warnings to zero. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-1104. Initialize RecoveryManager in JobTracker cstr called by - Mumak. (Hong Tang via cdouglas) - - MAPREDUCE-1061. Add unit test validating byte specifications for gridmix - jobs. (cdouglas) - - MAPREDUCE-1077. Fix Rumen so that truncated tasks do not mark the job as - successful. (Dick King via cdouglas) - - MAPREDUCE-1041. Make TaskInProgress::taskStatuses map package-private. - (Jothi Padmanabhan via cdouglas) - - MAPREDUCE-1070. Prevent a deadlock in the fair scheduler servlet. - (Todd Lipcon via cdouglas) - - MAPREDUCE-1086. Setup Hadoop logging environment for tasks to point to - task related parameters. (Ravi Gummadi via yhemanth) - - MAPREDUCE-1105. Remove max limit configuration in capacity scheduler in - favor of max capacity percentage thus allowing the limit to go over - queue capacity. (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-1016. Make the job history log format JSON. (cutting) - - MAPREDUCE-1038. Weave Mumak aspects only if related files have changed. - (Aaron Kimball via cdouglas) - - MAPREDUCE-1163. Remove unused, hard-coded paths from libhdfs. (Allen - Wittenauer via cdouglas) - - MAPREDUCE-962. Fix a NullPointerException while killing task process - trees. (Ravi Gummadi via yhemanth) - - MAPREDUCE-1177. Correct setup/cleanup inversion in - JobTracker::getTaskReports. (Vinod Kumar Vavilapalli via cdouglas) - - MAPREDUCE-1178. Fix ClassCastException in MultipleInputs by adding - a DelegatingRecordReader. (Amareshwari Sriramadasu and Jay Booth - via sharad) - - MAPREDUCE-1068. Fix streaming job to show proper message if file is - is not present. (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-1147. Add map output counters to new API. (Amar Kamat via - cdouglas) - - MAPREDUCE-915. The debug scripts are run as the job user. (ddas) - - MAPREDUCE-1007. Fix NPE in CapacityTaskScheduler.getJobs(). - (V.V.Chaitanya Krishna via sharad) - - MAPREDUCE-28. Refactor TestQueueManager and fix default ACLs. - (V.V.Chaitanya Krishna and Rahul K Singh via sharad) - - MAPREDUCE-1182. Fix overflow in reduce causing allocations to exceed the - configured threshold. (cdouglas) - - MAPREDUCE-1239. Fix contrib components build dependencies. - (Giridharan Kesavan and omalley) - - MAPREDUCE-787. Fix JobSubmitter to honor user given symlink path. - (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-1249. Update config default value for socket read timeout to - match code default. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1161. Remove ineffective synchronization in NotificationTestCase. - (Owen O'Malley via cdouglas) - - MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan) - - MAPREDUCE-1075. Fix JobTracker to not throw an NPE for a non-existent - queue. (V.V.Chaitanya Krishna via yhemanth) - - MAPREDUCE-754. Fix NPE in expiry thread when a TT is lost. (Amar Kamat - via sharad) - - MAPREDUCE-1074. Document Reducer mark/reset functionality. (Jothi - Padmanabhan via cdouglas) - - MAPREDUCE-1267. Fix typo in mapred-default.xml. (Todd Lipcon via cdouglas) - - MAPREDUCE-952. Remove inadvertently reintroduced Task.Counter enum. (Jothi - Padmanabhan via cdouglas) - - MAPREDUCE-1230. Fix handling of null records in VerticaInputFormat. (Omer - Trajman via cdouglas) - - MAPREDUCE-1171. Allow shuffle retries and read-error reporting to be - configurable. (Amareshwari Sriramadasu via acmurthy) - - MAPREDUCE-879. Fix broken unit test TestTaskTrackerLocalization on MacOS. - (Sreekanth Ramakrishnan via yhemanth) - - MAPREDUCE-1124. Fix imprecise byte counts in Gridmix. (cdouglas) - - MAPREDUCE-1222. Add an option to exclude numeric IP addresses in topologies - processed by Mumak. (Hong Tang via cdouglas) - - MAPREDUCE-1284. Fix fts_open() call in task-controller that was failing - LinuxTaskController unit tests. (Ravi Gummadi via yhemanth) - - MAPREDUCE-1143. Fix running task counters to be updated correctly - when speculative attempts are running for a TIP. - (Rahul Kumar Singh via yhemanth) - - MAPREDUCE-1241. Use a default queue configuration in JobTracker when - mapred-queues.xml is unavailable. (Todd Lipcon via cdouglas) - - MAPREDUCE-1301. Fix set up of permission checking script used in - localization tests. (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1286. Remove quoting from client opts in TaskRunner. (Yuri - Pradkin via cdouglas) - - MAPREDUCE-1059. Use distcp.bytes.per.map when adding sync markers in - distcp. (Aaron Kimball via cdouglas) - - MAPREDUCE-1009. Update forrest documentation describing hierarchical - queues. (Vinod Kumar Vavilapalli via yhemanth) - - MAPREDUCE-1342. Fixed deadlock in global blacklisting of tasktrackers. - (Amareshwari Sriramadasu via acmurthy) - - MAPREDUCE-1316. Fixes a memory leak of TaskInProgress instances in - the jobtracker. (Amar Kamat via yhemanth) - - MAPREDUCE-1359. TypedBytes TestIO doesn't mkdir its test dir first. - (Anatoli Fomenko via cos) - - MAPREDUCE-1314. Correct errant mapreduce.x.mapreduce.x replacements from - bulk change. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1365. Restore accidentally renamed test in - TestTaskTrackerBloacklisting. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1406. Fix spelling of JobContext.MAP_COMBINE_MIN_SPILLS. - (cdouglas) - - MAPREDUCE-1369. JUnit tests should never depend on anything in conf - (Anatoli Fomenko via cos) - - MAPREDUCE-1412. Fix timer granularity issue causing failures in - TestTaskTrackerBlacklisting. (cdouglas) - - MAPREDUCE-1448. Respect --config option in Mumak script. (Hong Tang via - cdouglas) - - MAPREDUCE-1251. c++ utils doesn't compile. (Eli Collins via tomwhite) - - MAPREDUCE-1522. FileInputFormat may use the default FileSystem for the - input path. (Tsz Wo (Nicholas), SZE via cdouglas) - - MAPREDUCE-1407. Update javadoc in mapreduce.{Mapper,Reducer} to match - actual usage. (Benoit Sigoure via cdouglas) - - MAPREDUCE-1258. Fix fair scheduler event log not logging job info. - (matei) - - MAPREDUCE-1089. Fix NPE in fair scheduler preemption when tasks are - scheduled but not running. (Todd Lipcon via matei) - - MAPREDUCE-1014. Fix the libraries for common and hdfs. (omalley) - - MAPREDUCE-1111. JT Jetty UI not working if we run mumak.sh - off packaged distribution directory. (hong tang via mahadev) - - MAPREDUCE-1133. Eclipse .classpath template has outdated jar files and is - missing some new ones. (cos) - - MAPREDUCE-1098. Fixed the distributed-cache to not do i/o while holding a - global lock. (Amareshwari Sriramadasu via acmurthy) - - MAPREDUCE-1158. Fix JT running maps and running reduces metrics. - (sharad) - - MAPREDUCE-1160. Reduce verbosity of log lines in some Map/Reduce classes - to avoid filling up jobtracker logs on a busy cluster. - (Ravi Gummadi and Hong Tang via yhemanth) - - MAPREDUCE-1153. Fix tasktracker metrics when trackers are decommissioned. - (sharad) - - MAPREDUCE-1128. Fix MRUnit to prohibit iterating over values twice. (Aaron - Kimball via cdouglas) - - MAPREDUCE-665. Move libhdfs to HDFS subproject. (Eli Collins via dhruba) - - MAPREDUCE-1196. Fix FileOutputCommitter to use the deprecated cleanupJob - api correctly. (acmurthy) - - MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan) - - MAPREDUCE-1140. Fix DistributedCache to not decrement reference counts for - unreferenced files in error conditions. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1245. Fix TestFairScheduler failures by instantiating lightweight - Jobtracker. (sharad) - - MAPREDUCE-1260. Update Eclipse configuration to match changes to Ivy - configuration. (Edwin Chan via cos) - - MAPREDUCE-1152. Distinguish between failed and killed tasks in - JobTrackerInstrumentation. (Sharad Agarwal via cdouglas) - - MAPREDUCE-1285. In DistCp.deleteNonexisting(..), get class from the - parameter instead of using FileStatus.class. (Peter Romianowski via - szetszwo) - - MAPREDUCE-1294. Build fails to pull latest hadoop-core-* artifacts (cos) - - MAPREDUCE-1213. TaskTrackers restart is faster because it deletes - distributed cache directory asynchronously. (Zheng Shao via dhruba) - - MAPREDUCE-1265. The task attempt error log prints the name of the - tasktracker machine. (Scott Chen via dhruba) - - MAPREDUCE-1201. ProcfsBasedProcessTree collects CPU usage information. - (Scott Chen via dhruba) - - MAPREDUCE-1326. fi tests don't use fi-site.xml (cos) - - MAPREDUCE-1165. Replace non-portable function name with C99 equivalent. - (Allen Wittenauer via cdouglas) - - MAPREDUCE-1331. Fixes a typo in a testcase (Devaraj Das) - - MAPREDUCE-1293. AutoInputFormat doesn't work with non-default FileSystems. - (Andrew Hitchcock via tomwhite) - - MAPREDUCE-1131. Using profilers other than hprof can cause JobClient to - report job failure. (Aaron Kimball via tomwhite) - - MAPREDUCE-1155. Streaming tests swallow exceptions. - (Todd Lipcon via tomwhite) - - MAPREDUCE-1212. Mapreduce contrib project ivy dependencies are not included - in binary target. (Aaron Kimball via tomwhite) - - MAPREDUCE-1388. Move the HDFS RAID package from HDFS to MAPREDUCE. - (Eli Collins via dhruba) - - MAPREDUCE-1322. Defines default value for staging directory to be user - based fixing a failing streaming test. - (Devaraj Das and Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-899. Modified LinuxTaskController to check that task-controller - has right permissions and ownership before performing any actions. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1443. DBInputFormat can leak connections. - (Aaron Kimball via tomwhite) - - MAPREDUCE-1457. Fixes JobTracker to get the FileSystem object within - getStagingAreaDir within a privileged block. Fixes Child.java to use the - appropriate UGIs while getting the TaskUmbilicalProtocol proxy and - while executing the task. (Jakob Homan via ddas) - - MAPREDUCE-1399. The archive command shows a null error message (nicholas - via mahadev) - - MAPREDUCE-1305. Improve efficiency of distcp -delete. (Peter Romianowski - via cdouglas) - - MAPREDUCE-1474. Update forrest documentation for Hadoop Archives. (Mahadev - Konar via cdouglas) - - MAPREDUCE-1400. Use tr rather than sed to effect literal substitution in - the build script. (Allen Wittenauer via cdouglas) - - MAPREDUCE-1358. Avoid false positives in OutputLogFilter. (Todd Lipcon via - cdouglas) - - MAPREDUCE-1490. Fix a NullPointerException that could occur during - instantiation and initialization of the DistributedRaidFileSystem. - (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1476. Fix the M/R framework to not call commit for special - tasks like job setup/cleanup and task cleanup. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1398. Fix TaskLauncher to stop waiting for slots on a TIP that - is killed / failed. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1491. The parity files created by the RAID are combined - using Hadoop Archive Files (HAR). (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1378. URL encode link in jobhistory.jsp to avoid errors caused by - unescaped characters. (E. Sammer via cdouglas) - - MAPREDUCE-1519. RaidNode fails to create new parity file - if an older version already exists. (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1537. Fixes a compilation problem in a testcase after commit - HDFS-984. (Jitendra Nath Pandey via ddas) - - MAPREDUCE-1537. The patch makes the job client call the getDelegationToken - only when security is enabled. (Jitendra Nath Pandey via ddas) - - MAPREDUCE-1510. RAID should regenerate parity files if they get deleted. - (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1421. Fix the LinuxTaskController tests failing on trunk after - the commit of MAPREDUCE-1385. (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1520. Fix TestMiniMRLocalFS failure caused by regression in - getting user working dir. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1512. RAID uses HarFileSystem directly instead of - FileSystem.get (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1435. Fix symlink handling in task work directory when - cleaning up, essentially to avoid following links. - (Ravi Gummadi via yhemanth) - - MAPREDUCE-1518. RaidNode does not run the deletion check on the - directory that stores the parity files. (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1573. TestStreamingAsDifferentUser fails if run as tt_user. - (Ravi Gummadi via vinodkv) - - MAPREDUCE-927. Cleanup of task-logs should happen in TaskTracker instead - of the Child. (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1578. Decouple HadoopArchives vesrion from HarFileSystem version. - (Rodrigo Schmidt via szetszwo) - - MAPREDUCE-1422. Fix cleanup of localized job directory to work if files - with non-deletable permissions are created within it. - (Amar Kamat via yhemanth) - - MAPREDUCE-1306. Randomize the arrival of heartbeat responses in Mumak. - (Tamas Sarlos via cdouglas) - - MAPREDUCE-1579. archive: check and possibly replace the space charater - in source paths. (szetszwo) - - MAPREDUCE-1536. DataDrivenDBInputFormat does not split date columns correctly. - (Aaron Kimball via enis) - - MAPREDUCE-890. After HADOOP-4491, the user who started mapred system is - not able to run job. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1615. Fix compilation of TestSubmitJob. (cdouglas) - - MAPREDUCE-1508. Protect against NPE in TestMultipleLevelCaching. (Aaron - Kimball via cdouglas) - - MAPREDUCE-1497. Suppress spurious findbugs warning about IndexCache - synchronization. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1420. Fix TestTTResourceReporting failure. (Scott Chen via - cdouglas) - - MAPREDUCE-1480. Correctly initialize child RecordReaders in - CombineFileInputFormat. (Aaron Kimball via cdouglas) - - MAPREDUCE-1348. Fix block forensics packaging. (Tom White via cdouglas) - - MAPREDUCE-1628. HarFileSystem shows incorrect replication numbers and - permissions. (szetszwo via mahadev) - - MAPREDUCE-1602. Fix the error message for the case that src does not - exist. (szetszwo) - - MAPREDUCE-1585. Create Hadoop Archives version 2 with filenames - URL-encoded (rodrigo via mahadev) - - MAPREDUCE-1523. Sometimes rumen trace generator fails to extract the job - finish time. (dick king via mahadev) - - MAPREDUCE-1635. ResourceEstimator does not work after MAPREDUCE-842. - (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-889. binary communication formats added to Streaming by - HADOOP-1722 should be documented. (Klaas Bosteels via tomwhite) - - MAPREDUCE-1031. ant tar target doens't seem to compile tests in contrib - projects. (Aaron Kimball via tomwhite) - - MAPREDUCE-1692. Removed unused testcase TestStreamedMerge. - (Sreekanth Ramakrishnan and Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1062. Fix ReliabilityTest to work with retired jobs. (Sreekanth - Ramakrishnan via cdouglas) - - MAPREDUCE-1409. IOExceptions thrown from FIleOutputCommitter::abortTask - should cause the task to fail. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1695. Include capacity scheduler in findbugs and javadoc-dev - targets and also fix existing warnings. (Hong Tang via yhemanth) - - MAPREDUCE-1494. Ensure TestJobDirCleanup verifies the correct paths. - (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1622. Include missing slf4j dependencies. (cdouglas) - - MAPREDUCE-1515. Accept java5.home from build.properties, not only from the - command line when building forrest docs. (Al Thompson via cdouglas) - - MAPREDUCE-1618. Add missing javadoc to JobStatus::*JobAcls. (Amareshwari - Sriramadasu via cdouglas) - - MAPREDUCE-1219. Remove job level metrics from jobtracker metrics to ease - undue load on jobtracker. (Sreekanth Ramakrishnan via sharad) - - MAPREDUCE-1604. Add Forrest documentation for Job ACLs. - (Amareshwari Sriramadasu via yhemanth) - - MAPREDUCE-1705. Archiving and Purging of HDFS parity files should - handle globbed policies accurately. (Rodrigo Schmidt via dhruba) - - MAPREDUCE-1612. job conf file is not accessible from job history web page. - (Ravi Gummadi and Sreekanth Ramakrishnan via vinodkv) - - MAPREDUCE-1397. NullPointerException observed during task failures. - (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1728. Oracle timezone strings do not match Java. - (Aaron Kimball via tomwhite) - - MAPREDUCE-1609. TaskTracker.localizeJob should not set permissions on - job log directory recursively. (Amareshwari Sriramadasu via vinodkv) - - MAPREDUCE-1657. After task logs directory is deleted, tasklog servlet - displays wrong error message about job ACLs. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1727. TestJobACLs fails after HADOOP-6686. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1611. Refresh nodes and refresh queues doesnt work with service - authorization enabled. (Amar Kamat via vinodkv) - - MAPREDUCE-1276. Correct flaws in the shuffle related to connection setup - and failure attribution. (Amareshwari Sriramadasu via cdouglas) - - MAPREDUCE-1372. ConcurrentModificationException in JobInProgress. - (Dick King and Amareshwari Sriramadasu via tomwhite) - - MAPREDUCE-118. Fix Job.getJobID(). (Amareshwari Sriramadasu via sharad) - - MAPREDUCE-913. TaskRunner crashes with NPE resulting in held up slots, - UNINITIALIZED tasks and hung TaskTracker. (Amareshwari Sriramadasu and - Sreekanth Ramakrishnan via vinodkv) - - MAPREDUCE-1725. Fix MapReduce API incompatibilities between 0.20 and 0.21. - (tomwhite) - - MAPREDUCE-1606. TestJobACLs may timeout as there are no slots for launching - JOB_CLEANUP task. (Ravi Gummadi via vinodkv) - - MAPREDUCE-1765. Correct streaming documentation for StreamXmlRecordReader. - (Corinne Chandel via amareshwari) - - MAPREDUCE-1880. Fix BigDecimal.divide(..) in the pi example. (szetszwo) - - MAPREDUCE-1885. Revert FileSystem create method that takes CreateFlags - (MapReduce part of HADOOP-6826). (Ravi Gummadi via tomwhite) - - MAPREDUCE-1870. Harmonize MapReduce JAR library versions with Common and - HDFS. (tomwhite) - - MAPREDUCE-1791. Remote cluster control functionality needs JavaDocs - improvement (Konstantin Boudnik) - - MAPREDUCE-1942. 'compile-fault-inject' should never be called directly. - (Konstantin Boudnik) - - MAPREDUCE-1876. Fixes TaskAttemptStartedEvent to correctly log event type - for all task types. (Amar Kamat via amareshwari) - - MAPREDUCE-1926. MapReduce distribution is missing build-utils.xml. - (tomwhite) - - MAPREDUCE-2012. Some contrib tests fail in branch 0.21 and trunk. - (Amareshwari Sriramadasu via tomwhite) - - MAPREDUCE-1980. Fixes TaskAttemptUnsuccessfulCompletionEvent and - TaskAttemptFinishedEvent to correctly log event type for all task types. - (Amar Kamat via amareshwari) - - MAPREDUCE-1856. Extract a subset of tests for smoke (DOA) validation (cos) - diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt deleted file mode 100644 index 4934e893aae..00000000000 --- a/hadoop-yarn-project/CHANGES.txt +++ /dev/null @@ -1,6905 +0,0 @@ -Hadoop YARN Change Log - -Trunk - Unreleased - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-2882. Add an OPPORTUNISTIC ExecutionType. - (Konstantinos Karanasos and Inigo Goiri via kasha) - - IMPROVEMENTS - - YARN-2438. yarn-env.sh cleanup (aw) - - YARN-2472. yarn-daemons.sh should jsut call yarn directly (Masatake Iwasaki - via aw) - - YARN-2437. start-yarn.sh/stop-yarn should give info (Varun Saxena via aw) - - YARN-2796. deprecate sbin/yarn-daemon.sh (aw) - - YARN-2980. Move health check script related functionality to hadoop-common - (Varun Saxena via aw) - - YARN-3168. Convert site documentation from apt to markdown (Gururaj Shetty - via aw) - - YARN-3199. Fair Scheduler documentation improvements (Rohit Agarwal via - aw) - - YARN-2280. Resource manager web service fields are not accessible - (Krisztian Horvath via aw) - - YARN-3261. rewrite resourcemanager restart doc to remove roadmap bits (Gururaj Shetty via aw) - - - OPTIMIZATIONS - - BUG FIXES - - YARN-4680. TimerTasks leak in ATS V1.5 Writer. (Xuan Gong via gtcarrera9) - - YARN-524 TestYarnVersionInfo failing if generated properties doesn't - include an SVN URL. (stevel) - - YARN-1471. The SLS simulator is not running the preemption policy - for CapacityScheduler (Carlo Curino via cdouglas) - - YARN-2436. [post-HADOOP-9902] yarn application help doesn't work (aw) - - YARN-2525. yarn logs command gives error on trunk (Akira AJISAKA via aw) - - YARN-3002. YARN documentation needs updating post-shell rewrite (aw) - - YARN-2428. LCE default banned user list should have yarn (Varun - Saxena via aw) - - YARN-2355. MAX_APP_ATTEMPTS_ENV may no longer be a useful env var - for a container (Darrell Taylor via aw) - - YARN-3915. scmadmin help message correction (Bibin A Chundatt via aw) - -Release 2.9.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-1856. Added cgroups based memory monitoring for containers as another - alternative to custom memory-monitoring. (Varun Vasudev via vinodkv) - - YARN-4462. FairScheduler: Disallow preemption from a queue. (Tao Jie via kasha) - - IMPROVEMENTS - - YARN-4072. ApplicationHistoryServer, WebAppProxyServer, NodeManager and - ResourceManager to support JvmPauseMonitor as a service. - (Sunil G via Stevel) - - YARN-4341. add doc about timeline performance tool usage (Chang Li via - sjlee) - - YARN-4417. Make RM and Timeline-server REST APIs more consistent. - (wtan via jianhe) - - YARN-3480. Remove attempts that are beyond max-attempt limit from state - store. (Jun Gong via jianhe) - - YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) - - YARN-4438. Implement RM leader election with curator. (Jian He via xgong) - - YARN-4571. Make app id/name available to the yarn authorizer provider for - better auditing. (Jian He via wangda) - - YARN-4551. Address the duplication between StatusUpdateWhenHealthy and - StatusUpdateWhenUnhealthy transitions. (Sunil G via kasha) - - YARN-4553. Add cgroups support for docker containers. - (Sidharta Seethana via vvasudev) - - YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. - (kasha) - - YARN-4559. Make leader elector and zk store share the same curator client. - (Jian He via xgong) - - YARN-4603. FairScheduler should mention user requested queuename in error - message when failed in queue ACL check. (Tao Jie via kasha) - - YARN-4496. Improve HA ResourceManager Failover detection on the client. - (Jian He via xgong) - - YARN-3542. Refactored existing CPU cgroups support to use the newer and - integrated ResourceHandler mechanism, and also deprecated the old - LCEResourceHandler inteface hierarchy. (Varun Vasudev via vinodkv) - - YARN-4647. Make RegisterNodeManagerRequestPBImpl thread-safe. (kasha) - - YARN-4649. Add additional logging to some NM state store operations. - (Sidharta Seethana via vvasudev) - - YARN-4625. Make ApplicationSubmissionContext and - ApplicationSubmissionContextInfo more consistent. - (Xuan Gong via vvasudev) - - YARN-4307. Display blacklisted nodes for AM container in the RM web UI. - (Naganarasimha G R via vvasudev) - - YARN-4628. Display application priority in yarn top. - (Bibin A Chundatt via vvasudev) - - YARN-4655. Log uncaught exceptions/errors in various thread pools in YARN. - (Sidharta Seethana via vvasudev) - - YARN-4689. FairScheduler: Cleanup preemptContainer to be more readable. - (Kai Sasaki via kasha) - - YARN-4708. Missing default mapper type in TimelineServer performance test tool - usage. (Kai Sasaki via ozawa) - - YARN-4648. Move preemption related tests from TestFairScheduler to - TestFairSchedulerPreemption. (Kai Sasaki via ozawa) - - YARN-4697. NM aggregation thread pool is not bound by - limits (haibochen via rkanter) - - YARN-4718. Rename variables in SchedulerNode to reduce ambiguity post - YARN-1011. (Inigo Goiri via kasha) - - OPTIMIZATIONS - - BUG FIXES - - YARN-4109. Exception on RM scheduler page loading with labels. - (Mohammad Shahid Khan via rohithsharmaks) - - YARN-4156. TestAMRestart#testAMBlacklistPreventsRestartOnSameNode - assumes CapacityScheduler. (Anubhav Dhoot via kasha) - - YARN-2934. Improve handling of container's stderr. - (Naganarasimha G R via gera) - - YARN-4530. LocalizedResource trigger a NPE Cause the NodeManager exit - (tangshangwen via rohithsharmaks) - - YARN-4535. Fix checkstyle error in CapacityScheduler.java - (Naganarasimha G R via rohithsharmaks) - - YARN-4393. Fix intermittent test failure for TestResourceLocalizationService - #testFailedDirsResourceRelease (Varun Saxana via rohithsharmaks) - - YARN-4550. Some tests in TestContainerLanch fails on non-english locale - environment. (Takashi Ohnishi via stevel) - - YARN-4567. javadoc failing on java 8. (Steve Loughran via aajisaka) - - YARN-3446. FairScheduler headroom calculation should exclude nodes in the - blacklist. (Zhihai Xu via kasha) - - YARN-4611. Fix scheduler load simulator to support multi-layer network - location. (Ming Ma via xgong) - - YARN-4584. RM startup failure when AM attempts greater than max-attempts. - (Bibin A Chundatt via rohithsharmaks) - - YARN-4578. Directories that are mounted in docker containers need to be more - restrictive/container-specific. (Sidharta Seethana via vvasudev) - - YARN-4497. RM might fail to restart when recovering apps whose attempts are missing. - (Jun Gong via rohithsharmaks) - - YARN-4574. Fix random failure in TestAMRMClientOnRMRestart. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4613. Fix test failure in TestClientRMService#testGetClusterNodes. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4612. Fix rumen and scheduler load simulator handle killed tasks properly. - (Ming Ma via xgong) - - YARN-4573. Fix test failure in TestRMAppTransitions#testAppRunningKill and - testAppKilledKilled. (Takashi Ohnishi via rohithsharmaks) - - YARN-4633. Fix random test failure in TestRMRestart#testRMRestartAfterPreemption - (Bibin A Chundatt via rohithsharmaks) - - YARN-4543. Fix random test failure in TestNodeStatusUpdater.testStopReentrant - (Akihiro Suda via rohithsharmaks) - - YARN-4615. Fix random test failure in TestAbstractYarnScheduler#testResource - RequestRecoveryToTheRightAppAttempt. (Sunil G via rohithsharmaks) - - YARN-4594. container-executor fails to remove directory tree when chmod - required (Colin Patrick McCabe via jlowe) - - YARN-4669. Fix logging statements in resource manager's Application class. - (Seethana Sidharta via vvasudev) - - YARN-4629. Distributed shell breaks under strong security. - (Daniel Templeton via stevel) - - YARN-4684. TestYarnCLI#testGetContainers failing in CN locale. - (Bibin A Chundatt via vvasudev) - - YARN-4569. Remove incorrect part of maxResources in FairScheduler - documentation. (Ray Chiang via kasha) - - YARN-4651. Document movetoqueue option in 'YARN Commands' - (Takashi Ohnishi via rohithsharmaks) - - YARN-4729. SchedulerApplicationAttempt#getTotalRequiredResources can throw - an NPE. (kasha) - - YARN-4701. When task logs are not available, port 8041 is referenced - instead of port 8042 (haibochen via rkanter) - - YARN-4566. Fix test failure in TestMiniYarnClusterNodeUtilization. - (Takashi Ohnishi via rohithsharmaks) - - YARN-4731. container-executor should not follow symlinks in - recursive_unlink_children (Colin Patrick McCabe via jlowe) - - YARN-4704. TestResourceManager#testResourceAllocation() fails when using - FairScheduler. (Yufei Gu via kasha) - -Release 2.8.0 - UNRELEASED - - INCOMPATIBLE CHANGES - - YARN-2336. Fair scheduler's REST API returns a missing '[' bracket JSON for - deep queue tree. (Kenji Kikushima and Akira Ajisaka via ozawa) - - NEW FEATURES - - YARN-3360. Add JMX metrics to TimelineDataManager (Jason Lowe via jeagles) - - YARN-3345. Add non-exclusive node label API. (Wangda Tan via jianhe) - - YARN-3365. Enhanced NodeManager to support using the 'tc' tool via - container-executor for outbound network traffic control. (Sidharta Seethana - via vinodkv) - - YARN-1376. NM need to notify the log aggregation status to RM through - heartbeat. (Xuan Gong via junping_du) - - YARN-3348. Add a 'yarn top' tool to help understand cluster usage. (Varun - Vasudev via jianhe) - - YARN-3347. Improve YARN log command to get AMContainer logs as well as - running containers logs. (Xuan Gong via junping_du) - - YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support - for new resource types on the NM. (Sidharta Seethana via junping_du) - - YARN-3361. CapacityScheduler side changes to support non-exclusive node - labels. (Wangda Tan via jianhe) - - YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. - (Craig Welch via wangda) - - YARN-3326. Support RESTful API for getLabelsToNodes. (Naganarasimha G R - via ozawa) - - YARN-3354. Add node label expression in ContainerTokenIdentifier to support - RM recovery. (Wangda Tan via jianhe) - - YARN-1402. Update related Web UI and CLI with exposing client API to check - log aggregation status. (Xuan Gong via junping_du) - - YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. - (Craig Welch via wangda) - - YARN-3410. YARN admin should be able to remove individual application - records from RMStateStore. (Rohith Sharmaks via wangda) - - YARN-3225. New parameter of CLI for decommissioning node gracefully in - RMAdmin CLI. (Devaraj K via junping_du) - - YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing - network bandwidth traffic originating from YARN containers (Sidharta Seethana - via vinodkv) - - YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) - - YARN-2498. Respect labels in preemption policy of capacity scheduler for - inter-queue preemption. (Wangda Tan via jianhe) - - YARN-2619. Added NodeManager support for disk io isolation through cgroups. - (Varun Vasudev and Wei Yan via vinodkv) - - YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. - (Jonathan Eagles via zjshen) - - YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in - RMApps. (Xuan Gong via junping_du) - - YARN-3541. Add version info on timeline service / generic history web UI - and REST API. (Zhijie Shen via xgong) - - YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from - underlying OS when configured to do so. (Varun Vasudev via vinodkv) - - YARN-41. The RM should handle the graceful shutdown of the NM. (Devaraj K via - junping_du) - - YARN-1012. Report NM aggregated container resource utilization in heartbeat. - (Inigo Goiri via kasha) - - YARN-3116. RM notifies NM whether a container is an AM container or normal - task container. (Giovanni Matteo Fumarola via zjshen) - - YARN-2003. Support for Application priority : Changes in RM and Capacity - Scheduler. (Sunil G via wangda) - - YARN-3656. LowCost: A Cost-Based Placement Agent for YARN Reservations. - (Jonathan Yaniv and Ishai Menache via curino) - - YARN-3852. Add docker container support to container-executor - (Abin Shahab via vvasudev) - - YARN-3853. Add docker container runtime support to LinuxContainterExecutor. - (Sidharta Seethana via vvasudev) - - YARN-3736. Add RMStateStore apis to store and load accepted reservations for - failover (adhoot via asuresh) - - YARN-3948. Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks) - - YARN-3873. PendingApplications in LeafQueue should also use OrderingPolicy. - (Sunil G via wangda) - - YARN-3887. Support changing Application priority during runtime. (Sunil G - via jianhe) - - YARN-4023. Publish Application Priority to TimelineServer. (Sunil G - via rohithsharmaks) - - YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) - - YARN-4055. Report node resource utilization in heartbeat. - (Inigo Goiri via kasha) - - YARN-2923. Support configuration based NodeLabelsProvider Service in Distributed - Node Label Configuration Setup. (Naganarasimha G R) - - YARN-221. NM should provide a way for AM to tell it not to aggregate logs. - (Ming Ma via xgong) - - YARN-4014. Support user cli interface in for Application Priority. - (Rohith Sharma K S via jianhe) - - YARN-3250. Support admin cli interface in for Application Priority. - (Rohith Sharma K S via jianhe) - - YARN-3970. Add REST api support for Application Priority. - (Naganarasimha G R via vvasudev) - - YARN-2884. Added a proxy service in NM to proxy the the communication - between AM and RM. (Kishore Chaliparambil via jianhe) - - YARN-313. Add Admin API for supporting node resource configuration in - command line. (Inigo Goiri, Kenji Kikushima and Junping Du - via junping_du) - - YARN-4034. Render cluster Max Priority in scheduler metrics in RM web - UI. (Rohith Sharma K S via jianhe) - - YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. - (Junping Du via wangda) - - YARN-3866. AM-RM protocol changes to support container resizing. (Meng Ding - via jianhe) - - YARN-1449. AM-NM protocol changes to support container resizing. - (Meng Ding & Wangda Tan via jianhe) - - YARN-1645. ContainerManager implementation to support container resizing. - (Meng Ding & Wangda Tan via jianhe) - - YARN-3867. ContainerImpl changes to support container resizing. (Meng Ding - via jianhe) - - YARN-1643. Make ContainersMonitor support changing monitoring size of an - allocated container. (Meng Ding and Wangda Tan) - - YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to - support container resizing. (Meng Ding via jianhe) - - YARN-3868. Recovery support for container resizing. (Meng Ding via jianhe) - - YARN-1651. CapacityScheduler side changes to support container resize. - (Wangda Tan via jianhe) - - YARN-1897. CLI and core support for signal container functionality. - (Ming Ma via xgong) - - YARN-261. Ability to fail AM attempts (Andrey Klochkov and - Rohith Sharma K S via jlowe) - - YARN-3964. Support NodeLabelsProvider at Resource Manager side. - (Dian Fu via devaraj) - - YARN-2556. Tool to measure the performance of the timeline server (Chang Li - via sjlee) - - YARN-4262. Allow whitelisted users to run privileged docker containers. - (Sidharta Seethana via vvasudev) - - YARN-3739. Add reservation system recovery to RM recovery process. - (Subru Krishnan via adhoot) - - YARN-2729. Support script based NodeLabelsProvider Interface in Distributed Node Label - Configuration Setup. (Naganarasimha G R via rohithsharmaks) - - YARN-1510. Make NMClient support change container resources. - (Meng Ding via wangda) - - YARN-1509. Make AMRMClient support send increase container request and - get increased/decreased containers. (Meng Ding via wangda) - - YARN-4184. Remove update reservation state api from state store as its not used by - ReservationSystem (Sean Po via asuresh) - - YARN-4349. Support CallerContext in YARN. (wtan via jianhe) - - YARN-3623. Add a new config to indicate the Timeline Service version. - (Xuan Gong via junping_du) - - YARN-3226. UI changes for decommissioning node. (Sunil G via - junping_du) - - YARN-3458. CPU resource monitoring in Windows. (Inigo Goiri via cnauroth) - - YARN-4234. New put APIs in TimelineClient for ats v1.5. (Xuan Gong via - junping_du) - - YARN-4265. Provide new timeline plugin storage to support fine-grained entity - caching. (Li Lu and Jason Lowe via junping_du) - - YARN-4219. New levelDB cache storage for timeline v1.5. (Li Lu via xgong) - - YARN-4411. RMAppAttemptImpl#createApplicationAttemptReport throws - IllegalArgumentException. (Bibin A Chundatt, yarntime via devaraj) - - YARN-3223. Resource update during NM graceful decommission. (Brook Zhou - via junping_du) - - IMPROVEMENTS - - YARN-644. Basic null check is not performed on passed in arguments before - using them in ContainerManagerImpl.startContainer (Varun Saxena via bobby) - - YARN-1880. Cleanup TestApplicationClientProtocolOnHA - (ozawa via harsh) - - YARN-3357. Move TestFifoScheduler to FIFO package. (Rohith Sharmaks - via devaraj) - - YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to - track used-resources-by-label. (Wangda Tan via jianhe) - - YARN-3350. YARN RackResolver spams logs with messages at info level. - (Wilfred Spiegelenburg via junping_du) - - YARN-2868. FairScheduler: Metric for latency to allocate first container - for an application. (Ray Chiang via kasha) - - YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha) - - YARN-3288. Document and fix indentation in the DockerContainerExecutor code - - YARN-2495. Allow admin specify labels from each NM (Distributed - configuration for node label). (Naganarasimha G R via wangda) - - YARN-3258. FairScheduler: Need to add more logging to investigate - allocations. (Anubhav Dhoot via ozawa) - - YARN-3428. Debug log resources to be localized for a container. (kasha) - - YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring - from info to debug. (Anubhav Dhoot via ozawa) - - YARN-2901. Add errors and warning metrics page to RM, NM web UI. - (Varun Vasudev via wangda) - - YARN-3294. Allow dumping of Capacity Scheduler debug logs via - web UI for a fixed time period. (Varun Vasudev via xgong) - - YARN-3293. Track and display capacity scheduler health metrics - in web UI. (Varun Vasudev via xgong) - - YARN-3394. Enrich WebApplication proxy documentation. (Naganarasimha G R - via jianhe) - - YARN-3404. Display queue name on application page. (Ryu Kobayashi via jianhe) - - YARN-2696. Queue sorting in CapacityScheduler should consider node label. - (Wangda Tan via jianhe) - - YARN-3451. Display attempt start time and elapsed time on the web UI. - (Rohith Sharmaks via jianhe) - - YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. (Rohith - Sharmaks via jianhe) - - YARN-3503. Expose disk utilization percentage and bad local and log dir - counts in NM metrics. (Varun Vasudev via jianhe) - - YARN-3511. Add errors and warnings page to ATS. (Varun Vasudev via xgong) - - YARN-3406. Display count of running containers in the RM's Web UI. - (Ryu Kobayashi via ozawa) - - YARN-3363. add localization and container launch time to ContainerMetrics - at NM to show these timing information for each active container. - (zxu via rkanter) - - YARN-3396. Handle URISyntaxException in ResourceLocalizationService. - (Brahma Reddy Battula via junping_du) - - YARN-3491. PublicLocalizer#addResource is too slow. (zxu via rkanter) - - YARN-3523. Cleanup ResourceManagerAdministrationProtocol interface audience. - (Naganarasimha G R via junping_du) - - YARN-3169. Drop YARN's overview document. (Brahma Reddy Battula via ozawa) - - YARN-2784. Make POM project names consistent. (Rohith via devaraj) - - YARN-20. More information for "yarn.resourcemanager.webapp.address" in - yarn-default.xml (Bartosz Ługowski vai tgraves) - - YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page. - (Naganarasimha G R via wangda) - - YARN-3018. Unified the default value for the configuration property - yarn.scheduler.capacity.node-locality-delay in code and default xml file. - (Nijel SF via vinodkv) - - YARN-2331. Distinguish shutdown during supervision vs. shutdown for - rolling upgrade. (Jason Lowe via xgong) - - YARN-1050. Document the Fair Scheduler REST API. - (Kenji Kikushima and Roman Shaposhnik via kasha) - - YARN-3271. FairScheduler: Move tests related to max-runnable-apps from - TestFairScheduler to TestAppRunnability. (nijel via kasha) - - YARN-1912. ResourceLocalizer started without any jvm memory control. - (Masatake Iwasaki via xgong) - - YARN-1287. Consolidate MockClocks. - (Sebastian Wong and Anubhav Dhoot via kasha) - - YARN-3395. FairScheduler: Trim whitespaces when using username for - queuename. (Zhihai Xu via kasha) - - YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. - projects. (Gabor Liptak via junping_du) - - YARN-3513. Remove unused variables in ContainersMonitorImpl and add debug - log for overall resource usage by all containers. (Naganarasimha G R via devaraj) - - YARN-3613. TestContainerManagerSecurity should init and start Yarn cluster in - setup instead of individual methods. (nijel via kasha) - - YARN-3579. CommonNodeLabelsManager should support NodeLabel instead of string - label name when getting node-to-label/label-to-label mappings. (Sunil G via wangda) - - YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda) - - YARN-3362. Add node label usage in RM CapacityScheduler web UI. - (Naganarasimha G R via wangda) - - YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use - NodeLabel object instead of String. (Naganarasimha G R via wangda) - - YARN-3583. Support of NodeLabel object instead of plain String - in YarnClient side. (Sunil G via wangda) - - YARN-3684. Changed ContainerExecutor's primary lifecycle methods to use a more - extensible mechanism of context objects. (Sidharta Seethana via vinodkv) - - YARN-3594. WintuilsProcessStubExecutor.startStreamReader leaks streams. - (Lars Francke via junping_du) - - YARN-3647. RMWebServices api's should use updated api from CommonNodeLabelsManager - to get NodeLabel object. (Sunil G via wangda) - - YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. - (Naganarasimha G R via wangda) - - YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils. - (Masatake Iwasaki via devaraj) - - YARN-3713. Remove duplicate function call storeContainerDiagnostics in - ContainerDiagnosticsUpdateTransition (zxu via rkanter) - - YARN-3467. Expose allocatedMB, allocatedVCores, and runningContainers metrics on - running Applications in RM Web UI. (Anubhav Dhoot via kasha) - - YARN-2392. Add more diags about app retry limits on AM failures. (Steve - Loughran via jianhe) - - YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. (xgong) - - YARN-2716. Refactor ZKRMStateStore retry code with Apache Curator. - (Karthik Kambatla via jianhe) - - YARN-3786. Document yarn class path options. - (Brahma Reddy Battula via cnauroth) - - YARN-3787. Allowed generic history service to load a number of applications whose - started time is within the given range. (Xuan Gong via zjshen) - - YARN-3785. Support for Resource as an argument during submitApp call in MockRM - test class. (Sunil G via xgong) - - YARN-3789. Improve logs for LeafQueue#activateApplications(). - (Bibin A Chundatt via devaraj) - - YARN-3148. Allow CORS related headers to passthrough in WebAppProxyServlet. - (Varun Saxena via devaraj) - - YARN-3834. Scrub debug logging of tokens during resource localization. - (Chris Nauroth via xgong) - - YARN-3827. Migrate YARN native build to new CMake framework (Alan Burlison - via Colin P. McCabe) - - YARN-3800. Reduce storage footprint for ReservationAllocation. (Anubhav Dhoot - via curino) - - YARN-3069. Document missing properties in yarn-default.xml. - (Ray Chiang via aajisaka) - - YARN-3844. Make hadoop-yarn-project Native code -Wall-clean (Alan Burlison - via Colin P. McCabe) - - YARN-3026. Move application-specific container allocation logic from - LeafQueue to FiCaSchedulerApp. (Wangda Tan via jianhe) - - YARN-3950. Add unique SHELL_ID environment variable to DistributedShell - (Robert Kanter via jlowe) - - YARN-3965. Add startup timestamp to nodemanager UI (Hong Zhiguo via jlowe) - - YARN-3543. ApplicationReport should be able to tell whether the Application - is AM managed or not. (Rohith Sharma K S via xgong) - - YARN-4004. container-executor should print output of docker logs if the docker - container exits with non-0 exit status. (Varun Vasudev via xgong) - - YARN-3961. Expose pending, running and reserved containers of a queue in REST - api and yarn top (adhoot via asuresh) - - YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. (Robert Kanter - via junping_du) - - YARN-3974. Refactor the reservation system test cases to use parameterized - base test. (subru via curino) - - YARN-3966. Fix excessive loggings in CapacityScheduler. (Jian He via wangda) - - YARN-4026. Refactored ContainerAllocator to accept a list of priorites - rather than a single priority. (Wangda Tan via jianhe) - - YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and - WebAppProxyServer (djp via rkanter) - - YARN-4057. If ContainersMonitor is not enabled, only print - related log info one time. (Jun Gong via zxu) - - YARN-1556. NPE getting application report with a null appId. (Weiwei Yang via - junping_du) - - YARN-4121. Fix typos in capacity scheduler documentation. - (Kai Sasaki via vvasudev) - - YARN-4086. Allow Aggregated Log readers to handle HAR files (rkanter) - - YARN-4145. Make RMHATestBase abstract so its not run when running all - tests under that namespace (adhoot via rkanter) - - YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha) - - YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / - CLI / REST-API. (Naganarasimha G R via wangda) - - YARN-4149. yarn logs -am should provide an option to fetch all the log files - (Varun Vasudev via xgong) - - YARN-2597. MiniYARNCluster should propagate reason for AHS not starting. - (stevel) - - MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup - failure during commitJob. (Junping Du via wangda) - - YARN-3920. FairScheduler container reservation on a node should be - configurable to limit it to large containers (adhoot via asuresh) - - HADOOP-12428. Fix inconsistency between log-level guards and statements. - (Jagadesh Kiran N and Jackie Chang via ozawa) - - YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator - between ShuffleHandler and LocalDirsHandlerService. (Zhihai Xu via jlowe) - - YARN-4176. Resync NM nodelabels with RM periodically for distributed nodelabels. - (Bibin A Chundatt via wangda) - - YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the - only modified Node Label Mappings in the request. (Naganarasimha G R via wangda) - - YARN-3943. Use separate threshold configurations for disk-full detection - and disk-not-full detection. (Zhihai Xu via jlowe) - - YARN-4252. Log container-executor invocation details when exit code is non-zero. - (Sidharta Seethana via vvasudev) - - YARN-4258. Add support for controlling capabilities for docker containers. - (Sidharta Seethana via vvasudev) - - YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity - by partition to REST API. (Naganarasimha G R via wangda) - - YARN-4170. AM need to be notified with priority in AllocateResponse. - (Sunil G via jianhe) - - YARN-4267. Add additional logging to container launch implementations in - container-executor. (Sidharta Seethana via vvasudev) - - YARN-3985. Make ReservationSystem persist state using RMStateStore - reservation APIs. (adhoot via asuresh) - - YARN-4243. Add retry on establishing Zookeeper conenction in - EmbeddedElectorService#serviceInit. (Xuan Gong via junping_du) - - YARN-2913. Fair scheduler should have ability to set MaxResourceDefault for - each queue. (Siqi Li via mingma) - - YARN-4296. DistributedShell Log.info is not friendly. - (Xiaowei Wang via stevel) - - YARN-3738. Add support for recovery of reserved apps running under dynamic - queues (subru via asuresh) - - YARN-4285. Display resource usage as percentage of queue and cluster in the - RM UI (Varun Vasudev via wangda) - - YARN-3216. Max-AM-Resource-Percentage should respect node labels. - (Sunil G via wangda) - - YARN-4310. FairScheduler: Log skipping reservation messages at DEBUG level (asuresh) - - YARN-4279. Mark ApplicationId and ApplicationAttemptId static methods as @Public, - @Unstable. (stevel) - - YARN-3454. Add efficient merge operation to RLESparseResourceAllocation - (Carlo Curino via asuresh) - - YARN-3980. Plumb resource-utilization info in node heartbeat through to the - scheduler. (Inigo Goiri via kasha) - - YARN-4132. Separate configs for nodemanager to resourcemanager connection - timeout and retries (Chang Li via jlowe) - - YARN-4292. ResourceUtilization should be a part of NodeInfo REST API. - (Sunil G via wangda) - - YARN-4405. Support node label store in non-appendable file system. (Wangda - Tan via jianhe) - - YARN-4358. Reservation System: Improve relationship between SharingPolicy - and ReservationAgent. (Carlo Curino via asuresh) - - YARN-3456. Improve handling of incomplete TimelineEntities. (Varun Saxena - via rohithsharmaks) - - YARN-4248. REST API for submit/update/delete Reservations. (curino) - - YARN-3946. Update exact reason as to why a submitted app is in ACCEPTED state to - app's diagnostic message. (Naganarasimha G R via wangda) - - YARN-4309. Add container launch related debug information to container logs - when a container fails. (Varun Vasudev via wangda) - - YARN-4418. AM Resource Limit per partition can be updated to ResourceUsage as well. - (Sunil G via wangda) - - YARN-4207. Add a non-judgemental YARN app completion status. (Rich Haase via sseth) - - YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) - - YARN-4164. Changed updateApplicationPriority API to return the updated - application priority. (Rohith Sharma K S via jianhe) - - YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh) - - YARN-4290. Add -showDetails option to YARN Nodes CLI to print all nodes reports - information. (Sunil G via wangda) - - YARN-4098. Document ApplicationPriority feature. (Rohith Sharma K S via jianhe) - - YARN-4544. All the log messages about rolling monitoring interval are - shown with WARN level. (Takashi Ohnishi via aajisaka) - - YARN-4537. Pull out priority comparison from fifocomparator and use compound - comparator for FifoOrdering policy. (Rohith Sharma K S via jianhe) - - YARN-4582. Label-related invalid resource request exception should be able to - properly handled by application. (Bibin A Chundatt via wangda) - - YARN-4304. AM max resource configuration per partition to be displayed/updated - correctly in UI and in various partition related metrics. (Sunil G via wangda) - - YARN-4524. Cleanup AppSchedulingInfo. (Karthik Kambatla via wangda) - - YARN-4371. "yarn application -kill" should take multiple application ids - (Sunil G via jlowe) - - YARN-4340. Add "list" API to reservation system. (Sean Po via wangda) - - YARN-4100. Add Documentation for Distributed and Delegated-Centralized - Node Labels feature. (Naganarasimha G R via devaraj) - - YARN-4662. Document some newly added metrics. (Jian He via xgong) - - YARN-4360. Improve GreedyReservationAgent to support "early" allocations, - and performance improvements (curino via asuresh) - - YARN-4420. Add REST API for List Reservations. (Sean Po via curino) - - YARN-4519. Potential deadlock of CapacityScheduler between decrease container - and assign containers. (Meng Ding via jianhe) - - YARN-4138. Roll back container resource allocation after resource - increase token expires. (Meng Ding via jianhe) - - YARN-2575. Create separate ACLs for Reservation create/update/delete/list - ops (Sean Po via asuresh) - - YARN-4682. AMRM client to log when AMRM token updated. - (Prabhu Joseph via stevel) - - YARN-4579. Allow DefaultContainerExecutor container log directory - permissions to be configurable (rchiang via rkanter) - - OPTIMIZATIONS - - YARN-3339. TestDockerContainerExecutor should pull a single image and not - the entire centos repository. (Ravindra Kumar Naik via raviprak) - - YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. - (Jun Gong via kasha) - - YARN-3006. Improve the error message when attempting manual failover with - auto-failover enabled. (Akira AJISAKA via wangda) - - YARN-3547. FairScheduler: Apps that have no resource demand should not participate - scheduling. (Xianyin Xin via kasha) - - YARN-3259. FairScheduler: Trigger fairShare updates on node events. - (Anubhav Dhoot via kasha) - - YARN-2768. Avoid cloning Resource in FSAppAttempt#updateDemand. - (Hong Zhiguo via kasha) - - YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend - container allocation logic. (Wangda Tan via jianhe) - - YARN-3635. Refactored current queue mapping implementation in CapacityScheduler - to use a generic PlacementManager framework. (Wangda Tan via jianhe) - - YARN-4066. Large number of queues choke fair scheduler. - (Johan Gustavsson via kasha) - - YARN-4720. Skip unnecessary NN operations in log aggregation. - (Jun Gong via mingma) - - BUG FIXES - - YARN-3197. Confusing log generated by CapacityScheduler. (Varun Saxena - via devaraj) - - YARN-3305. Normalize AM resource request on app submission. (Rohith Sharmaks - via jianhe) - - YARN-3205 FileSystemRMStateStore should disable FileSystem Cache to avoid - get a Filesystem with an old configuration. (Zhihai Xu via ozawa) - - YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to - fully qualified path. (Xuan Gong via junping_du) - - YARN-3241. FairScheduler handles "invalid" queue names inconsistently. - (Zhihai Xu via kasha) - - YARN-3383. AdminService should use "warn" instead of "info" to log exception - when operation fails. (Li Lu via wangda) - - YARN-3400. [JDK 8] Build Failure due to unreported exceptions in - RPCUtil (rkanter) - - YARN-3412. RM tests should use MockRM where possible. (kasha) - - YARN-3425. NPE from RMNodeLabelsManager.serviceStop when - NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda) - - YARN-3415. Non-AM containers can be counted towards amResourceUsage of a - Fair Scheduler queue (Zhihai Xu via Sandy Ryza) - - YARN-3435. AM container to be allocated Appattempt AM container shown as null. - (Bibin A Chundatt via xgong) - - YARN-2666. TestFairScheduler.testContinuousScheduling fails Intermittently. - (Zhihai Xu via ozawa) - - YARN-3429. TestAMRMTokens.testTokenExpiry fails Intermittently with - error message:Invalid AMRMToken (zxu via rkanter) - - YARN-3110. Few issues in ApplicationHistory web ui. (Naganarasimha G R via xgong) - - YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. - (Varun Vasudev via wangda) - - YARN-3266. RMContext#inactiveNodes should have NodeId as map key. - (Chengbing Liu via jianhe) - - YARN-3436. Fix URIs in documantion of YARN web service REST APIs. - (Bibin A Chundatt via ozawa) - - YARN-3021. YARN's delegation-token handling disallows certain trust setups - to operate properly over DistCp. (Yongjun Zhang via jianhe) - - YARN-3136. Fixed a synchronization problem of - AbstractYarnScheduler#getTransferredContainers. (Sunil G via jianhe) - - YARN-3495. Confusing log generated by FairScheduler. - (Brahma Reddy Battula via ozawa) - - YARN-3387. Previous AM's container completed status couldn't pass to current - AM if AM and RM restarted during the same time. (sandflee via jianhe) - - YARN-3444. Fix typo capabililty. (Gabor Liptak via aajisaka) - - YARN-3530. ATS throws exception on trying to filter results without otherinfo. - (zhijie shen via xgong) - - YARN-2740. Fix NodeLabelsManager to properly handle node label modifications - when distributed node label configuration enabled. (Naganarasimha G R via wangda) - - YARN-3517. RM web ui for dumping scheduler logs should be for admins only - (Varun Vasudev via tgraves) - - YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be scheduled. - (Anubhav Dhoot via jianhe) - - YARN-3564. Fix TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable - fails randomly. (Jian He via wangda) - - YARN-2893. AMLaucher: sporadic job failures due to EOFException in - readTokenStorageStream. (Zhihai Xu via gera) - - YARN-2454. Fix compareTo of variable UNBOUNDED in o.a.h.y.util.resource.Resources. - (Xu Yang via junping_du) - - YARN-1993. Cross-site scripting vulnerability in TextView.java. (Kenji Kikushima - via ozawa) - - YARN-3097. Logging of resource recovery on NM restart has redundancies - (Eric Payne via jlowe) - - YARN-3375. NodeHealthScriptRunner.shouldRun() check is performing 3 times for - starting NodeHealthScriptRunner. (Devaraj K via wangda) - - YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore. - (Tsuyoshi Ozawa via jianhe) - - YARN-3552. RM Web UI shows -1 running containers for completed apps - (Rohith via jlowe) - - YARN-2123. Progress bars in Web UI always at 100% due to non-US locale. - (Akira AJISAKA via xgong) - - YARN-3343. Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate - timeout. (Rohith Sharmaks via jianhe) - - YARN-3582. NPE in WebAppProxyServlet. (jian he via xgong) - - YARN-3577. Misspelling of threshold in log4j.properties for tests. - (Brahma Reddy Battula via aajisaka) - - YARN-3584. Fixed attempt diagnostics format shown on the UI. (nijel via - jianhe) - - YARN-1832. Fix wrong MockLocalizerStatus#equals implementation. - (Hong Zhiguo via aajisaka) - - YARN-3572. Correct typos in WritingYarnApplications.md. - (Gabor Liptak via aajisaka) - - YARN-3592. Fix typos in RMNodeLabelsManager. (Sunil G via devaraj) - - YARN-3589. RM and AH web UI display DOCTYPE wrongly. (Rohith via ozawa) - - YARN-3600. AM container link is broken (Naganarasimha G R via tgraves) - - YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch. - (zhihai xu via jianhe) - - YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter) - - YARN-2206. Updated document for applications REST API response examples. (Kenji - Kikushima and Brahma Reddy Battula via zjshen) - - YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir - fails Intermittently due to IOException from cleanup. (zhihai xu via xgong) - - YARN-3629. NodeID is always printed as "null" in node manager initialization log. - (nijel via devaraj) - - YARN-2921. Fix MockRM/MockAM#waitForState sleep too long. - (Tsuyoshi Ozawa via wangda) - - YARN-1519. Check in container-executor if sysconf is implemented before - using it (Radim Kolar and Eric Payne via raviprak) - - YARN-2421. RM still allocates containers to an app in the FINISHING - state (Chang Li via jlowe) - - YARN-3302. TestDockerContainerExecutor should run automatically if it can - detect docker in the usual place (Ravindra Kumar Naik via raviprak) - - YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted. - (Varun Vasudev via jianhe) - - YARN-3654. ContainerLogsPage web UI should not have meta-refresh. (Xuan Gong - via jianhe) - - YARN-3707. RM Web UI queue filter doesn't work. (Wangda Tan via jianhe) - - YARN-3632. Ordering policy should be allowed to reorder an application when - demand changes. (Craig Welch via jianhe) - - YARN-3716. Node-label-expression should be included by - ResourceRequestPBImpl.toString. (Xianyin Xin via wangda) - - YARN-3751. Fixed AppInfo to check if used resources are null. (Sunil G via - zjshen) - - YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha) - - YARN-3749. We should make a copy of configuration when init MiniYARNCluster - with multiple RMs. (Chun Chen via xgong) - - YARN-3766. Fixed the apps table column error of generic history web UI. - (Xuan Gong via zjshen) - - YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation - and container reservation. (Zhihai Xu via kasha) - - YARN-3747. TestLocalDirsHandlerService should delete the created test directory logDir2. - (David Moore via devaraj) - - YARN-3778. Fix Yarn resourcemanger CLI usage. (Brahma Reddy Battula via xgong) - - YARN-3794. TestRMEmbeddedElector fails because of ambiguous LOG reference. - (Chengbing Liu via devaraj) - - YARN-3714. AM proxy filter can not get RM webapp address from - yarn.resourcemanager.hostname.rm-id. (Masatake Iwasaki via xgong) - - YARN-3617. Fix WindowsResourceCalculatorPlugin.getCpuFrequency() - returning always -1. (J.Andreina via devaraj) - - YARN-3824. Fix two minor nits in member variable properties - of YarnConfiguration. (Ray Chiang via devaraj) - - YARN-3835. hadoop-yarn-server-resourcemanager test package bundles - core-site.xml, yarn-site.xml (vamsee via rkanter) - - YARN-3790. usedResource from rootQueue metrics may get stale data for FS - scheduler after recovering the container (Zhihai Xu via rohithsharmaks) - - YARN-3826. Race condition in ResourceTrackerService leads to - wrong diagnostics messages. (Chengbing Liu via devaraj) - - YARN-3745. SerializedException should also try to instantiate internal - exception with the default constructor. (Lavkesh Lahngir via devaraj) - - YARN-2871. TestRMRestart#testRMRestartGetApplicationList sometime fails in trunk. - (zhihai xu via xgong) - - YARN-3859. LeafQueue doesn't print user properly for application add. - (Varun Saxena via devaraj) - - YARN-3860. rmadmin -transitionToActive should check the state of non-target node. - (Masatake Iwasaki via junping_du) - - YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non - network exception. (Raju Bairishetti via jianhe) - - YARN-3770. SerializedException should also handle java.lang.Error on - de-serialization. (Lavkesh Lahngir via jianhe) - - YARN-3768. ArrayIndexOutOfBoundsException with empty environment variables. - (Zhihai Xu via gera) - - YARN-3823. Fix mismatch in default values for - yarn.scheduler.maximum-allocation-vcores property. (Ray Chiang via devaraj) - - YARN-3830. AbstractYarnScheduler.createReleaseCache may try to clean a null - attempt. (nijel via devaraj) - - YARN-3875. FSSchedulerNode#reserveResource() doesn't print Application Id - properly in log. (Bibin A Chundatt via devaraj) - - YARN-3882. AggregatedLogFormat should close aclScanner and ownerScanner - after create them. (zhihai xu via xgong) - - YARN-3837. javadocs of TimelineAuthenticationFilterInitializer give wrong - prefix for auth options. (Bibin A Chundatt via devaraj) - - YARN-2194. Fix bug causing CGroups functionality to fail on RHEL7. - (Wei Yan via vvasudev) - - YARN-3892. Fixed NPE on RMStateStore#serviceStop when - CapacityScheduler#serviceInit fails. (Bibin A Chundatt via jianhe) - - YARN-3888. ApplicationMaster link is broken in RM WebUI when appstate is NEW. - (Bibin A Chundatt via xgong) - - YARN-3917. getResourceCalculatorPlugin for the default should intercept all - exceptions. (gera) - - YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity - configuration. (Bibin A Chundatt via wangda) - - YARN-3381. Fix typo InvalidStateTransitonException. - (Brahma Reddy Battula via aajisaka) - - YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator - in DRF queues to prevent unnecessary thrashing. (asuresh) - - YARN-3174. Consolidate the NodeManager and NodeManagerRestart documentation - into one. (Masatake Iwasaki via ozawa) - - YARN-3805. Update the documentation of Disk Checker based on YARN-90. - (Masatake Iwasaki via ozawa) - - YARN-3930. FileSystemNodeLabelsStore should make sure edit log file closed when - exception is thrown. (Dian Fu via wangda) - - YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is - more than 2 level. (Ajith S via wangda) - - YARN-3932. SchedulerApplicationAttempt#getResourceUsageReport and UserInfo - should based on total-used-resources. (Bibin A Chundatt via wangda) - - YARN-3954. Fix TestYarnConfigurationFields#testCompareConfigurationClassAgainstXml. - (varun saxena via rohithsharmaks) - - YARN-3956. Fix TestNodeManagerHardwareUtils fails on Mac (Varun Vasudev via wangda) - - YARN-3941. Proportional Preemption policy should try to avoid sending duplicate - PREEMPT_CONTAINER event to scheduler. (Sunil G via wangda) - - YARN-3900. Protobuf layout of yarn_security_token causes errors in other protos - that include it (adhoot via rkanter) - - YARN-3845. Scheduler page does not render RGBA color combinations in IE11. - (Contributed by Mohammad Shahid Khan) - - YARN-3957. FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to - return 500. (Anubhav Dhoot via kasha) - - YARN-3973. Recent changes to application priority management break - reservation system from YARN-1051. (Carlo Curino via wangda) - - YARN-3958. TestYarnConfigurationFields should be moved to hadoop-yarn-api - module. (Varun Saxena via aajisaka) - - YARN-3846. RM Web UI queue filter is not working for sub queue. - (Mohammad Shahid Khan via jianhe) - - YARN-3982. container-executor parsing of container-executor.cfg broken in - trunk and branch-2. (Varun Vasudev via xgong) - - YARN-3919. NPEs' while stopping service after exception during - CommonNodeLabelsManager#start. (varun saxane via rohithsharmaks) - - YARN-3963. AddNodeLabel on duplicate label addition shows success. - (Bibin A Chundatt via wangda) - - YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue - on nodelabel recovery. (Bibin A Chundatt via wangda) - - YARN-433. When RM is catching up with node updates then it should not expire - acquired containers. (Xuan Gong via zxu) - - YARN-3992. TestApplicationPriority.testApplicationPriorityAllocation fails - intermittently. (Contributed by Sunil G) - - YARN-3987. Send AM container completed msg to NM once AM finishes. - (sandflee via jianhe) - - YARN-4028. AppBlock page key update and diagnostics value null on recovery - (Bibin A Chundatt via xgong) - - YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present - in YarnScheduler interface instead. (Varun Saxena via rohithsharmaks) - - YARN-4082. Container shouldn't be killed when node's label updated. - (Wangda Tan via vvasudev) - - YARN-4073. Removed unused ApplicationACLsManager in ContainerManagerImpl constructor. - (Naganarasimha G R via rohithsharmaks) - - YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. - (Hong Zhiguo via wangda) - - YARN-3591. Resource localization on a bad disk causes subsequent containers failure. - (Lavkesh Lahngir via vvasudev) - - YARN-4106. NodeLabels for NM in distributed mode is not updated even after - clusterNodelabel addition in RM. (Bibin A Chundatt via wangda) - - YARN-4115. Reduce loglevel of ContainerManagementProtocolProxy to Debug - (adhoot via rkanter) - - YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. - (Meng Ding via wangda) - - YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. - (Naganarasimha G R via jianhe) - - YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state. - (Nijel S F via rohithsharmaks) - - YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks) - - YARN-4113. RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER. - (Sunil G via wangda) - - YARN-4188. Make MoveApplicationAcrossQueues abstract, newInstance static. - (Giovanni Matteo Fumarola via cdouglas) - - YARN-4171. Fix findbugs warnings in YARN-1197 branch. (Wangda Tan via jianhe) - - YARN-4152. NodeManager crash with NPE when LogAggregationService#stopContainer called for - absent container. (Bibin A Chundatt via rohithsharmaks) - - YARN-4044. Running applications information changes such as movequeue is not published to - TimeLine server. (Sunil G via rohithsharmaks) - - YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot) - - YARN-4141. Runtime Application Priority change should not throw exception - for applications at finishing states (Sunil G via jlowe) - - YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks) - - YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned - for a user. (Anubhav Dhoot via rohithsharmaks) - - YARN-4140. RM container allocation delayed incase of app submitted to - Nodelabel partition. (Bibin A Chundatt via wangda) - - YARN-4201. AMBlacklist does not work for minicluster. (Jun Gong via zxu) - - YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. - (Meng Ding via jianhe) - - YARN-4017. container-executor overuses PATH_MAX. (Sidharta Seethana via vvasudev) - - YARN-4253. Standardize on using PrivilegedOperationExecutor for all - invocations of container-executor in LinuxContainerExecutor. (Sidharta Seethana via vvasudev) - - YARN-4255. container-executor does not clean up docker operation command files. - (Sidharta Seethana via vvasudev) - - YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy Battula via rohithsharmaks) - - YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart. - (Varun Saxena via jianhe) - - YARN-4155. TestLogAggregationService.testLogAggregationServiceWithInterval failing - (Bibin A Chundatt via stevel) - - YARN-4270. Limit application resource reservation on nodes for non-node/rack - specific requests (asuresh) - - YARN-4256. YARN fair scheduler vcores with decimal values. (Jun Gong via zxu) - - YARN-4294. [JDK8] Fix javadoc errors caused by wrong reference and illegal - tag. (aajisaka) - - YARN-4289. TestDistributedShell failing with bind exception. - (Brahma Reddy Battula via stevel) - - YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) - - YARN-4246. NPE while listing app attempt. (Nijel S F via rohithsharmaks) - - YARN-3528. Tests with 12345 as hard-coded port break jenkins. - (Brahma Reddy Battula via ozawa) - - YARN-4223. Fixed findbugs warnings in hadoop-yarn-server-nodemanager project - (varun saxena via rohithsharmaks) - - YARN-4284. condition for AM blacklisting is too narrow (Sangjin Lee via - jlowe) - - YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. - (Naganarasimha G R via wangda) - - YARN-4300. [JDK8] Fix javadoc errors caused by wrong tags. (aajisaka) - - YARN-3573. MiniMRYarnCluster constructor that starts the timeline server - using a boolean should be marked deprecated. (Brahma Reddy Battula via ozawa) - - YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt - (Bibin A Chundatt via vvasudev) - - YARN-4251. TestAMRMClientOnRMRestart#testAMRMClientOnAMRMTokenRollOverOnRMRestart - is failing. (Brahma Reddy Battula via ozawa) - - YARN-4130. Duplicate declaration of ApplicationId in RMAppManager#submitApplication method. - (Kai Sasaki via rohithsharmaks) - - YARN-4288. Fixed RMProxy to retry on IOException from local host. - (Junping Du via jianhe) - - YARN-4127. RM fail with noAuth error if switched from failover to non-failover. - (Varun Saxena via jianhe) - - YARN-4345. yarn rmadmin -updateNodeResource doesn't work (Junping Du via - jlowe) - - YARN-4367. SLS webapp doesn't load. (kasha). - - YARN-4298. Fix findbugs warnings in hadoop-yarn-common. - (Sunil G via aajisaka) - - YARN-4387. Fix typo in FairScheduler log message. (Xin Wang via ozawa) - - YARN-4384. updateNodeResource CLI should not accept negative values for resource. - (Junping Du via wangda) - - YARN-4408. Fix issue that NodeManager reports negative running containers. - (Robert Kanter via junping_du) - - YARN-4392. ApplicationCreatedEvent event time resets after RM restart/failover. - (Naganarasimha G R and Xuan Gong via xgong) - - YARN-4431. Not necessary to do unRegisterNM() if NM get stop due to failed to connect - to RM. (Junping Du via rohithsharmaks) - - YARN-4421. Remove dead code in RmAppImpl.RMAppRecoveredTransition. - (Daniel Templeton via rohithsharmaks) - - YARN-4403. (AM/NM/Container)LivelinessMonitor should use monotonic time - when calculating period. (Junping Du via jianhe) - - YARN-4402. TestNodeManagerShutdown And TestNodeManagerResync fails with - bind exception. (Brahma Reddy Battula via jianhe) - - YARN-4440. FSAppAttempt#getAllowedLocalityLevelByTime should init the - lastScheduler time. (Lin Yiqun via zxu) - - YARN-4452. NPE when submit Unmanaged application. (Naganarasimha G R - via junping_du) - - YARN-4225. Add preemption status to yarn queue -status for capacity scheduler. - (Eric Payne via wangda) - - YARN-4416. Deadlock due to synchronised get Methods in AbstractCSQueue. - (Naganarasimha G R via wangda) - - YARN-4461. Redundant nodeLocalityDelay log in LeafQueue (Eric Payne via - jlowe) - - YARN-4454. NM to nodelabel mapping going wrong after RM restart. - (Bibin A Chundatt via wangda) - - YARN-4477. FairScheduler: Handle condition which can result in an - infinite loop in attemptScheduling. (Tao Jie via asuresh) - - YARN-4400. AsyncDispatcher.waitForDrained should be final. (Daniel Templeton - via junping_du) - - YARN-1382. Remove unusableRMNodesConcurrentSet (never used) in NodeListManager - to get rid of memory leak. (Rohith Sharma K S via junping_du) - - YARN-4546. ResourceManager crash due to scheduling opportunity overflow. - (Jason Lowe via junping_du) - - YARN-4479. Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps. - (Rohith Sharma K S via jianhe) - - YARN-4534. Remove the redundant symbol in yarn rmadmin help msg. - (Lin Yiqun via aajisaka) - - YARN-4389. "yarn.am.blacklisting.enabled" and "yarn.am.blacklisting.disable- - failure-threshold" should be app specific rather than a setting for whole - YARN cluster. (Sunil G via rohithsharmaks) - - YARN-4581. AHS writer thread leak makes RM crash while RM is recovering. - (sandflee via junping_du) - - YARN-4538. QueueMetrics pending cores and memory metrics wrong. - (Bibin A Chundatt via wangda) - - YARN-4596. SystemMetricPublisher should not swallow error messages from - TimelineClient#putEntities. (Li Lu via jianhe) - - YARN-4502. Fix two AM containers get allocated when AM restart. - (Vinod Kumar Vavilapalli via wangda) - - YARN-4565. Fix a bug that leads to AM resource limit not hornored when - sizeBasedWeight enabled for FairOrderingPolicy. (wtan via jianhe) - - YARN-4557. Fix improper Queues sorting in PartitionedQueueComparator - when accessible-node-labels=*. (Naganarasimha G R via wangda) - - YARN-4608. Redundant code statement in WritingYarnApplications. - (Kai Sasaki via aajisaka) - - YARN-4605. Spelling mistake in the help message of "yarn applicationattempt" - command. (Weiwei Yang via aajisaka) - - YARN-4614. Fix random failure in TestApplicationPriority#testApplicationPriority - AllocationWithChangeInPriority. (Sunil G via rohithsharmaks) - - YARN-4592. Remove unused GetContainerStatus proto. (Chang Li via aajisaka) - - YARN-4520. Finished app info is unnecessarily persisted in NM state-store - if container is acquired but not lunched on this node. (sandflee via jianeh) - - YARN-4643. Container recovery is broken with delegating container runtime - (Sidharta Seethana via jlowe) - - YARN-4617. LeafQueue#pendingOrderingPolicy should always use fixed ordering - policy instead of using same as active applications ordering policy. - (Rohith Sharma K S via jianhe) - - YARN-4667. RM Admin CLI for refreshNodesResources throws NPE when nothing - is configured. (Naganarasimha G R via devaraj) - - YARN-4654. Yarn node label CLI should parse "=" correctly when trying to - remove all labels on a node. (Naganarasimha G R via rohithsharmaks) - - YARN-4386. refreshNodesGracefully() should send recommission event to active - RMNodes only. (Kuhu Shukla via junping_du) - - YARN-4709. NMWebServices produces incorrect JSON for containers. - (Varun Saxena via vvasudev) - - YARN-4748. ApplicationHistoryManagerOnTimelineStore should not - swallow exceptions on generateApplicationReport. (Li Lu via jianhe) - - YARN-4671. There is no need to acquire CS lock when completing a container. - (Meng Ding via jianhe) - -Release 2.7.3 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-4287. Capacity Scheduler: Rack Locality improvement (Nathan Roberts via wangda) - - YARN-4492. Add documentation for preemption supported in Capacity - scheduler (Naganarasimha G R via jlowe) - - YARN-4653. Document YARN security model from the perspective of - Application Developers. (Steve Loughran via jianhe) - - OPTIMIZATIONS - - BUG FIXES - - YARN-4326. Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer - binds to default port 8188. (Meng Ding via wangda) - - YARN-4347. Resource manager fails with Null pointer exception. (Jian He via wangda) - - YARN-4374. RM capacity scheduler UI rounds user limit factor (Chang Li via - jlowe) - - YARN-3849. Too much of preemption activity causing continuos killing of - containers across queues. (Sunil G via wangda) - - YARN-3769. Consider user limit when calculating total pending resource for - preemption policy in Capacity Scheduler. (Eric Payne via wangda) - - YARN-4344. NMs reconnecting with changed capabilities can lead to wrong - cluster resource calculations (Varun Vasudev via jlowe) - - YARN-4365. FileSystemNodeLabelStore should check for root dir existence on - startup (Kuhu Shukla via jlowe) - - YARN-4380. TestResourceLocalizationService.testDownloadingResourcesOnContainerKill - fails intermittently. (Varun Saxena via ozawa) - - YARN-4398. Remove unnecessary synchronization in RMStateStore. (Ning Ding via jianhe) - - YARN-3840. Resource Manager web ui issue when sorting application by id - (with application having id > 9999) （Mohammad Shahid Khan & Varun Saxena - via jianhe) - - YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page - (Eric Payne via jeagles) - - YARN-4439. Clarify NMContainerStatus#toString method. (Jian He via xgong) - - YARN-4452. NPE when submit Unmanaged application. (Naganarasimha G R via - junping_du) - - YARN-4315. NaN in Queue percentage for cluster apps page. (Bibin A Chundatt via wangda) - - YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) - - YARN-4546. ResourceManager crash due to scheduling opportunity overflow. - (Jason Lowe via junping_du) - - YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non - network exception. (Raju Bairishetti via jianhe) - - YARN-4414. Nodemanager connection errors are retried at multiple levels - (Chang Li via jlowe) - - YARN-4581. AHS writer thread leak makes RM crash while RM is recovering. - (sandflee via junping_du) - - YARN-4610. Reservations continue looking for one app causes other apps to - starve (jlowe) - - YARN-4598. Invalid event: RESOURCE_FAILED at - CONTAINER_CLEANEDUP_AFTER_KILL (tangshangwen via jlowe) - - YARN-4428. Redirect RM page to AHS page when AHS turned on and RM page is - not available (Chang Li via jlowe) - - YARN-3102. Decommisioned Nodes not listed in Web UI (Kuhu Shukla via - jlowe) - - YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir - fails Intermittently due to IOException from cleanup. (zhihai xu via xgong) - - YARN-4707. Remove the extra char (>) from SecureContainer.md. - (Brahma Reddy Battula via aajisaka) - - YARN-2046. Out of band heartbeats are sent only on container kill and - possibly too early (Ming Ma via jlowe) - - YARN-4723. NodesListManager$UnknownNodeId ClassCastException (Kuhu Shukla - via jlowe) - -Release 2.7.2 - 2016-01-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles) - - YARN-3170. YARN architecture document needs updating. (Brahma Reddy Battula - via ozawa) - - YARN-3967. Fetch the application report from the AHS if the RM does not know about it. - (Mit Desai via xgong) - - YARN-2801. Add documentation for node labels feature. (Wangda Tan and Naganarasimha - G R via ozawa) - - YARN-2513. Host framework UIs in YARN for use with the ATS (jeagles) - - OPTIMIZATIONS - - BUG FIXES - - YARN-3793. Several NPEs when deleting local files on NM recovery (Varun - Saxena via jlowe) - - YARN-3508. Prevent processing preemption events on the main RM dispatcher. - (Varun Saxena via wangda) - - YARN-3690. [JDK8] 'mvn site' fails. (Brahma Reddy Battula via aajisaka) - - YARN-3905. Application History Server UI NPEs when accessing apps run after - RM restart (Eric Payne via jeagles) - - YARN-3535. Scheduler must re-request container resources when RMContainer transitions - from ALLOCATED to KILLED (rohithsharma and peng.zhang via asuresh) - - YARN-3878. AsyncDispatcher can hang while stopping if it is configured for - draining events on stop. (Varun Saxena via jianhe) - - YARN-3969. Allow jobs to be submitted to reservation that is active - but does not have any allocations. (subru via curino) - - YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container - log files from full disks. (zhihai xu via jlowe) - - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. - (mujunchao via zxu) - - YARN-3893. Both RM in active state when Admin#transitionToActive failure - from refeshAll() (Bibin A Chundatt via rohithsharmaks) - - YARN-4103. RM WebServices missing scheme for appattempts logLinks. - (Jonathan Eagles via vvasudeb) - - YARN-4105. Capacity Scheduler headroom for DRF is wrong (Chang Li via - jlowe) - - YARN-4096. App local logs are leaked if log aggregation fails to initialize - for the app. (Jason Lowe via zxu) - - YARN-4136. LinuxContainerExecutor loses info when forwarding - ResourceHandlerException. (Bibin A Chundatt via vvasudev) - - YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. - (Zhihai Xu via kasha) - - YARN-4126. RM should not issue delegation tokens in unsecure mode. - (Bibin A Chundatt via jianhe) - - YARN-4158. Remove duplicate close for LogWriter in - AppLogAggregatorImpl#uploadLogsForContainers (Zhihai Xu via jlowe) - - YARN-3433. Jersey tests failing with Port in Use -again. - (Brahma Reddy Battula) - - YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is - enabled (Mit Desai via jlowe) - - YARN-3624. ApplicationHistoryServer should not reverse the order of the - filters it gets. (Mit Desai via xgong) - - YARN-4180. AMLauncher does not retry on failures when talking to NM. - (adhoot) - - YARN-3619. ContainerMetrics unregisters during getMetrics and leads to - ConcurrentModificationException (Zhihai Xu via jlowe) - - YARN-4209. RMStateStore FENCED state doesn’t work due to updateFencedState called - by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks) - - YARN-4041. Slow delegation token renewal can severely prolong RM recovery - (Sunil G via jlowe) - - YARN-2902. Killing a container that is localizing can orphan resources in - the DOWNLOADING state (Varun Saxena via jlowe) - - YARN-4313. Race condition in MiniMRYarnCluster when getting history server - address. (Jian He via xgong) - - YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. (Robert Kanter - via junping_du) - - YARN-4312. TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 - as some of the test cases time out. (Varun Saxena via ozawa) - - YARN-4320. TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer - binds to default port 8188. (Varun Saxena via ozawa) - - YARN-4354. Public resource localization fails with NPE. (Jason Lowe via - junping_du) - -Release 2.7.1 - 2015-07-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-3243. CapacityScheduler should pass headroom from parent to children - to make sure ParentQueue obey its capacity limits. (Wangda Tan via jianhe) - - YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue - info once. (Varun Saxena via wangda) - - YARN-3539. Updated timeline server documentation and marked REST APIs evolving. - (Steve Loughran via zjshen) - - YARN-3723. Need to clearly document primaryFilter and otherInfo value type. - (Zhijie Shen via xgong) - - YARN-3711. Documentation of ResourceManager HA should explain configurations - about listen addresses. (Masatake Iwasaki via ozawa) - - OPTIMIZATIONS - - BUG FIXES - - YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf - without making a copy. (Jason Lowe via jianhe) - - YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. - (Xuan Gong via stevel) - - YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the - correct user. (Zhijie Shen via jianhe) - - YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) - - YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root - queue metrics. (Rohit Agarwal via jianhe) - - YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. - (Rohith Sharmaks via jianhe) - - YARN-3465. Use LinkedHashMap to preserve order of resource requests. - (Zhihai Xu via kasha) - - YARN-3516. killing ContainerLocalizer action doesn't take effect when - private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) - - YARN-3485. FairScheduler headroom calculation doesn't consider - maxResources for Fifo and FairShare policies. (kasha) - - YARN-3301. Fixed the format issue of the new RM attempt web page. - (Xuan Gong via jianhe) - - YARN-3385. Fixed a race-condition in ResourceManager's ZooKeeper based - state-store to avoid crashing on duplicate deletes. (Zhihai Xu via vinodkv) - - YARN-3358. Audit log not present while refreshing Service ACLs. - (Varun Saxena via devaraj) - - YARN-3476. Nodemanager can fail to delete local logs if log aggregation - fails (Rohith via jlowe) - - YARN-3434. Interaction between reservations and userlimit can result in - significant ULF violation (tgraves) - - YARN-3626. On Windows localized resources are not moved to the front - of the classpath when they should be. (Craig Welch via xgong) - - YARN-3457. NPE when NodeManager.serviceInit fails and stopRecoveryStore called. - (Bibin A Chundatt via ozawa) - - YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore - invoked (Brahma Reddy Battula via jlowe) - - YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) - - YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. - (Vinod Kumar Vavilapalli via ozawa) - - YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. - (Craig Welch and Varun Saxena via xgong) - - YARN-3609. Load node labels from storage inside RM serviceStart. (Wangda - Tan via jianhe) - - YARN-3694. Fix dead link for TimelineServer REST API. - (Jagadesh Kiran N via aajisaka) - - YARN-3646. Applications are getting stuck some times in case of retry - policy forever. (Raju Bairishetti via devaraj) - - YARN-3675. FairScheduler: RM quits when node removal races with - continuous-scheduling on the same node. (Anubhav Dhoot via kasha) - - YARN-3701. Isolating the error of generating a single app report when - getting all apps from generic history service. (Zhijie Shen via xgong) - - YARN-2238. filtering on UI sticks even if I move away from the page. - (Jian He via xgong) - - YARN-3686. CapacityScheduler should trim default_node_label_expression. - (Sunil G via wangda) - - YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent - to another. (Wangda Tan via jianhe) - - YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl - (Varun Saxena via xgong) - - YARN-3842. NMProxy should retry on NMNotYetReadyException. - (Robert Kanter via kasha) - - YARN-3809. Failed to launch new attempts because - ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) - -Release 2.7.0 - 2015-04-20 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-2179. [YARN-1492] Initial cache manager structure and context. - (Chris Trezzo via kasha) - - YARN-2180. [YARN-1492] In-memory backing store for cache manager. - (Chris Trezzo via kasha) - - YARN-2183. [YARN-1492] Cleaner service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2186. [YARN-1492] Node Manager uploader service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2236. [YARN-1492] Shared Cache uploader service on the Node - Manager. (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2188. [YARN-1492] Client service for cache manager. - (Chris Trezzo and Sangjin Lee via kasha) - - YARN-2189. [YARN-1492] Admin service for cache manager. - (Chris Trezzo via kasha) - - YARN-2765. Added leveldb-based implementation for RMStateStore. (Jason Lowe - via jianhe) - - YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha) - - YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. - (Anubhav Dhoot via kasha) - - YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. - (Anubhav Dhoot via kasha) - - YARN-2427. Added the API of moving apps between queues in RM web services. - (Varun Vasudev via zjshen) - - YARN-2217. [YARN-1492] Shared cache client side changes. - (Chris Trezzo via kasha) - - YARN-2616 [YARN-913] Add CLI client to the registry to list, view - and manipulate entries. (Akshay Radia via stevel) - - YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) - - YARN-2786. Created a yarn cluster CLI and seeded with one command for listing - node-labels collection. (Wangda Tan via vinodkv) - - IMPROVEMENTS - - YARN-3005. [JDK7] Use switch statement for String instead of if-else - statement in RegistrySecurity.java (Kengo Seki via aajisaka) - - YARN-2950. Change message to mandate, not suggest JS requirement on UI. - (Dustin Cote via harsh) - - YARN-2891. Failed Container Executor does not provide a clear error - message. (Dustin Cote via harsh) - - YARN-1979. TestDirectoryCollection fails when the umask is unusual. - (Vinod Kumar Vavilapalli and Tsuyoshi OZAWA via junping_du) - - YARN-2641. Decommission nodes on -refreshNodes instead of next - NM-RM heartbeat. (Zhihai Xu via kasha) - - YARN-2742. FairSchedulerConfiguration should allow extra spaces - between value and unit. (Wei Yan via kasha) - - YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with - queue and headroom checks. (Tsuyoshi Ozawa via kasha) - - YARN-2735. diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff - are initialized twice in DirectoryCollection. (Zhihai Xu via kasha) - - YARN-570. Time strings are formated in different timezone. - (Akira Ajisaka and Peng Zhang via kasha) - - YARN-2780. Log aggregated resource allocation in rm-appsummary.log (Eric - Payne via jlowe) - - YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes - independent of Scheduler type. (Anubhav Dhoot via kasha) - - YARN-2157. Added YARN metrics in the documentaion. (Akira AJISAKA via - jianhe) - - YARN-2802. ClusterMetrics to include AM launch and register delays. - (Zhihai Xu via kasha) - - YARN-2375. Allow enabling/disabling timeline server per framework. - (Mit Desai via jeagles) - - YARN-2604. Scheduler should consider max-allocation-* in conjunction - with the largest node. (Robert Kanter via kasha) - - YARN-2679. Add metric for container launch duration. (Zhihai Xu via kasha) - - YARN-2669. FairScheduler: queue names shouldn't allow periods - (Wei Yan via Sandy Ryza) - - YARN-2404. Removed ApplicationAttemptState and ApplicationState class in - RMStateStore. (Tsuyoshi OZAWA via jianhe) - - YARN-2165. Added the sanity check for the numeric configuration values of - the timeline service. (Vasanth kumar RJ via zjshen) - - YARN-2907. SchedulerNode#toString should print all resource detail instead - of only memory. (Rohith via junping_du) - - YARN-2136. Changed RMStateStore to ignore store opearations when fenced. - (Varun Saxena via jianhe) - - YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics - for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) - - YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) - - YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments - before sending to RM. (Rohith Sharmaks via jianhe) - - YARN-2972. DelegationTokenRenewer thread pool never expands. (Jason Lowe - via junping_du) - - YARN-2949. Add documentation for CGroups (Varun Vasudev via junping_du) - - YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command. - (Varun Saxena via junping_du) - - YARN-2837. Support TimeLine server to recover delegation token when - restarting. (Zhijie Shen via jianhe) - - YARN-2993. Several fixes (missing acl check, error log msg ...) and some - refinement in AdminService. (Yi Liu via junping_du) - - YARN-2943. Added node-labels page on RM web UI. (Wangda Tan via jianhe) - - YARN-2998. Abstract out scheduler independent PlanFollower components. - (Anubhav Dhoot via kasha) - - YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the - scheduler page. (Ashwin Shankar and Wei Yan via kasha) - - YARN-2880. Added a test to make sure node labels will be recovered - if RM restart is enabled. (Rohith Sharmaks via jianhe) - - YARN-2996. Improved synchronization and I/O operations of FS- and Mem- - RMStateStore. (Yi Liu via zjshen) - - YARN-2956. Added missing links in YARN documentation. (Masatake Iwasaki via - jianhe) - - YARN-2957. Create unit test to automatically compare YarnConfiguration - and yarn-default.xml. (rchiang via rkanter) - - YARN-2643. Don't create a new DominantResourceCalculator on every - FairScheduler.allocate call. (kasha via rkanter) - - YARN-3019. Make work-preserving-recovery the default mechanism for RM - recovery. (Jian He via junping_du) - - YARN-2807. Option "--forceactive" not works as described in usage of - "yarn rmadmin -transitionToActive". (Masatake Iwasaki via xgong) - - YARN-2984. Metrics for container's actual memory usage. (kasha) - - YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable - node labels feature. (Wangda Tan via ozawa) - - YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. - (Robert Metzger via ozawa) - - YARN-2897. CrossOriginFilter needs more log statements (Mit Desai via - jeagles) - - YARN-3028. Better syntax for replaceLabelsOnNode in RMAdmin CLI - (Rohith Sharmaks via wangda) - - YARN-2932. Add entry for "preemptable" status (enabled/disabled) to - scheduler web UI and queue initialize/refresh logging. - (Eric Payne via wangda) - - YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Chang Li - via jeagles) - - YARN-2808. Made YARN CLI list attempt’s finished containers of a running - application. (Naganarasimha G R via zjshen) - - YARN-3085. Application summary should include the application type (Rohith - via jlowe) - - YARN-3022. Expose Container resource information from NodeManager for - monitoring (adhoot via ranter) - - YARN-3075. NodeLabelsManager implementation to retrieve label to node - mapping (Varun Saxena via wangda) - - YARN-1393. SLS: Add how-to-use instructions. (Wei Yan via kasha) - - YARN-1723. AMRMClientAsync missing blacklist addition and removal - functionality. (Bartosz Ługowski via sseth) - - YARN-3123. Made YARN CLI show a single completed container even if the app - is running. (Naganarasimha G R via zjshen) - - YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per - queue (Thomas Graves via jlowe) - - YARN-1904. Ensure exceptions thrown in ClientRMService & - ApplicationHistoryClientService are uniform when application-attempt is - not found. (zjshen via acmurthy) - - YARN-3144. Configuration for making delegation token failures to timeline - server not-fatal (Jonathan Eagles via jlowe) - - YARN-3155. Refactor the exception handling code for TimelineClientImpl's - retryOn method (Li Lu via wangda) - - YARN-3100. Made YARN authorization pluggable. (Jian He via zjshen) - - YARN-2683. [YARN-913] registry config options: document and move to - core-default. (stevel) - - YARN-1237. Description for yarn.nodemanager.aux-services in - yarn-default.xml is misleading. (Brahma Reddy Battula via ozawa) - - YARN-3157. Refactor the exception handling in ConverterUtils#to*Id. - (Bibin A Chundatt via ozawa) - - YARN-3147. Clean up RM web proxy code. (Steve Loughran via xgong) - - YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager - restart. (Jason Lowe via junping_du) - - YARN-3158. Correct log messages in ResourceTrackerService. - (Varun Saxena via xgong) - - YARN-3179. Update use of Iterator to Iterable in RMAdminCLI and - CommonNodeLabelsManager. (Ray Chiang via xgong) - - YARN-3182. Cleanup switch statement in ApplicationMasterLauncher#handle(). - (Ray Chiang via ozawa) - - YARN-3203. Correct a log message in AuxServices. (Brahma Reddy Battula - via ozawa) - - YARN-1299. Improve a log message in AppSchedulingInfo by adding application - id. (Ashutosh Jindal and Devaraj K via ozawa) - - YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. - (Tsuyoshi OZAWA via jianhe) - - YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node - mapping. (Varun Saxena via wangda) - - YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. - (Zhihai Xu via junping_du) - - YARN-3237. AppLogAggregatorImpl fails to log error cause. - (Rushabh S Shah via xgong) - - YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY. - (zhihai xu via xgong) - - YARN-2797. TestWorkPreservingRMRestart should use ParametrizedSchedulerTestBase - (Karthik Kambatla via xgong) - - YARN-2797. Add -help to yarn logs and nodes CLI command. - (Jagadesh Kiran N via devaraj) - - YARN-3217. Remove httpclient dependency from hadoop-yarn-server-web-proxy. - (Brahma Reddy Battula via ozawa). - - YARN-3255. RM, NM, JobHistoryServer, and WebAppProxyServer's main() - should support generic options. (shv) - - YARN-2820. Retry in FileSystemRMStateStore when FS's operations fail - due to IOException. (Zhihai Xu via ozawa) - - YARN-3262. Surface application outstanding resource requests table - in RM web UI. (Jian He via wangda) - - YARN-3281. Added RMStateStore to StateMachine visualization list. - (Chengbing Liu via jianhe) - - YARN-3272. Surface container locality info in RM web UI. - (Jian He via wangda) - - YARN-3122. Metrics for container's actual CPU usage. - (Anubhav Dhoot via kasha) - - YARN-2190. Added CPU and memory limit options to the default container - executor for Windows containers. (Chuan Liu via jianhe) - - YARN-3296. Mark ResourceCalculatorProcessTree class as Public for configurable - resource monitoring. (Hitesh Shah via junping_du) - - YARN-3187. Documentation of Capacity Scheduler Queue mapping based on user - or group. (Gururaj Shetty via jianhe) - - YARN-2854. Updated the documentation of the timeline service and the generic - history service. (Naganarasimha G R via zjshen) - - YARN-2777. Mark the end of individual log in aggregated log. - (Varun Saxena via xgong) - - YARN-3273. Improve scheduler UI to facilitate scheduling analysis and - debugging. (Rohith Sharmaks via jianhe) - - OPTIMIZATIONS - - YARN-2990. FairScheduler's delay-scheduling always waits for node-local and - rack-local delays, even for off-rack-only requests. (kasha) - - BUG FIXES - - YARN-3071. Remove invalid char from sample conf in doc of FairScheduler. - (Masatake Iwasaki via aajisaka) - - YARN-2254. TestRMWebServicesAppsModification should run against both - CS and FS. (Zhihai Xu via kasha) - - YARN-2713. "RM Home" link in NM should point to one of the RMs in an - HA setup. (kasha) - - YARN-2857. ConcurrentModificationException in ContainerLogAppender - (Mohammad Kamrul Islam via jlowe) - - YARN-2432. RMStateStore should process the pending events before close. - (Varun Saxena via jianhe) - - YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. - (Rohith Sharma via vinodkv) - - YARN-2870. Updated the command to run the timeline server in the document. - (Masatake Iwasaki via zjshen) - - YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via - jianhe) - - YARN-2315. FairScheduler: Set current capacity in addition to capacity. - (Zhihai Xu via kasha) - - YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) - - YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in - YarnConfiguration. (rchiang via rkanter) - - YARN-2869. CapacityScheduler should trim sub queue names when parse - configuration. (Wangda Tan via jianhe) - - YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. - (Ray Chiang via kasha) - - YARN-2931. PublicLocalizer may fail until directory is initialized by - LocalizeRunner. (Anubhav Dhoot via kasha) - - YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager - intermittent failure. (Wangda Tan via jianhe) - - YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. - (Wangda Tan via jianhe) - - YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in - SchedulerApplicationAttempt ctor. (devaraj) - - YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) - - YARN-2356. yarn status command for non-existent application/application - attempt/container is too verbose. (Sunil G via devaraj) - - YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of - SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha) - - YARN-2945. FSLeafQueue#assignContainer - document the reason for using both write and - read locks. (Tsuyoshi Ozawa via kasha) - - YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. - (Chris Trezzo via kasha) - - YARN-2675. containersKilled metrics is not updated when the container is killed - during localization. (Zhihai Xu via kasha) - - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) - - YARN-2977. Fixed intermittent TestNMClient failure. - (Junping Du via ozawa) - - YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) - - YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu - via junping_du) - - YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. (Varun Saxena - via zjshen) - - YARN-2946. Fixed potential deadlock in RMStateStore. (Rohith Sharmaks via - jianhe) - - YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) - - YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and - hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) - - YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and - app ACLs. (Varun Saxena via jianhe) - - YARN-2991. Fixed DrainDispatcher to reuse the draining code path in - AsyncDispatcher. (Rohith Sharmaks via zjshen) - - YARN-2958. Made RMStateStore not update the last sequence number when updating the - delegation token. (Varun Saxena via zjshen) - - YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat - via jianhe) - - YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. (Yi Liu via - jianhe) - - YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on - getProto method. (Varun Saxena via jianhe) - - YARN-3014. Replaces labels on a host should update all NM's labels on that - host. (Wangda Tan via jianhe) - - YARN-3027. Scheduler should use totalAvailable resource from node instead of - availableResource for maxAllocation. (adhoot via rkanter) - - YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. - (Zhijie Shen via jianhe) - - YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync - failure with allocation timeout. (Jian He via junping_du) - - YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common. - (Ferdinand Xu via zjshen) - - YARN-3070. TestRMAdminCLI#testHelp fails for transitionToActive command. - (Contributed by Junping Du) - - YARN-3015. yarn classpath command should support same options as hadoop - classpath. (Contributed by Varun Saxena) - - YARN-2933. Capacity Scheduler preemption policy should only consider capacity - without labels temporarily. (Mayank Bansal via wangda) - - YARN-2731. Fixed RegisterApplicationMasterResponsePBImpl to properly invoke - maybeInitBuilder. (Carlo Curino via wangda) - - YARN-3078. LogCLIHelpers lacks of a blank space before string 'does not exist'. - (Sam Liu via ozawa) - - YARN-3082. Non thread safe access to systemCredentials in NodeHeartbeatResponse - processing. (Anubhav Dhoot via ozawa) - - YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native - executor returns an error (Eric Payne via jlowe) - - YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. - (Zhihai Xu via wangda) - - YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not - work everywhere. (Varun Saxena via ozawa) - - YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe) - - YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal) - - YARN-3056. Add verification for containerLaunchDuration - in TestNodeManagerMetrics. (zhihai xu via xgong) - - YARN-2543. Made resource usage be published to the timeline server too. - (Naganarasimha G R via zjshen) - - YARN-3058. Fix error message of tokens' activation delay configuration. - (Yi Liu via ozawa) - - YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding - max share (Anubhav Dhoot via Sandy Ryza) - - YARN-3149. Fix typo in message for invalid application id. - (Bibin A Chundatt via xgong) - - YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler - ParentQueue#getQueueUserAclInfo. (Tsuyoshi OZAWA via jianhe) - - YARN-1537. Fix race condition in - TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy) - - YARN-3089. LinuxContainerExecutor does not handle file arguments to - deleteAsUser (Eric Payne via jlowe) - - YARN-3143. RM Apps REST API can return NPE or entries missing id and other - fields (jlowe) - - YARN-2971. RM uses conf instead of token service address to renew timeline - delegation tokens (jeagles) - - YARN-3090. DeletionService can silently ignore deletion task failures - (Varun Saxena via jlowe) - - YARN-2809. Implement workaround for linux kernel panic when removing - cgroup (Nathan Roberts via jlowe) - - YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. - (Chengbing Liu via junping_du) - - YARN-3074. Nodemanager dies when localizer runner tries to write to a full - disk (Varun Saxena via jlowe) - - YARN-3151. On Failover tracking url wrong in application cli for - KILLED application (Rohith via xgong) - - YARN-1580. Documentation error regarding "container-allocation.expiry-interval-ms" - (Brahma Reddy Battula via junping_du) - - YARN-3104. Fixed RM to not generate new AMRM tokens on every heartbeat - between rolling and activation. (Jason Lowe via jianhe) - - YARN-3191. Log object should be initialized with its own class. (Rohith via - aajisaka) - - YARN-3164. RMAdmin command usage prints incorrect command name. - (Bibin A Chundatt via xgong) - - YARN-2847. Linux native container executor segfaults if default banned - user detected (Olaf Flebbe via jlowe) - - YARN-2899. Run TestDockerContainerExecutorWithMocks on Linux only. - (Ming Ma via cnauroth) - - YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. - (Xuan Gong via junping_du) - - YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping - when node becomes deactivated. (Wangda Tan via jianhe) - - YARN-1615. Fix typos in description about delay scheduling. (Akira Ajisaka via - ozawa) - - YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in - RMApp. (Rohith Sharmaks via jianhe) - - YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. - (Zhihai Xu via ozawa) - - YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against - all Schedulers even when using ParameterizedSchedulerTestBase. - (Anubhav Dhoot via devaraj) - - YARN-3270. Fix node label expression not getting set in - ApplicationSubmissionContext (Rohit Agarwal via wangda) - - YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's - available resource-limit from the parent queue. (Wangda Tan via vinodkv) - - YARN-3131. YarnClientImpl should check FAILED and KILLED state in - submitApplication (Chang Li via jlowe) - - YARN-3275. CapacityScheduler: Preemption happening on non-preemptable - queues (Eric Payne via jlowe) - - YARN-3300. Outstanding_resource_requests table should not be shown in AHS. - (Xuan Gong via jianhe) - - YARN-3295. Fix documentation nits found in markdown conversion. - (Masatake Iwasaki via ozawa) - - YARN-3338. Exclude jline dependency from YARN. (Zhijie Shen via xgong) - - YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating - running logs of application when rolling is enabled. (Xuan Gong via vinodkv) - - YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in - doc comments. (Akira AJISAKA, Andrew Purtell, and Allen Wittenauer via ozawa) - - YARN-3349. Treat all exceptions as failure in - TestFSRMStateStore#testFSRMStateStoreClientRetry. (Zhihai Xu via ozawa) - - YARN-3379. Fixed missing data in localityTable and ResourceRequests table - in RM WebUI. (Xuan Gong via jianhe) - - YARN-3384. TestLogAggregationService.verifyContainerLogs fails after - YARN-2777. (Naganarasimha G R via ozawa) - - YARN-3336. FileSystem memory leak in DelegationTokenRenewer. - (Zhihai Xu via cnauroth) - - YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. - (Varun Saxena via xgong) - - YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and - removing inconsistencies in the default values. (Junping Du and Karthik - Kambatla via vinodkv) - - YARN-3430. Made headroom data available on app attempt page of RM WebUI. - (Xuan Gong via zjshen) - - YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers - and node-label column (Jason Lowe via wangda) - -Release 2.6.5 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - YARN-4690. Skip object allocation in FSAppAttempt#getResourceUsage when - possible (Ming Ma via sjlee) - - BUG FIXES - - YARN-2046. Out of band heartbeats are sent only on container kill and - possibly too early (Ming Ma via jlowe) - - YARN-4722. AsyncDispatcher logs redundant event queue sizes (Jason Lowe via - sjlee) - -Release 2.6.4 - 2016-02-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) - - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. - (mujunchao via zxu) - - YARN-3535. Scheduler must re-request container resources when RMContainer transitions - from ALLOCATED to KILLED (rohithsharma and peng.zhang via asuresh) - - YARN-4452. NPE when submit Unmanaged application. (Naganarasimha G R - via junping_du) - - YARN-3893. Both RM in active state when Admin#transitionToActive failure - from refeshAll() (Bibin A Chundatt via rohithsharmaks) - - YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. - (Zhihai Xu via kasha) - - YARN-4546. ResourceManager crash due to scheduling opportunity overflow. - (Jason Lowe via junping_du) - - YARN-2902. Killing a container that is localizing can orphan resources in - the DOWNLOADING state (Varun Saxena via jlowe) - - YARN-4354. Public resource localization fails with NPE. (Jason Lowe via - junping_du) - - YARN-3849. Too much of preemption activity causing continuos killing of - containers across queues. (Sunil G via wangda) - - YARN-3842. NMProxy should retry on NMNotYetReadyException. - (Robert Kanter via kasha) - - YARN-3695. ServerProxy (NMProxy, etc.) shouldn't retry forever for non - network exception. (Raju Bairishetti via jianhe) - - YARN-4414. Nodemanager connection errors are retried at multiple levels - (Chang Li via jlowe) - - YARN-4581. AHS writer thread leak makes RM crash while RM is recovering. - (sandflee via junping_du) - - YARN-4598. Invalid event: RESOURCE_FAILED at - CONTAINER_CLEANEDUP_AFTER_KILL (tangshangwen via jlowe) - -Release 2.6.3 - 2015-12-17 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - YARN-4241. Fix typo of property name in yarn-default.xml. - (Anthony Rojas via aajisaka) - - YARN-2859. ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster. - (Vinod Kumar Vavilapalli via xgong) - - YARN-4344. NMs reconnecting with changed capabilities can lead to wrong - cluster resource calculations (Varun Vasudev via jlowe) - - YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container - log files from full disks. (zhihai xu via jlowe) - - YARN-4365. FileSystemNodeLabelStore should check for root dir existence on - startup (Kuhu Shukla via jlowe) - - YARN-4348. ZKRMStateStore.syncInternal shouldn't wait for sync completion for - avoiding blocking ZK's event thread. (ozawa) - - YARN-4424. Fix deadlock in RMAppImpl. (Jian he via wangda) - - YARN-4434. NodeManager Disk Checker parameter documentation is not correct. - (Weiwei Yang via aajisaka) - -Release 2.6.2 - 2015-10-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-3727. For better error recovery, check if the directory exists before - using it for localization. (Zhihai Xu via jlowe) - - YARN-4092. Fixed UI redirection to print useful messages when both RMs are - in standby mode. (Xuan Gong via jianhe) - - YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager - gets connection issue. (Xuan Gong via jianhe) - - OPTIMIZATIONS - - BUG FIXES - - YARN-2019. Retrospect on decision of making RM crashed if any exception throw - in ZKRMStateStore. (Jian He via junping_du) - - YARN-4087. Followup fixes after YARN-2019 regarding RM behavior when - state-store error occurs. (Jian He via xgong) - - YARN-3554. Default value for maximum nodemanager connect wait time is too - high (Naganarasimha G R via jlowe) - - YARN-4005. Completed container whose app is finished is possibly not - removed from NMStateStore. (Jun Gong via jianhe) - - YARN-3780. Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransition. - (zhihai xu via devaraj) - - YARN-3802. Two RMNodes for the same NodeId are used in RM sometimes - after NM is reconnected. (zhihai xu via xgong) - - YARN-3194. RM should handle NMContainerStatuses sent by NM while - registering if NM is Reconnected node (Rohith via jlowe) - - YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id - has not been reset synchronously. (Jun Gong via rohithsharmaks) - - YARN-3798. ZKRMStateStore shouldn't create new session without occurrance of - SESSIONEXPIED. (ozawa and Varun Saxena) - -Release 2.6.1 - 2015-09-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. - (Ryu Kobayashi via ozawa) - - IMPROVEMENTS - - YARN-3230. Clarify application states on the web UI. (Jian He via wangda) - - YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and - Xuan Gong via jianhe) - - YARN-3092. Created a common ResourceUsage class to track labeled resource - usages in Capacity Scheduler. (Wangda Tan via jianhe) - - YARN-3098. Created common QueueCapacities class in Capacity Scheduler to - track capacities-by-labels of queues. (Wangda Tan via jianhe) - - YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) - - YARN-3978. Configurably turn off the saving of container info in Generic AHS - (Eric Payne via jeagles) - - YARN-3248. Display count of nodes blacklisted by apps in the web UI. - (Varun Vasudev via xgong) - - OPTIMIZATIONS - - BUG FIXES - - YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state - on app recovery. (Rohith Sharmaks via jianhe) - - YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu - via jlowe) - - YARN-2414. RM web UI: app page will crash if app is failed before any - attempt has been created (Wangda Tan via jlowe) - - YARN-2865. Fixed RM to always create a new RMContext when transtions from - StandBy to Active. (Rohith Sharmaks via jianhe) - - YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. - (Jason Lowe via jianhe) - - YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated - log file is corrupted (Varun Saxena via jlowe) - - YARN-2890. MiniYARNCluster should start the timeline server based on the - configuration. (Mit Desai via zjshen) - - YARN-2894. Fixed a bug regarding application view acl when RM fails over. - (Rohith Sharmaks via jianhe) - - YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute - any further apps. (Naganarasimha G R via kasha) - - YARN-2910. FSLeafQueue can throw ConcurrentModificationException. - (Wilfred Spiegelenburg via kasha) - - YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher - (Rohith Sharmaks via jianhe) - - YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) - (Jian He via jlowe) - - YARN-1984. LeveldbTimelineStore does not handle db exceptions properly - (Varun Saxena via jlowe) - - YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks - via jianhe) - - YARN-2340. Fixed NPE when queue is stopped during RM restart. - (Rohith Sharmaks via jianhe) - - YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla - via jianhe) - - YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. - (Rohith Sharmaks via ozawa) - - YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed - container statuses on heartbeat. (Chengbing Liu via jianhe) - - YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService - might lead NM to crash. (Varun Saxena via jianhe) - - YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe - via jianhe) - - YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong - via jianhe) - - YARN-2246. Made the proxy tracking URL always be - http(s)://proxy addr:port/proxy/ to avoid duplicate sections. (Devaraj - K via zjshen) - - YARN-3207. Secondary filter matches entites which do not have the key being - filtered for. (Zhijie Shen via xgong) - - YARN-3238. Connection timeouts to nodemanagers are retried at - multiple levels (Jason Lowe via xgong) - - YARN-3239. WebAppProxy does not support a final tracking url which has - query fragments and params (Jian He via jlowe) - - YARN-3222. Fixed RMNode to send scheduler events in sequential order when a - node reconnects. (Rohith Sharma K S via jianhe) - - YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending - jobs. (Siqi Li via kasha) - - YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving - events for old client. (Zhihai Xu via kasha) - - YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired - (Zhijie Shen via xgong) - - YARN-3287. Made TimelineClient put methods do as the correct login context. - (Daryn Sharp and Jonathan Eagles via zjshen) - - YARN-3267. Timelineserver applies the ACL rules after applying the limit on - the number of records (Chang Li via jeagles) - - YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. - (Brahma Reddy Battula via wangda) - - YARN-3393. Getting application(s) goes wrong when app finishes before - starting the attempt. (Zhijie Shen via xgong) - - YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token - renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) - - YARN-3493. RM fails to come up with error "Failed to load/recover state" - when mem settings are changed. (Jian He via wangda) - - YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when - calling getQueue (Jason Lowe via wangda) - - YARN-3024. LocalizerRunner should give DIE action when all resources are - localized. (Chengbing Liu via xgong) - - YARN-3464. Race condition in LocalizerRunner kills localizer before - localizing all resources. (Zhihai Xu via kasha) - - YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when - exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) - - YARN-3526. ApplicationMaster tracking URL is incorrectly redirected - on a QJM cluster. (Weiwei Yang via xgong) - - YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, - attempts and containers. (Robert Kanter via zjshen) - - YARN-3700. Made generic history service load a number of latest applications - according to the parameter or the configuration. (Xuan Gong via zjshen) - - YARN-2900. Application (Attempt and Container) Not Found in AHS results - in InternalServer Error (500). (Zhijie Shen and Mit Desai via xgong) - - YARN-3725. App submission via REST API is broken in secure mode due to - Timeline DT service address is empty. (Zhijie Shen via wangda) - - YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM - recovery is enabled (Rohith Sharmaks via jlowe) - - YARN-3832. Resource Localization fails on a cluster due to existing cache - directories (Brahma Reddy Battula via jlowe) - - YARN-3850. NM fails to read files from full disks which can lead to - container logs being lost and other issues (Varun Saxena via jlowe) - - YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when - Node is connected/disconnected (Bibin A Chundatt via jlowe) - - YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler - when activating applications. (Craig Welch via jianhe) - - YARN-3733. Fix DominantRC#compare() does not work as expected if - cluster resource is empty. (Rohith Sharmaks via wangda) - - YARN-2920. Changed CapacityScheduler to kill containers on nodes where - node labels are changed. (Wangda Tan via jianhe) - - YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via - jianhe) - - YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage - to track used-resources-by-label.(Wangda Tan via jianhe) - - YARN-2694. Ensure only single node label specified in ResourceRequest. - (Wangda Tan via jianhe) - - YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track - capacities-by-label. (Wangda Tan via jianhe) - - YARN-2918. RM should not fail on startup if queue's configured labels do - not exist in cluster-node-labels. (Wangda Tan via jianhe) - - YARN-3999. RM hangs on draing events. (Jian He via xgong) - - YARN-4047. ClientRMService getApplications has high scheduler lock contention. - (Jason Lowe via jianhe) - - YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM - web page. (Xuan Gong via zjshen) - - YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work - in ATS / RM web ui. (Naganarasimha G R via xgong) - - YARN-3740. Fixed the typo in the configuration name: - APPLICATION_HISTORY_PREFIX_MAX_APPS. (Xuan Gong via zjshen) - - YARN-3544. Got back AM logs link on the RM web UI for a completed app. - (Xuan Gong via zjshen) - -Release 2.6.0 - 2014-11-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN. (Abin - Shahab via raviprak) - - YARN-2131. Add a way to format the RMStateStore. (Robert Kanter via kasha) - - YARN-1367. Changed NM to not kill containers on NM resync if RM work-preserving - restart is enabled. (Anubhav Dhoot via jianhe) - - YARN-1366. Changed AMRMClient to re-register with RM and send outstanding requests - back to RM on work-preserving RM restart. (Rohith via jianhe) - - YARN-2181. Added preemption info to logs and RM web UI. (Wangda Tan via - jianhe) - - YARN-1354. Recover applications upon nodemanager restart. (Jason Lowe via - junping_du) - - YARN-1337. Recover containers upon nodemanager restart. (Jason Lowe via - junping_du) - - YARN-2277. Added cross-origin support for the timeline server web services. - (Jonathan Eagles via zjshen) - - YARN-2378. Added support for moving applications across queues in - CapacityScheduler. (Subramaniam Venkatraman Krishnan via jianhe) - - YARN-2411. Support simple user and group mappings to queues. (Ram Venkatesh - via jianhe) - - YARN-2174. Enable HTTPs for the writer REST API of TimelineServer. - (Zhijie Shen via jianhe) - - YARN-2393. FairScheduler: Add the notion of steady fair share. - (Wei Yan via kasha) - - YARN-2395. FairScheduler: Preemption timeout should be configurable per - queue. (Wei Yan via kasha) - - YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. - (Wei Yan via kasha) - - YARN-415. Capture aggregate memory allocation at the app-level for chargeback. - (Eric Payne & Andrey Klochkov via jianhe) - - YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all - containers to a preconfigured limit. (Varun Vasudev via vinodkv) - - YARN-2033. Merging generic-history into the Timeline Store - (Zhijie Shen via junping_du) - - YARN-611. Added an API to let apps specify an interval beyond which AM - failures should be ignored towards counting max-attempts. (Xuan Gong via - vinodkv) - - YARN-2531. Added a configuration for admins to be able to override app-configs - and enforce/not-enforce strict control of per-container cpu usage. (Varun - Vasudev via vinodkv) - - YARN-1250. Generic history service should support application-acls. (Zhijie Shen - via junping_du) - - YARN-2569. Added the log handling APIs for the long running services. (Xuan - Gong via zjshen) - - YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs - on Timeline service event data. (Zhijie Shen via vinodkv) - - YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. - (Xuan Gong via zjshen) - - YARN-1063. Augmented Hadoop common winutils to have the ability to create - containers as domain users. (Remus Rusanu via vinodkv) - - YARN-1972. Added a secure container-executor for Windows. (Remus Rusanu via - vinodkv) - - YARN-2613. Support retry in NMClient for rolling-upgrades. (Jian He via - junping_du) - - YARN-2446. Augmented Timeline service APIs to start taking in domains as a - parameter while posting entities and events. (Zhijie Shen via vinodkv) - - YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for - use by long running services. (Xuan Gong via vinodkv) - - YARN-1051. Add a system for creating reservations of cluster capacity. - (see breakdown below) - - YARN-913. Add a way to register long-lived services in a YARN cluster. - (stevel) - - YARN-2493. Added user-APIs for using node-labels. (Wangda Tan via vinodkv) - - YARN-2544. Added admin-API objects for using node-labels. (Wangda Tan via - vinodkv) - - YARN-2494. Added NodeLabels Manager internal API and implementation. (Wangda - Tan via vinodkv) - - YARN-2501. Enhanced AMRMClient library to support requests against node - labels. (Wangda Tan via vinodkv) - - YARN-2656. Made RM web services authentication filter support proxy user. - (Varun Vasudev and Zhijie Shen via zjshen) - - YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating - resources based on node-labels. (Wangda Tan via vinodkv) - - YARN-2500. Enhaced ResourceManager to support schedulers allocating resources - based on node-labels. (Wangda Tan via vinodkv) - - YARN-2504. Enhanced RM Admin CLI to support management of node-labels. - (Wangda Tan via vinodkv) - - YARN-2198. Remove the need to run NodeManager as privileged account for - Windows Secure Container Executor. (Remus Rusanu via jianhe) - - YARN-2647. Added a queue CLI for getting queue information. (Sunil Govind via - vinodkv) - - YARN-2632. Document NM Restart feature. (Junping Du and Vinod Kumar - Vavilapalli via jlowe) - - YARN-2505. Supported get/add/remove/change labels in RM REST API. (Craig Welch - via zjshen) - - YARN-2811. In Fair Scheduler, reservation fulfillments shouldn't ignore max - share (Siqi Li via Sandy Ryza) - - YARN-3445. Cache runningApps in RMNode for getting running apps on given - NodeId. (Junping Du via mingma) - - IMPROVEMENTS - - YARN-2197. Add a link to YARN CHANGES.txt in the left side of doc - (Akira AJISAKA via aw) - - YARN-1918. Typo in description and error message for - 'yarn.resourcemanager.cluster-id' (Anandha L Ranganathan via aw) - - YARN-2242. Improve exception information on AM launch crashes. (Li Lu - via junping_du) - - YARN-2274. FairScheduler: Add debug information about cluster capacity, - availability and reservations. (kasha) - - YARN-2228. Augmented TimelineServer to load pseudo authentication filter when - authentication = simple. (Zhijie Shen via vinodkv) - - YARN-1341. Recover NMTokens upon nodemanager restart. (Jason Lowe via - junping_du) - - YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. (xgong) - - YARN-2323. FairShareComparator creates too many Resource objects (Hong Zhiguo - via Sandy Ryza) - - YARN-2045. Data persisted in NM should be versioned (Junping Du via jlowe) - - YARN-2013. The diagnostics is always the ExitCodeException stack when the container - crashes. (Tsuyoshi OZAWA via junping_du) - - YARN-2295. Refactored DistributedShell to use public APIs of protocol records. - (Li Lu via jianhe) - - YARN-1342. Recover container tokens upon nodemanager restart. (Jason Lowe via - devaraj) - - YARN-2214. FairScheduler: preemptContainerPreCheck() in FSParentQueue delays - convergence towards fairness. (Ashwin Shankar via kasha) - - YARN-2211. Persist AMRMToken master key in RMStateStore for RM recovery. - (Xuan Gong via jianhe) - - YARN-2328. FairScheduler: Verify update and continuous scheduling threads are - stopped when the scheduler is stopped. (kasha) - - YARN-2347. Consolidated RMStateVersion and NMDBSchemaVersion into Version in - yarn-server-common. (Junping Du via zjshen) - - YARN-1994. Expose YARN/MR endpoints on multiple interfaces. (Craig Welch, - Milan Potocnik, Arpit Agarwal via xgong) - - YARN-2343. Improve NMToken expire exception message. (Li Lu via jianhe) - - YARN-2370. Fix comment in o.a.h.y.server.resourcemanager.schedulerAppSchedulingInfo - (Wenwu Peng via junping_du) - - YARN-2298. Move TimelineClient to yarn-common project (Zhijie Shen via - junping_du) - - YARN-2288. Made persisted data in LevelDB timeline store be versioned. (Junping Du - via zjshen) - - YARN-2352. FairScheduler: Collect metrics on duration of critical methods that - affect performance. (kasha) - - YARN-2212. ApplicationMaster needs to find a way to update the AMRMToken - periodically. (xgong) - - YARN-2026. Fair scheduler: Consider only active queues for computing fairshare. - (Ashwin Shankar via kasha) - - YARN-1954. Added waitFor to AMRMClient(Async). (Tsuyoshi Ozawa via zjshen) - - YARN-2302. Refactor TimelineWebServices. (Zhijie Shen via junping_du) - - YARN-2337. ResourceManager sets ClientRMService in RMContext multiple times. - (Zhihai Xu via kasha) - - YARN-2138. Cleaned up notifyDone* APIs in RMStateStore. (Varun Saxena via - jianhe) - - YARN-2373. Changed WebAppUtils to use Configuration#getPassword for - accessing SSL passwords. (Larry McCay via jianhe) - - YARN-2317. Updated the document about how to write YARN applications. (Li Lu via - zjshen) - - YARN-2399. FairScheduler: Merge AppSchedulable and FSSchedulerApp into - FSAppAttempt. (kasha) - - YARN-1370. Fair scheduler to re-populate container allocation state. - (Anubhav Dhoot via kasha) - - YARN-2389. Added functionality for schedulers to kill all applications in a - queue. (Subramaniam Venkatraman Krishnan via jianhe) - - YARN-1326. RM should log using RMStore at startup time. - (Tsuyoshi Ozawa via kasha) - - YARN-2182. Updated ContainerId#toString() to append RM Epoch number. - (Tsuyoshi OZAWA via jianhe) - - YARN-2406. Move RM recovery related proto to - yarn_server_resourcemanager_recovery.proto. (Tsuyoshi Ozawa via jianhe) - - YARN-1506. Changed RMNode/SchedulerNode to update resource with event - notification. (Junping Du via jianhe) - - YARN-2509. Enable Cross Origin Filter for timeline server only and not all - Yarn servers (Mit Desai via jeagles) - - YARN-2511. Allowed all origins by default when CrossOriginFilter is - enabled. (Jonathan Eagles via zjshen) - - YARN-2508. Cross Origin configuration parameters prefix are not honored - (Mit Desai via jeagles) - - YARN-2512. Allowed pattern matching for origins in CrossOriginFilter. - (Jonathan Eagles via zjshen) - - YARN-2507. Documented CrossOriginFilter configurations for the timeline - server. (Jonathan Eagles via zjshen) - - YARN-2515. Updated ConverterUtils#toContainerId to parse epoch. - (Tsuyoshi OZAWA via jianhe) - - YARN-2448. Changed ApplicationMasterProtocol to expose RM-recognized resource - types to the AMs. (Varun Vasudev via vinodkv) - - YARN-2538. Added logs when RM sends roll-overed AMRMToken to AM. (Xuan Gong - via zjshen) - - YARN-2229. Changed the integer field of ContainerId to be long type. - (Tsuyoshi OZAWA via jianhe) - - YARN-2547. Cross Origin Filter throws UnsupportedOperationException upon - destroy (Mit Desai via jeagles) - - YARN-2557. Add a parameter "attempt_Failures_Validity_Interval" into - DistributedShell (xgong) - - YARN-2001. Added a time threshold for RM to wait before starting container - allocations after restart/failover. (Jian He via vinodkv) - - YARN-1372. Ensure all completed containers are reported to the AMs across - RM restart. (Anubhav Dhoot via jianhe) - - YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. - (Wei Yan via kasha) - - YARN-1959. Fix headroom calculation in FairScheduler. - (Anubhav Dhoot via kasha) - - YARN-2577. Clarify ACL delimiter and how to configure ACL groups only - (Miklos Christine via aw) - - YARN-2372. There are Chinese Characters in the FairScheduler's document - (Fengdong Yu via aw) - - YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier - to use protobuf object as the payload. (Junping Du via jianhe) - - YARN-1769. CapacityScheduler: Improve reservations (Thomas Graves via - jlowe) - - YARN-2627. Added the info logs of attemptFailuresValidityInterval and number - of previous failed attempts. (Xuan Gong via zjshen) - - YARN-2562. Changed ContainerId#toString() to be more readable. (Tsuyoshi - OZAWA via jianhe) - - YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier - to use protobuf as payload. (Junping Du via jianhe) - - YARN-2629. Made the distributed shell use the domain-based timeline ACLs. - (zjshen) - - YARN-2583. Modified AggregatedLogDeletionService to be able to delete rolling - aggregated logs. (Xuan Gong via zjshen) - - YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to - use ContainerId#getContainerId instead. (Tsuyoshi OZAWA via jianhe) - - YARN-2621. Simplify the output when the user doesn't have the access for - getDomain(s). (Zhijie Shen via jianhe) - - YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol - for RM fail over. (Tsuyoshi OZAWA via jianhe) - - YARN-2676. Enhanced Timeline auth-filter to support proxy users. (Zhijie Shen - via vinodkv) - - YARN-2673. Made timeline client put APIs retry if ConnectException happens. - (Li Lu via zjshen) - - YARN-2582. Fixed Log CLI and Web UI for showing aggregated logs of LRS. (Xuan - Gong via zjshen) - - YARN-90. NodeManager should identify failed disks becoming good again - (Varun Vasudev via jlowe) - - YARN-2709. Made timeline client getDelegationToken API retry if ConnectException - happens. (Li Lu via zjshen) - - YARN-2682. Updated WindowsSecureContainerExecutor to not use - DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() - instead. (Zhihai Xu via jianhe) - - YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and - made related MR changes. (Jian He via zjshen) - - YARN-2703. Added logUploadedTime into LogValue for better display. (Xuan Gong - via zjshen) - - YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the - sake of localization and log-aggregation for long-running services. (Jian He - via vinodkv) - - YARN-2502. Changed DistributedShell to support node labels. (Wangda Tan via - jianhe) - - YARN-2760. Remove 'experimental' from FairScheduler docs. (Harsh J via kasha) - - YARN-2503. Added node lablels in web UI. (Wangda Tan via jianhe) - - YARN-2779. Fixed ResourceManager to not require delegation tokens for - communicating with Timeline Service. (Zhijie Shen via vinodkv) - - YARN-2778. Moved node-lables' reports to the yarn nodes CLI from the admin - CLI. (Wangda Tan via vinodkv) - - YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. - (Zhijie Shen via jianhe) - - YARN-2818. Removed the now unnecessary user entity injection from Timeline - service given we now have domains. (Zhijie Shen via vinodkv) - - YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run - with both CS and FS. (Wei Yan and kasha via kasha) - - OPTIMIZATIONS - - BUG FIXES - - YARN-2251. Avoid negative elapsed time in JHS/MRAM web UI and services. - (Zhijie Shen via junping_du) - - YARN-2088. Fixed a bug in GetApplicationsRequestPBImpl#mergeLocalToBuilder. - (Binglin Chang via jianhe) - - YARN-2260. Fixed ResourceManager's RMNode to correctly remember containers - when nodes resync during work-preserving RM restart. (Jian He via vinodkv) - - YARN-2264. Fixed a race condition in DrainDispatcher which may cause random - test failures. (Li Lu via jianhe) - - YARN-2219. Changed ResourceManager to avoid AMs and NMs getting exceptions - after RM recovery but before scheduler learns about apps and app-attempts. - (Jian He via vinodkv) - - YARN-2244. FairScheduler missing handling of containers for unknown - application attempts. (Anubhav Dhoot via kasha) - - YARN-2321. NodeManager web UI can incorrectly report Pmem enforcement - (Leitao Guo via jlowe) - - YARN-2273. NPE in ContinuousScheduling thread when we lose a node. - (Wei Yan via kasha) - - YARN-2313. Livelock can occur in FairScheduler when there are lots of - running apps (Tsuyoshi Ozawa via Sandy Ryza) - - YARN-2147. client lacks delegation token exception details when - application submit fails (Chen He via jlowe) - - YARN-1796. container-executor shouldn't require o-r permissions (atm) - - YARN-2354. DistributedShell may allocate more containers than client - specified after AM restarts. (Li Lu via jianhe) - - YARN-2051. Fix bug in PBimpls and add more unit tests with reflection. - (Binglin Chang via junping_du) - - YARN-2374. Fixed TestDistributedShell#testDSShell failure due to hostname - dismatch. (Varun Vasudev via jianhe) - - YARN-2359. Application hangs when it fails to launch AM container. - (Zhihai Xu via kasha) - - YARN-2388. Fixed TestTimelineWebServices failure due to HADOOP-10791. (zjshen) - - YARN-2008. Fixed CapacityScheduler to calculate headroom based on max available - capacity instead of configured max capacity. (Craig Welch via jianhe) - - YARN-2400. Fixed TestAMRestart fails intermittently. (Jian He via xgong) - - YARN-2361. RMAppAttempt state machine entries for KILLED state has duplicate - event entries. (Zhihai Xu via kasha) - - YARN-2070. Made DistributedShell publish the short user name to the timeline - server. (Robert Kanter via zjshen) - - YARN-2397. Avoided loading two authentication filters for RM and TS web - interfaces. (Varun Vasudev via zjshen) - - YARN-2409. RM ActiveToStandBy transition missing stoping previous rmDispatcher. - (Rohith via jianhe) - - YARN-2249. Avoided AM release requests being lost on work preserving RM - restart. (Jian He via zjshen) - - YARN-2034. Description for yarn.nodemanager.localizer.cache.target-size-mb - is incorrect (Chen He via jlowe) - - YARN-1919. Potential NPE in EmbeddedElectorService#stop. - (Tsuyoshi Ozawa via kasha) - - YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas - via aw) - - YARN-2434. RM should not recover containers from previously failed attempt - when AM restart is not enabled (Jian He via jlowe) - - YARN-2035. FileSystemApplicationHistoryStore should not make working dir - when it already exists. (Jonathan Eagles via zjshen) - - YARN-2405. NPE in FairSchedulerAppsBlock. (Tsuyoshi Ozawa via kasha) - - YARN-2449. Fixed the bug that TimelineAuthenticationFilterInitializer - is not automatically added when hadoop.http.filter.initializers is not - configured. (Varun Vasudev via zjshen) - - YARN-2450. Fix typos in log messages. (Ray Chiang via hitesh) - - YARN-2447. RM web service app submission doesn't pass secrets correctly. - (Varun Vasudev via jianhe) - - YARN-2462. TestNodeManagerResync#testBlockNewContainerRequestsOnStartAndResync - should have a test timeout (Eric Payne via jlowe) - - YARN-2431. NM restart: cgroup is not removed for reacquired containers - (jlowe) - - YARN-2519. Credential Provider related unit tests failed on Windows. - (Xiaoyu Yao via cnauroth) - - YARN-2526. SLS can deadlock when all the threads are taken by AMSimulators. - (Wei Yan via kasha) - - YARN-1458. FairScheduler: Zero weight can lead to livelock. - (Zhihai Xu via kasha) - - YARN-2459. RM crashes if App gets rejected for any reason - and HA is enabled. (Jian He and Mayank Bansal via xgong) - - YARN-2158. Fixed TestRMWebServicesAppsModification#testSingleAppKill test - failure. (Varun Vasudev via jianhe) - - YARN-2534. FairScheduler: Potential integer overflow calculating totalMaxShare. - (Zhihai Xu via kasha) - - YARN-2541. Fixed ResourceManagerRest.apt.vm table syntax error. (jianhe) - - YARN-2484. FileSystemRMStateStore#readFile/writeFile should close - FSData(In|Out)putStream in final block (Tsuyoshi OZAWA via jlowe) - - YARN-2456. Possible livelock in CapacityScheduler when RM is recovering apps. - (Jian He via xgong) - - YARN-2542. Fixed NPE when retrieving ApplicationReport from TimeLineServer. - (Zhijie Shen via jianhe) - - YARN-2528. Relaxed http response split vulnerability protection for the origins - header and made it accept multiple origins in CrossOriginFilter. (Jonathan - Eagles via zjshen) - - YARN-2549. TestContainerLaunch fails due to classpath problem with hamcrest - classes. (cnauroth) - - YARN-2529. Generic history service RPC interface doesn't work when service - authorization is enabled. (Zhijie Shen via jianhe) - - YARN-2558. Updated ContainerTokenIdentifier#read/write to use - ContainerId#getContainerId. (Tsuyoshi OZAWA via jianhe) - - YARN-2559. Fixed NPE in SystemMetricsPublisher when retrieving - FinalApplicationStatus. (Zhijie Shen via jianhe) - - YARN-1779. Fixed AMRMClient to handle AMRMTokens correctly across - ResourceManager work-preserving-restart or failover. (Jian He via vinodkv) - - YARN-2363. Submitted applications occasionally lack a tracking URL (jlowe) - - YARN-2561. MR job client cannot reconnect to AM after NM restart. (Junping - Du via jlowe) - - YARN-2563. Fixed YarnClient to call getTimeLineDelegationToken only if the - Token is not present. (Zhijie Shen via jianhe) - - YARN-2568. Fixed the potential test failures due to race conditions when RM - work-preserving recovery is enabled. (Jian He via zjshen) - - YARN-2565. Fixed RM to not use FileSystemApplicationHistoryStore unless - explicitly set. (Zhijie Shen via jianhe) - - YARN-2460. Remove obsolete entries from yarn-default.xml (Ray Chiang via - aw) - - YARN-2452. TestRMApplicationHistoryWriter fails with FairScheduler. - (Zhihai Xu via kasha) - - YARN-2453. TestProportionalCapacityPreemptionPolicy fails with - FairScheduler. (Zhihai Xu via kasha) - - YARN-2540. FairScheduler: Queue filters not working on scheduler page in - RM UI. (Ashwin Shankar via kasha) - - YARN-2584. TestContainerManagerSecurity fails on trunk. (Jian He via - junping_du) - - YARN-2252. Intermittent failure of - TestFairScheduler.testContinuousScheduling. - (Ratandeep Ratti and kasha via kasha) - - YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) - - YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) - - YARN-2546. Made REST API for application creation/submission use numeric and - boolean types instead of the string of them. (Varun Vasudev via zjshen) - - YARN-2523. ResourceManager UI showing negative value for "Decommissioned - Nodes" field (Rohith via jlowe) - - YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and - clock access. (Wei Yan via kasha) - - YARN-2606. Application History Server tries to access hdfs before doing - secure login (Mit Desai via jeagles) - - YARN-2610. Hamlet should close table tags. (Ray Chiang via kasha) - - YARN-2387. Resource Manager crashes with NPE due to lack of - synchronization (Mit Desai via jlowe) - - YARN-2594. Potential deadlock in RM when querying - ApplicationResourceUsageReport. (Wangda Tan via kasha) - - YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. - (Zhijie Shen via jianhe) - - YARN-2630. Prevented previous AM container status from being acquired by the - current restarted AM. (Jian He via zjshen) - - YARN-2617. Fixed NM to not send duplicate container status whose app is not - running. (Jun Gong via jianhe) - - YARN-2624. Resource Localization fails on a cluster due to existing cache - directories (Anubhav Dhoot via jlowe) - - YARN-2527. Fixed the potential NPE in ApplicationACLsManager and added test - cases for it. (Benoy Antony via zjshen) - - YARN-2628. Capacity scheduler with DominantResourceCalculator carries out - reservation even though slots are free. (Varun Vasudev via jianhe) - - YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource - tracking per label when a host runs multiple node-managers. (Wangda Tan via - vinodkv) - - YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail - when using ephemeral ports on NodeIDs. (Wangda Tan via vinodkv) - - YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing - test-failures: added a dummy in-memory labels-manager. (Wangda Tan via - vinodkv) - - YARN-2715. Fixed ResourceManager to respect common configurations for proxy - users/groups beyond just the YARN level config. (Zhijie Shen via vinodkv) - - YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken - identifiers to be tampered and thus causing app submission failures in - secure mode. (Jian He via vinodkv) - - BREAKDOWN OF YARN-1051 SUBTASKS AND RELATED JIRAS - - YARN-1707. Introduce APIs to add/remove/resize queues in the - CapacityScheduler. (Carlo Curino and Subru Krishnan via curino) - - YARN-2475. Logic for responding to capacity drops for the - ReservationSystem. (Carlo Curino and Subru Krishnan via curino) - - YARN-1708. Public YARN APIs for creating/updating/deleting - reservations. (Subru Krishnan and Carlo Curino via subru) - - YARN-1709. In-memory data structures used to track resources over - time to enable reservations. (Subru Krishnan and Carlo Curino via - subru) - - YARN-1710. Logic to find allocations within a Plan that satisfy - user ReservationRequest(s). (Carlo Curino and Subru Krishnan via - curino) - - YARN-1711. Policy to enforce instantaneous and over-time quotas - on user reservations. (Carlo Curino and Subru Krishnan via curino) - - YARN-1712. Plan follower that synchronizes the current state of reservation - subsystem with the scheduler. (Subru Krishnan and Carlo Curino via subru) - - YARN-2080. Integrating reservation system with ResourceManager and - client-RM protocol. (Subru Krishnan and Carlo Curino via subru) - - MAPREDUCE-6103. Adding reservation APIs to MR resource manager - delegate. (Subru Krishnan and Carlo Curino via subru) - - YARN-2576. Fixing compilation, javadocs and audit issues to pass - test patch in branch. (Subru Krishnan and Carlo Curino via subru) - - YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched - for branch YARN-1051. (Subru Krishnan and Carlo Curino via subru) - - YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when - AM allocates. (Craig Welch via jianhe) - - YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. - (Chen He and Craig Welch via jianhe) - - YARN-2649. Fixed TestAMRMRPCNodeUpdates test failure. (Ming Ma via jianhe) - - YARN-2662. TestCgroupsLCEResourcesHandler leaks file descriptors. (cnauroth) - - BREAKDOWN OF YARN-913 SUBTASKS AND RELATED JIRAS - - YARN-2652 Add hadoop-yarn-registry package under hadoop-yarn. (stevel) - - YARN-2668 yarn-registry JAR won't link against ZK 3.4.5. (stevel) - - YARN-2689 TestSecureRMRegistryOperations failing on windows: - secure ZK won't start (stevel) - - YARN-2692 ktutil test hanging on some machines/ktutil versions (stevel) - - YARN-2700 TestSecureRMRegistryOperations failing on windows: auth problems - (stevel) - - YARN-2677 registry punycoding of usernames doesn't fix all usernames to be - DNS-valid (stevel) - - YARN-2768 Improved Yarn Registry service record structure (stevel) - - --- - - YARN-2598 GHS should show N/A instead of null for the inaccessible information - (Zhijie Shen via mayank) - - YARN-2671. Fixed ApplicationSubmissionContext to still set resource for - backward compatibility. (Wangda Tan via zjshen) - - YARN-2667. Fix the release audit warning caused by hadoop-yarn-registry - (Yi Liu via jlowe) - - YARN-2651. Spun off LogRollingInterval from LogAggregationContext. (Xuan Gong - via zjshen) - - YARN-2377. Localization exception stack traces are not passed as - diagnostic info (Gera Shegalov via jlowe) - - YARN-2308. Changed CapacityScheduler to explicitly throw exception if the - queue to which the apps were submitted is changed across RM restart. - (Craig Welch & Chang Li via jianhe) - - YARN-2566. DefaultContainerExecutor should pick a working directory randomly. - (Zhihai Xu via kasha) - - YARN-2588. Standby RM fails to transitionToActive if previous - transitionToActive failed with ZK exception. (Rohith Sharmaks via jianhe) - - YARN-2701. Potential race condition in startLocalizer when using - LinuxContainerExecutor. (Xuan Gong via jianhe) - - YARN-2717. Avoided duplicate logging when container logs are not found. (Xuan - Gong via zjshen) - - YARN-2720. Windows: Wildcard classpath variables not expanded against - resources contained in archives. (Craig Welch via cnauroth) - - YARN-2721. Suppress NodeExist exception thrown by ZKRMStateStore when it - retries creating znode. (Jian He via zjshen) - - YARN-2732. Fixed syntax error in SecureContainer.apt.vm. (Jian He via zjshen) - - YARN-2724. Skipped uploading a local log file to HDFS if exception is raised - when opening it. (Xuan Gong via zjshen) - - YARN-1915. Fixed a race condition that client could use the ClientToAMToken - to contact with AM before AM actually receives the ClientToAMTokenMasterKey. - (Jason Lowe via jianhe) - - YARN-2314. Disable ContainerManagementProtocolProxy cache by default to - prevent creating thousands of threads in a large cluster. (Jason Lowe via - jianhe) - - YARN-2723. Fix rmadmin -replaceLabelsOnNode does not correctly parse port. - (Naganarasimha G R via xgong) - - YARN-2734. Skipped sub-folders in the local log dir when aggregating logs. - (Xuan Gong via zjshen) - - YARN-2726. CapacityScheduler should explicitly log when an accessible - label has no capacity. (Wangda Tan via xgong) - - YARN-2591. Fixed AHSWebServices to return FORBIDDEN(403) if the request user - doesn't have access to the history data. (Zhijie Shen via jianhe) - - YARN-2279. Add UTs to cover timeline server authentication. - (Zhijie Shen via xgong) - - YARN-2758. Update TestApplicationHistoryClientService to use the new generic - history store. (Zhijie Shen via xgong) - - YARN-2741. Made NM web UI serve logs on the drive other than C: on Windows. (Craig - Welch via zjshen) - - YARN-2747. Fixed the test failure of TestAggregatedLogFormat when native I/O is - enabled. (Xuan Gong via zjshen) - - YARN-2769. Fixed the problem that timeline domain is not set in distributed shell - AM when using shell_command on Windows. (Varun Vasudev via zjshen) - - YARN-2755. NM fails to clean up usercache_DEL_ dirs after - YARN-661 (Siqi Li via jlowe) - - YARN-2698. Moved some node label APIs to be correctly placed in client - protocol. (Wangda Tan via vinodkv) - - YARN-2789. Re-instated the NodeReport.newInstance private unstable API - modified in YARN-2698 so that tests in YARN frameworks don't break. (Wangda - Tan via vinodkv) - - YARN-2707. Potential null dereference in FSDownload (Gera Shegalov via - jlowe) - - YARN-2711. Fixed TestDefaultContainerExecutor#testContainerLaunchError failure on - Windows. (Varun Vasudev via zjshen) - - YARN-2790. Fixed a NodeManager bug that was causing log-aggregation to fail - beyond HFDS delegation-token expiry even when RM is a proxy-user (YARN-2704). - (Jian He via vinodkv) - - YARN-2785. Fixed intermittent TestContainerResourceUsage failure. (Varun Vasudev - via zjshen) - - YARN-2730. DefaultContainerExecutor runs only one localizer at a time - (Siqi Li via jlowe) - - YARN-2798. Fixed YarnClient to populate the renewer correctly for Timeline - delegation tokens. (Zhijie Shen via vinodkv) - - YARN-2788. Fixed backwards compatiblity issues with log-aggregation feature - that were caused when adding log-upload-time via YARN-2703. (Xuan Gong via - vinodkv) - - YARN-2795. Fixed ResourceManager to not crash loading node-label data from - HDFS in secure mode. (Wangda Tan via vinodkv) - - YARN-1922. Fixed NodeManager to kill process-trees correctly in the presence - of races between the launch and the stop-container call and when root - processes crash. (Billie Rinaldi via vinodkv) - - YARN-2010. Handle app-recovery failures gracefully. - (Jian He and Karthik Kambatla via kasha) - - YARN-2804. Fixed Timeline service to not fill the logs with JAXB bindings - exceptions. (Zhijie Shen via vinodkv) - - YARN-2767. Added a test case to verify that http static user cannot kill or submit - apps in the secure mode. (Varun Vasudev via zjshen) - - YARN-2805. Fixed ResourceManager to load HA configs correctly before kerberos - login. (Wangda Tan via vinodkv) - - YARN-2579. Fixed a deadlock issue when EmbeddedElectorService and - FatalEventDispatcher try to transition RM to StandBy at the same time. - (Rohith Sharmaks via jianhe) - - YARN-2813. Fixed NPE from MemoryTimelineStore.getDomains. (Zhijie Shen via xgong) - - YARN-2812. TestApplicationHistoryServer is likely to fail on less powerful machine. - (Zhijie Shen via xgong) - - YARN-2744. Fixed CapacityScheduler to validate node-labels correctly against - queues. (Wangda Tan via vinodkv) - - YARN-2823. Fixed ResourceManager app-attempt state machine to inform - schedulers about previous finished attempts of a running appliation to avoid - expectation mismatch w.r.t transferred containers. (Jian He via vinodkv) - - YARN-2810. TestRMProxyUsersConf fails on Windows VMs. (Varun Vasudev via xgong) - - YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are - not mapped to queues by making default capacities per label to be zero. - (Wangda Tan via vinodkv) - - YARN-2827. Fixed bugs in "yarn queue" CLI. (Wangda Tan via vinodkv) - - YARN-2803. MR distributed cache not working correctly on Windows after - NodeManager privileged account changes. (Craig Welch via cnauroth) - - YARN-2753. Fixed a bunch of bugs in the NodeLabelsManager classes. (Zhihai xu - via vinodkv) - - YARN-2825. Container leak on NM (Jian He via jlowe) - - YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. - (Zhijie Shen via xgong) - - YARN-2826. Fixed user-groups mappings' refresh bug caused by YARN-2826. - (Wangda Tan via vinodkv) - - YARN-2607. Fixed issues in TestDistributedShell. (Wangda Tan via vinodkv) - - YARN-2830. Add backwords compatible ContainerId.newInstance constructor. - (jeagles via acmurthy) - - YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery - consistent with the (somewhat incorrect) behaviour in the non-recovery case. - (Jian He via vinodkv) - - YARN-2841. RMProxy should retry EOFException. (Jian He via xgong) - - YARN-2843. Fixed NodeLabelsManager to trim inputs for hosts and labels so - as to make them work correctly. (Wangda Tan via vinodkv) - - YARN-2794. Fixed log messages about distributing system-credentials. (Jian He via - zjshen) - - YARN-2846. Incorrect persist exit code for running containers in - reacquireContainer() that interrupted by NodeManager restart. (Junping Du - via jlowe) - - YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user - kill request races with ApplicationMaster finish. (Jian He via vinodkv) - -Release 2.5.2 - 2014-11-19 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - -Release 2.5.1 - 2014-09-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.5.0 - 2014-08-11 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-1757. NM Recovery. Auxiliary service support. (Jason Lowe via kasha) - - YARN-1864. Fair Scheduler Dynamic Hierarchical User Queues (Ashwin Shankar - via Sandy Ryza) - - YARN-1362. Distinguish between nodemanager shutdown for decommission vs shutdown - for restart. (Jason Lowe via junping_du) - - YARN-1338. Recover localized resource cache state upon nodemanager restart - (Jason Lowe via junping_du) - - YARN-1368. Added core functionality of recovering container state into - schedulers after ResourceManager Restart so as to preserve running work in - the cluster. (Jian He via vinodkv) - - YARN-1702. Added kill app functionality to RM web services. (Varun Vasudev - via vinodkv) - - YARN-1339. Recover DeletionService state upon nodemanager restart. (Jason Lowe - via junping_du) - - YARN-1365. Changed ApplicationMasterService to allow an app to re-register - after RM restart. (Anubhav Dhoot via jianhe) - - YARN-2052. Embedded an epoch number in container id to ensure the uniqueness - of container id after RM restarts. (Tsuyoshi OZAWA via jianhe) - - YARN-1713. Added get-new-app and submit-app functionality to RM web services. - (Varun Vasudev via vinodkv) - - YARN-2233. Implemented ResourceManager web-services to create, renew and - cancel delegation tokens. (Varun Vasudev via vinodkv) - - YARN-2247. Made RM web services authenticate users via kerberos and delegation - token. (Varun Vasudev via zjshen) - - IMPROVEMENTS - - YARN-1479. Invalid NaN values in Hadoop REST API JSON response (Chen He via - jeagles) - - YARN-1736. FS: AppSchedulable.assignContainer's priority argument is - redundant. (Naren Koneru via kasha) - - YARN-1678. Fair scheduler gabs incessantly about reservations (Sandy Ryza) - - YARN-1561. Fix a generic type warning in FairScheduler. (Chen He via junping_du) - - YARN-1429. *nix: Allow a way for users to augment classpath of YARN daemons. - (Jarek Jarcec Cecho via kasha) - - YARN-1520. update capacity scheduler docs to include necessary parameters - (Chen He via jeagles) - - YARN-1845. Elapsed time for failed tasks that never started is wrong - (Rushabh S Shah via jeagles) - - YARN-1136. Replace junit.framework.Assert with org.junit.Assert (Chen He - via jeagles) - - YARN-1889. In Fair Scheduler, avoid creating objects on each call to - AppSchedulable comparator (Hong Zhiguo via Sandy Ryza) - - YARN-1923. Make Fair Scheduler resource ratio calculations terminate faster - (Anubhav Dhoot via Sandy Ryza) - - YARN-1870. FileInputStream is not closed in ProcfsBasedProcessTree#constructProcessSMAPInfo. - (Fengdong Yu via junping_du) - - YARN-1970. Prepare YARN codebase for JUnit 4.11. (cnauroth) - - YARN-483. Improve documentation on log aggregation in yarn-default.xml - (Akira Ajisaka via Sandy Ryza) - - YARN-2036. Document yarn.resourcemanager.hostname in ClusterSetup (Ray - Chiang via Sandy Ryza) - - YARN-766. TestNodeManagerShutdown in branch-2 should use Shell to form the output path and a format - issue in trunk. (Contributed by Siddharth Seth) - - YARN-1982. Renamed the daemon name to be TimelineServer instead of History - Server and deprecated the old usage. (Zhijie Shen via vinodkv) - - YARN-1987. Wrapper for leveldb DBIterator to aid in handling database exceptions. - (Jason Lowe via kasha) - - YARN-1751. Improve MiniYarnCluster for log aggregation testing (Ming Ma - via jlowe) - - YARN-1981. Nodemanager version is not updated when a node reconnects (Jason - Lowe via jeagles) - - YARN-1938. Added kerberos login for the Timeline Server. (Zhijie Shen via - vinodkv) - - YARN-2017. Merged some of the common scheduler code. (Jian He via vinodkv) - - YARN-2049. Added delegation-token support for the Timeline Server. (Zhijie - Shen via vinodkv) - - YARN-1936. Added security support for the Timeline Client. (Zhijie Shen via - vinodkv) - - YARN-1937. Added owner-only ACLs support for Timeline Client and server. - (Zhijie Shen via vinodkv) - - YARN-2012. Fair Scheduler: allow default queue placement rule to take an - arbitrary queue (Ashwin Shankar via Sandy Ryza) - - YARN-2059. Added admin ACLs support to Timeline Server. (Zhijie Shen via - vinodkv) - - YARN-2073. Fair Scheduler: Add a utilization threshold to prevent preempting - resources when cluster is free (Karthik Kambatla via Sandy Ryza) - - YARN-2071. Modified levelDB store permissions to be readable only by the - server user. (Zhijie Shen via vinodkv) - - YARN-2107. Refactored timeline classes into o.a.h.y.s.timeline package. (Vinod - Kumar Vavilapalli via zjshen) - - YARN-596. Use scheduling policies throughout the queue hierarchy to decide - which containers to preempt (Wei Yan via Sandy Ryza) - - YARN-2054. Better defaults for YARN ZK configs for retries and retry-inteval - when HA is enabled. (kasha) - - YARN-1877. Document yarn.resourcemanager.zk-auth and its scope. - (Robert Kanter via kasha) - - YARN-2115. Replaced RegisterNodeManagerRequest's ContainerStatus with a new - NMContainerStatus which has more information that is needed for - work-preserving RM-restart. (Jian He via vinodkv) - - YARN-1474. Make schedulers services. (Tsuyoshi Ozawa via kasha) - - YARN-1913. With Fair Scheduler, cluster can logjam when all resources are - consumed by AMs (Wei Yan via Sandy Ryza) - - YARN-2061. Revisit logging levels in ZKRMStateStore. (Ray Chiang via kasha) - - YARN-1977. Add tests on getApplicationRequest with filtering start time range. (junping_du) - - YARN-2122. In AllocationFileLoaderService, the reloadThread should be created - in init() and started in start(). (Robert Kanter via kasha) - - YARN-2132. ZKRMStateStore.ZKAction#runWithRetries doesn't log the exception - it encounters. (Vamsee Yarlagadda via kasha) - - YARN-2030. Augmented RMStateStore with state machine.(Binglin Chang via jianhe) - - YARN-1424. RMAppAttemptImpl should return the - DummyApplicationResourceUsageReport for all invalid accesses. - (Ray Chiang via kasha) - - YARN-2091. Add more values to ContainerExitStatus and pass it from NM to - RM and then to app masters (Tsuyoshi OZAWA via bikas) - - YARN-2125. Changed ProportionalCapacityPreemptionPolicy to log CSV in debug - level. (Wangda Tan via jianhe) - - YARN-2159. Better logging in SchedulerNode#allocateContainer. - (Ray Chiang via kasha) - - YARN-2191. Added a new test to ensure NM will clean up completed applications - in the case of RM restart. (Wangda Tan via jianhe) - - YARN-2195. Clean a piece of code in ResourceRequest. (Wei Yan via devaraj) - - YARN-2074. Changed ResourceManager to not count AM preemptions towards app - failures. (Jian He via vinodkv) - - YARN-2192. TestRMHA fails when run with a mix of Schedulers. - (Anubhav Dhoot via kasha) - - YARN-2109. Fix TestRM to work with both schedulers. (Anubhav Dhoot via kasha) - - YARN-2072. RM/NM UIs and webservices are missing vcore information. - (Nathan Roberts via tgraves) - - YARN-2152. Added missing information into ContainerTokenIdentifier so that - NodeManagers can report the same to RM when RM restarts. (Jian He via vinodkv) - - YARN-2171. Improved CapacityScheduling to not lock on nodemanager-count when - AMs heartbeat in. (Jason Lowe via vinodkv) - - YARN-614. Changed ResourceManager to not count disk failure, node loss and - RM restart towards app failures. (Xuan Gong via jianhe) - - YARN-2224. Explicitly enable vmem check in - TestContainersMonitor#testContainerKillOnMemoryOverflow. - (Anubhav Dhoot via kasha) - - YARN-2022. Preempting an Application Master container can be kept as least priority - when multiple applications are marked for preemption by - ProportionalCapacityPreemptionPolicy (Sunil G via mayank) - - YARN-2241. ZKRMStateStore: On startup, show nicer messages if znodes already - exist. (Robert Kanter via kasha) - - YARN-1408 Preemption caused Invalid State Event: ACQUIRED at KILLED and - caused a task timeout for 30mins. (Sunil G via mayank) - - YARN-2300. Improved the documentation of the sample requests for RM REST API - - submitting an app. (Varun Vasudev via zjshen) - - OPTIMIZATIONS - - BUG FIXES - - YARN-1718. Fix a couple isTerminals in Fair Scheduler queue placement rules - (Sandy Ryza) - - YARN-1790. Fair Scheduler UI not showing apps table (bc Wong via Sandy Ryza) - - YARN-1784. TestContainerAllocation assumes CapacityScheduler. - (Robert Kanter via kasha) - - YARN-1940. deleteAsUser() terminates early without deleting more files on - error (Rushabh S Shah via jlowe) - - YARN-1865. ShellScriptBuilder does not check for some error conditions. - (Remus Rusanu via ivanmi) - - YARN-738. TestClientRMTokens is failing irregularly while running all yarn - tests (Ming Ma via jlowe) - - YARN-2018. TestClientRMService.testTokenRenewalWrongUser fails after - HADOOP-10562 (Ming Ma via Arpit Agarwal) - - YARN-2011. Fix typo and warning in TestLeafQueue (Chen He via junping_du) - - - YARN-2042. String shouldn't be compared using == in - QueuePlacementRule#NestedUserQueue#getQueueForApp (Chen He via Sandy Ryza) - - YARN-2050. Fix LogCLIHelpers to create the correct FileContext (Ming Ma - via jlowe) - - YARN-2089. FairScheduler: QueuePlacementPolicy and QueuePlacementRule - are missing audience annotations. (Zhihai Xu via kasha) - - YARN-2096. Race in TestRMRestart#testQueueMetricsOnRMRestart. - (Anubhav Dhoot via kasha) - - YARN-2105. Fix TestFairScheduler after YARN-2012. (Ashwin Shankar via - Sandy Ryza) - - YARN-2112. Fixed yarn-common's pom.xml to include jackson dependencies so - that both Timeline Server and client can access them. (Zhijie Shen via - vinodkv) - - YARN-1868. YARN status web ui does not show correctly in IE 11. - (Chuan Liu via cnauroth) - - YARN-2103. Inconsistency between viaProto flag and initial value of - SerializedExceptionProto.Builder (Binglin Chang via junping_du) - - YARN-1550. NPE in FairSchedulerAppsBlock#render. (Anubhav Dhoot via kasha) - - YARN-2119. DEFAULT_PROXY_ADDRESS should use DEFAULT_PROXY_PORT. - (Anubhav Dhoot via kasha) - - YARN-2118. Fixed the type mismatch in Map#containsKey check of - TimelineWebServices#injectOwnerInfo. (Ted Yu via zjshen) - - YARN-2117. Fixed the issue that secret file reader is potentially not - closed in TimelineAuthenticationFilterInitializer. (Chen He via zjshen) - - YARN-2121. Fixed NPE handling in Timeline Server's TimelineAuthenticator. - (Zhijie Shen via vinodkv) - - YARN-2128. FairScheduler: Incorrect calculation of amResource usage. - (Wei Yan via kasha) - - YARN-2124. Fixed NPE in ProportionalCapacityPreemptionPolicy. (Wangda Tan - via jianhe) - - YARN-2148. TestNMClient failed due more exit code values added and passed - to AM (Wangda Tan via bikas) - - YARN-2075. Fixed the test failure of TestRMAdminCLI. (Kenji Kikushima via - zjshen) - - YARN-2155. FairScheduler: Incorrect threshold check for preemption. - (Wei Yan via kasha) - - YARN-1885. Fixed a bug that RM may not send application-clean-up signal - to NMs where the completed applications previously ran in case of RM restart. - (Wangda Tan via jianhe) - - YARN-2167. LeveldbIterator should get closed in - NMLeveldbStateStoreService#loadLocalizationState() within finally block - (Junping Du via jlowe) - - YARN-2187. FairScheduler: Disable max-AM-share check by default. - (Robert Kanter via kasha) - - YARN-2111. In FairScheduler.attemptScheduling, we don't count containers - as assigned if they have 0 memory but non-zero cores (Sandy Ryza) - - YARN-2204. TestAMRestart#testAMRestartWithExistingContainers assumes - CapacityScheduler. (Robert Kanter via kasha) - - YARN-2163. WebUI: Order of AppId in apps table should be consistent with - ApplicationId.compareTo(). (Wangda Tan via raviprak) - - YARN-2104. Scheduler queue filter failed to work because index of queue - column changed. (Wangda Tan via jlowe) - - YARN-2201. Made TestRMWebServicesAppsModification be independent of the - changes on yarn-default.xml. (Varun Vasudev via zjshen) - - YARN-2216 YARN-2065 AM cannot create new containers after restart - (Jian He via stevel) - - YARN-2232. Fixed ResourceManager to allow DelegationToken owners to be able - to cancel their own tokens in secure mode. (Varun Vasudev via vinodkv) - - YARN-2250. FairScheduler.findLowestCommonAncestorQueue returns null when - queues not identical (Krisztian Horvath via Sandy Ryza) - - YARN-2158. Improved assertion messages of TestRMWebServicesAppsModification. - (Varun Vasudev via zjshen) - - YARN-2269. Remove external links from YARN UI. (Craig Welch via xgong) - - YARN-2270. Made TestFSDownload#testDownloadPublicWithStatCache be skipped - when there’s no ancestor permissions. (Akira Ajisaka via zjshen) - - YARN-2319. Made the MiniKdc instance start/close before/after the class of - TestRMWebServicesDelegationTokens. (Wenwu Peng via zjshen) - - YARN-2335. Annotate all hadoop-sls APIs as @Private. (Wei Yan via kasha) - - YARN-1726. ResourceSchedulerWrapper broken due to AbstractYarnScheduler. - (Wei Yan via kasha) - - YARN-2216. TestRMApplicationHistoryWriter sometimes fails in trunk. - (Zhijie Shen via xgong) - -Release 2.4.1 - 2014-06-23 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-1892. Improved some logs in the scheduler. (Jian He via zjshen) - - YARN-1696. Added documentation for ResourceManager fail-over. (Karthik - Kambatla, Masatake Iwasaki, Tsuyoshi OZAWA via vinodkv) - - YARN-1701. Improved default paths of the timeline store and the generic - history store. (Tsuyoshi Ozawa via zjshen) - - YARN-1962. Changed Timeline Service client configuration to be off by default - given the non-readiness of the feature yet. (Mohammad Kamrul Islam via - vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - YARN-1898. Made Standby RM links conf, stacks, logLevel, metrics, jmx, logs - and static not be redirected to Active RM. (Xuan Gong via zjshen) - - YARN-1837. Fixed TestMoveApplication#testMoveRejectedByScheduler failure. - (Hong Zhiguo via jianhe) - - YARN-1905. TestProcfsBasedProcessTree must only run on Linux. (cnauroth) - - YARN-1883. TestRMAdminService fails due to inconsistent entries in - UserGroups (Mit Desai via jeagles) - - YARN-1908. Fixed DistributedShell to not fail in secure clusters. (Vinod - Kumar Vavilapalli and Jian He via vinodkv) - - YARN-1910. Fixed a race condition in TestAMRMTokens that causes the test to - fail more often on Windows. (Xuan Gong via vinodkv) - - YARN-1920. Fixed TestFileSystemApplicationHistoryStore failure on windows. - (Vinod Kumar Vavilapalli via zjshen) - - YARN-1914. Fixed resource-download on NodeManagers to skip permission - verification of public cache files in Windows+local file-system environment. - (Varun Vasudev via vinodkv) - - YARN-1903. Set exit code and diagnostics when container is killed at - NEW/LOCALIZING state. (Zhijie Shen via jianhe) - - YARN-1924. Made ZKRMStateStore updateApplication(Attempt)StateInternal work - when Application(Attempt) state hasn't been stored before. (Jian He via - zjshen) - - YARN-1926. Changed DistributedShell to use appIDs as unique identifiers for - HDFS paths and thus avoid test failures on Windows. (Varun Vasudev via - vinodkv) - - YARN-1833. TestRMAdminService Fails in trunk and branch-2 (Mit Desai via - jeagles) - - YARN-1907. TestRMApplicationHistoryWriter#testRMWritingMassiveHistory - intermittently fails. (Mit Desai via kihwal) - - YARN-1933. Fixed test issues with TestAMRestart and TestNodeHealthService. - (Jian He via vinodkv) - - YARN-1928. Fixed a race condition in TestAMRMRPCNodeUpdates which caused it - to fail occassionally. (Zhijie Shen via vinodkv) - - YARN-1934. Fixed a potential NPE in ZKRMStateStore caused by handling - Disconnected event from ZK. (Karthik Kambatla via jianhe) - - YARN-1931. Private API change in YARN-1824 in 2.4 broke compatibility - with previous releases (Sandy Ryza via tgraves) - - YARN-1750. TestNodeStatusUpdater#testNMRegistration is incorrect in test - case. (Wangda Tan via junping_du) - - YARN-1947. TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey - is failing intermittently. (Jian He via junping_du) - - YARN-1281. Fixed TestZKRMStateStoreZKClientConnections to not fail - intermittently due to ZK-client timeouts. (Tsuyoshi Ozawa via vinodkv) - - YARN-1932. Javascript injection on the job status page (Mit Desai via - jlowe) - - YARN-1975. Used resources shows escaped html in CapacityScheduler and - FairScheduler page (Mit Desai via jlowe) - - YARN-1929. Fixed a deadlock in ResourceManager that occurs when failover - happens right at the time of shutdown. (Karthik Kambatla via vinodkv) - - YARN-1201. TestAMAuthorization fails with local hostname cannot be resolved. - (Wangda Tan via junping_du) - - YARN-1861. Fixed a bug in RM to reset leader-election on fencing that was - causing both RMs to be stuck in standby mode when automatic failover is - enabled. (Karthik Kambatla and Xuan Gong via vinodkv) - - YARN-1957. Consider the max capacity of the queue when computing the ideal - capacity for preemption. (Carlo Curino via cdouglas) - - YARN-1986. In Fifo Scheduler, node heartbeat in between creating app and - attempt causes NPE (Hong Zhiguo via Sandy Ryza) - - YARN-1976. Fix yarn application CLI to print the scheme of the tracking url - of failed/killed applications. (Junping Du via jianhe) - - YARN-2016. Fix a bug in GetApplicationsRequestPBImpl to add the missed fields - to proto. (Junping Du via jianhe) - - YARN-2053. Fixed a bug in AMS to not add null NMToken into NMTokens list from - previous attempts for work-preserving AM restart. (Wangda Tan via jianhe) - - YARN-2066. Wrong field is referenced in GetApplicationsRequestPBImpl#mergeLocalToBuilder() - (Hong Zhiguo via junping_du) - - YARN-2081. Fixed TestDistributedShell failure after YARN-1962. (Zhiguo Hong - via zjshen) - -Release 2.4.0 - 2014-04-07 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-930. Bootstrapping ApplicationHistoryService module. (vinodkv) - - YARN-947. Implementing the data objects to be used by the History reader - and writer interfaces. (Zhijie Shen via vinodkv) - - YARN-934. Defined a Writer Interface for HistoryStorage. (Zhijie Shen via - vinodkv) - - YARN-925. Defined a Reader Interface for HistoryStorage. (Mayank Bansal via - vinodkv) - - YARN-978. Created ApplicationAttemptReport. (Mayank Bansal via vinodkv) - - YARN-956. Added a testable in-memory HistoryStorage. (Mayank Bansal via - vinodkv) - - YARN-975. Added a file-system implementation for HistoryStorage. (Zhijie Shen - via vinodkv) - - YARN-1123. Added a new ContainerReport and its Protobuf implementation. (Mayank - Bansal via vinodkv) - - YARN-979. Added more APIs for getting information about ApplicationAttempts - and Containers from ApplicationHistoryProtocol. (Mayank Bansal and Zhijie Shen - via vinodkv) - - YARN-953. Changed ResourceManager to start writing history data. (Zhijie Shen - via vinodkv) - - YARN-1266. Implemented PB service and client wrappers for - ApplicationHistoryProtocol. (Mayank Bansal via vinodkv) - - YARN-955. Implemented ApplicationHistoryProtocol handler. (Mayank Bansal via - vinodkv) - - YARN-1242. Changed yarn scripts to be able to start ApplicationHistoryServer - as an individual process. (Mayank Bansal via vinodkv) - - YARN-954. Implemented web UI for the ApplicationHistoryServer and wired it into - the HistoryStorage. (Zhijie Shen via vinodkv) - - YARN-967. Added the client and CLI interfaces for obtaining ApplicationHistory - data. (Mayank Bansal via vinodkv) - - YARN-1023. Added Webservices REST APIs support for Application History. (Zhijie - Shen via vinodkv) - - YARN-1413. Implemented serving of aggregated-logs in the ApplicationHistory - server. (Mayank Bansal via vinodkv) - - YARN-1633. Defined user-facing entity, entity-info and event objects related - to Application Timeline feature. (Zhijie Shen via vinodkv) - - YARN-1611. Introduced the concept of a configuration provider which can be - used by ResourceManager to read configuration locally or from remote systems - so as to help RM failover. (Xuan Gong via vinodkv) - - YARN-1659. Defined the ApplicationTimelineStore store as an abstraction for - implementing different storage impls for storing timeline information. - (Billie Rinaldi via vinodkv) - - YARN-1634. Added a testable in-memory implementation of - ApplicationTimelineStore. (Zhijie Shen via vinodkv) - - YARN-1461. Added tags for YARN applications and changed RM to handle them. - (Karthik Kambatla via zjshen) - - YARN-1636. Augmented Application-history server's web-services to also expose - new APIs for retrieving and storing timeline information. (Zhijie Shen via - vinodkv) - - YARN-1490. Introduced the ability to make ResourceManager optionally not kill - all containers when an ApplicationMaster exits. (Jian He via vinodkv) - - YARN-1041. Added the ApplicationMasterProtocol API for applications to use the - ability in ResourceManager to optionally not kill containers when the - ApplicationMaster exits. (Jian He via vinodkv) - - YARN-1566. Changed Distributed Shell to retain containers across application - attempts. (Jian He via vinodkv) - - YARN-1635. Implemented a Leveldb based ApplicationTimelineStore. (Billie - Rinaldi via zjshen) - - YARN-1637. Implemented a client library for Java users to post timeline - entities and events. (zjshen) - - YARN-1496. Protocol additions to allow moving apps between queues (Sandy - Ryza) - - YARN-1498. Common scheduler changes for moving apps between queues (Sandy - Ryza) - - YARN-1504. RM changes for moving apps between queues (Sandy Ryza) - - YARN-1499. Fair Scheduler changes for moving apps between queues (Sandy - Ryza) - - YARN-1497. Command line additions for moving apps between queues (Sandy - Ryza) - - YARN-1588. Enhanced RM and the scheduling protocol to also send NMTokens of - transferred containers from previous app-attempts to new AMs after YARN-1490. - (Jian He via vinodkv) - - YARN-1717. Enabled periodically discarding old data in LeveldbTimelineStore. - (Billie Rinaldi via zjshen) - - YARN-1690. Made DistributedShell send timeline entities+events. (Mayank Bansal - via zjshen) - - YARN-1775. Enhanced ProcfsBasedProcessTree to optionally add the ability to - use smaps for obtaining used memory information. (Rajesh Balamohan via - vinodkv) - - YARN-1838. Enhanced timeline service getEntities API to get entities from a - given entity ID or insertion timestamp. (Billie Rinaldi via zjshen) - - IMPROVEMENTS - - YARN-1007. Enhance History Reader interface for Containers. (Mayank Bansal via - devaraj) - - YARN-974. Added more information to RMContainer to be collected and recorded in - Application-History. (Zhijie Shen via vinodkv) - - YARN-987. Added ApplicationHistoryManager responsible for exposing reports to - all clients. (Mayank Bansal via vinodkv) - - YARN-1630. Introduce timeout for async polling operations in YarnClientImpl - (Aditya Acharya via Sandy Ryza) - - YARN-1617. Remove ancient comment and surround LOG.debug in - AppSchedulingInfo.allocate (Sandy Ryza) - - YARN-1639. Modified RM HA configuration handling to have a way of not - requiring separate configuration files for each RM. (Xuan Gong via vinodkv) - - YARN-1668. Modified RM HA handling of admin-acls to be available across RM - failover by making using of a remote configuration-provider. (Xuan Gong via - vinodkv) - - YARN-1667. Modified RM HA handling of super users (with proxying ability) to - be available across RM failover by making using of a remote - configuration-provider. (Xuan Gong via vinodkv) - - YARN-1285. Changed the default value of yarn.acl.enable in yarn-default.xml - to be consistent with what exists (false) in the code and documentation. - (Kenji Kikushima via vinodkv) - - YARN-1669. Modified RM HA handling of protocol level service-ACLS to - be available across RM failover by making using of a remote - configuration-provider. (Xuan Gong via vinodkv) - - YARN-1665. Simplify the configuration of RM HA by having better default - values. (Xuan Gong via vinodkv) - - YARN-1660. Simplified the RM HA configuration to accept and be able to simply - depend just on configuration properties of the form - yarn.resourcemanager.hostname.RMID and use the default ports for all service - addresses. (Xuan Gong via vinodkv) - - YARN-1493. Changed ResourceManager and Scheduler interfacing to recognize - app-attempts separately from apps. (Jian He via vinodkv) - - YARN-1459. Changed ResourceManager to depend its service initialization - on the configuration-provider mechanism during startup too. (Xuan Gong via - vinodkv) - - YARN-1706. Created an utility method to dump timeline records to JSON - strings. (zjshen) - - YARN-1641. ZK store should attempt a write periodically to ensure it is - still Active. (kasha) - - YARN-1531. True up yarn command documentation (Akira Ajisaka via kasha) - - YARN-1345. Remove FINAL_SAVING state from YarnApplicationAttemptState - (Zhijie Shen via jianhe) - - YARN-1676. Modified RM HA handling of user-to-group mappings to - be available across RM failover by making using of a remote - configuration-provider. (Xuan Gong via vinodkv) - - YARN-1666. Modified RM HA handling of include/exclude node-lists to be - available across RM failover by making using of a remote - configuration-provider. (Xuan Gong via vinodkv) - - YARN-1171. Add default queue properties to Fair Scheduler documentation - (Naren Koneru via Sandy Ryza) - - YARN-1470. Add audience annotations to MiniYARNCluster. (Anubhav Dhoot - via kasha) - - YARN-1732. Changed types of related-entities and primary-filters in the - timeline-service to be sets instead of maps. (Billie Rinaldi via vinodkv) - - YARN-1687. Renamed user-facing records for the timeline-service to be simply - named after 'timeline' instead of 'apptimeline'. (Zhijie Shen via vinodkv) - - YARN-1749. Updated application-history related configs to reflect the latest - reality and to be consistently named. (Zhijie Shen via vinodkv) - - YARN-1301. Added the INFO level log of the non-empty blacklist additions - and removals inside ApplicationMasterService. (Tsuyoshi Ozawa via zjshen) - - YARN-1528. Allow setting auth for ZK connections. (kasha) - - YARN-1704. Modified LICENSE and NOTICE files to reflect newly used levelDB - related libraries. (Billie Rinaldi via vinodkv) - - YARN-1765. Added test cases to verify that killApplication API works across - ResourceManager failover. (Xuan Gong via vinodkv) - - YARN-1730. Implemented simple write-locking in the LevelDB based timeline- - store. (Billie Rinaldi via vinodkv) - - YARN-986. Changed client side to be able to figure out the right RM Delegation - token for the right ResourceManager when HA is enabled. (Karthik Kambatla via - vinodkv) - - YARN-1761. Modified RMAdmin CLI to check whether HA is enabled or not before - it executes any of the HA admin related commands. (Xuan Gong via vinodkv) - - YARN-1780. Improved logging in the Timeline client and server. (Zhijie Shen - via vinodkv) - - YARN-1525. Web UI should redirect to active RM when HA is enabled. (Cindy Li - via kasha) - - YARN-1781. Modified NodeManagers to allow admins to specify max disk - utilization for local disks so as to be able to offline full disks. (Varun - Vasudev via vinodkv) - - YARN-1410. Added tests to validate that clients can fail-over to a new RM - after getting an application-ID but before submission and can still submit to - the newly active RM with no issues. (Xuan Gong via vinodkv) - - YARN-1764. Modified YarnClient to correctly handle failover of ResourceManager - after the submitApplication call goes through. (Xuan Gong via vinodkv) - - YARN-1389. Made ApplicationClientProtocol and ApplicationHistoryProtocol - expose analogous getApplication(s)/Attempt(s)/Container(s) APIs. (Mayank - Bansal via zjshen) - - YARN-1658. Modified web-app framework to let standby RMs redirect - web-service calls to the active RM. (Cindy Li via vinodkv) - - YARN-1824. Improved NodeManager and clients to be able to handle cross - platform application submissions. (Jian He via vinodkv) - - YARN-1512. Enhanced CapacityScheduler to be able to decouple scheduling from - node-heartbeats. (Arun C Murthy via vinodkv) - - YARN-1570. Fixed formatting of the lines in YarnCommands.apt.vm docs source. - (Akira Ajisaka via vinodkv) - - YARN-1536. Cleanup: Get rid of ResourceManager#get*SecretManager() methods - and use the RMContext methods instead. (Anubhav Dhoot via kasha) - - YARN-1850. Introduced the ability to optionally disable sending out timeline- - events in the TimelineClient. (Zhijie Shen via vinodkv) - - YARN-1452. Added documentation about the configuration and usage of generic - application history and the timeline data service. (Zhijie Shen via vinodkv) - - YARN-1891. Added documentation for NodeManager health-monitoring. (Varun - Vasudev via vinodkv) - - YARN-1017. Added documentation for ResourceManager Restart.(jianhe) - - OPTIMIZATIONS - - YARN-1771. Reduce the number of NameNode operations during localization of - public resources using a cache. (Sangjin Lee via cdouglas) - - BUG FIXES - - YARN-935. Correcting pom.xml to build applicationhistoryserver module - successfully. (Zhijie Shen via vinodkv) - - YARN-962. Fixed bug in application-history proto file and renamed it be just - a client proto file. (Zhijie Shen via vinodkv) - - YARN-984. Renamed the incorrectly named applicationhistoryservice.records.pb.impl - package to be applicationhistoryservice.records.impl.pb. (Devaraj K via vinodkv) - - YARN-1534. Fixed failure of test TestAHSWebApp. (Shinichi Yamashita via vinodkv) - - YARN-1555. Fixed test failures in applicationhistoryservice.* (Vinod Kumar - Vavilapalli via mayank) - - YARN-1594. Updated pom.xml of applicationhistoryservice sub-project according to - YARN-888. (Vinod Kumar Vavilapalli via zjshen) - - YARN-1596. Fixed Javadoc warnings on branch YARN-321. (Vinod Kumar Vavilapalli - via zjshen) - - YARN-1597. Fixed Findbugs warnings on branch YARN-321. (Vinod Kumar Vavilapalli - via zjshen) - - YARN-1595. Made enabling history service configurable and fixed test failures on - branch YARN-321. (Vinod Kumar Vavilapalli via zjshen) - - YARN-1605. Fixed formatting issues in the new module on branch YARN-321. (Vinod - Kumar Vavilapalli via zjshen) - - YARN-1625. Fixed RAT warnings after YARN-321 merge. (Shinichi Yamashita via - vinodkv) - - YARN-1613. Fixed the typo with the configuration name - YARN_HISTORY_SERVICE_ENABLED. (Akira Ajisaka via vinodkv) - - YARN-1618. Fix invalid RMApp transition from NEW to FINAL_SAVING (kasha) - - YARN-1600. RM does not startup when security is enabled without spnego - configured (Haohui Mai via jlowe) - - YARN-1642. RMDTRenewer#getRMClient should use ClientRMProxy (kasha) - - YARN-1632. TestApplicationMasterServices should be under - org.apache.hadoop.yarn.server.resourcemanager package (Chen He via jeagles) - - YARN-1673. Fix option parsing in YARN's application CLI after it is broken - by YARN-967. (Mayank Bansal via vinodkv) - - YARN-1684. Fixed history server heap size in yarn script. (Billie Rinaldi - via zjshen) - - YARN-1166. Fixed app-specific and attempt-specific QueueMetrics to be - triggered by accordingly app event and attempt event. - - YARN-1689. Made RMAppAttempt get killed when RMApp is at ACCEPTED. (Vinod - Kumar Vavilapalli via zjshen) - - YARN-1661. Fixed DS ApplicationMaster to write the correct exit log. (Vinod - Kumar Vavilapalli via zjshen) - - YARN-1672. YarnConfiguration is missing a default for - yarn.nodemanager.log.retain-seconds (Naren Koneru via kasha) - - YARN-1698. Fixed default TimelineStore in code to match what is documented - in yarn-default.xml (Zhijie Shen via vinodkv) - - YARN-1697. NodeManager reports negative running containers (Sandy Ryza) - - YARN-1719. Fixed the root path related Jersey warnings produced in - ATSWebServices. (Billie Rinaldi via zjshen) - - YARN-1692. ConcurrentModificationException in fair scheduler AppSchedulable - (Sangjin Lee via Sandy Ryza) - - YARN-1578. Fixed reading incomplete application attempt and container data - in FileSystemApplicationHistoryStore. (Shinichi Yamashita via zjshen) - - YARN-1417. Modified RM to generate container-tokens not at creation time, but - at allocation time so as to prevent RM from shelling out containers with - expired tokens. (Omkar Vinit Joshi and Jian He via vinodkv) - - YARN-1553. Modified YARN and MR to stop using HttpConfig.isSecure() and - instead rely on the http policy framework. And also fix some bugs related - to https handling in YARN web-apps. (Haohui Mai via vinodkv) - - YARN-1721. When moving app between queues in Fair Scheduler, grab lock on - FSSchedulerApp (Sandy Ryza) - - YARN-1724. Race condition in Fair Scheduler when continuous scheduling is - turned on (Sandy Ryza) - - YARN-1590. Fixed ResourceManager, web-app proxy and MR JobHistoryServer to - expand _HOST properly in their kerberos principles. (Mohammad Kamrul Islam - va vinodkv) - - YARN-1428. Fixed RM to write the final state of RMApp/RMAppAttempt to the - application history store in the transition to the final state. (Contributed - by Zhijie Shen) - - YARN-713. Fixed ResourceManager to not crash while building tokens when DNS - issues happen transmittently. (Jian He via vinodkv) - - YARN-1398. Fixed a deadlock in ResourceManager between users requesting - queue-acls and completing containers. (vinodkv) - - YARN-1071. Enabled ResourceManager to recover cluster metrics - numDecommissionedNMs after restarting. (Jian He via zjshen) - - YARN-1742. Fixed javadoc of configuration parameter - DEFAULT_NM_MIN_HEALTHY_DISKS_FRACTION. (Akira Ajisaka via vinodkv) - - YARN-1686. Fixed NodeManager to properly handle any errors during - re-registration after a RESYNC and thus avoid hanging. (Rohith Sharma via - vinodkv) - - YARN-1734. Fixed ResourceManager to update the configurations when it - transits from standby to active mode so as to assimilate any changes that - happened while it was in standby mode. (Xuan Gong via vinodkv) - - YARN-1760. TestRMAdminService assumes CapacityScheduler. (kasha) - - YARN-1758. Fixed ResourceManager to not mandate the presence of site specific - configuration files and thus fix failures in downstream tests. (Xuan Gong via - vinodkv) - - YARN-1748. Excluded core-site.xml from hadoop-yarn-server-tests package's jar - and thus avoid breaking downstream tests. (Sravya Tirukkovalur via vinodkv) - - YARN-1729. Made TimelineWebServices deserialize the string primary- and - secondary-filters param into the JSON-compatible object. (Billie Rinaldi via - zjshen) - - YARN-1766. Fixed a bug in ResourceManager to use configuration loaded from the - configuration-provider when booting up. (Xuan Gong via vinodkv) - - YARN-1768. Fixed error message being too verbose when killing a non-existent - application. (Tsuyoshi OZAWA via raviprak) - - YARN-1774. FS: Submitting to non-leaf queue throws NPE. (Anubhav Dhoot and - Karthik Kambatla via kasha) - - YARN-1783. Fixed a bug in NodeManager's status-updater that was losing - completed container statuses when NodeManager is forced to resync by the - ResourceManager. (Jian He via vinodkv) - - YARN-1787. Fixed help messages for applicationattempt and container - sub-commands in bin/yarn. (Zhijie Shen via vinodkv) - - YARN-1793. Fixed ClientRMService#forceKillApplication not killing unmanaged - application. (Karthik Kambatla via jianhe) - - YARN-1788. Fixed a bug in ResourceManager to set the apps-completed and - apps-killed metrics correctly for killed applications. (Varun Vasudev via - vinodkv) - - YARN-1821. NPE on registerNodeManager if the request has containers for - UnmanagedAMs. (kasha) - - YARN-1800. Fixed NodeManager to gracefully handle RejectedExecutionException - in the public-localizer thread-pool. (Varun Vasudev via vinodkv) - - YARN-1444. Fix CapacityScheduler to deal with cases where applications - specify host/rack requests without off-switch request. (Wangda Tan via - acmurthy) - - YARN-1812. Fixed ResourceManager to synchrously renew tokens after recovery - and thus recover app itself synchronously and avoid races with resyncing - NodeManagers. (Jian He via vinodkv) - - YARN-1816. Fixed ResourceManager to get RMApp correctly handle - ATTEMPT_FINISHED event at ACCEPTED state that can happen after RM restarts. - (Jian He via vinodkv) - - YARN-1789. ApplicationSummary does not escape newlines in the app name - (Tsuyoshi OZAWA via jlowe) - - YARN-1830. Fixed TestRMRestart#testQueueMetricsOnRMRestart failure due to - race condition when app is submitted. (Zhijie Shen via jianhe) - - YARN-1685. Fixed few bugs related to handling of containers' log-URLs on - ResourceManager and history-service. (Zhijie Shen via vinodkv) - - YARN-1206. Fixed AM container log to show on NM web page after application - finishes if log-aggregation is disabled. (Rohith Sharmaks via jianhe) - - YARN-1591. Fixed AsyncDispatcher to handle interrupts on shutdown in a sane - manner and thus fix failure of TestResourceTrackerService. (Tsuyoshi Ozawa - via vinodkv) - - YARN-1839. Fixed handling of NMTokens in ResourceManager such that containers - launched by AMs running on the same machine as the AM are correctly - propagated. (Jian He via vinodkv) - - YARN-1640. Fixed manual failover of ResourceManagers to work correctly in - secure clusters. (Xuan Gong via vinodkv) - - YARN-1855. Made Application-history server to be optional in MiniYARNCluster - and thus avoid the failure of TestRMFailover#testRMWebAppRedirect. (Zhijie - Shen via vinodkv) - - YARN-1859. Fixed WebAppProxyServlet to correctly handle applications absent - on the ResourceManager. (Zhijie Shen via vinodkv) - - YARN-1811. Fixed AMFilters in YARN to correctly accept requests from either - web-app proxy or the RMs when HA is enabled. (Robert Kanter via vinodkv) - - YARN-1670. Fixed a bug in log-aggregation that can cause the writer to write - more log-data than the log-length that it records. (Mit Desai via vinodk) - - YARN-1849. Fixed NPE in ResourceTrackerService#registerNodeManager for UAM - (Karthik Kambatla via jianhe ) - - YARN-1863. Fixed test failure in TestRMFailover after YARN-1859. (Xuan Gong - via vinodkv) - - YARN-1854. Fixed test failure in TestRMHA#testStartAndTransitions. (Rohith - Sharma KS via vinodkv) - - YARN-1776. Fixed DelegationToken renewal to survive RM failover. (Zhijie - Shen via jianhe) - - YARN-1577. Made UnmanagedAMLauncher do launchAM after the attempt reaches - the LAUNCHED state. (Jian He via zjshen) - - YARN-1785. FairScheduler treats app lookup failures as ERRORs. - (bc Wong via kasha) - - YARN-1752. Fixed ApplicationMasterService to reject unregister request if - AM did not register before. (Rohith Sharma via jianhe) - - YARN-1846. TestRM#testNMTokenSentForNormalContainer assumes CapacityScheduler. - (Robert Kanter via kasha) - - YARN-1705. Reset cluster-metrics on transition to standby. (Rohith via kasha) - - YARN-1852. Fixed RMAppAttempt to not resend AttemptFailed/AttemptKilled - events to already recovered Failed/Killed RMApps. (Rohith via jianhe) - - YARN-1866. Fixed an issue with renewal of RM-delegation tokens on restart or - fail-over. (Jian He via vinodkv) - - YARN-1867. Fixed a bug in ResourceManager that was causing invalid ACL checks - in the web-services after fail-over. (Vinod Kumar Vavilapalli) - - YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in - ApplicationClientProtcol, ResourceManagerAdministrationProtocol and - ResourceTrackerProtocol so that they work in HA scenario. (Xuan Gong - via jianhe) - - YARN-1873. Fixed TestDistributedShell failure when the test cases are out of - order. (Mit Desai via zjshen) - - YARN-1893. Mark AtMostOnce annotation to ApplicationMasterProtocol#allocate. - (Xuan Gong via jianhe) - -Release 2.3.1 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.3.0 - 2014-02-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-649. Added a new NM web-service to serve container logs in plain text - over HTTP. (Sandy Ryza via vinodkv) - - YARN-1021. Yarn Scheduler Load Simulator. (ywskycn via tucu) - - YARN-1010. FairScheduler: decouple container scheduling from nodemanager - heartbeats. (Wei Yan via Sandy Ryza) - - YARN-1253. Changes to LinuxContainerExecutor to run containers as a single - dedicated user in non-secure mode. (rvs via tucu) - - YARN-1027. Implement RMHAProtocolService (Karthik Kambatla via bikas) - - YARN-1068. Add admin support for HA operations (Karthik Kambatla via - bikas) - - YARN-311. RM/scheduler support for dynamic resource configuration. - (Junping Du via llu) - - YARN-1392. Allow sophisticated app-to-queue placement policies in the Fair - Scheduler (Sandy Ryza) - - YARN-1447. Common PB type definitions for container resizing. (Wangda Tan - via Sandy Ryza) - - YARN-1448. AM-RM protocol changes to support container resizing (Wangda Tan - via Sandy Ryza) - - YARN-312. Introduced ResourceManagerAdministrationProtocol changes to support - changing resources on node. (Junping Du via vinodkv) - - YARN-1028. Added FailoverProxyProvider capability to ResourceManager to help - with RM failover. (Karthik Kambatla via vinodkv) - - YARN-1029. Added embedded leader election in the ResourceManager. (Karthik - Kambatla via vinodkv) - - YARN-1033. Expose RM active/standby state to Web UI and REST API (kasha) - - IMPROVEMENTS - - YARN-305. Fair scheduler logs too many "Node offered to app" messages. - (Lohit Vijayarenu via Sandy Ryza) - - YARN-1258. Allow configuring the Fair Scheduler root queue (Sandy Ryza) - - YARN-1288. Make Fair Scheduler ACLs more user friendly (Sandy Ryza) - - YARN-1315. TestQueueACLs should also test FairScheduler (Sandy Ryza) - - YARN-1335. Move duplicate code from FSSchedulerApp and FiCaSchedulerApp - into SchedulerApplication (Sandy Ryza) - - YARN-1333. Support blacklisting in the Fair Scheduler (Tsuyoshi Ozawa via - Sandy Ryza) - - YARN-1109. Demote NodeManager "Sending out status for container" logs to - debug (haosdent via Sandy Ryza) - - YARN-1321. Changed NMTokenCache to support both singleton and an instance - usage. (Alejandro Abdelnur via vinodkv) - - YARN-1388. Fair Scheduler page always displays blank fair share (Liyin Liang - via Sandy Ryza) - - YARN-7. Support CPU resource for DistributedShell. (Junping Du via llu) - - YARN-905. Add state filters to nodes CLI (Wei Yan via Sandy Ryza) - - YARN-1098. Separate out RM services into Always On and Active (Karthik - Kambatla via bikas) - - YARN-353. Add Zookeeper-based store implementation for RMStateStore. - (Bikas Saha, Jian He and Karthik Kambatla via hitesh) - - YARN-819. ResourceManager and NodeManager should check for a minimum allowed - version (Robert Parker via jeagles) - - YARN-425. coverage fix for yarn api (Aleksey Gorshkov via jeagles) - - YARN-1199. Make NM/RM Versions Available (Mit Desai via jeagles) - - YARN-1232. Configuration to support multiple RMs (Karthik Kambatla via - bikas) - - YARN-465. fix coverage org.apache.hadoop.yarn.server.webproxy (Aleksey - Gorshkov and Andrey Klochkov via jlowe) - - YARN-976. Document the meaning of a virtual core. (Sandy Ryza) - - YARN-1182. MiniYARNCluster creates and inits the RM/NM only on start() - (Karthik Kambatla via Sandy Ryza) - - HADOOP-9598. Improve code coverage of RMAdminCLI (Aleksey Gorshkov and - Andrey Klochkov via jeagles) - - YARN-1306. Clean up hadoop-sls sample-conf according to YARN-1228 (Wei Yan - via Sandy Ryza) - - YARN-891. Modified ResourceManager state-store to remember completed - applications so that clients can get information about them post RM-restart. - (Jian He via vinodkv) - - YARN-1290. Let continuous scheduling achieve more balanced task assignment - (Wei Yan via Sandy Ryza) - - YARN-786. Expose application resource usage in RM REST API (Sandy Ryza) - - YARN-1323. Set HTTPS webapp address along with other RPC addresses in HAUtil - (Karthik Kambatla via Sandy Ryza) - - YARN-1121. Changed ResourceManager's state-store to drain all events on - shut-down. (Jian He via vinodkv) - - YARN-1387. RMWebServices should use ClientRMService for filtering - applications (Karthik Kambatla via Sandy Ryza) - - YARN-1222. Make improvements in ZKRMStateStore for fencing (Karthik - Kambatla via bikas) - - YARN-709. Added tests to verify validity of delegation tokens and logging of - appsummary after RM restart. (Jian He via vinodkv) - - YARN-1210. Changed RM to start new app-attempts on RM restart only after - ensuring that previous AM exited or after expiry time. (Omkar Vinit Joshi via - vinodkv) - - YARN-674. Fixed ResourceManager to renew DelegationTokens on submission - asynchronously to work around potential slowness in state-store. (Omkar Vinit - Joshi via vinodkv) - - YARN-584. In scheduler web UIs, queues unexpand on refresh. (Harshit - Daga via Sandy Ryza) - - YARN-1303. Fixed DistributedShell to not fail with multiple commands separated - by a semi-colon as shell-command. (Xuan Gong via vinodkv) - - YARN-1423. Support queue placement by secondary group in the Fair Scheduler - (Ted Malaska via Sandy Ryza) - - YARN-1314. Fixed DistributedShell to not fail with multiple arguments for a - shell command separated by spaces. (Xuan Gong via vinodkv) - - YARN-1239. Modified ResourceManager state-store implementations to start - storing version numbers. (Jian He via vinodkv) - - YARN-1241. In Fair Scheduler, maxRunningApps does not work for non-leaf - queues. (Sandy Ryza) - - YARN-1318. Promoted AdminService to an Always-On service and merged it into - RMHAProtocolService. (Karthik Kambatla via vinodkv) - - YARN-1332. In TestAMRMClient, replace assertTrue with assertEquals where - possible (Sebastian Wong via Sandy Ryza) - - YARN-1403. Separate out configuration loading from QueueManager in the Fair - Scheduler (Sandy Ryza) - - YARN-1181. Augment MiniYARNCluster to support HA mode (Karthik Kambatla) - - YARN-546. Allow disabling the Fair Scheduler event log (Sandy Ryza) - - YARN-807. When querying apps by queue, iterating over all apps is - inefficient and limiting (Sandy Ryza) - - YARN-1378. Implemented a cleaner of old finished applications from the RM - state-store. (Jian He via vinodkv) - - YARN-1481. Move internal services logic from AdminService to ResourceManager. - (vinodkv via kasha) - - YARN-1491. Upgrade JUnit3 TestCase to JUnit 4 (Chen He via jeagles) - - YARN-408. Change CapacityScheduler to not disable delay-scheduling by default. - (Mayank Bansal via vinodkv) - - YARN-1325. Modified RM HA configuration validation to also ensure that - multiple RMs are configured. (Xuan Gong via vinodkv) - - YARN-1311. Fixed app specific scheduler-events' names to be app-attempt - based. (vinodkv via jianhe) - - YARN-1485. Modified RM HA configuration validation to also ensure that - service-address configuration are configured for every RM. (Xuan Gong via - vinodkv) - - YARN-1435. Modified Distributed Shell to accept either the command or the - custom script. (Xuan Gong via zjshen) - - YARN-1446. Changed client API to retry killing application till RM - acknowledges so as to account for RM crashes/failover. (Jian He via vinodkv) - - YARN-1307. Redesign znode structure for Zookeeper based RM state-store for - better organization and scalability. (Tsuyoshi OZAWA via vinodkv) - - YARN-1172. Convert SecretManagers in RM to services (Tsuyoshi OZAWA via kasha) - - YARN-1523. Use StandbyException instead of RMNotYetReadyException (kasha) - - YARN-1541. Changed ResourceManager to invalidate ApplicationMaster host/port - information once an AM crashes. (Jian He via vinodkv) - - YARN-1482. Modified WebApplicationProxy to make it work across ResourceManager - fail-over. (Xuan Gong via vinodkv) - - YARN-1568. Rename clusterid to clusterId in ActiveRMInfoProto (kasha) - - YARN-1579. ActiveRMInfoProto fields should be optional (kasha) - - YARN-888. Cleaned up POM files so that non-leaf modules don't include any - dependencies and thus compact the dependency list for leaf modules. - (Alejandro Abdelnur via vinodkv) - - YARN-1567. In Fair Scheduler, allow empty queues to change between leaf and - parent on allocation file reload (Sandy Ryza) - - YARN-1616. RMFatalEventDispatcher should log the cause of the event (kasha) - - YARN-1624. QueuePlacementPolicy format is not easily readable via a JAXB - parser (Aditya Acharya via Sandy Ryza) - - YARN-1623. Include queue name in RegisterApplicationMasterResponse (Sandy - Ryza) - - YARN-1573. ZK store should use a private password for root-node-acls. - (kasha). - - OPTIMIZATIONS - - BUG FIXES - - YARN-1284. LCE: Race condition leaves dangling cgroups entries for killed - containers. (Alejandro Abdelnur via Sandy Ryza) - - YARN-1283. Fixed RM to give a fully-qualified proxy URL for an application - so that clients don't need to do scheme-mangling. (Omkar Vinit Joshi via - vinodkv) - - YARN-879. Fixed tests w.r.t o.a.h.y.server.resourcemanager.Application. - (Junping Du via devaraj) - - YARN-1265. Fair Scheduler chokes on unhealthy node reconnect (Sandy Ryza) - - YARN-1044. used/min/max resources do not display info in the scheduler page - (Sangjin Lee via Sandy Ryza) - - YARN-1259. In Fair Scheduler web UI, queue num pending and num active apps - switched. (Robert Kanter via Sandy Ryza) - - YARN-1295. In UnixLocalWrapperScriptBuilder, using bash -c can cause Text - file busy errors (Sandy Ryza) - - YARN-1185. Fixed FileSystemRMStateStore to not leave partial files that - prevent subsequent ResourceManager recovery. (Omkar Vinit Joshi via vinodkv) - - YARN-1331. yarn.cmd exits with NoClassDefFoundError trying to run rmadmin or - logs. (cnauroth) - - YARN-1330. Fair Scheduler: defaultQueueSchedulingPolicy does not take effect - (Sandy Ryza) - - YARN-1022. Unnecessary INFO logs in AMRMClientAsync (haosdent via bikas) - - YARN-1349. yarn.cmd does not support passthrough to any arbitrary class. - (cnauroth) - - YARN-1357. TestContainerLaunch.testContainerEnvVariables fails on Windows. - (Chuan Liu via cnauroth) - - YARN-1358. TestYarnCLI fails on Windows due to line endings. (Chuan Liu via - cnauroth) - - YARN-1343. NodeManagers additions/restarts are not reported as node updates - in AllocateResponse responses to AMs. (tucu) - - YARN-1381. Same relaxLocality appears twice in exception message of - AMRMClientImpl#checkLocalityRelaxationConflict() (Ted Yu via Sandy Ryza) - - YARN-1407. RM Web UI and REST APIs should uniformly use - YarnApplicationState (Sandy Ryza) - - YARN-1438. Ensure container diagnostics includes exception from container - launch. (stevel via acmurthy) - - YARN-1138. yarn.application.classpath is set to point to $HADOOP_CONF_DIR - etc., which does not work on Windows. (Chuan Liu via cnauroth) - - YARN-461. Fair scheduler should not accept apps with empty string queue name. - (ywskycn via tucu) - - YARN-1060. Two tests in TestFairScheduler are missing @Test annotation - (Niranjan Singh via Sandy Ryza) - - YARN-1188. The context of QueueMetrics becomes default when using - FairScheduler (Tsuyoshi Ozawa via Sandy Ryza) - - YARN-1268. TestFairScheduler.testContinuousScheduling is flaky (Sandy Ryza) - - YARN-1300. SLS tests fail because conf puts YARN properties in - fair-scheduler.xml (Ted Yu via Sandy Ryza) - - YARN-1183. MiniYARNCluster shutdown takes several minutes intermittently - (Andrey Klochkov via jeagles) - - YARN-1305. RMHAProtocolService#serviceInit should handle HAUtil's - IllegalArgumentException (Tsuyoshi Ozawa via bikas) - - YARN-1374. Changed ResourceManager to start the preemption policy monitors - as active services. (Karthik Kambatla via vinodkv) - - YARN-1395. Distributed shell application master launched with debug flag can - hang waiting for external ls process. (cnauroth) - - YARN-1400. yarn.cmd uses HADOOP_RESOURCEMANAGER_OPTS. Should be - YARN_RESOURCEMANAGER_OPTS. (Raja Aluri via cnauroth) - - YARN-1401. With zero sleep-delay-before-sigkill.ms, no signal is ever sent - (Gera Shegalov via Sandy Ryza) - - YARN-1411. HA config shouldn't affect NodeManager RPC addresses (Karthik - Kambatla via bikas) - - YARN-1419. TestFifoScheduler.testAppAttemptMetrics fails intermittently - under jdk7 (Jonathan Eagles via jlowe) - - YARN-744. Race condition in ApplicationMasterService.allocate .. It might - process same allocate request twice resulting in additional containers - getting allocated. (Omkar Vinit Joshi via bikas) - - YARN-1425. TestRMRestart fails because MockRM.waitForState(AttemptId) uses - current attempt instead of the attempt passed as argument (Omkar Vinit - Joshi via bikas) - - YARN-1053. Diagnostic message from ContainerExitEvent is ignored in - ContainerImpl (Omkar Vinit Joshi via bikas) - - YARN-1320. Fixed Distributed Shell application to respect custom log4j - properties file. (Xuan Gong via vinodkv) - - YARN-1416. Fixed a few invalid transitions in RMApp, RMAppAttempt and in some - tests. (Jian He via vinodkv) - - YARN-895. Changed RM state-store to not crash immediately if RM restarts while - the state-store is down. (Jian He via vinodkv) - - YARN-1454. Fixed test failure issue with TestRMRestart. (Karthik Kambatla - via vinodkv) - - YARN-1450. Fixed test failure in TestUnmanagedAMLauncher by removing its - dependency on distributed-shell. (Binglin Chang via vinodkv) - - YARN-1405. Fixed ResourceManager to not hang when init/start fails with an - exception w.r.t state-store. (Jian He via vinodkv) - - YARN-1505. Fixed Webapplication proxy server to not hardcode its bind - address. (Xuan Gong via vinodkv) - - YARN-1145. Fixed a potential file-handle leak in the web interface for - displaying aggregated logs. (Rohith Sharma via vinodkv) - - YARN-1451. TestResourceManager relies on the scheduler assigning multiple - containers in a single node update. (Sandy Ryza via kasha) - - YARN-1527. Fix yarn rmadmin command to print the correct usage info. - (Akira AJISAKA via jianhe) - - YARN-1522. Fixed a race condition in the test TestApplicationCleanup that was - causing it to randomly fail. (Liyin Liang via vinodkv) - - YARN-1549. Fixed a bug in ResourceManager's ApplicationMasterService that - was causing unamanged AMs to not finish correctly. (haosdent via vinodkv) - - YARN-1559. Race between ServerRMProxy and ClientRMProxy setting - RMProxy#INSTANCE. (kasha and vinodkv via kasha) - - YARN-1560. Fixed TestYarnClient#testAMMRTokens failure with null AMRM token. - (Ted Yu via jianhe) - - YARN-1409. NonAggregatingLogHandler can throw RejectedExecutionException - (Tsuyoshi OZAWA via jlowe) - - YARN-1293. Fixed TestContainerLaunch#testInvalidEnvSyntaxDiagnostics failure - caused by non-English system locale. (Tsuyoshi OZAWA via jianhe) - - YARN-1574. RMDispatcher should be reset on transition to standby. (Xuan Gong - via kasha) - - YARN-1598. HA-related rmadmin commands don't work on a secure cluster (kasha) - - YARN-1603. Remove two *.orig files which were unexpectedly committed. - (Zhijie Shen via junping_du) - - YARN-1601. 3rd party JARs are missing from hadoop-dist output. (tucu) - - YARN-1351. Invalid string format in Fair Scheduler log warn message - (Konstantin Weitz via Sandy Ryza) - - YARN-1608. LinuxContainerExecutor has a few DEBUG messages at INFO level - (kasha) - - YARN-1606. Fix the default value of yarn.resourcemanager.zk-timeout-ms - in yarn-default.xml (kasha) - - YARN-1607. TestRM relies on the scheduler assigning multiple containers in - a single node update (Sandy Ryza) - - YARN-1575. Public localizer crashes with "Localized unkown resource" - (jlowe) - - YARN-1629. IndexOutOfBoundsException in MaxRunningAppsEnforcer (Sandy Ryza) - - YARN-1628. Fixed the test failure in TestContainerManagerSecurity. (Vinod - Kumar Vavilapalli via zjshen) - -Release 2.2.0 - 2013-10-13 - - INCOMPATIBLE CHANGES - - YARN-1229. Define constraints on Auxiliary Service names. Change - ShuffleHandler service name from mapreduce.shuffle to - mapreduce_shuffle (Xuan Gong via sseth) - - NEW FEATURES - - IMPROVEMENTS - - YARN-1246. Added application finish-status to ApplicationSummary for the sake - of testing given ApplicationHistoryServer is not yet ready. (Arpit Gupta via - vinodkv) - - YARN-899. Added back queue level administrator-acls so that there is no - regression w.r.t 1.x. (Xuan Gong via vinodkv) - - YARN-1228. Clean up Fair Scheduler configuration loading. (Sandy Ryza) - - YARN-1213. Restore config to ban submitting to undeclared pools in the - Fair Scheduler. (Sandy Ryza) - - YARN-1277. Added a policy based configuration for http/https in common - HttpServer and using the same in YARN - related to per project https config - support via HADOOP-10022. (Suresh Srinivas and Omkar Vinit Joshi via vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - YARN-1128. FifoPolicy.computeShares throws NPE on empty list of Schedulables - (Karthik Kambatla via Sandy Ryza) - - YARN-1214. Register ClientToken MasterKey in SecretManager after it is - saved (Jian He via bikas) - - YARN-49. Improve distributed shell application to work on a secure cluster. - (Vinod Kumar Vavilapalli via hitesh) - - YARN-1157. Fixed ResourceManager UI to behave correctly when apps like - distributed-shell do not set tracking urls. (Xuan Gong via vinodkv) - - YARN-1221. With Fair Scheduler, reserved MB reported in RM web UI increases - indefinitely (Siqi Li via Sandy Ryza) - - YARN-1247. test-container-executor has gotten out of sync with the changes to - container-executor. (rvs via tucu) - - YARN-1070. Fixed race conditions in NodeManager during container-kill. - (Zhijie Shen via vinodkv) - - YARN-1215. Yarn URL should include userinfo. (Chuan Liu via cnauroth) - - YARN-1262. TestApplicationCleanup relies on all containers assigned in a - single heartbeat (Karthik Kambatla via Sandy Ryza) - - YARN-1260. Added webapp.http.address to yarn-default.xml so that default - install with https enabled doesn't have broken link on NM UI. (Omkar Vinit - Joshi via vinodkv) - - YARN-1141. Updating resource requests should be decoupled with updating - blacklist (Zhijie Shen via bikas) - - YARN-876. Node resource is added twice when node comes back from unhealthy - to healthy. (Peng Zhang via Sandy Ryza) - - YARN-890. Ensure CapacityScheduler doesn't round-up metric for available - resources. (Xuan Gong & Hitesh Shah via acmurthy) - - YARN-621. Changed YARN web app to not add paths that can cause duplicate - additions of authenticated filters there by causing kerberos replay errors. - (Omkar Vinit Joshi via vinodkv) - - YARN-1236. FairScheduler setting queue name in RMApp is not working. - (Sandy Ryza) - - YARN-1256. NM silently ignores non-existent service in - StartContainerRequest (Xuan Gong via bikas) - - YARN-1149. NM throws InvalidStateTransitonException: Invalid event: - APPLICATION_LOG_HANDLING_FINISHED at RUNNING (Xuan Gong via hitesh) - - YARN-1271. "Text file busy" errors launching containers again - (Sandy Ryza) - - YARN-1131. $yarn logs command should return an appropriate error message if - YARN application is still running. (Siddharth Seth via hitesh) - - YARN-1219. FSDownload changes file suffix making FileUtil.unTar() throw - exception. (Shanyu Zhao via cnauroth) - - YARN-1251. TestDistributedShell#TestDSShell failed with timeout. (Xuan Gong - via hitesh) - - YARN-1167. Fixed Distributed Shell to not incorrectly show empty hostname - on RM UI. (Xuan Gong via vinodkv) - - YARN-1254. Fixed NodeManager to not pollute container's credentials. (Omkar - Vinit Joshi via vinodkv) - - YARN-1273. Fixed Distributed-shell to account for containers that failed - to start. (Hitesh Shah via vinodkv) - - YARN-1032. Fixed NPE in RackResolver. (Lohit Vijayarenu via acmurthy) - - YARN-1090. Fixed CS UI to better reflect applications as non-schedulable - and not as pending. (Jian He via acmurthy) - - YARN-1274. Fixed NodeManager's LinuxContainerExecutor to create user, app-dir - and log-dirs correctly even when there are no resources to localize for the - container. (Siddharth Seth via vinodkv) - - YARN-1278. Fixed NodeManager to not delete local resources for apps on resync - command from RM - a bug caused by YARN-1149. (Hitesh Shah via vinodkv) - - YARN-1463. Tests should avoid starting http-server where possible or creates - spnego keytab/principals (vinodkv via kasha) - -Release 2.1.1-beta - 2013-09-23 - - INCOMPATIBLE CHANGES - - YARN-707. Added user information also in the YARN ClientToken so that AMs - can implement authorization based on incoming users. (Jason Lowe via vinodkv) - - YARN-1170. YARN & MapReduce proto definitions fixed to specify protobuf - package as hadoop.yarn and hadoop.mapreduce respectively. (Binglin Chang - via acmurthy) - - NEW FEATURES - - IMPROVEMENTS - - YARN-589. Expose a REST API for monitoring the fair scheduler (Sandy Ryza). - - YARN-1074. Cleaned up YARN CLI application list to only display running - applications by default. (Xuan Gong via vinodkv) - - YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via - Sandy Ryza) - - YARN-942. In Fair Scheduler documentation, inconsistency on which - properties have prefix (Akira Ajisaka via Sandy Ryza) - - YARN-1083. Changed ResourceManager to fail when the expiry interval is less - than the configured node-heartbeat interval. (Zhijie Shen via vinodkv) - - YARN-1081. Made a trivial change to YARN node CLI header to avoid potential - confusion. (Akira AJISAKA via vinodkv) - - YARN-1034. Remove "experimental" in the Fair Scheduler documentation. - (Karthik Kambatla via Sandy Ryza) - - YARN-1080. Improved help message for "yarn logs" command. (Xuan Gong via - vinodkv) - - YARN-771. AMRMClient support for resource blacklisting (Junping Du via - bikas) - - YARN-1117. Improved help messages for "yarn application" and "yarn node" - commands. (Xuan Gong via vinodkv) - - YARN-1120. Made ApplicationConstants.Environment.USER definition OS neutral - as the corresponding value is now set correctly end-to-end. (Chuan Liu via - vinodkv) - - YARN-1124. Modified YARN CLI application list to display new and submitted - applications together with running apps by default, following up YARN-1074. - (Xuan Gong via vinodkv) - - YARN-1065. NM should provide AuxillaryService data to the container (Xuan - Gong via bikas) - - YARN-758. Augment MockNM to use multiple cores (Karthik Kambatla via - Sandy Ryza) - - YARN-696. Changed RMWebservice apps call to take in multiple application - states. (Trevor Lorimer via vinodkv) - - YARN-910. Augmented auxiliary services to listen for container starts and - completions in addition to application events. (Alejandro Abdelnur via - vinodkv) - - YARN-1137. Add support whitelist for system users to Yarn - container-executor.c. (rvs via tucu) - - YARN-1001. Added a web-service to get statistics about per application-type - per state for consumption by downstream projects. (Zhijie Shen via vinodkv) - - YARN-1203. Changed YARN web-app proxy to handle http and https URLs from - AM registration and finish correctly. (Omkar Vinit Joshi via vinodkv) - - YARN-1204. Added separate configuration properties for https for RM and NM - without which servers enabled with https will also start on http ports. - (Omkar Vinit Joshi via vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - YARN-948. Changed ResourceManager to validate the release container list - before actually releasing them. (Omkar Vinit Joshi via vinodkv) - - YARN-966. Fixed ContainerLaunch to not fail quietly when there are no - localized resources due to some other failure. (Zhijie Shen via vinodkv) - - YARN-502. Fixed a state machine issue with RMNode inside ResourceManager - which was crashing scheduler. (Mayank Bansal via vinodkv) - - YARN-573. Shared data structures in Public Localizer and Private Localizer - are not Thread safe. (Omkar Vinit Joshi via jlowe) - - YARN-903. Changed ContainerManager to suppress unnecessary warnings when - stopping already stopped containers. (Omkar Vinit Joshi via vinodkv) - - YARN-906. Fixed a bug in NodeManager where cancelling ContainerLaunch at - KILLING state causes that the container to hang. (Zhijie Shen via vinodkv) - - YARN-994. HeartBeat thread in AMRMClientAsync does not handle runtime - exception correctly (Xuan Gong via bikas) - - YARN-337. RM handles killed application tracking URL poorly (jlowe) - - YARN-107. Fixed ResourceManager and clients to better handle - forceKillApplication on non-running and finished applications. (Xuan Gong - via vinodkv) - - YARN-643. Fixed ResourceManager to remove all tokens consistently on app - finish. (Xuan Gong via vinodkv) - - YARN-1006. Fixed broken rendering in the Nodes list web page on the RM web - UI. (Xuan Gong via vinodkv) - - YARN-881. Priority#compareTo method seems to be wrong. (Jian He via bikas) - - YARN-1082. Create base directories on HDFS after RM login to ensure RM - recovery doesn't fail in secure mode. (vinodkv via acmurthy) - - YARN-1085. Modified YARN and MR2 web-apps to do HTTP authentication in - secure setup with kerberos. (Omkar Vinit Joshi via vinodkv) - - YARN-1094. Fixed a blocker with RM restart code because of which RM crashes - when try to recover an existing app. (vinodkv) - - YARN-1008. MiniYARNCluster with multiple nodemanagers, all nodes have same - key for allocations. (tucu) - - YARN-981. Fixed YARN webapp so that /logs servlet works like before. (Jian He - via vinodkv) - - YARN-602. Fixed NodeManager to not let users override some mandatory - environmental variables. (Kenji Kikushima via vinodkv) - - YARN-1101. Active nodes can be decremented below 0 (Robert Parker - via tgraves) - - YARN-1077. Fixed TestContainerLaunch test failure on Windows. (Chuan Liu via - vinodkv) - - YARN-957. Fixed a bug in CapacityScheduler because of which requests that - need more than a node's total capability were incorrectly allocated on that - node causing apps to hang. (Omkar Vinit Joshi via vinodkv) - - YARN-1107. Fixed a bug in ResourceManager because of which RM in secure mode - fails to restart. (Omkar Vinit Joshi via vinodkv) - - YARN-1049. ContainerExistStatus should define a status for preempted - containers. (tucu) - - YARN-1144. Unmanaged AMs registering a tracking URI should not be - proxy-fied. (tucu) - - YARN-1152. Fixed a bug in ResourceManager that was causing clients to get - invalid client token key errors when an appliation is about to finish. - (Jason Lowe via vinodkv) - - YARN-292. Fixed FifoScheduler and FairScheduler to make their applications - data structures thread safe to avoid RM crashing with - ArrayIndexOutOfBoundsException. (Zhijie Shen via vinodkv) - - YARN-1025. ResourceManager and NodeManager do not load native libraries on - Windows. (cnauroth) - - YARN-1176. RM web services ClusterMetricsInfo total nodes doesn't include - unhealthy nodes (Jonathan Eagles via tgraves) - - YARN-1078. TestNodeManagerResync, TestNodeManagerShutdown, and - TestNodeStatusUpdater fail on Windows. (Chuan Liu via cnauroth) - - YARN-1194. TestContainerLogsPage fails with native builds (Roman Shaposhnik - via jlowe) - - YARN-1116. Populate AMRMTokens back to AMRMTokenSecretManager after RM - restarts (Jian He via bikas) - - YARN-1189. NMTokenSecretManagerInNM is not being told when applications - have finished (Omkar Vinit Joshi via jlowe) - - YARN-540. Race condition causing RM to potentially relaunch already - unregistered AMs on RM restart (Jian He via bikas) - - YARN-1184. ClassCastException during preemption enforcement. (cdouglas) - -Release 2.1.0-beta - 2013-08-22 - - INCOMPATIBLE CHANGES - - YARN-396. Rationalize AllocateResponse in RM Scheduler API. (Zhijie Shen - via hitesh) - - YARN-439. Flatten NodeHeartbeatResponse. (Xuan Gong via sseth) - - YARN-440. Flatten RegisterNodeManagerResponse. (Xuan Gong via sseth) - - YARN-536. Removed the unused objects ContainerStatus and ContainerStatus from - Container which also don't belong to the container. (Xuan Gong via vinodkv) - - YARN-486. Changed NM's startContainer API to accept Container record given by - RM as a direct parameter instead of as part of the ContainerLaunchContext - record. (Xuan Gong via vinodkv) - - YARN-444. Moved special container exit codes from YarnConfiguration to API - where they belong. (Sandy Ryza via vinodkv) - - YARN-441. Removed unused utility methods for collections from two API - records. (Xuan Gong via vinodkv) - - YARN-561. Modified NodeManager to set key information into the environment - of every container that it launches. (Xuan Gong via vinodkv) - - YARN-579. Stop setting the Application Token in the AppMaster env, in - favour of the copy present in the container token field. - (Vinod Kumar Vavilapalli via sseth) - - YARN-629. Make YarnRemoteException not be rooted at IOException. (Xuan Gong - via vinodkv) - - YARN-633. Changed RMAdminProtocol api to throw IOException and - YarnRemoteException. (Xuan Gong via vinodkv) - - YARN-632. Changed ContainerManager api to throw IOException and - YarnRemoteException. (Xuan Gong via vinodkv) - - YARN-631. Changed ClientRMProtocol api to throw IOException and - YarnRemoteException. (Xuan Gong via vinodkv) - - YARN-630. Changed AMRMProtocol api to throw IOException and - YarnRemoteException. (Xuan Gong via vinodkv) - - YARN-615. Rename ContainerLaunchContext.containerTokens to tokens. - (Vinod Kumar Vavilapalli via sseth) - - YARN-571. Remove user from ContainerLaunchContext. (Omkar Vinit Joshi via - vinodkv) - - YARN-716. Making ApplicationID immutable. (Siddharth Seth via vinodkv) - - YARN-684. ContainerManager.startContainer should use - ContainerTokenIdentifier instead of the entire Container. - (Vinod Kumar Vavilapalli via sseth) - - YARN-735. Make ApplicationAttemptId, ContaienrId and NodeId immutable. - (Jian He via sseth) - - YARN-749. Rename ResourceRequest.(get,set)HostName to - ResourceRequest.(get,set)ResourceName. (acmurthy) - - YARN-720. container-log4j.properties should not refer to mapreduce - property names. (Zhijie Shen via sseth) - - YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for - eventual retirement. (Jian He via vinodkv) - - YARN-635. Renamed YarnRemoteException to YarnException. (Siddharth Seth via - vinodkv) - - YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. (Bikas - Saha via vinodkv) - - YARN-753. Added individual factory methods for all api protocol records and - converted the records to be abstract classes. (Jian He via vinodkv) - - YARN-724. Moved ProtoBase from api.records to api.records.impl.pb. (Jian He - via vinodkv) - - YARN-759. Create Command enum in AllocateResponse (bikas) - - YARN-777. Removed unreferenced objects from .proto files. (Jian He via - vinodkv) - - YARN-642. Removed health parameter from ResourceManager /nodes web-service - and cleaned the behaviour of the status parameter. (Sandy Ryza vid vinodkv) - - YARN-530. Defined Service model strictly, implemented AbstractService for - robust subclassing and migrated yarn-common services. (Steve Loughran via - vinodkv) - - YARN-746. Renamed Service.register() and Service.unregister() to - registerServiceListener() & unregisterServiceListener() respectively. - (Steve Loughran via vinodkv) - - YARN-792. Moved NodeHealthStatus from yarn.api.record to - yarn.server.api.record. (Jian He via vinodkv) - - YARN-806. Moved ContainerExitStatus from yarn.api to yarn.api.records. (Jian - He via vinodkv) - - YARN-821. Renamed setFinishApplicationStatus to setFinalApplicationStatus in - FinishApplicationMasterRequest for consistency. (Jian He via vinodkv) - - YARN-787. Removed minimum resource from RegisterApplicationMasterResponse. - (tucu via acmurthy) - - YARN-829. Renamed RMTokenSelector to be RMDelegationTokenSelector. (Zhijie - Shen via vinodkv) - - YARN-828. Removed the unsed YarnVersionAnnotation. (Zhijie Shen via vinodkv) - - YARN-823. Moved RMAdmin from yarn.client to yarn.client.cli and renamed it to - be RMAdminCLI. (Jian He via vinodkv) - - YARN-387. Renamed YARN protocols for consistency. - ClientRMProtocol -> ApplicationClientProtocol - AMRMProtocol -> ApplicationMasterProtocol - ContainerManager -> ContainerManagementProtocol - (vinodkv via acmurthy) - - YARN-831. Removed minimum resource from GetNewApplicationResponse as a - follow-up to YARN-787. (Jian He via acmurthy) - - YARN-824. Added static factory methods to hadoop-yarn-client interfaces. - (Jian He via acmurthy) - - YARN-826. Moved Clock and SystemClock into yarn.util package. (Zhijie Shen - via vinodkv) - - YARN-837. Moved yarn.ClusterInfo into MapReduce project as it doesn't belong - to YARN. (Zhijie Shen via vinodkv) - - YARN-822. Renamed ApplicationToken to be AMRMToken, and similarly the - corresponding TokenSelector and SecretManager. (Omkar Vinit Joshi via vinodkv) - - YARN-610. ClientToken is no longer set in the environment of the Containers. - (Omkar Vinit Joshi via vinodkv) - - YARN-834. Fixed annotations for yarn-client module, reorganized packages and - clearly differentiated *Async apis. (Arun C Murthy and Zhijie Shen via - vinodkv) - - YARN-840. Moved ProtoUtils to yarn.api.records.pb.impl. (Jian He via - acmurthy) - - YARN-841. Move Auxiliary service to yarn-api, annotate and document it. - (vinodkv) - - YARN-850. Rename getClusterAvailableResources to getAvailableResources in - AMRMClients (Jian He via bikas) - - YARN-694. Starting to use NMTokens to authenticate all communication with - NodeManagers. (Omkar Vinit Joshi via vinodkv) - - YARN-553. Replaced YarnClient.getNewApplication with - YarnClient.createApplication which provides a directly usable - ApplicationSubmissionContext to simplify the api. (Karthik Kambatla via - acmurthy) - - YARN-851. Share NMTokens using NMTokenCache (api-based) between AMRMClient - and NMClient instead of memory based approach which is used currently. (Omkar - Vinit Joshi via vinodkv) - - YARN-869. Move ResourceManagerAdministrationProtocol out of main YARN api. - (vinodkv via acmurthy) - - YARN-791. Changed RM APIs and web-services related to nodes to ensure that - both are consistent with each other. (Sandy Ryza via vinodkv) - - YARN-727. ClientRMProtocol.getAllApplications should accept ApplicationType as - a parameter. (Xuan Gong via hitesh) - - YARN-701. Use application tokens irrespective of secure or non-secure - mode. (vinodkv via acmurthy) - - YARN-918. Remove ApplicationAttemptId from - RegisterApplicationMasterRequestProto. (vinodkv via acmurthy) - - YARN-926. Modified ContainerManagerProtcol APIs to take in requests for - multiple containers. (Jian He via vinodkv) - - NEW FEATURES - - YARN-482. FS: Extend SchedulingMode to intermediate queues. - (kkambatl via tucu) - - YARN-45. Add protocol for schedulers to request containers back from - ApplicationMasters. (Carlo Curino, cdouglas) - - YARN-563. Add the concept of an application-type for each application. - (Mayank Bansal via vinodkv) - - HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows - Azure environments. (See breakdown of tasks below for subtasks and - contributors) - - YARN-422. Add a NM Client library to help application-writers. (Zhijie Shen - via vinodkv) - - YARN-392. Make it possible to specify hard locality constraints in resource - requests. (sandyr via tucu) - - YARN-326. Add multi-resource scheduling to the fair scheduler. - (sandyr via tucu) - - YARN-398. Make it possible to specify hard locality constraints in resource - requests for CapacityScheduler. (acmurthy) - - YARN-781. Exposing LOGDIR in all containers' environment which should be used - by containers for logging purposes. (Jian He via vinodkv) - - IMPROVEMENTS - - YARN-347. Node CLI should show CPU info besides memory in node status. - (Junping Du via llu) - - YARN-365. Change NM heartbeat handling to not generate a scheduler event - on each heartbeat. (Xuan Gong via sseth) - - YARN-380. Fix yarn node -status output to be better readable. (Omkar Vinit - Joshi via vinodkv) - - YARN-410. Fixed RM UI so that the new lines diagnostics for a failed app on - the per-application page are translated to html line breaks. (Omkar Vinit - Joshi via vinodkv) - - YARN-198. Added a link to RM pages from the NodeManager web app. (Jian He - via vinodkv) - - YARN-237. Refreshing the RM page forgets how many rows I had in my - Datatables (jian he via bobby) - - YARN-481. Add AM Host and RPC Port to ApplicationCLI Status Output - (Chris Riccomini via bikas) - - YARN-297. Improve hashCode implementations for PB records. (Xuan Gong via - hitesh) - - YARN-417. Create AMRMClient wrapper that provides asynchronous callbacks. - (Sandy Ryza via bikas) - - YARN-497. Yarn unmanaged-am launcher jar does not define a main class in - its manifest (Hitesh Shah via bikas) - - YARN-469. Make scheduling mode in FS pluggable. (kkambatl via tucu) - - YARN-450. Define value for * in the scheduling protocol (Zhijie Shen via - bikas) - - YARN-475. Remove a unused constant in the public API - - ApplicationConstants.AM_APP_ATTEMPT_ID_ENV. (Hitesh Shah via vinodkv) - - YARN-309. Changed NodeManager to obtain heart-beat interval from the - ResourceManager. (Xuan Gong via vinodkv) - - YARN-447. Move ApplicationComparator in CapacityScheduler to use comparator - in ApplicationId. (Nemon Lou via vinodkv) - - YARN-381. Improve fair scheduler docs. (Sandy Ryza via tomwhite) - - YARN-458. YARN daemon addresses must be placed in many different configs. - (sandyr via tucu) - - YARN-193. Scheduler.normalizeRequest does not account for allocation - requests that exceed maximumAllocation limits (Zhijie Shen via bikas) - - YARN-479. NM retry behavior for connection to RM should be similar for - lost heartbeats (Jian He via bikas) - - YARN-495. Changed NM reboot behaviour to be a simple resync - kill all - containers and re-register with RM. (Jian He via vinodkv) - - YARN-514. Delayed store operations should not result in RM unavailability - for app submission (Zhijie Shen via bikas) - - YARN-586. Fixed a typo in ApplicationSubmissionContext#setApplicationId. - (Zhijie Shen via vinodkv) - - YARN-542. Changed the default global AM max-attempts value to be not one. - (Zhijie Shen via vinodkv) - - YARN-583. Moved application level local resources to be localized under the - filecache sub-directory under application directory. (Omkar Vinit Joshi via - vinodkv) - - YARN-581. Added a test to verify that app delegation tokens are restored - after RM restart. (Jian He via vinodkv) - - YARN-577. Add application-progress also to ApplicationReport. (Hitesh Shah - via vinodkv) - - YARN-595. Refactor fair scheduler to use common Resources. (Sandy Ryza - via tomwhite) - - YARN-562. Modified NM to reject any containers allocated by a previous - ResourceManager. (Jian He via vinodkv) - - YARN-591. Moved RM recovery related records out of public API as they do not - belong there. (vinodkv) - - YARN-599. Refactoring submitApplication in ClientRMService and RMAppManager - to separate out various validation checks depending on whether they rely on - RM configuration or not. (Zhijie Shen via vinodkv) - - YARN-618. Modified RM_INVALID_IDENTIFIER to be -1 instead of zero. (Jian He - via vinodkv) - - YARN-625. Move the utility method unwrapAndThrowException from - YarnRemoteExceptionPBImpl to RPCUtil. (Siddharth Seth via vinodkv) - - YARN-645. Moved RMDelegationTokenSecretManager from yarn-server-common to - yarn-server-resourcemanager where it really belongs. (Jian He via vinodkv) - - YARN-651. Changed PBClientImpls of ContainerManager and RMAdmin to throw - IOExceptions also. (Xuan Gong via vinodkv) - - YARN-582. Changed ResourceManager to recover Application token and client - tokens for app attempt so that RM can be restarted while preserving current - applications. (Jian He via vinodkv) - - YARN-568. Add support for work preserving preemption to the FairScheduler. - (Carlo Curino and Sandy Ryza via cdouglas) - - YARN-598. Add virtual cores to queue metrics. (sandyr via tucu) - - YARN-634. Modified YarnRemoteException to be not backed by PB and introduced - a separate SerializedException record. (Siddharth Seth via vinodkv) - - YARN-663. Changed ResourceTracker API and LocalizationProtocol API to throw - YarnRemoteException and IOException. (Xuan Gong via vinodkv) - - YARN-590. Added an optional mesage to be returned by ResourceMaanger when RM - asks an RM to shutdown/resync etc so that NMs can log this message locally - for better debuggability. (Mayank Bansal via vinodkv) - - YARN-617. Made ContainerTokens to be used for validation at NodeManager - also in unsecure mode to prevent AMs from faking resource requirements in - unsecure mode. (Omkar Vinit Joshi via vinodkv) - - YARN-708. Moved RecordFactory classes to hadoop-yarn-api, and put some - miscellaneous fixes to the interfaces. (Siddharth Seth via vinodkv) - - YARN-711. Copied BuilderUtil methods in individual API records as - BuilderUtils is going to be dismantled. (Jian He via vinodkv) - - YARN-714. Added NMTokens to be sent to AMs as part of heart-beat response. - (Omkar Vinit Joshi via vinodkv) - - YARN-638. Modified ResourceManager to restore RMDelegationTokens after - restarting. (Jian He via vinodkv) - - YARN-660. Improve AMRMClient with matching requests (bikas) - - YARN-717. Put object creation factories for Token in the class itself and - remove useless derivations for specific tokens. (Jian He via vinodkv) - - YARN-756. Move Preemption* records to yarn.api where they really belong. - (Jian He via vinodkv) - - YARN-750. Allow for black-listing resources in YARN API and Impl in CS - (acmurthy via bikas) - - YARN-877. Support resource blacklisting for FifoScheduler. - (Junping Du via llu) - - YARN-686. Flatten NodeReport. (sandyr via tucu) - - YARN-737. Throw some specific exceptions directly instead of wrapping them - in YarnException. (Jian He via sseth) - - YARN-731. RPCUtil.unwrapAndThrowException should unwrap remote - RuntimeExceptions. (Zhijie Shen via sseth) - - YARN-600. Hook up cgroups CPU settings to the number of virtual cores - allocated. (sandyr via tucu) - - YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu) - - YARN-773. Moved YarnRuntimeException from package api.yarn to - api.yarn.exceptions. (Jian He via vinodkv) - - YARN-692. Creating NMToken master key on RM and sharing it with NM as a part - of RM-NM heartbeat. (Omkar Vinit Joshi via vinodkv) - - YARN-782. vcores-pcores ratio functions differently from vmem-pmem ratio in - misleading way. (sandyr via tucu) - - YARN-803. factor out scheduler config validation from the ResourceManager - to each scheduler implementation. (tucu) - - YARN-789. Enable zero capabilities resource requests in fair scheduler. - (tucu) - - YARN-639. Modified Distributed Shell application to start using the new - NMClient library. (Zhijie Shen via vinodkv) - - YARN-693. Modified RM to send NMTokens on allocate call so that AMs can then - use them for authentication with NMs. (Omkar Vinit Joshi via vinodkv) - - YARN-752. In AMRMClient, automatically add corresponding rack requests for - requested nodes. (sandyr via tucu) - - YARN-825. Fixed javadoc and annotations for yarn-common module. (vinodkv) - - YARN-833. Moved Graph and VisualizeStateMachine into yarn.state package. - (Zhijie Shen via vinodkv) - - YARN-805. Fix javadoc and annotations on classes in the yarn-api - package. (Jian He via sseth) - - YARN-846. Move pb Impl classes from yarn-api to yarn-common. (Jian He via - vinodkv) - - YARN-827. Need to make Resource arithmetic methods accessible (Jian He via - bikas) - - YARN-866. Add test for class ResourceWeights. (ywskycn via tucu) - - YARN-736. Add a multi-resource fair sharing metric. (sandyr via tucu) - - YARN-883. Expose Fair Scheduler-specific queue metrics. (sandyr via tucu) - - YARN-569. Add support for requesting and enforcing preemption requests via - a capacity monitor. (Carlo Curino, cdouglas) - - YARN-521. Augment AM - RM client module to be able to request containers - only at specific locations (Sandy Ryza via bikas) - - YARN-513. Create common proxy client for communicating with RM. (Xuan Gong - & Jian He via bikas) - - YARN-927. Change ContainerRequest to not have more than 1 container count - and remove StoreContainerRequest (bikas) - - YARN-922. Change FileSystemRMStateStore to use directories (Jian He via - bikas) - - YARN-865. RM webservices can't query based on application Types. (Xuan Gong - via hitesh) - - YARN-912. Move client facing exceptions to yarn-api module. (Mayank Bansal - via vinodkv) - - YARN-84. Use Builder to build RPC server. (Brandon Li via suresh) - - YARN-1046. Disable mem monitoring by default in MiniYARNCluster. (Karthik - Kambatla via Sandy Ryza) - - YARN-1045. Improve toString implementation for PBImpls. (Jian He via sseth) - - OPTIMIZATIONS - - YARN-512. Log aggregation root directory check is more expensive than it - needs to be. (Maysam Yabandeh via jlowe) - - YARN-719. Move RMIdentifier from Container to ContainerTokenIdentifier. - (Vinod Kumar Vavilapalli via sseth) - - BUG FIXES - - YARN-383. AMRMClientImpl should handle null rmClient in stop() - (Hitesh Shah via sseth) - - YARN-385. Add missing fields - location and #containers to - ResourceRequestPBImpl's toString(). (Sandy Ryza via sseth) - - YARN-377. Use the new StringUtils methods added by HADOOP-9252 and fix - TestContainersMonitor. (Chris Nauroth via szetszwo) - - YARN-391. Formatting fixes for LCEResourceHandler classes. - (Steve Loughran via sseth) - - YARN-390. ApplicationCLI and NodeCLI hard-coded platform-specific line - separator causes test failures on Windows. (Chris Nauroth via suresh) - - YARN-406. Fix TestRackResolver to function in networks where "host1" - resolves to a valid host. (Hitesh Shah via sseth) - - YARN-376. Fixes a bug which would prevent the NM knowing about completed - containers and applications. (Jason Lowe via sseth) - - YARN-196. Nodemanager should be more robust in handling connection failure - to ResourceManager when a cluster is started (Xuan Gong via hitesh) - - YARN-485. TestProcfsProcessTree#testProcessTree() doesn't wait long enough - for the process to die. (kkambatl via tucu) - - YARN-71. Fix the NodeManager to clean up local-dirs on restart. - (Xuan Gong via sseth) - - YARN-378. Fix RM to make the AM max attempts/retries to be configurable - per application by clients. (Zhijie Shen via vinodkv) - - YARN-498. Unmanaged AM launcher does not set various constants in env for - an AM, also does not handle failed AMs properly. (Hitesh Shah via bikas) - - YARN-496. Fair scheduler configs are refreshed inconsistently in - reinitialize. (Sandy Ryza via tomwhite) - - YARN-474. Fix CapacityScheduler to trigger application-activation when - am-resource-percent configuration is refreshed. (Zhijie Shen via vinodkv) - - YARN-209. Fix CapacityScheduler to trigger application-activation when - the cluster capacity changes. (Zhijie Shen via vinodkv) - - YARN-24. Nodemanager fails to start if log aggregation enabled and - namenode unavailable. (sandyr via tucu) - - YARN-515. Node Manager not getting the master key. (Robert Joseph Evans - via jlowe) - - YARN-382. SchedulerUtils improve way normalizeRequest sets the resource - capabilities. (Zhijie Shen via bikas) - - YARN-467. Modify public distributed cache to localize files such that no - local directory hits unix file count limits and thus prevent job failures. - (Omkar Vinit Joshi via vinodkv) - - YARN-101. Fix NodeManager heartbeat processing to not lose track of completed - containers in case of dropped heartbeats. (Xuan Gong via vinodkv) - - YARN-538. RM address DNS lookup can cause unnecessary slowness on every JHS - page load. (sandyr via tucu) - - YARN-532. Change RMAdmin and Localization client protocol PB implementations - to implement closeable so that they can be stopped when needed via - RPC.stopProxy(). (Siddharth Seth via vinodkv) - - YARN-99. Modify private distributed cache to localize files such that no - local directory hits unix file count limits and thus prevent job failures. - (Omkar Vinit Joshi via vinodkv) - - YARN-112. Fixed a race condition during localization that fails containers. - (Omkar Vinit Joshi via vinodkv) - - YARN-534. Change RM restart recovery to also account for AM max-attempts - configuration after the restart. (Jian He via vinodkv) - - YARN-539. Addressed memory leak of LocalResource objects NM when a resource - localization fails. (Omkar Vinit Joshi via vinodkv) - - YARN-319. Submitting a job to a fair scheduler queue for which the user - does not have permission causes the client to wait forever. - (shenhong via tomwhite) - - YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather - than its host:port during scheduling which caused incorrect locality for - containers. (Roger Hoover via acmurthy) - - YARN-500. Fixed YARN webapps to not roll-over ports when explicitly asked - to use non-ephemeral ports. (Kenji Kikushima via vinodkv) - - YARN-518. Fair Scheduler's document link could be added to the hadoop 2.x - main doc page. (sandyr via tucu) - - YARN-476. ProcfsBasedProcessTree info message confuses users. - (sandyr via tucu) - - YARN-585. Fix failure in TestFairScheduler#testNotAllowSubmitApplication - caused by YARN-514. (Zhijie Shen via vinodkv) - - YARN-547. Fixed race conditions in public and private resource localization - which used to cause duplicate downloads. (Omkar Vinit Joshi via vinodkv) - - YARN-594. Update test and add comments in YARN-534 (Jian He via bikas) - - YARN-549. YarnClient.submitApplication should wait for application to be - accepted by the RM (Zhijie Shen via bikas) - - YARN-605. Fix failing unit test in TestNMWebServices when versionInfo has - parantheses like when running on a git checkout. (Hitesh Shah via vinodkv) - - YARN-289. Fair scheduler allows reservations that won't fit on node. - (Sandy Ryza via tomwhite) - - YARN-576. Modified ResourceManager to reject NodeManagers that don't satisy - minimum resource requirements. (Kenji Kikushima via vinodkv) - - YARN-646. Fix two typos in Fair Scheduler user guide. (Dapeng Sun via atm) - - YARN-507. Add interface visibility and stability annotations to FS - interfaces/classes. (kkambatl via tucu) - - YARN-637. FS: maxAssign is not honored. (kkambatl via tucu) - - YARN-655. Fair scheduler metrics should subtract allocated memory from - available memory. (sandyr via tucu) - - YARN-628. Fix the way YarnRemoteException is being unrolled to extract out - the underlying exception. (Siddharth Seth via vinodkv) - - YARN-695. Remove masterContainer and status unused fields from - ApplicationReportProto and fix bugs in ApplicationReportPBImpl. (Zhijie Shen - via vinodkv) - - YARN-706. Fixed race conditions in TestFSDownload. (Zhijie Shen via vinodkv). - - YARN-715. Fixed unit test failures - TestDistributedShell and - TestUnmanagedAMLauncher. (Vinod Kumar Vavilapalli via sseth) - - YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. - (Omkar Vinit Joshi via vinodkv) - - YARN-733. Fixed TestNMClient from failing occasionally. (Zhijie Shen via - vinodkv) - - YARN-730. Fix NMClientAsync to remove completed containers. (Zhijie Shen - via acmurthy) - - YARN-726. Fix queue & finish time fields in web-ui for ResourceManager. - (Mayank Bansal via acmurthy) - - YARN-757. Changed TestRMRestart to use the default scheduler to avoid test - failures. (Bikas Saha via vinodkv) - - YARN-742. Log aggregation causes a lot of redundant setPermission calls. - (jlowe via kihwal) - - YARN-764. blank Used Resources on Capacity Scheduler page (Nemon Lou via - tgraves) - - YARN-761. TestNMClientAsync fails sometimes (Zhijie Shen via bikas) - - YARN-760. NodeManager throws AvroRuntimeException on failed start. - (Niranjan Singh via jlowe) - - YARN-767. Initialize application metrics at RM bootup. (Jian He via - acmurthy) - - YARN-700. TestInfoBlock fails on Windows because of line ending missmatch. - (Ivan Mitic via cnauroth) - - YARN-117. Migrated rest of YARN to the new service model. (Steve Louhran via - vinodkv) - - YARN-812. Set default logger for application summary logger to - hadoop.root.logger. (sseth via acmurthy) - - YARN-848. Nodemanager does not register with RM using the fully qualified - hostname. (Hitesh Shah via sseth) - - YARN-854. Fixing YARN bugs that are failing applications in secure - environment. (Omkar Vinit Joshi via vinodkv) - - YARN-861. TestContainerManager is failing. (Vinod Kumar Vavilapalli via - hitesh) - - YARN-874. Making common RPC to switch to not switch to simple when other - mechanisms are enabled and thus fix YARN/MR test failures after HADOOP-9421. - (Daryn Sharp and Vinod Kumar Vavilapalli via vinodkv) - - YARN-845. RM crash with NPE on NODE_UPDATE (Mayank Bansal via bikas) - - YARN-369. Handle ( or throw a proper error when receiving) status updates - from application masters that have not registered (Mayank Bansal & - Abhishek Kapoor via bikas) - - YARN-541. getAllocatedContainers() is not returning all the allocated - containers (bikas) - - YARN-763. AMRMClientAsync should stop heartbeating after receiving - shutdown from RM (Xuan Gong via bikas) - - YARN-654. AMRMClient: Perform sanity checks for parameters of public - methods (Xuan Gong via bikas)" - - YARN-919. Document setting default heap sizes in yarn-env.sh (Mayank - Bansal via hitesh) - - YARN-795. Fair scheduler queue metrics should subtract allocated vCores from - available vCores. (ywskycn via tucu) - - YARN-799. Fix CgroupsLCEResourcesHandler to use /tasks instead of - /cgroup.procs. (Chris Riccomini via acmurthy) - - YARN-333. Schedulers cannot control the queue-name of an - application. (sandyr via tucu) - - YARN-368. Fixed a typo in error message in Auxiliary services. (Albert Chu - via vinodkv) - - YARN-295. Fixed a race condition in ResourceManager RMAppAttempt state - machine. (Mayank Bansal via vinodkv) - - YARN-523. Modified a test-case to validate container diagnostics on - localization failures. (Jian He via vinodkv) - - YARN-661. Fixed NM to cleanup users' local directories correctly when - starting up. (Omkar Vinit Joshi via vinodkv) - - YARN-820. Fixed an invalid state transition in NodeManager caused by failing - resource localization. (Mayank Bansal via vinodkv) - - YARN-62. Modified NodeManagers to avoid AMs from abusing container tokens for - repetitive container launches. (Omkar Vinit Joshi via vinodkv) - - YARN-814. Improving diagnostics when containers fail during launch due to - various reasons like invalid env etc. (Jian He via vinodkv) - - YARN-897. Ensure child queues are ordered correctly to account for - completed containers. (Djellel Eddine Difallah via acmurthy) - - YARN-853. Fixed CapacityScheduler's maximum-am-resource-percent to properly - work beyond refreshing queues. (Devaraj K via vinodkv) - - YARN-873. YARNClient.getApplicationReport(unknownAppId) returns a null - report (Xuan Gong via bikas) - - YARN-875. Application can hang if AMRMClientAsync callback thread has - exception (Xuan Gong via bikas) - - YARN-968. RM admin commands don't work. (vinodkv via kihwal) - - YARN-688. Fixed NodeManager to properly cleanup containers when it is shut - down. (Jian He via vinodkv) - - YARN-960. Fixed ResourceManager to propagate client-submitted credentials - irrespective of security. (Daryn Sharp via vinodkv) - - YARN-937. Fix unmanaged AM in non-secure/secure setup post YARN-701. (tucu) - - YARN-932. TestResourceLocalizationService.testLocalizationInit can fail on - JDK7. (Karthik Kambatla via Sandy Ryza) - - YARN-961. Changed ContainerManager to enforce Token auth irrespective of - security. (Omkar Vinit Joshi via vinodkv) - - YARN-945. Removed setting of AMRMToken's service from ResourceManager - and changed client libraries do it all the time and correctly. (vinodkv) - - YARN-656. In scheduler UI, including reserved memory in Memory Total can - make it exceed cluster capacity. (Sandy Ryza) - - BREAKDOWN OF HADOOP-8562/YARN-191 SUBTASKS AND RELATED JIRAS - - YARN-158. Yarn creating package-info.java must not depend on sh. - (Chris Nauroth via suresh) - - YARN-176. Some YARN tests fail to find winutils. (Chris Nauroth via suresh) - - YARN-207. YARN distribution build fails on Windows. (Chris Nauroth via - suresh) - - YARN-199. Yarn cmd line scripts for windows. (Ivan Mitic via suresh) - - YARN-213. YARN build script would be more readable using abspath. - (Chris Nauroth via suresh) - - YARN-233. Added support for running containers in MS Windows to YARN. (Chris - Nauroth via acmurthy) - - YARN-234. Added support for process tree and resource calculator in MS Windows - to YARN. (Chris Nauroth via acmurthy) - - YARN-259. Fix LocalDirsHandlerService to use Path rather than URIs. (Xuan - Gong via acmurthy) - - YARN-316. YARN container launch may exceed maximum Windows command line - length due to long classpath. (Chris Nauroth via suresh) - - YARN-359. Fixing commands for container signalling in Windows. (Chris Nauroth - via vinodkv) - - YARN-506. Move to common utils FileUtil#setReadable/Writable/Executable and - FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh) - - YARN-488. TestContainerManagerSecurity fails on Windows. (Chris Nauroth - via hitesh) - - YARN-490. TestDistributedShell fails on Windows. (Chris Nauroth via hitesh) - - YARN-491. TestContainerLogsPage fails on Windows. (Chris Nauroth via hitesh) - - YARN-487. Modify path manipulation in LocalDirsHandlerService to let - TestDiskFailures pass on Windows. (Chris Nauroth via vinodkv) - - YARN-593. container launch on Windows does not correctly populate - classpath with new process's environment variables and localized resources - (Chris Nauroth via bikas) - - YARN-493. Fixed some shell related flaws in YARN on Windows. (Chris Nauroth - via vinodkv) - - YARN-839. TestContainerLaunch.testContainerEnvVariables fails on Windows. - (Chuan Liu via cnauroth) - - YARN-597. TestFSDownload fails on Windows due to dependencies on - tar/gzip/jar tools. (Ivan Mitic via acmurthy) - - YARN-852. TestAggregatedLogFormat.testContainerLogsFileAccess fails on - Windows. (Chuan Liu via cnauroth) - - YARN-894. NodeHealthScriptRunner timeout checking is inaccurate on Windows. - (Chuan Liu via cnauroth) - - YARN-909. Disable TestLinuxContainerExecutorWithMocks on Windows. (Chuan Liu - via cnauroth) - - YARN-1043. Push all metrics consistently. (Jian He via acmurthy) - - YARN-1056. Remove dual use of string 'resourcemanager' in - yarn.resourcemanager.connect.{max.wait.secs|retry_interval.secs} - (Karthik Kambatla via acmurthy) - -Release 2.0.6-alpha - 08/22/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - YARN-854. Fixing YARN bugs that are failing applications in secure - environment. (Omkar Vinit Joshi and shv) - -Release 2.0.5-alpha - 06/06/2013 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - -Release 2.0.4-alpha - 2013-04-25 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - YARN-429. capacity-scheduler config missing from yarn-test artifact. - (sseth via hitesh) - - YARN-470. Support a way to disable resource monitoring on the NodeManager. - (Siddharth Seth via hitesh) - -Release 2.0.3-alpha - 2013-02-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - YARN-145. Add a Web UI to the fair share scheduler. (Sandy Ryza via tomwhite) - - YARN-3. Add support for CPU isolation/monitoring of containers. - (adferguson via tucu) - - YARN-230. RM Restart phase 1 - includes support for saving/restarting all - applications on an RM bounce. (Bikas Saha via acmurthy) - - YARN-103. Add a yarn AM-RM client module. (Bikas Saha via sseth) - - YARN-286. Add a YARN ApplicationClassLoader. (tomwhite) - - YARN-2. Enhanced CapacityScheduler to account for CPU alongwith memory for - multi-dimensional resource scheduling. (acmurthy) - - YARN-328. Use token request messages defined in hadoop common. (suresh) - - YARN-231. RM Restart - Add FS-based persistent store implementation for - RMStateStore (Bikas Saha via hitesh) - - IMPROVEMENTS - - YARN-223. Update process tree instead of getting new process trees. - (Radim Kolar via llu) - - YARN-57. Allow process-tree based resource calculation et al. to be - pluggable to support it on multiple platforms. (Radim Kolar via acmurthy) - - YARN-78. Changed UnManagedAM application to use YarnClient. (Bikas Saha via - vinodkv) - - YARN-93. Fixed RM to propagate diagnostics from applications that have - finished but failed (Jason Lowe via vinodkv). - - YARN-28. Fixed TestCompositeService to not depend on test-order and thus - made it pass on JDK7 (Thomas Graves via vinodkv). - - YARN-82. Change the default local and log dirs to be based on - hadoop.tmp.dir and yarn.log.dir. (Hemanth Yamijala via sseth) - - YARN-53. Added the missing getGroups API to ResourceManager. (Bo Wang via - vinodkv) - - YARN-116. Add the ability to change the RM include/exclude file without - a restart. (xieguiming and Harsh J via sseth) - - YARN-23. FairScheduler: FSQueueSchedulable#updateDemand() - potential - redundant aggregation. (kkambatl via tucu) - - YARN-127. Move RMAdmin tool to its correct location - the client module. - (vinodkv) - - YARN-40. Provided support for missing YARN commands (Devaraj K and Vinod - Kumar Vavilapalli via vinodkv) - - YARN-33. Change LocalDirsHandlerService to validate the configured local and - log dirs. (Mayank Bansal via sseth) - - YARN-94. Modify DistributedShell to point to main-class by default, clean up - the help message, and hard-code the AM class. (Hitesh Shah via vinodkv) - - YARN-146. Add unit tests for computing fair share in the fair scheduler. - (Sandy Ryza via tomwhite) - - HADOOP-8911. CRLF characters in source and text files. - (Raja Aluri via suresh) - - YARN-136. Make ClientToAMTokenSecretManager part of RMContext (Vinod Kumar - Vavilapalli via sseth) - - YARN-183. Clean up fair scheduler code. (Sandy Ryza via tomwhite) - - YARN-129. Simplify classpath construction for mini YARN tests. (tomwhite) - - YARN-254. Update fair scheduler web UI for hierarchical queues. - (sandyr via tucu) - - YARN-315. Using the common security token protobuf definition from hadoop - common. (Suresh Srinivas via vinodkv) - - YARN-170. Change NodeManager stop to be reentrant. (Sandy Ryza via vinodkv) - - YARN-331. Fill in missing fair scheduler documentation. (sandyr via tucu) - - YARN-277. Use AMRMClient in DistributedShell to exemplify the approach. - (Bikas Saha via hitesh) - - YARN-360. Allow apps to concurrently register tokens for renewal. - (Daryn Sharp via sseth) - - OPTIMIZATIONS - - BUG FIXES - - YARN-131. Fix incorrect ACL properties in capacity scheduler documentation. - (Ahmed Radwan via sseth) - - YARN-102. Move the apache header to the top of the file in MemStore.java. - (Devaraj K via sseth) - - YARN-134. ClientToAMSecretManager creates keys without checking for - validity of the appID. (Vinod Kumar Vavilapalli via sseth) - - YARN-30. Fixed tests verifying web-services to work on JDK7. (Thomas Graves - via vinodkv) - - YARN-150. Fixes AppRejectedTransition does not unregister a rejected - app-attempt from the ApplicationMasterService (Bikas Saha via sseth) - - YARN-140. Add capacity-scheduler-default.xml to provide a default set of - configurations for the capacity scheduler. (ahmed via tucu) - - YARN-179. Fix some unit test failures. (Vinod Kumar Vavilapalli via sseth) - - YARN-181. Fixed eclipse settings broken by capacity-scheduler.xml move via - YARN-140. (Siddharth Seth via vinodkv) - - YARN-169. Update log4j.appender.EventCounter to use - org.apache.hadoop.log.metrics.EventCounter (Anthony Rojas via tomwhite) - - YARN-184. Remove unnecessary locking in fair scheduler, and address - findbugs excludes. (sandyr via tucu) - - YARN-224. Fair scheduler logs too many nodeUpdate INFO messages. - (Sandy Ryza via tomwhite) - - YARN-222. Fair scheduler should create queue for each user by default. - (Sandy Ryza via tomwhite) - - MAPREDUCE-4778. Fair scheduler event log is only written if directory - exists on HDFS. (Sandy Ryza via tomwhite) - - YARN-229. Remove old unused RM recovery code. (Bikas Saha via acmurthy) - - YARN-187. Add hierarchical queues to the fair scheduler. - (Sandy Ryza via tomwhite) - - YARN-72. NM should handle cleaning up containers when it shuts down. - (Sandy Ryza via tomwhite) - - YARN-267. Fix fair scheduler web UI. (Sandy Ryza via tomwhite) - - YARN-264. y.s.rm.DelegationTokenRenewer attempts to renew token even - after removing an app. (kkambatl via tucu) - - YARN-271. Fair scheduler hits IllegalStateException trying to reserve - different apps on same node. (Sandy Ryza via tomwhite) - - YARN-272. Fair scheduler log messages try to print objects without - overridden toString methods. (sandyr via tucu) - - YARN-278. Fair scheduler maxRunningApps config causes no apps to make - progress. (sandyr via tucu) - - YARN-282. Fair scheduler web UI double counts Apps Submitted. - (sandyr via tucu) - - YARN-283. Fair scheduler fails to get queue info without root prefix. - (sandyr via tucu) - - YARN-192. Node update causes NPE in the fair scheduler. - (Sandy Ryza via tomwhite) - - YARN-288. Fair scheduler queue doesn't accept any jobs when ACLs are - configured. (Sandy Ryza via tomwhite) - - YARN-300. After YARN-271, fair scheduler can infinite loop and not - schedule any application. (Sandy Ryza via tomwhite) - - YARN-301. Fair scheduler throws ConcurrentModificationException when - iterating over app's priorities. (Sandy Ryza via tomwhite) - - YARN-217. Fix RMAdmin protocol description to make it work in secure mode - also. (Devaraj K via vinodkv) - - YARN-253. Fixed container-launch to not fail when there are no local - resources to localize. (Tom White via vinodkv) - - YARN-330. Fix flakey test: TestNodeManagerShutdown#testKillContainersOnShutdown. - (Sandy Ryza via hitesh) - - YARN-335. Fair scheduler doesn't check whether rack needs containers - before assigning to node. (Sandy Ryza via tomwhite) - - YARN-336. Fair scheduler FIFO scheduling within a queue only allows 1 - app at a time. (Sandy Ryza via tomwhite) - - YARN-135. Client tokens should be per app-attempt, and should be - unregistered on App-finish. (vinodkv via sseth) - - YARN-302. Fair scheduler assignmultiple should default to false. (sandyr via tucu) - - YARN-372. Move InlineDispatcher from hadoop-yarn-server-resourcemanager to - hadoop-yarn-common (sseth via hitesh) - - YARN-370. Fix SchedulerUtils to correctly round up the resource for - containers. (Zhijie Shen via acmurthy) - - YARN-355. Fixes a bug where RM app submission could jam under load. - (Daryn Sharp via sseth) - -Release 2.0.2-alpha - 2012-09-07 - - INCOMPATIBLE CHANGES - - YARN-9. Rename YARN_HOME to HADOOP_YARN_HOME. (vinodkv via acmurthy) - - NEW FEATURES - - YARN-1. Promote YARN to be a sub-project of Apache Hadoop. (acmurthy) - - IMPROVEMENTS - - YARN-29. Add a yarn-client module. (Vinod Kumar Vavilapalli via sseth) - - YARN-10. Fix DistributedShell module to not have a dependency on - hadoop-mapreduce-client-core. (Hitesh Shah via vinodkv) - - YARN-80. Add support for delaying rack-local containers in - CapacityScheduler. (acmurthy) - - YARN-137. Change the default YARN scheduler to be the CapacityScheduler. - (sseth via acmurthy) - - OPTIMAZATIONS - - BUG FIXES - - YARN-12. Fix findbugs warnings in FairScheduler. (Junping Du via acmurthy) - - YARN-22. Fix ContainerLogs to work if the log-dir is specified as a URI. - (Mayank Bansal via sseth) - - YARN-37. Change TestRMAppTransitions to use the DrainDispatcher. - (Mayank Bansal via sseth) - - YARN-79. Implement close on all clients to YARN so that RPC clients don't - throw exceptions on shut-down. (Vinod Kumar Vavilapalli) - - YARN-42. Modify NM's non-aggregating logs' handler to stop properly so that - NMs don't get NPEs on startup errors. (Devaraj K via vinodkv) - - YARN-15. Updated default classpath for YARN applications to reflect split of - YARN into a sub-project. (Arun C Murthy via vinodkv) - - YARN-75. Modified ResourceManager's RMContainer to handle a valid RELEASE - event at RUNNING state. (Siddharth Seth via vinodkv) - - MAPREDUCE-2374. "Text File Busy" errors launching MR tasks. (Andy Isaacson - via atm) - - YARN-138. Ensure default values for minimum/maximum container sizes is - sane. (harsh & sseth via acmurthy) - -Release 0.23.11 - UNRELEASED - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - BUG FIXES - - YARN-1180. Update capacity scheduler docs to include types on the configs - (Chen He via jeagles) - -Release 0.23.10 - 2013-12-09 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-985. Nodemanager should log where a resource was localized (Ravi - Prakash via jeagles) - - YARN-1119. Add ClusterMetrics checks to tho TestRMNodeTransitions tests - (Mit Desai via jeagles) - - OPTIMIZATIONS - - BUG FIXES - - YARN-337. RM handles killed application tracking URL poorly (jlowe) - - YARN-1101. Active nodes can be decremented below 0 (Robert Parker - via tgraves) - - YARN-1176. RM web services ClusterMetricsInfo total nodes doesn't include - unhealthy nodes (Jonathan Eagles via tgraves) - - YARN-1386. NodeManager mistakenly loses resources and relocalizes them - (Jason Lowe via jeagles) - -Release 0.23.9 - 2013-07-08 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-427. Coverage fix for org.apache.hadoop.yarn.server.api.* (Aleksey - Gorshkov via jeagles) - - YARN-478. fix coverage org.apache.hadoop.yarn.webapp.log (Aleksey Gorshkov - via jeagles) - - OPTIMIZATIONS - - BUG FIXES - -Release 0.23.8 - 2013-06-05 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMIZATIONS - - YARN-548. Add tests for YarnUncaughtExceptionHandler (Vadim Bondarev via - jeagles) - - BUG FIXES - - YARN-363. Add webapps/proxy directory without which YARN proxy-server fails - when started in stand-alone mode. (Kenji Kikushima via vinodkv) - - YARN-690. RM exits on token cancel/renew problems (daryn via bobby) - -Release 0.23.7 - 2013-04-18 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-133 Update web services docs for RM clusterMetrics (Ravi Prakash via - kihwal) - - YARN-249. Capacity Scheduler web page should show list of active users per - queue like it used to (in 1.x) (Ravi Prakash via tgraves) - - YARN-236. RM should point tracking URL to RM web page when app fails to - start (Jason Lowe via jeagles) - - YARN-269. Resource Manager not logging the health_check_script result when - taking it out (Jason Lowe via kihwal) - - YARN-227. Application expiration difficult to debug for end-users - (Jason Lowe via jeagles) - - YARN-443. allow OS scheduling priority of NM to be different than the - containers it launches (tgraves) - - YARN-468. coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter - (Aleksey Gorshkov via bobby) - - YARN-200. yarn log does not output all needed information, and is in a - binary format (Ravi Prakash via jlowe) - - YARN-525. make CS node-locality-delay refreshable (Thomas Graves via jlowe) - - OPTIMIZATIONS - - YARN-357. App submission should not be synchronized (daryn) - - BUG FIXES - - YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan - Gong via tgraves) - - YARN-364. AggregatedLogDeletionService can take too long to delete logs - (jlowe) - - YARN-362. Unexpected extra results when using webUI table search (Ravi - Prakash via jlowe) - - YARN-400. RM can return null application resource usage report leading to - NPE in client (Jason Lowe via tgraves) - - YARN-426. Failure to download a public resource prevents further downloads - (Jason Lowe via bobby) - - YARN-448. Remove unnecessary hflush from log aggregation (Kihwal Lee via - bobby) - - YARN-345. Many InvalidStateTransitonException errors for ApplicationImpl - in Node Manager (Robert Parker via jlowe) - - YARN-109. .tmp file is not deleted for localized archives (Mayank Bansal - via bobby) - - YARN-460. CS user left in list of active users for the queue even when - application finished (tgraves) - -Release 0.23.6 - 2013-02-06 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-285. Added a temporary plugin interface for RM to be able to redirect - to JobHistory server for apps that it no longer tracks. (Derek Dagit via - vinodkv) - - OPTIMIZATIONS - - BUG FIXES - - YARN-188. Coverage fixing for CapacityScheduler (Aleksey Gorshkov via - bobby) - - YARN-214. RMContainerImpl does not handle event EXPIRE at state RUNNING - (jeagles via bobby) - - YARN-151. Browser thinks RM main page JS is taking too long - (Ravi Prakash via bobby) - - YARN-204. test coverage for org.apache.hadoop.tools (Aleksey Gorshkov via - bobby) - - YARN-251. Proxy URI generation fails for blank tracking URIs (Tom White - via jlowe) - - YARN-258. RM web page UI shows Invalid Date for start and finish times - (Ravi Prakash via jlowe) - - YARN-266. RM and JHS Web UIs are blank because AppsBlock is not escaping - string properly (Ravi Prakash via jlowe) - - YARN-280. RM does not reject app submission with invalid tokens - (Daryn Sharp via tgraves) - - YARN-225. Proxy Link in RM UI thows NPE in Secure mode - (Devaraj K via bobby) - - YARN-293. Node Manager leaks LocalizerRunner object for every Container - (Robert Joseph Evans via jlowe) - - YARN-50. Implement renewal / cancellation of Delegation Tokens - (Siddharth Seth via tgraves) - - YARN-320. RM should always be able to renew its own tokens. - (Daryn Sharp via sseth) - - YARN-325. RM CapacityScheduler can deadlock when getQueueInfo() is - called and a container is completing (Arun C Murthy via tgraves) - - YARN-334. Maven RAT plugin is not checking all source files (tgraves) - - YARN-354. WebAppProxyServer exits immediately after startup (Liang Xie via - jlowe) - -Release 0.23.5 - 2012-11-28 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - YARN-161. Fix multiple compiler warnings for unchecked operations in YARN - common. (Chris Nauroth via vinodkv) - - YARN-43. Fix TestResourceTrackerService to not depend on test order and thus - pass on JDK7. (Thomas Graves via vinodkv) - - YARN-32. Fix TestApplicationTokens to not depend on test order and thus pass - on JDK7. (vinodkv) - - YARN-186. Coverage fixing LinuxContainerExecutor (Aleksey Gorshkov via - bobby) - - YARN-216. Remove jquery theming support. (Robert Joseph Evans via jlowe) - - OPTIMIZATIONS - - BUG FIXES - - YARN-163. Retrieving container log via NM webapp can hang with multibyte - characters in log (jlowe via bobby) - - YARN-174. Modify NodeManager to pass the user's configuration even when - rebooting. (vinodkv) - - YARN-177. CapacityScheduler - adding a queue while the RM is running has - wacky results (acmurthy vai tgraves) - - YARN-178. Fix custom ProcessTree instance creation (Radim Kolar via bobby) - - YARN-180. Capacity scheduler - containers that get reserved create - container token to early (acmurthy and bobby) - - YARN-139. Interrupted Exception within AsyncDispatcher leads to user - confusion. (Vinod Kumar Vavilapalli via jlowe) - - YARN-165. RM should point tracking URL to RM web page for app when AM fails - (jlowe via bobby) - - YARN-159. RM web ui applications page should be sorted to display last app - first (tgraves via bobby) - - YARN-166. capacity scheduler doesn't allow capacity < 1.0 (tgraves via - bobby) - - YARN-189. Fixed a deadlock between RM's ApplicationMasterService and the - dispatcher. (Thomas Graves via vinodkv) - - YARN-202. Log Aggregation generates a storm of fsync() for namenode - (Kihwal Lee via bobby) - - YARN-201. Fix CapacityScheduler to be less conservative for starved - off-switch requests. (jlowe via acmurthy) - - YARN-206. TestApplicationCleanup.testContainerCleanup occasionally fails. - (jlowe via jeagles) - - YARN-212. NM state machine ignores an APPLICATION_CONTAINER_FINISHED event - when it shouldn't (Nathan Roberts via jlowe) - - YARN-219. NM should aggregate logs when application finishes. (bobby) - -Release 0.23.4 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - Change package of YarnClient to org.apache.hadoop. (Bikas Saha via vinodkv) - - YARN-108. FSDownload can create cache directories with the wrong - permissions (Jason Lowe via bobby) - - OPTIMIZATIONS - - BUG FIXES - - YARN-88. DefaultContainerExecutor can fail to set proper permissions. - (Jason Lowe via sseth) - - YARN-106. Nodemanager needs to set permissions of local directories (jlowe - via bobby) - -Release 0.23.3 - - INCOMPATIBLE CHANGES - - NEW FEATURES - - IMPROVEMENTS - - OPTIMAZATIONS - - BUG FIXES - - YARN-14. Symlinks to peer distributed cache files no longer work - (Jason Lowe via bobby) - - YARN-25. remove old aggregated logs (Robert Evans via tgraves) - - YARN-27. Failed refreshQueues due to misconfiguration prevents further - refreshing of queues (Arun Murthy via tgraves) - - YARN-58. NM leaks filesystems (Jason Lowe via jeagles) - - YARN-39. RM-NM secret-keys should be randomly generated and rolled every - so often. (vinodkv and sseth via sseth) - - YARN-31. Fix TestDelegationTokenRenewer to not depend on test order so as to - pass tests on jdk7. (Thomas Graves via vinodkv) - - YARN-63. RMNodeImpl is missing valid transitions from the UNHEALTHY state - (Jason Lowe via bobby) - - YARN-60. Fixed a bug in ResourceManager which causes all NMs to get NPEs and - thus causes all containers to be rejected. (vinodkv) - - YARN-66. aggregated logs permissions not set properly (tgraves via bobby) - - YARN-68. NodeManager will refuse to shutdown indefinitely due to container - log aggregation (daryn via bobby) - - YARN-87. NM ResourceLocalizationService does not set permissions of local - cache directories (Jason Lowe via tgraves)