diff --git a/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html b/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
index 2044ee87b43..8ac55c1ce62 100644
--- a/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
+++ b/hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
@@ -1,7649 +1,10726 @@
-
-
-
-Hadoop 0.23.1 Release Notes
+Hadoop 2.0.2-alpha Release Notes
+ H1 {font-family: sans-serif}
+ H2 {font-family: sans-serif; margin-left: 7mm}
+ TABLE {margin-left: 7mm}
+
-Hadoop 0.23.1 Release Notes
- These release notes include new developer and user-facing incompatibilities, features, and major improvements.
-
+Hadoop 2.0.2-alpha Release Notes
+These release notes include new developer and user-facing incompatibilities, features, and major improvements.
-Changes since Hadoop 0.23.0
-
-Jiras with Release Notes (describe major or incompatible changes)
+Changes since Hadoop 2.0.1-alpha
-
-- HADOOP-7348.
- Major improvement reported by xiexianshan and fixed by xiexianshan (fs)
- Modify the option of FsShell getmerge from [addnl] to [-nl] for more comprehensive
- The 'fs -getmerge' tool now uses a -nl flag to determine if adding a newline at end of each file is required, in favor of the 'addnl' boolean flag that was used earlier.
-
-
-
-
-- HADOOP-7802.
- Major bug reported by bmahe and fixed by bmahe
- Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
- Here is a patch to enable this behavior
-
-
-
-
-- HADOOP-7963.
- Blocker bug reported by tgraves and fixed by sseth
- test failures: TestViewFileSystemWithAuthorityLocalFileSystem and TestViewFileSystemLocalFileSystem
- Fix ViewFS to catch a null canonical service-name and pass tests TestViewFileSystem*
-
-
-
-
-- HADOOP-7986.
- Major bug reported by mahadev and fixed by mahadev
- Add config for History Server protocol in hadoop-policy for service level authorization.
- Adding config for MapReduce History Server protocol in hadoop-policy.xml for service level authorization.
-
-
-
-
-- HDFS-1314.
- Minor bug reported by karims and fixed by sho.shimauchi
- dfs.blocksize accepts only absolute value
- The default blocksize property 'dfs.blocksize' now accepts unit symbols to be used instead of byte length. Values such as "10k", "128m", "1g" are now OK to provide instead of just no. of bytes as was before.
-
-
-
-
-- HDFS-2129.
- Major sub-task reported by tlipcon and fixed by tlipcon (hdfs client, performance)
- Simplify BlockReader to not inherit from FSInputChecker
- BlockReader has been reimplemented to use direct byte buffers. If you use a custom socket factory, it must generate sockets that have associated Channels.
-
-
-
-
-- HDFS-2130.
- Major sub-task reported by tlipcon and fixed by tlipcon (hdfs client)
- Switch default checksum to CRC32C
- The default checksum algorithm used on HDFS is now CRC32C. Data from previous versions of Hadoop can still be read backwards-compatibly.
-
-
-
-
-- HDFS-2246.
- Major improvement reported by sanjay.radia and fixed by jnp
- Shortcut a local client reads to a Datanodes files directly
- 1. New configurations
-
-a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read.
-
-b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration.
-
-c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side.
-
-2. By default none of the above are enabled and short circuit read will not kick in.
-
-3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general.
-
-
-
-
-- HDFS-2316.
- Major new feature reported by szetszwo and fixed by szetszwo
- [umbrella] WebHDFS: a complete FileSystem implementation for accessing HDFS over HTTP
- Provide WebHDFS as a complete FileSystem implementation for accessing HDFS over HTTP.
-
-Previous hftp feature was a read-only FileSystem and does not provide "write" accesses.
-
-
-- MAPREDUCE-778.
- Major new feature reported by hong.tang and fixed by amar_kamat (tools/rumen)
- [Rumen] Need a standalone JobHistory log anonymizer
- Added an anonymizer tool to Rumen. Anonymizer takes a Rumen trace file and/or topology as input. It supports persistence and plugins to override the default behavior.
-
-
-
-
-- MAPREDUCE-2733.
- Major task reported by vinaythota and fixed by vinaythota
- Gridmix v3 cpu emulation system tests.
- Adds system tests for the CPU emulation feature in Gridmix3.
-
-
-
-
-- MAPREDUCE-2765.
- Major new feature reported by mithun and fixed by mithun (distcp, mrv2)
- DistCp Rewrite
- DistCpV2 added to hadoop-tools.
-
-
-
-
-- MAPREDUCE-2784.
- Major bug reported by amar_kamat and fixed by amar_kamat (contrib/gridmix)
- [Gridmix] TestGridmixSummary fails with NPE when run in DEBUG mode.
- Fixed bugs in ExecutionSummarizer and ResourceUsageMatcher.
-
-
-
-
-- MAPREDUCE-2863.
- Blocker improvement reported by acmurthy and fixed by tgraves (mrv2, nodemanager, resourcemanager)
- Support web-services for RM & NM
- Support for web-services in YARN and MR components.
-
-
-
-
-- MAPREDUCE-2950.
- Major bug reported by amar_kamat and fixed by ravidotg (contrib/gridmix)
- [Gridmix] TestUserResolve fails in trunk
- Fixes bug in TestUserResolve.
-
-
-
-
-- MAPREDUCE-3102.
- Major sub-task reported by vinodkv and fixed by hitesh (mrv2, security)
- NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
- Changed NodeManager to fail fast when LinuxContainerExecutor has wrong configuration or permissions.
-
-
-
-
-- MAPREDUCE-3215.
- Minor sub-task reported by hitesh and fixed by hitesh (mrv2)
- org.apache.hadoop.mapreduce.TestNoJobSetupCleanup failing on trunk
- Reneabled and fixed bugs in the failing test TestNoJobSetupCleanup.
-
-
-
-
-- MAPREDUCE-3217.
- Minor sub-task reported by hitesh and fixed by devaraj.k (mrv2, test)
- ant test TestAuditLogger fails on trunk
- Reenabled and fixed bugs in the failing ant test TestAuditLogger.
-
-
-
-
-- MAPREDUCE-3219.
- Minor sub-task reported by hitesh and fixed by hitesh (mrv2, test)
- ant test TestDelegationToken failing on trunk
- Reenabled and fixed bugs in the failing test TestDelegationToken.
-
-
-
-
-- MAPREDUCE-3221.
- Minor sub-task reported by hitesh and fixed by devaraj.k (mrv2, test)
- ant test TestSubmitJob failing on trunk
- Fixed a bug in TestSubmitJob.
-
-
-
-
-- MAPREDUCE-3280.
- Major bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
- MR AM should not read the username from configuration
- Removed the unnecessary job user-name configuration in mapred-site.xml.
-
-
-
-
-- MAPREDUCE-3297.
- Major task reported by sseth and fixed by sseth (mrv2)
- Move Log Related components from yarn-server-nodemanager to yarn-common
- Moved log related components into yarn-common so that HistoryServer and clients can use them without depending on the yarn-server-nodemanager module.
-
-
-
-
-- MAPREDUCE-3299.
- Minor improvement reported by sseth and fixed by jeagles (mrv2)
- Add AMInfo table to the AM job page
- Added AMInfo table to the MR AM job pages to list all the job-attempts when AM restarts and recovers.
-
-
-
-
-- MAPREDUCE-3312.
- Major bug reported by revans2 and fixed by revans2 (mrv2)
- Make MR AM not send a stopContainer w/o corresponding start container
- Modified MR AM to not send a stop-container request for a container that isn't launched at all.
-
-
-
-
-- MAPREDUCE-3325.
- Major improvement reported by tgraves and fixed by tgraves (mrv2)
- Improvements to CapacityScheduler doc
- document changes only.
-
-
-
-
-- MAPREDUCE-3333.
- Blocker bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
- MR AM for sort-job going out of memory
- Fixed bugs in ContainerLauncher of MR AppMaster due to which per-container connections to NodeManager were lingering long enough to hit the ulimits on number of processes.
-
-
-
-
-- MAPREDUCE-3339.
- Blocker bug reported by ramgopalnaali and fixed by sseth (mrv2)
- Job is getting hanged indefinitely,if the child processes are killed on the NM. KILL_CONTAINER eventtype is continuosly sent to the containers that are not existing
- Fixed MR AM to stop considering node blacklisting after the number of nodes blacklisted crosses a threshold.
-
-
-
-
-- MAPREDUCE-3342.
- Critical bug reported by tgraves and fixed by jeagles (jobhistoryserver, mrv2)
- JobHistoryServer doesn't show job queue
- Fixed JobHistoryServer to also show the job's queue name.
-
-
-
-
-- MAPREDUCE-3345.
- Major bug reported by vinodkv and fixed by hitesh (mrv2, resourcemanager)
- Race condition in ResourceManager causing TestContainerManagerSecurity to fail sometimes
- Fixed a race condition in ResourceManager that was causing TestContainerManagerSecurity to fail sometimes.
-
-
-
-
-- MAPREDUCE-3349.
- Blocker bug reported by vinodkv and fixed by amar_kamat (mrv2)
- No rack-name logged in JobHistory for unsuccessful tasks
- Unsuccessful tasks now log hostname and rackname to job history.
-
-
-
-
-- MAPREDUCE-3355.
- Blocker bug reported by vinodkv and fixed by vinodkv (applicationmaster, mrv2)
- AM scheduling hangs frequently with sort job on 350 nodes
- Fixed MR AM's ContainerLauncher to handle node-command timeouts correctly.
-
-
-
-
-- MAPREDUCE-3360.
- Critical improvement reported by kam_iitkgp and fixed by kamesh (mrv2)
- Provide information about lost nodes in the UI.
- Added information about lost/rebooted/decommissioned nodes on the webapps.
-
-
-
-
-- MAPREDUCE-3368.
- Critical bug reported by rramya and fixed by hitesh (build, mrv2)
- compile-mapred-test fails
- Fixed ant test compilation.
-
-
-
-
-- MAPREDUCE-3375.
- Major task reported by vinaythota and fixed by vinaythota
- Memory Emulation system tests.
- Added system tests to test the memory emulation feature in Gridmix.
-
-
-
-
-- MAPREDUCE-3379.
- Major bug reported by sseth and fixed by sseth (mrv2, nodemanager)
- LocalResourceTracker should not tracking deleted cache entries
- Fixed LocalResourceTracker in NodeManager to remove deleted cache entries correctly.
-
-
-
-
-- MAPREDUCE-3382.
- Critical bug reported by vinodkv and fixed by raviprak (applicationmaster, mrv2)
- Network ACLs can prevent AMs to ping the Job-end notification URL
- Enhanced MR AM to use a proxy to ping the job-end notification URL.
-
-
-
-
-- MAPREDUCE-3387.
- Critical bug reported by revans2 and fixed by revans2 (mrv2)
- A tracking URL of N/A before the app master is launched breaks oozie
- Fixed AM's tracking URL to always go through the proxy, even before the job started, so that it works properly with oozie throughout the job execution.
-
-
-
-
-- MAPREDUCE-3392.
- Blocker sub-task reported by johnvijoe and fixed by johnvijoe
- Cluster.getDelegationToken() throws NPE if client.getDelegationToken() returns null.
- Fixed Cluster's getDelegationToken's API to return null when there isn't a supported token.
-
-
-
-
-- MAPREDUCE-3398.
- Blocker bug reported by sseth and fixed by sseth (mrv2, nodemanager)
- Log Aggregation broken in Secure Mode
- Fixed log aggregation to work correctly in secure mode. Contributed by Siddharth Seth.
-
-
-
-
-- MAPREDUCE-3399.
- Blocker sub-task reported by sseth and fixed by sseth (mrv2, nodemanager)
- ContainerLocalizer should request new resources after completing the current one
- Modified ContainerLocalizer to send a heartbeat to NM immediately after downloading a resource instead of always waiting for a second.
-
-
-
-
-- MAPREDUCE-3404.
- Critical bug reported by patwhitey2007 and fixed by eepayne (job submission, mrv2)
- Speculative Execution: speculative map tasks launched even if -Dmapreduce.map.speculative=false
- Corrected MR AM to honor speculative configuration and enable speculating either maps or reduces.
-
-
-
-
-- MAPREDUCE-3407.
- Minor bug reported by hitesh and fixed by hitesh (mrv2)
- Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
- Fixed pom files to refer to the correct MR app-jar needed by the integration tests.
-
-
-
-
-- MAPREDUCE-3412.
- Major bug reported by amar_kamat and fixed by amar_kamat
- 'ant docs' is broken
- Fixes 'ant docs' by removing stale references to capacity-scheduler docs.
-
-
-
-
-- MAPREDUCE-3417.
- Blocker bug reported by tgraves and fixed by jeagles (mrv2)
- job access controls not working app master and job history UI's
- Fixed job-access-controls to work with MR AM and JobHistoryServer web-apps.
-
-
-
-
-- MAPREDUCE-3426.
- Blocker sub-task reported by hitesh and fixed by hitesh (mrv2)
- uber-jobs tried to write outputs into wrong dir
- Fixed MR AM in uber mode to write map intermediate outputs in the correct directory to work properly in secure mode.
-
-
-
-
-- MAPREDUCE-3462.
- Blocker bug reported by amar_kamat and fixed by raviprak (mrv2, test)
- Job submission failing in JUnit tests
- Fixed failing JUnit tests in Gridmix.
-
-
-
-
-- MAPREDUCE-3481.
- Major improvement reported by amar_kamat and fixed by amar_kamat (contrib/gridmix)
- [Gridmix] Improve STRESS mode locking
- Modified Gridmix STRESS mode locking structure. The submitted thread and the polling thread now run simultaneously without blocking each other.
-
-
-
-
-- MAPREDUCE-3484.
- Major bug reported by raviprak and fixed by raviprak (mr-am, mrv2)
- JobEndNotifier is getting interrupted before completing all its retries.
- Fixed JobEndNotifier to not get interrupted before completing all its retries.
-
-
-
-
-- MAPREDUCE-3487.
- Critical bug reported by tgraves and fixed by jlowe (mrv2)
- jobhistory web ui task counters no longer links to singletakecounter page
- Fixed JobHistory web-UI to display links to single task's counters' page.
-
-
-
-
-- MAPREDUCE-3490.
- Blocker bug reported by sseth and fixed by sharadag (mr-am, mrv2)
- RMContainerAllocator counts failed maps towards Reduce ramp up
- Fixed MapReduce AM to count failed maps also towards Reduce ramp up.
-
-
-
-
-- MAPREDUCE-3511.
- Blocker sub-task reported by sseth and fixed by vinodkv (mr-am, mrv2)
- Counters occupy a good part of AM heap
- Removed a multitude of cloned/duplicate counters in the AM thereby reducing the AM heap size and preventing full GCs.
-
-
-
-
-- MAPREDUCE-3512.
- Blocker sub-task reported by sseth and fixed by sseth (mr-am, mrv2)
- Batch jobHistory disk flushes
- Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM.
-
-
-
-
-- MAPREDUCE-3519.
- Blocker sub-task reported by ravidotg and fixed by ravidotg (mrv2, nodemanager)
- Deadlock in LocalDirsHandlerService and ShuffleHandler
- Fixed a deadlock in NodeManager LocalDirectories's handling service.
-
-
-
-
-- MAPREDUCE-3528.
- Major bug reported by sseth and fixed by sseth (mr-am, mrv2)
- The task timeout check interval should be configurable independent of mapreduce.task.timeout
- Fixed TaskHeartBeatHandler to use a new configuration for the thread loop interval separate from task-timeout configuration property.
-
-
-
-
-- MAPREDUCE-3530.
- Blocker bug reported by karams and fixed by acmurthy (mrv2, resourcemanager, scheduler)
- Sometimes NODE_UPDATE to the scheduler throws an NPE causing the scheduling to stop
- Fixed an NPE occuring during scheduling in the ResourceManager.
-
-
-
-
-- MAPREDUCE-3532.
- Critical bug reported by karams and fixed by kamesh (mrv2, nodemanager)
- When 0 is provided as port number in yarn.nodemanager.webapp.address, NMs webserver component picks up random port, NM keeps on Reporting 0 port to RM
- Modified NM to report correct http address when an ephemeral web port is configured.
-
-
-
-
-- MAPREDUCE-3549.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- write api documentation for web service apis for RM, NM, mapreduce app master, and job history server
- new files added: A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm
-
-A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/NodeManagerRest.apt.vm
-
-A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm
-
-A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm
-
-A hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
-
-
-
-The hadoop-project/src/site/site.xml is split into separate patch.
-
-
-- MAPREDUCE-3564.
- Blocker bug reported by mahadev and fixed by sseth (mrv2)
- TestStagingCleanup and TestJobEndNotifier are failing on trunk.
- Fixed failures in TestStagingCleanup and TestJobEndNotifier tests.
-
-
-
-
-- MAPREDUCE-3568.
- Critical sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2, performance)
- Optimize Job's progress calculations in MR AM
- Optimized Job's progress calculations in MR AM.
-
-
-
-
-- MAPREDUCE-3586.
- Blocker bug reported by vinodkv and fixed by vinodkv (mr-am, mrv2)
- Lots of AMs hanging around in PIG testing
- Modified CompositeService to avoid duplicate stop operations thereby solving race conditions in MR AM shutdown.
-
-
-
-
-- MAPREDUCE-3597.
- Major improvement reported by ravidotg and fixed by ravidotg (tools/rumen)
- Provide a way to access other info of history file from Rumentool
- Rumen now provides {{Parsed*}} objects. These objects provide extra information that are not provided by {{Logged*}} objects.
-
-
-
-
-- MAPREDUCE-3618.
- Major sub-task reported by sseth and fixed by sseth (mrv2, performance)
- TaskHeartbeatHandler holds a global lock for all task-updates
- Fixed TaskHeartbeatHandler to not hold a global lock for all task-updates.
-
-
-
-
-- MAPREDUCE-3630.
- Critical task reported by amolkekre and fixed by mahadev (mrv2)
- NullPointerException running teragen
- Committed to trunk and branch-0.23. Thanks Mahadev.
-
-
-
-
-- MAPREDUCE-3639.
- Blocker bug reported by sseth and fixed by sseth (mrv2)
- TokenCache likely broken for FileSystems which don't issue delegation tokens
- Fixed TokenCache to work with absent FileSystem canonical service-names.
-
-
-
-
-- MAPREDUCE-3641.
- Blocker sub-task reported by acmurthy and fixed by acmurthy (mrv2, scheduler)
- CapacityScheduler should be more conservative assigning off-switch requests
- Making CapacityScheduler more conservative so as to assign only one off-switch container in a single scheduling iteration.
-
-
-
-
-- MAPREDUCE-3656.
- Blocker bug reported by karams and fixed by sseth (applicationmaster, mrv2, resourcemanager)
- Sort job on 350 scale is consistently failing with latest MRV2 code
- Fixed a race condition in MR AM which is failing the sort benchmark consistently.
-
-
-
-
-- MAPREDUCE-3699.
- Major bug reported by vinodkv and fixed by hitesh (mrv2)
- Default RPC handlers are very low for YARN servers
- Increased RPC handlers for all YARN servers to reasonable values for working at scale.
-
-
-
-
-- MAPREDUCE-3703.
- Critical bug reported by eepayne and fixed by eepayne (mrv2, resourcemanager)
- ResourceManager should provide node lists in JMX output
- New JMX Bean in ResourceManager to provide list of live node managers:
-
-
-
-Hadoop:service=ResourceManager,name=RMNMInfo LiveNodeManagers
-
-
-- MAPREDUCE-3710.
- Major bug reported by sseth and fixed by sseth (mrv1, mrv2)
- last split generated by FileInputFormat.getSplits may not have the best locality
- Improved FileInputFormat to return better locality for the last split.
-
-
-
-
-- MAPREDUCE-3711.
- Blocker sub-task reported by sseth and fixed by revans2 (mrv2)
- AppMaster recovery for Medium to large jobs take long time
- Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time.
-
-
-
-
-- MAPREDUCE-3713.
- Blocker bug reported by sseth and fixed by acmurthy (mrv2, resourcemanager)
- Incorrect headroom reported to jobs
- Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application.
-
-
-
-
-- MAPREDUCE-3714.
- Blocker bug reported by vinodkv and fixed by vinodkv (mrv2, task)
- Reduce hangs in a corner case
- Fixed EventFetcher and Fetcher threads to shut-down properly so that reducers don't hang in corner cases.
-
-
-
-
-- MAPREDUCE-3716.
- Blocker bug reported by jeagles and fixed by jeagles (mrv2)
- java.io.File.createTempFile fails in map/reduce tasks
- Fixing YARN+MR to allow MR jobs to be able to use java.io.File.createTempFile to create temporary files as part of their tasks.
-
-
-
-
-- MAPREDUCE-3720.
- Major bug reported by vinodkv and fixed by vinodkv (client, mrv2)
- Command line listJobs should not visit each AM
- Changed bin/mapred job -list to not print job-specific information not available at RM.
-
-
-
-Very minor incompatibility in cmd-line output, inevitable due to MRv2 architecture.
-
-
-- MAPREDUCE-3732.
- Blocker bug reported by acmurthy and fixed by acmurthy (mrv2, resourcemanager, scheduler)
- CS should only use 'activeUsers with pending requests' for computing user-limits
- Modified CapacityScheduler to use only users with pending requests for computing user-limits.
-
-
-
-
-- MAPREDUCE-3752.
- Blocker bug reported by acmurthy and fixed by acmurthy (mrv2)
- Headroom should be capped by queue max-cap
- Modified application limits to include queue max-capacities besides the usual user limits.
-
-
-
-
-- MAPREDUCE-3754.
- Major bug reported by vinodkv and fixed by vinodkv (mrv2, webapps)
- RM webapp should have pages filtered based on App-state
- Modified RM UI to filter applications based on state of the applications.
-
-
-
-
-- MAPREDUCE-3760.
- Major bug reported by rramya and fixed by vinodkv (mrv2)
- Blacklisted NMs should not appear in Active nodes list
- Changed active nodes list to not contain unhealthy nodes on the webUI and metrics.
-
-
-
-
-- MAPREDUCE-3774.
- Major bug reported by mahadev and fixed by mahadev (mrv2)
- yarn-default.xml should be moved to hadoop-yarn-common.
-
-
-
-- MAPREDUCE-3784.
- Major bug reported by rramya and fixed by acmurthy (mrv2)
- maxActiveApplications(|PerUser) per queue is too low for small clusters
- Fixed CapacityScheduler so that maxActiveApplication and maxActiveApplicationsPerUser per queue are not too low for small clusters.
-
-
-
-
-- MAPREDUCE-3804.
- Major bug reported by davet and fixed by davet (jobhistoryserver, mrv2, resourcemanager)
- yarn webapp interface vulnerable to cross scripting attacks
- fix cross scripting attacks vulnerability through webapp interface.
-
-
-
-
-- MAPREDUCE-3808.
- Blocker bug reported by sseth and fixed by revans2 (mrv2)
- NPE in FileOutputCommitter when running a 0 reduce job
- Fixed an NPE in FileOutputCommitter for jobs with maps but no reduces.
-
-
-
-
-- MAPREDUCE-3815.
- Critical sub-task reported by sseth and fixed by sseth (mrv2)
- Data Locality suffers if the AM asks for containers using IPs instead of hostnames
- Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly.
-
-
-
-
-
-
-Other Jiras (describe bug fixes and minor changes)
-
-
-- HADOOP-4515.
- Minor improvement reported by abagri and fixed by sho.shimauchi
- conf.getBoolean must be case insensitive
- Currently, if xx is set to "TRUE", conf.getBoolean("xx", false) would return false.
conf.getBoolean should do an equalsIgnoreCase() instead of equals()
I am marking the change as incompatible because it does change semantics as pointed by Steve in HADOOP-4416
-
-- HADOOP-6490.
- Minor bug reported by zshao and fixed by umamaheswararao (fs)
- Path.normalize should use StringUtils.replace in favor of String.replace
- in our environment, we are seeing that the JobClient is going out of memory because Path.normalizePath(String) is called several tens of thousands of times, and each time it calls "String.replace" twice.
java.lang.String.replace compiles a regex to do the job which is very costly.
We should use org.apache.commons.lang.StringUtils.replace which is much faster and consumes almost no extra memory.
-
-- HADOOP-6614.
- Minor improvement reported by stevel@apache.org and fixed by jmhsieh (util)
- RunJar should provide more diags when it can't create a temp file
- When you see a stack trace about permissions, it is better if the trace included the file/directory at fault:
{code}
Exception in thread "main" java.io.IOException: Permission denied
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.checkAndCreate(File.java:1704)
at java.io.File.createTempFile(File.java:1792)
at org.apache.hadoop.util.RunJar.main(RunJar.java:147)
{code}
As it is, you need to go into the code, discover that it's {{${hadoop.tmp.dir}/hadoop-unja...
-
-- HADOOP-6840.
- Minor improvement reported by nspiegelberg and fixed by nspiegelberg (fs, io)
- Support non-recursive create() in FileSystem & SequenceFile.Writer
- The proposed solution for HBASE-2312 requires the sequence file to handle a non-recursive create. This is already supported by HDFS, but needs to have an equivalent FileSystem & SequenceFile.Writer API.
-
-- HADOOP-6886.
- Minor improvement reported by nspiegelberg and fixed by nspiegelberg (fs)
- LocalFileSystem Needs createNonRecursive API
- While running sanity check tests for HBASE-2312, I noticed that HDFS-617 did not include createNonRecursive() support for the LocalFileSystem. This is a problem for HBase, which allows the user to run over the LocalFS instead of HDFS for local cluster testing. I think this only affects 0.20-append, but may affect the trunk based upon how exactly FileContext handles non-recursive creates.
-
-- HADOOP-7424.
- Major improvement reported by eli and fixed by umamaheswararao
- Log an error if the topology script doesn't handle multiple args
- ScriptBasedMapping#resolve currently warns and returns null if it passes n arguments to the topology script and gets back a different number of resolutions. This indicates a bug in the topology script (or it's input) and therefore should be an error.
{code}
// invalid number of entries returned by the script
LOG.warn("Script " + scriptName + " returned "
+ Integer.toString(m.size()) + " values when "
+ Integer.toString(names.size()) + " were expected.");
return null;
{code}
There's on...
-
-- HADOOP-7470.
- Minor improvement reported by stevel@apache.org and fixed by enis (util)
- move up to Jackson 1.8.8
- I see that hadoop-core still depends on Jackson 1.0.1 -but that project is now up to 1.8.2 in releases. Upgrading will make it easier for other Jackson-using apps that are more up to date to keep their classpath consistent.
The patch would be updating the ivy file to pull in the later version; no test
-
-- HADOOP-7504.
- Trivial improvement reported by eli and fixed by qwertymaniac (metrics)
- hadoop-metrics.properties missing some Ganglia31 options
- The "jvm", "rpc", and "ugi" sections of hadoop-metrics.properties should have Ganglia31 options like "dfs" and "mapred"
-
-- HADOOP-7574.
- Trivial improvement reported by xiexianshan and fixed by xiexianshan (fs)
- Improvement for FSshell -stat
- Add two optional formats for FSshell -stat, one is %G for group name of owner and the other is %U for user name.
-
-- HADOOP-7590.
- Major sub-task reported by tucu00 and fixed by tucu00 (build)
- Mavenize streaming and MR examples
- MR1 code is still available in MR2 for testing contribs.
While this is a temporary until contribs tests are ported to MR2.
As a follow up the contrib projects themselves should be mavenized.
-
-- HADOOP-7657.
- Major improvement reported by mrbsd and fixed by decster
- Add support for LZ4 compression
- According to several benchmark sites, LZ4 seems to overtake other fast compression algorithms, especially in the decompression speed area. The interface is also trivial to integrate (http://code.google.com/p/lz4/source/browse/trunk/lz4.h) and there is no license issue.
-
-- HADOOP-7736.
- Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (fs)
- Remove duplicate call of Path#normalizePath during initialization.
- Found during code reading on HADOOP-6490, there seems to be an unnecessary call of {{normalizePath(...)}} being made in the constructor {{Path(Path, Path)}}. Since {{initialize(...)}} normalizes its received path string already, its unnecessary to do it to the path parameter in the constructor's call of the same.
-
-- HADOOP-7758.
- Major improvement reported by tucu00 and fixed by tucu00 (fs)
- Make GlobFilter class public
- Currently the GlobFilter class is package private.
As a generic filter it is quite useful (and I've found myself doing cut&paste of it a few times)
-
-- HADOOP-7761.
- Major improvement reported by tlipcon and fixed by tlipcon (io, performance, util)
- Improve performance of raw comparisons
- Guava has a nice implementation of lexicographical byte-array comparison that uses sun.misc.Unsafe to compare unsigned byte arrays long-at-a-time. Their benchmarks show it as being 2x more CPU-efficient than the equivalent pure-Java implementation. We can easily integrate this into WritableComparator.compareBytes to improve CPU performance in the shuffle.
-
-- HADOOP-7777.
- Major improvement reported by stevel@apache.org and fixed by stevel@apache.org (util)
- Implement a base class for DNSToSwitchMapping implementations that can offer extra topology information
- HDFS-2492 has identified a need for DNSToSwitchMapping implementations to provide a bit more topology information (e.g. whether or not there are multiple switches). This could be done by writing an extended interface, querying its methods if present and coming up with a default action if there is no extended interface.
Alternatively, we have a base class that all the standard mappings implement, with a boolean isMultiRack() method; all the standard subclasses would extend this, as could any...
-
-- HADOOP-7787.
- Major bug reported by bmahe and fixed by bmahe (build)
- Make source tarball use conventional name.
- When building binary and source tarballs, I get the following artifacts:
Binary tarball: hadoop-0.23.0-SNAPSHOT.tar.gz
Source tarball: hadoop-dist-0.23.0-SNAPSHOT-src.tar.gz
Notice the "-dist" right between "hadoop" and the version in the source tarball name.
-
-- HADOOP-7801.
- Major bug reported by bmahe and fixed by bmahe (build)
- HADOOP_PREFIX cannot be overriden
- hadoop-config.sh forces HADOOP_prefix to a specific value:
export HADOOP_PREFIX=`dirname "$this"`/..
It would be nice to make this overridable.
-
-- HADOOP-7804.
- Major improvement reported by arpitgupta and fixed by arpitgupta (conf)
- enable hadoop config generator to set dfs.block.local-path-access.user to enable short circuit read
- we have a new config that allows to select which user can have access for short circuit read. We should make that configurable through the config generator scripts.
-
-- HADOOP-7808.
- Major new feature reported by daryn and fixed by daryn (fs, security)
+ - YARN-137.
+ Major improvement reported by Siddharth Seth and fixed by Siddharth Seth (scheduler)
+ Change the default scheduler to the CapacityScheduler
+ There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues.
+That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues.
+
+- YARN-108.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ FSDownload can create cache directories with the wrong permissions
+ When the cluster is configured with a restrictive umask, e.g.: {{fs.permissions.umask-mode=0077}}, the nodemanager can end up creating directory entries in the public cache with the wrong permissions. The permissions can end up where only the nodemanager user can access files in the public cache, preventing jobs from running properly.
+- YARN-106.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ Nodemanager needs to set permissions of local directories
+ If the nodemanager process is running with a restrictive default umask (e.g.: 0077) then it will create its local directories with permissions that are too restrictive to allow containers from other users to run.
+- YARN-88.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ DefaultContainerExecutor can fail to set proper permissions
+ {{DefaultContainerExecutor}} can fail to set the proper permissions on its local directories if the cluster has been configured with a restrictive umask, e.g.: fs.permissions.umask-mode=0077. The configured umask ends up defeating the permissions requested by {{DefaultContainerExecutor}} when it creates directories.
+- YARN-87.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ NM ResourceLocalizationService does not set permissions of local cache directories
+ {{ResourceLocalizationService}} creates a file cache and user cache directory when it starts up but doesn't specify the permissions for them when they are created. If the cluster configs are set to limit the default permissions (e.g.: fs.permissions.umask-mode=0077 instead of the default 0022), then the cache directories are created with too-restrictive permissions and no jobs are able to run.
+- YARN-83.
+ Major bug reported by Bikas Saha and fixed by Bikas Saha (client)
+ Change package of YarnClient to include apache
+ Currently its org.hadoop.* instead of org.apache.hadoop.*
+- YARN-80.
+ Major improvement reported by Todd Lipcon and fixed by Arun C Murthy (capacityscheduler)
+ Support delay scheduling for node locality in MR2's capacity scheduler
+ The capacity scheduler in MR2 doesn't support delay scheduling for achieving node-level locality. So, jobs exhibit poor data locality even if they have good rack locality. Especially on clusters where disk throughput is much better than network capacity, this hurts overall job performance. We should optionally support node-level delay scheduling heuristics similar to what the fair scheduler implements in MR1.
+- YARN-79.
+ Major bug reported by Bikas Saha and fixed by Vinod Kumar Vavilapalli (client)
+ Calling YarnClientImpl.close throws Exception
+ The following exception is thrown
+===========
+*org.apache.hadoop.HadoopIllegalArgumentException: Cannot close proxy - is not Closeable or does not provide closeable invocation handler class org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl*
+ *at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:624)*
+ *at org.hadoop.yarn.client.YarnClientImpl.stop(YarnClientImpl.java:102)*
+ at org.apache.hadoop.yarn.applications.unmanagedamlauncher.UnmanagedAMLauncher.run(UnmanagedAMLauncher.java:336)
+ at org.apache.hadoop.yarn.applications.unmanagedamlauncher.TestUnmanagedAMLauncher.testDSShell(TestUnmanagedAMLauncher.java:156)
+ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+ at java.lang.reflect.Method.invoke(Method.java:597)
+ at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
+ at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
+ at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
+ at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
+ at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
+ at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
+ at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
+ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
+ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
+ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
+ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
+ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
+ at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
+ at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
+ at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
+ at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
+ at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
+ at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
+ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
+ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
+ at java.lang.reflect.Method.invoke(Method.java:597)
+ at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
+ at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
+ at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
+ at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
+ at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
+===========
+- YARN-75.
+ Major bug reported by Siddharth Seth and fixed by Siddharth Seth
+ RMContainer should handle a RELEASE event while RUNNING
+ An AppMaster can send a container release at any point. Currently this results in an exception, if this is done while the RM considers the container to be RUNNING.
+The event not being processed correctly also implies that these containers do not show up in the Completed Container List seen by the AM (AMRMProtocol). MR-3902 depends on this set being complete.
+- YARN-68.
+ Major bug reported by patrick white and fixed by Daryn Sharp (nodemanager)
+ NodeManager will refuse to shutdown indefinitely due to container log aggregation
+ The nodemanager is able to get into a state where containermanager.logaggregation.AppLogAggregatorImpl will apparently wait
+indefinitely for log aggregation to complete for an application, even if that application has abnormally terminated and is no longer present.
+
+Observed behavior is that an attempt to stop the nodemanager daemon will return but have no effect, the nm log continually displays messages similar to this:
+
+[Thread-1]2012-08-21 17:44:07,581 INFO
+org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl:
+Waiting for aggregation to complete for application_1345221477405_2733
+
+The only recovery we found to work was to 'kill -9' the nm process.
+
+What exactly causes the NM to enter this state is unclear but we do see this behavior reliably when the NM has run a task which failed, for example when debugging oozie distcp actions and having a distcp map task fail, the NM that was running the container will now enter this state where a shutdown on said NM will never complete, 'never' in this case was waiting for 2 hours before killing the nodemanager process.
+
+- YARN-66.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (nodemanager)
+ aggregated logs permissions not set properly
+ If the default file permissions are set to something restrictive - like 700, application logs get aggregated and created with those restrictive file permissions which doesn't allow the history server to serve them up.
+
+
+They need to be created with group readable similar to how log aggregation sets up the directory permissions.
+
+- YARN-63.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
+ RMNodeImpl is missing valid transitions from the UNHEALTHY state
+ The ResourceManager isn't properly handling nodes that have been marked UNHEALTHY when they are lost or are decommissioned.
+- YARN-60.
+ Blocker sub-task reported by Daryn Sharp and fixed by Vinod Kumar Vavilapalli (nodemanager)
+ NMs rejects all container tokens after secret key rolls
+ The NM's token secret manager will reject all container tokens after the secret key is activated which means the NM will not launch _any_ containers including AMs. The whole yarn cluster becomes inoperable in 1d.
+- YARN-58.
+ Critical bug reported by Daryn Sharp and fixed by Jason Lowe (nodemanager)
+ NM leaks filesystems
+ The NM is exhausting its fds because it's not closing fs instances when the app is finished.
+- YARN-42.
+ Major bug reported by Devaraj K and fixed by Devaraj K (nodemanager)
+ Node Manager throws NPE on startup
+ NM throws NPE on startup if it doesn't have persmission's on nm local dir's
+
+
+{code:xml}
+2012-05-14 16:32:13,468 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
+org.apache.hadoop.yarn.YarnException: Failed to initialize LocalizationService
+ at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:202)
+ at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58)
+ at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.init(ContainerManagerImpl.java:183)
+ at org.apache.hadoop.yarn.service.CompositeService.init(CompositeService.java:58)
+ at org.apache.hadoop.yarn.server.nodemanager.NodeManager.init(NodeManager.java:166)
+ at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:268)
+ at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:284)
+Caused by: java.io.IOException: mkdir of /mrv2/tmp/nm-local-dir/usercache failed
+ at org.apache.hadoop.fs.FileSystem.primitiveMkdir(FileSystem.java:907)
+ at org.apache.hadoop.fs.DelegateToFileSystem.mkdir(DelegateToFileSystem.java:143)
+ at org.apache.hadoop.fs.FilterFs.mkdir(FilterFs.java:189)
+ at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:706)
+ at org.apache.hadoop.fs.FileContext$4.next(FileContext.java:703)
+ at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2325)
+ at org.apache.hadoop.fs.FileContext.mkdir(FileContext.java:703)
+ at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.init(ResourceLocalizationService.java:188)
+ ... 6 more
+2012-05-14 16:32:13,472 INFO org.apache.hadoop.yarn.service.CompositeService: Error stopping org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler
+java.lang.NullPointerException
+ at org.apache.hadoop.yarn.server.nodemanager.containermanager.loghandler.NonAggregatingLogHandler.stop(NonAggregatingLogHandler.java:82)
+ at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
+ at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
+ at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.stop(ContainerManagerImpl.java:266)
+ at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
+ at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
+ at org.apache.hadoop.yarn.server.nodemanager.NodeManager.stop(NodeManager.java:182)
+ at org.apache.hadoop.yarn.service.CompositeService$CompositeServiceShutdownHook.run(CompositeService.java:122)
+ at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
+{code}
+
+- YARN-39.
+ Critical sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
+ RM-NM secret-keys should be randomly generated and rolled every so often
+ - RM should generate the master-key randomly
+ - The master-key should roll every so often
+ - NM should remember old expired keys so that already doled out container-requests can be satisfied.
+- YARN-37.
+ Minor bug reported by Jason Lowe and fixed by Mayank Bansal (resourcemanager)
+ TestRMAppTransitions.testAppSubmittedKilled passes for the wrong reason
+ TestRMAppTransitions#testAppSubmittedKilled causes an invalid event exception but the test doesn't catch the error since the final app state is still killed. Killed for the wrong reason, but the final state is the same.
+- YARN-36.
+ Blocker bug reported by Eli Collins and fixed by Radim Kolar
+ branch-2.1.0-alpha doesn't build
+ branch-2.1.0-alpha doesn't build due to the following. Per YARN-1 I updated the mvn version to be 2.1.0-SNAPSHOT, before I hit this issue it didn't compile due to the bogus version.
+
+{noformat}
+hadoop-branch-2.1.0-alpha $ mvn compile
+[INFO] Scanning for projects...
+[ERROR] The build could not read 1 project -> [Help 1]
+[ERROR]
+[ERROR] The project org.apache.hadoop:hadoop-yarn-project:2.1.0-SNAPSHOT (/home/eli/src/hadoop-branch-2.1.0-alpha/hadoop-yarn-project/pom.xml) has 1 error
+[ERROR] 'dependencies.dependency.version' for org.hsqldb:hsqldb:jar is missing. @ line 160, column 17
+{noformat}
+- YARN-31.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves
+ TestDelegationTokenRenewer fails on jdk7
+ TestDelegationTokenRenewer fails when run with jdk7.
+
+With JDK7, test methods run in an undefined order. Here it is expecting that testDTRenewal runs first but it no longer is.
+- YARN-29.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (client)
+ Add a yarn-client module
+ I see that we are duplicating (some) code for talking to RM via client API. In this light, a yarn-client module will be useful so that clients of all frameworks can use/extend it.
+
+And that same module can be the destination for all the YARN's command line tools.
+- YARN-27.
+ Major bug reported by Ramya Sunil and fixed by Arun C Murthy
+ Failed refreshQueues due to misconfiguration prevents further refreshing of queues
+ Stumbled upon this problem while refreshing queues with incorrect configuration. The exact scenario was:
+1. Added a new queue "newQueue" without defining its capacity.
+2. "bin/mapred queue -refreshQueues" fails correctly with "Illegal capacity of -1 for queue root.newQueue"
+3. However, after defining the capacity of "newQueue" followed by a second "bin/mapred queue -refreshQueues" throws "org.apache.hadoop.metrics2.MetricsException: Metrics source QueueMetrics,q0=root,q1=newQueue already exists!" Also see Hadoop:name=QueueMetrics,q0=root,q1=newQueue,service=ResourceManager metrics being available even though the queue was not added.
+
+The expected behavior would be to refresh the queues correctly and allow addition of "newQueue".
+- YARN-25.
+ Major bug reported by Thomas Graves and fixed by Robert Joseph Evans
+ remove old aggregated logs
+ Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
+
+It might make sense for job history server to remove them.
+- YARN-22.
+ Minor bug reported by Eli Collins and fixed by Mayank Bansal
+ Using URI for yarn.nodemanager log dirs fails
+ If I use URIs (eg file:///home/eli/hadoop/dirs) for yarn.nodemanager.log-dirs or yarn.nodemanager.remote-app-log-dir the container log servlet fails with an NPE (works if I remove the "file" scheme). Using a URI for yarn.nodemanager.local-dirs works.
+- YARN-15.
+ Critical bug reported by Alejandro Abdelnur and fixed by Arun C Murthy (nodemanager)
+ YarnConfiguration DEFAULT_YARN_APPLICATION_CLASSPATH should be updated
+
+{code}
+ /**
+ * Default CLASSPATH for YARN applications. A comma-separated list of
+ * CLASSPATH entries
+ */
+ public static final String[] DEFAULT_YARN_APPLICATION_CLASSPATH = {
+ "$HADOOP_CONF_DIR", "$HADOOP_COMMON_HOME/share/hadoop/common/*",
+ "$HADOOP_COMMON_HOME/share/hadoop/common/lib/*",
+ "$HADOOP_HDFS_HOME/share/hadoop/hdfs/*",
+ "$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*",
+ "$YARN_HOME/share/hadoop/mapreduce/*",
+ "$YARN_HOME/share/hadoop/mapreduce/lib/*"};
+{code}
+
+It should have {{share/yarn/}} and MR should add the {{share/mapreduce/}} (another JIRA?)
+- YARN-14.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
+ Symlinks to peer distributed cache files no longer work
+ Trying to create a symlink to another file that is specified for the distributed cache will fail to create the link. For example:
+
+hadoop jar ... -files "x,y,x#z"
+
+will localize the files x and y as x and y, but the z symlink for x will not be created. This is a regression from 1.x behavior.
+- YARN-13.
+ Critical bug reported by Todd Lipcon and fixed by
+ Merge of yarn reorg into branch-2 copied trunk tree
+ When the move of yarn from inside MR to the project root was merged into branch-2, it seems like the trunk code base was actually copied into the branch-2 branch, instead of a parallel move occurring. So, the poms in branch-2 show the version as 3.0.0-SNAPSHOT instead of a 2.x snapshot version. This is breaking the branch-2 build.
+- YARN-12.
+ Major bug reported by Junping Du and fixed by Junping Du (scheduler)
+ Several Findbugs issues with new FairScheduler in YARN
+ The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
+org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
+Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
+Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
+Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
+The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
+
+
+- YARN-10.
+ Major improvement reported by Arun C Murthy and fixed by Hitesh Shah
+ dist-shell shouldn't have a (test) dependency on hadoop-mapreduce-client-core
+ dist-shell shouldn't have a (test) dependency on hadoop-mapreduce-client-core, this should be removed.
+- YARN-9.
+ Major improvement reported by Arun C Murthy and fixed by Vinod Kumar Vavilapalli
+ Rename YARN_HOME to HADOOP_YARN_HOME
+ We should rename YARN_HOME to HADOOP_YARN_HOME to be consistent with rest of Hadoop sub-projects.
+- YARN-1.
+ Major task reported by Arun C Murthy and fixed by Arun C Murthy
+ Move YARN out of hadoop-mapreduce
+ Move YARN out of hadoop-mapreduce-project into hadoop-yarn-project in hadoop trunk
+- MAPREDUCE-4691.
+ Critical bug reported by Jason Lowe and fixed by Robert Joseph Evans (jobhistoryserver , mrv2)
+ Historyserver can report "Unknown job" after RM says job has completed
+
+- MAPREDUCE-4689.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (client)
+ JobClient.getMapTaskReports on failed job results in NPE
+
+- MAPREDUCE-4649.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (jobhistoryserver)
+ mr-jobhistory-daemon.sh needs to be updated post YARN-1
+
+- MAPREDUCE-4647.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ We should only unjar jobjar if there is a lib directory in it.
+
+- MAPREDUCE-4646.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ client does not receive job diagnostics for failed jobs
+
+- MAPREDUCE-4642.
+ Major bug reported by Robert Kanter and fixed by Robert Kanter (test)
+ MiniMRClientClusterFactory should not use job.setJar()
+
+- MAPREDUCE-4641.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ Exception in commitJob marks job as successful in job history
+
+- MAPREDUCE-4638.
+ Major improvement reported by Arun C Murthy and fixed by Arun C Murthy
+ MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars
+
+- MAPREDUCE-4635.
+ Major bug reported by Bikas Saha and fixed by Bikas Saha
+ MR side of YARN-83. Changing package of YarnClient
+
+- MAPREDUCE-4633.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (jobhistoryserver)
+ history server doesn't set permissions on all subdirs
+
+- MAPREDUCE-4629.
+ Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla
+ Remove JobHistory.DEBUG_MODE
+
+- MAPREDUCE-4614.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (client , task)
+ Simplify debugging a job's tokens
+
+- MAPREDUCE-4612.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves
+ job summary file permissions not set when its created
+
+- MAPREDUCE-4611.
+ Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ MR AM dies badly when Node is decomissioned
+
+- MAPREDUCE-4610.
+ Major bug reported by Tom White and fixed by Tom White (mrv2)
+ Support deprecated mapreduce.job.counters.limit property in MR2
+
+- MAPREDUCE-4608.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
+ hadoop-mapreduce-client is missing some dependencies
+
+- MAPREDUCE-4604.
+ Critical bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
+ In mapred-default, mapreduce.map.maxattempts & mapreduce.reduce.maxattempts defaults are set to 4 as well as mapreduce.job.maxtaskfailures.per.tracker.
+
+- MAPREDUCE-4600.
+ Critical bug reported by Robert Joseph Evans and fixed by Daryn Sharp
+ TestTokenCache.java from MRV1 no longer compiles
+
+- MAPREDUCE-4580.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
+ Change MapReduce to use the yarn-client module
+
+- MAPREDUCE-4579.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves
+ TestTaskAttempt fails jdk7
+
+- MAPREDUCE-4577.
+ Minor bug reported by Alejandro Abdelnur and fixed by Aaron T. Myers (test)
+ HDFS-3672 broke TestCombineFileInputFormat.testMissingBlocks() test
+
+- MAPREDUCE-4572.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (tasktracker , webapps)
+ Can not access user logs - Jetty is not configured by default to serve aliases/symlinks
+
+- MAPREDUCE-4570.
+ Minor bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ ProcfsBasedProcessTree#constructProcessInfo() prints a warning if procfsDir/<pid>/stat is not found.
+
+- MAPREDUCE-4569.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves
+ TestHsWebServicesJobsQuery fails on jdk7
+
+- MAPREDUCE-4562.
+ Major bug reported by Jarek Jarcec Cecho and fixed by Jarek Jarcec Cecho
+ Support for "FileSystemCounter" legacy counter group name for compatibility reasons is creating incorrect counter name
+
+- MAPREDUCE-4511.
+ Major improvement reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv1 , mrv2 , performance)
+ Add IFile readahead
+
+- MAPREDUCE-4504.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ SortValidator writes to wrong directory
+
+- MAPREDUCE-4503.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ Should throw InvalidJobConfException if duplicates found in cacheArchives or cacheFiles
+
+- MAPREDUCE-4498.
+ Critical bug reported by Robert Kanter and fixed by Robert Kanter (build , examples)
+ Remove hsqldb jar from Hadoop runtime classpath
+
+- MAPREDUCE-4496.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
+ AM logs link is missing user name
+
+- MAPREDUCE-4494.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2 , test)
+ TestFifoScheduler failing with Metrics source QueueMetrics,q0=default already exists!
+
+- MAPREDUCE-4493.
+ Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ Distibuted Cache Compatability Issues
+
+- MAPREDUCE-4492.
+ Minor bug reported by Nishan Shetty and fixed by Mayank Bansal (mrv2)
+ Configuring total queue capacity between 100.5 and 99.5 at perticular level is sucessfull
+
+- MAPREDUCE-4484.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration
+
+- MAPREDUCE-4483.
+ Major bug reported by John George and fixed by John George
+ 2.0 build does not work
+
+- MAPREDUCE-4470.
+ Major bug reported by Kihwal Lee and fixed by Ilya Katsov (test)
+ Fix TestCombineFileInputFormat.testForEmptyFile
+
+- MAPREDUCE-4467.
+ Critical bug reported by Andrey Klochkov and fixed by Kihwal Lee (nodemanager)
+ IndexCache failures due to missing synchronization
+
+- MAPREDUCE-4465.
+ Trivial bug reported by Bo Wang and fixed by Bo Wang
+ Update description of yarn.nodemanager.address property
+
+- MAPREDUCE-4457.
+ Critical bug reported by Thomas Graves and fixed by Robert Joseph Evans (mrv2)
+ mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at FAILED
+
+- MAPREDUCE-4456.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ LocalDistributedCacheManager can get an ArrayIndexOutOfBounds when creating symlinks
+
+- MAPREDUCE-4449.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig
+
+- MAPREDUCE-4448.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2 , nodemanager)
+ Nodemanager crashes upon application cleanup if aggregation failed to start
+
+- MAPREDUCE-4447.
+ Major bug reported by Eli Collins and fixed by Eli Collins (build)
+ Remove aop from cruft from the ant build
+
+- MAPREDUCE-4444.
+ Blocker bug reported by Nathan Roberts and fixed by Jason Lowe (nodemanager)
+ nodemanager fails to start when one of the local-dirs is bad
+
+- MAPREDUCE-4441.
+ Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla
+ Fix build issue caused by MR-3451
+
+- MAPREDUCE-4440.
+ Major bug reported by Arun C Murthy and fixed by Arun C Murthy
+ Change SchedulerApp & SchedulerNode to be a minimal interface
+
+- MAPREDUCE-4437.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
+ Race in MR ApplicationMaster can cause reducers to never be scheduled
+
+- MAPREDUCE-4432.
+ Trivial bug reported by Gabriel Reid and fixed by
+ Confusing warning message when GenericOptionsParser is not used
+
+- MAPREDUCE-4427.
+ Major improvement reported by Bikas Saha and fixed by Bikas Saha
+ Enable the RM to work with AM's that are not managed by it
+
+- MAPREDUCE-4423.
+ Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ Potential infinite fetching of map output
+
+- MAPREDUCE-4422.
+ Major improvement reported by Arun C Murthy and fixed by Ahmed Radwan (nodemanager)
+ YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration
+
+- MAPREDUCE-4419.
+ Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
+ ./mapred queue -info <queuename> -showJobs displays all the jobs irrespective of <queuename>
+
+- MAPREDUCE-4417.
+ Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (mrv2 , security)
+ add support for encrypted shuffle
+
+- MAPREDUCE-4416.
+ Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (client , mrv2)
+ Some tests fail if Clover is enabled
+
+- MAPREDUCE-4408.
+ Major improvement reported by Alejandro Abdelnur and fixed by Robert Kanter (mrv1 , mrv2)
+ allow jobs to set a JAR that is in the distributed cached
+
+- MAPREDUCE-4407.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (build , mrv2)
+ Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist package
+
+- MAPREDUCE-4406.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2 , test)
+ Users should be able to specify the MiniCluster ResourceManager and JobHistoryServer ports
+
+- MAPREDUCE-4402.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (test)
+ TestFileInputFormat fails intermittently
+
+- MAPREDUCE-4395.
+ Critical bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (distributed-cache , job submission , mrv2)
+ Possible NPE at ClientDistributedCacheManager#determineTimestamps
+
+- MAPREDUCE-4392.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ Counters.makeCompactString() changed behavior from 0.20
+
+- MAPREDUCE-4387.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (resourcemanager)
+ RM gets fatal error and exits during TestRM
+
+- MAPREDUCE-4384.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (nodemanager)
+ Race conditions in IndexCache
+
+- MAPREDUCE-4383.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (pipes)
+ HadoopPipes.cc needs to include unistd.h
+
+- MAPREDUCE-4380.
+ Minor bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
+ Empty Userlogs directory is getting created under logs directory
+
+- MAPREDUCE-4379.
+ Blocker bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
+ Node Manager throws java.lang.OutOfMemoryError: Java heap space due to org.apache.hadoop.fs.LocalDirAllocator.contexts
+
+- MAPREDUCE-4376.
+ Major bug reported by Jason Lowe and fixed by Kihwal Lee (mrv2 , test)
+ TestClusterMRNotification times out
+
+- MAPREDUCE-4375.
+ Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
+ Show Configuration Tracability in MR UI
+
+- MAPREDUCE-4372.
+ Major bug reported by Devaraj K and fixed by Devaraj K (mrv2 , resourcemanager)
+ Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor and Shutdown hook manager
+
+- MAPREDUCE-4361.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ Fix detailed metrics for protobuf-based RPC on 0.23
+
+- MAPREDUCE-4355.
+ Major new feature reported by Karthik Kambatla and fixed by Karthik Kambatla (mrv1 , mrv2)
+ Add RunningJob.getJobStatus()
+
+- MAPREDUCE-4341.
+ Major bug reported by Thomas Graves and fixed by Karthik Kambatla (capacity-sched , mrv2)
+ add types to capacity scheduler properties documentation
+
+- MAPREDUCE-4336.
+ Major bug reported by Siddharth Seth and fixed by Ahmed Radwan (mrv2)
+ Distributed Shell fails when used with the CapacityScheduler
+
+- MAPREDUCE-4320.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (contrib/gridmix)
+ gridmix mainClass wrong in pom.xml
+
+- MAPREDUCE-4313.
+ Blocker bug reported by Eli Collins and fixed by Robert Joseph Evans (build , test)
+ TestTokenCache doesn't compile due TokenCache.getDelegationToken compilation error
+
+- MAPREDUCE-4311.
+ Major bug reported by Thomas Graves and fixed by Karthik Kambatla (capacity-sched , mrv2)
+ Capacity scheduler.xml does not accept decimal values for capacity and maximum-capacity settings
+
+- MAPREDUCE-4307.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ TeraInputFormat calls FileSystem.getDefaultBlockSize() without a Path - Failure when using ViewFileSystem
+
+- MAPREDUCE-4306.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ Problem running Distributed Shell applications as a user other than the one started the daemons
+
+- MAPREDUCE-4302.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (nodemanager)
+ NM goes down if error encountered during log aggregation
+
+- MAPREDUCE-4301.
+ Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
+ Dedupe some strings in MRAM for memory savings
+
+- MAPREDUCE-4300.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (applicationmaster)
+ OOM in AM can turn it into a zombie.
+
+- MAPREDUCE-4299.
+ Major bug reported by Tom White and fixed by Tom White (mrv2)
+ Terasort hangs with MR2 FifoScheduler
+
+- MAPREDUCE-4297.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash (contrib/gridmix)
+ Usersmap file in gridmix should not fail on empty lines
+
+- MAPREDUCE-4295.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2 , resourcemanager)
+ RM crashes due to DNS issue
+
+- MAPREDUCE-4290.
+ Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
+ JobStatus.getState() API is giving ambiguous values
+
+- MAPREDUCE-4283.
+ Major improvement reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver , mrv2)
+ Display tail of aggregated logs by default
+
+- MAPREDUCE-4276.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ Allow setting yarn.nodemanager.delete.debug-delay-sec property to "-1" for easier container debugging.
+
+- MAPREDUCE-4270.
+ Major bug reported by Brock Noland and fixed by Thomas Graves (mrv2)
+ data_join test classes are in the wrong packge
+
+- MAPREDUCE-4269.
+ Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ documentation: Gridmix has javadoc warnings in StressJobFactory
+
+- MAPREDUCE-4267.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ mavenize pipes
+
+- MAPREDUCE-4264.
+ Blocker bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ Got ClassCastException when using mapreduce.history.server.delegationtoken.required=true
+
+- MAPREDUCE-4262.
+ Minor bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
+ NM gives wrong log message saying "Connected to ResourceManager" before trying to connect
+
+- MAPREDUCE-4252.
+ Major bug reported by Tom White and fixed by Tom White (mrv2)
+ MR2 job never completes with 1 pending task
+
+- MAPREDUCE-4250.
+ Major bug reported by Patrick Hunt and fixed by Patrick Hunt (nodemanager)
+ hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
+
+- MAPREDUCE-4238.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ mavenize data_join
+
+- MAPREDUCE-4237.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ TestNodeStatusUpdater can fail if localhost has a domain associated with it
+
+- MAPREDUCE-4233.
+ Critical bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ NPE can happen in RMNMNodeInfo.
+
+- MAPREDUCE-4228.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (applicationmaster , mrv2)
+ mapreduce.job.reduce.slowstart.completedmaps is not working properly to delay the scheduling of the reduce tasks
+
+- MAPREDUCE-4226.
+ Major bug reported by Tom White and fixed by Tom White (mrv2)
+ ConcurrentModificationException in FileSystemCounterGroup
+
+- MAPREDUCE-4224.
+ Major bug reported by Devaraj K and fixed by Devaraj K (mrv2 , scheduler , test)
+ TestFifoScheduler throws org.apache.hadoop.metrics2.MetricsException
+
+- MAPREDUCE-4220.
+ Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ RM apps page starttime/endtime sorts are incorrect
+
+- MAPREDUCE-4215.
+ Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ RM app page shows 500 error on appid parse error
+
+- MAPREDUCE-4212.
+ Major test reported by Daryn Sharp and fixed by Daryn Sharp (test)
+ TestJobClientGetJob sometimes fails
+
+- MAPREDUCE-4211.
+ Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ Error conditions (missing appid, appid not found) are masked in the RM app page
+
+- MAPREDUCE-4210.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (webapps)
+ Expose listener address for WebApp
+
+- MAPREDUCE-4209.
+ Major bug reported by Radim Kolar and fixed by (build)
+ junit dependency in hadoop-mapreduce-client is missing scope test
+
+- MAPREDUCE-4206.
+ Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ Sorting by Last Health-Update on the RM nodes page sorts does not work correctly
+
+- MAPREDUCE-4205.
+ Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (mrv2)
+ retrofit all JVM shutdown hooks to use ShutdownHookManager
+
+- MAPREDUCE-4197.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash
+ Include the hsqldb jar in the hadoop-mapreduce tar file
+
+- MAPREDUCE-4194.
+ Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ ConcurrentModificationError in DirectoryCollection
+
+- MAPREDUCE-4190.
+ Major improvement reported by Thomas Graves and fixed by Thomas Graves (mrv2 , webapps)
+ Improve web UI for task attempts userlog link
+
+- MAPREDUCE-4189.
+ Critical bug reported by Devaraj K and fixed by Devaraj K (mrv2)
+ TestContainerManagerSecurity is failing
+
+- MAPREDUCE-4169.
+ Minor bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ Container Logs appear in unsorted order
+
+- MAPREDUCE-4165.
+ Trivial bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ Committing is misspelled as commiting in task logs
+
+- MAPREDUCE-4163.
+ Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (mrv2)
+ consistently set the bind address
+
+- MAPREDUCE-4162.
+ Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (client , mrv2)
+ Correctly set token service
+
+- MAPREDUCE-4161.
+ Major sub-task reported by Daryn Sharp and fixed by Daryn Sharp (client , mrv2)
+ create sockets consistently
+
+- MAPREDUCE-4160.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (test)
+ some mrv1 ant tests fail with timeout - due to 4156
+
+- MAPREDUCE-4159.
+ Major bug reported by Nishan Shetty and fixed by Devaraj K (mrv2)
+ Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
+
+- MAPREDUCE-4157.
+ Major improvement reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ ResourceManager should not kill apps that are well behaved
+
+- MAPREDUCE-4156.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (build)
+ ant build fails compiling JobInProgress
+
+- MAPREDUCE-4152.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ map task left hanging after AM dies trying to connect to RM
+
+- MAPREDUCE-4151.
+ Major improvement reported by Jason Lowe and fixed by Jason Lowe (mrv2 , webapps)
+ RM scheduler web page should filter apps to those that are relevant to scheduling
+
+- MAPREDUCE-4148.
+ Major bug reported by Tom White and fixed by Tom White (mrv2)
+ MapReduce should not have a compile-time dependency on HDFS
+
+- MAPREDUCE-4146.
+ Major improvement reported by Tom White and fixed by Ahmed Radwan
+ Support limits on task status string length and number of block locations in branch-2
+
+- MAPREDUCE-4144.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ ResourceManager NPE while handling NODE_UPDATE
+
+- MAPREDUCE-4140.
+ Major bug reported by Patrick Hunt and fixed by Patrick Hunt (client , mrv2)
+ mapreduce classes incorrectly importing "clover.org.apache.*" classes
+
+- MAPREDUCE-4139.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ Potential ResourceManager deadlock when SchedulerEventDispatcher is stopped
+
+- MAPREDUCE-4134.
+ Major task reported by Ravi Prakash and fixed by Ravi Prakash (mrv2)
+ Remove references of mapred.child.ulimit etc. since they are not being used any more
+
+- MAPREDUCE-4133.
+ Major bug reported by John George and fixed by John George
+ MR over viewfs is broken
+
+- MAPREDUCE-4129.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (mrv2)
+ Lots of unneeded counters log messages
+
+- MAPREDUCE-4128.
+ Major bug reported by Bikas Saha and fixed by Bikas Saha (mrv2)
+ AM Recovery expects all attempts of a completed task to also be completed.
+
+- MAPREDUCE-4117.
+ Critical bug reported by Devaraj K and fixed by Devaraj K (client , mrv2)
+ mapred job -status throws NullPointerException
+
+- MAPREDUCE-4102.
+ Major bug reported by Thomas Graves and fixed by Bhallamudi Venkata Siva Kamesh (webapps)
+ job counters not available in Jobhistory webui for killed jobs
+
+- MAPREDUCE-4099.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (mrv2)
+ ApplicationMaster may fail to remove staging directory
+
+- MAPREDUCE-4097.
+ Major bug reported by Alejandro Abdelnur and fixed by Roman Shaposhnik (build)
+ tools testcases fail because missing mrapp-generated-classpath file in classpath
+
+- MAPREDUCE-4092.
+ Blocker bug reported by Jonathan Eagles and fixed by Jonathan Eagles (mrv2)
+ commitJob Exception does not fail job (regression in 0.23 vs 0.20)
+
+- MAPREDUCE-4091.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build , test)
+ tools testcases failing because of MAPREDUCE-4082
+
+- MAPREDUCE-4089.
+ Blocker bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ Hung Tasks never time out.
+
+- MAPREDUCE-4082.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
+ hadoop-mapreduce-client-app's mrapp-generated-classpath file should not be in the module JAR
+
+- MAPREDUCE-4079.
+ Blocker improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mr-am , mrv2)
+ Allow MR AppMaster to limit ephemeral port range.
+
+- MAPREDUCE-4074.
+ Major bug reported by Devaraj K and fixed by xieguiming
+ Client continuously retries to RM When RM goes down before launching Application Master
+
+- MAPREDUCE-4073.
+ Critical bug reported by Siddharth Seth and fixed by Siddharth Seth (mrv2 , scheduler)
+ CS assigns multiple off-switch containers when using multi-level-queues
+
+- MAPREDUCE-4072.
+ Major bug reported by Anupam Seth and fixed by Anupam Seth (mrv2)
+ User set java.library.path seems to overwrite default creating problems native lib loading
+ -Djava.library.path in mapred.child.java.opts can cause issues with native libraries. LD_LIBRARY_PATH through mapred.child.env should be used instead.
+- MAPREDUCE-4068.
+ Blocker bug reported by Ahmed Radwan and fixed by Robert Kanter (mrv2)
+ Jars in lib subdirectory of the submittable JAR are not added to the classpath
+
+- MAPREDUCE-4062.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ AM Launcher thread can hang forever
+
+- MAPREDUCE-4060.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (build)
+ Multiple SLF4J binding warning
+
+- MAPREDUCE-4059.
+ Major improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ The history server should have a separate pluggable storage/query interface
+
+- MAPREDUCE-4053.
+ Major bug reported by Alejandro Abdelnur and fixed by Robert Joseph Evans (mrv2)
+ Counters group names deprecation is wrong, iterating over group names deprecated names don't show up
+
+- MAPREDUCE-4051.
+ Major task reported by Ravi Prakash and fixed by Ravi Prakash
+ Remove the empty hadoop-mapreduce-project/assembly/all.xml file
+
+- MAPREDUCE-4050.
+ Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
+ Invalid node link
+
+- MAPREDUCE-4048.
+ Major bug reported by Devaraj K and fixed by Devaraj K (mrv2)
+ NullPointerException exception while accessing the Application Master UI
+
+- MAPREDUCE-4040.
+ Minor bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (jobhistoryserver , mrv2)
+ History links should use hostname rather than IP address.
+
+- MAPREDUCE-4031.
+ Critical bug reported by Devaraj K and fixed by Devaraj K (mrv2 , nodemanager)
+ Node Manager hangs on shut down
+
+- MAPREDUCE-4024.
+ Major bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ RM webservices can't query on finalStatus
+
+- MAPREDUCE-4020.
+ Major bug reported by Jason Lowe and fixed by Anupam Seth (mrv2 , webapps)
+ Web services returns incorrect JSON for deep queue tree
+
+- MAPREDUCE-4017.
+ Trivial improvement reported by Koji Noguchi and fixed by Thomas Graves (jobhistoryserver , jobtracker)
+ Add jobname to jobsummary log
+ The Job Summary log may contain commas in values that are escaped by a '\' character. This was true before, but is more likely to be exposed now.
+- MAPREDUCE-4012.
+ Minor bug reported by Koji Noguchi and fixed by Thomas Graves
+ Hadoop Job setup error leaves no useful info to users (when LinuxTaskController is used)
+
+- MAPREDUCE-4010.
+ Critical bug reported by Jason Lowe and fixed by Alejandro Abdelnur (mrv2)
+ TestWritableJobConf fails on trunk
+
+- MAPREDUCE-4002.
+ Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (examples)
+ MultiFileWordCount job fails if the input path is not from default file system
+
+- MAPREDUCE-3999.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash (mrv2 , webapps)
+ Tracking link gives an error if the AppMaster hasn't started yet
+
+- MAPREDUCE-3993.
+ Major bug reported by Todd Lipcon and fixed by Karthik Kambatla (mrv1 , mrv2)
+ Graceful handling of codec errors during decompression
+
+- MAPREDUCE-3992.
+ Major bug reported by Todd Lipcon and fixed by Todd Lipcon (mrv1)
+ Reduce fetcher doesn't verify HTTP status code of response
+
+- MAPREDUCE-3988.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Eric Payne (mrv2)
+ mapreduce.job.local.dir doesn't point to a single directory on a node.
+
+- MAPREDUCE-3983.
+ Major test reported by Robert Joseph Evans and fixed by Ravi Prakash (mrv1)
+ TestTTResourceReporting can fail, and should just be deleted
+
+- MAPREDUCE-3972.
+ Major sub-task reported by Robert Joseph Evans and fixed by Robert Joseph Evans (mrv2)
+ Locking and exception issues in JobHistory Server.
+
+- MAPREDUCE-3947.
+ Minor bug reported by Todd Lipcon and fixed by Devaraj K
+ yarn.app.mapreduce.am.resource.mb not documented
+
+- MAPREDUCE-3942.
+ Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (mrv2 , security)
+ Randomize master key generation for ApplicationTokenSecretManager and roll it every so often
+
+- MAPREDUCE-3940.
+ Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli (mrv2 , security)
+ ContainerTokens should have an expiry interval
+ ContainerTokens now have an expiry interval so that stale tokens cannot be used for launching containers.
+- MAPREDUCE-3932.
+ Critical bug reported by Vinod Kumar Vavilapalli and fixed by Robert Joseph Evans (mr-am , mrv2)
+ MR tasks failing and crashing the AM when available-resources/headRoom becomes zero
+
+- MAPREDUCE-3927.
+ Critical bug reported by MengWang and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
+ Shuffle hang when set map.failures.percent
+
+- MAPREDUCE-3907.
+ Minor improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation)
+ Document entries mapred-default.xml for the jobhistory server.
+
+- MAPREDUCE-3906.
+ Trivial improvement reported by Eugene Koontz and fixed by Eugene Koontz (documentation)
+ Fix inconsistency in documentation regarding mapreduce.jobhistory.principal
+
+- MAPREDUCE-3893.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
+
+- MAPREDUCE-3889.
+ Critical bug reported by Thomas Graves and fixed by Devaraj K (mrv2)
+ job client tries to use /tasklog interface, but that doesn't exist anymore
+
+- MAPREDUCE-3873.
+ Minor bug reported by Nishan Shetty and fixed by xieguiming (mrv2 , nodemanager)
+ Nodemanager is not getting decommisioned if the absolute ip is given in exclude file.
+ Fixed NodeManagers' decommissioning at RM to accept IP addresses also.
+- MAPREDUCE-3871.
+ Major improvement reported by Tom White and fixed by Tom White (distributed-cache)
+ Allow symlinking in LocalJobRunner DistributedCache
+
+- MAPREDUCE-3870.
+ Major bug reported by Bhallamudi Venkata Siva Kamesh and fixed by Bhallamudi Venkata Siva Kamesh (mrv2)
+ Invalid App Metrics
+
+- MAPREDUCE-3850.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
+ Avoid redundant calls for tokens in TokenCache
+
+- MAPREDUCE-3842.
+ Critical improvement reported by Alejandro Abdelnur and fixed by Thomas Graves (mrv2 , webapps)
+ stop webpages from automatic refreshing
+
+- MAPREDUCE-3812.
+ Major sub-task reported by Vinod Kumar Vavilapalli and fixed by Harsh J (mrv2 , performance)
+ Lower default allocation sizes, fix allocation configurations and document them
+ Removes two sets of previously available config properties:
+
+1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
+2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )
+
+In favor of two new, generically named properties:
+
+1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
+2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.
+
+Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.
+
+Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
+- MAPREDUCE-3782.
+ Critical bug reported by Arpit Gupta and fixed by Jason Lowe (mrv2)
+ teragen terasort jobs fail when using webhdfs://
+
+- MAPREDUCE-3773.
+ Major new feature reported by Owen O'Malley and fixed by Owen O'Malley (jobtracker)
+ Add queue metrics with buckets for job run times
+
+- MAPREDUCE-3728.
+ Critical bug reported by Roman Shaposhnik and fixed by Giridharan Kesavan (mrv2 , nodemanager)
+ ShuffleHandler can't access results when configured in a secure mode
+
+- MAPREDUCE-3682.
+ Major bug reported by David Capwell and fixed by Ravi Prakash (mrv2)
+ Tracker URL says AM tasks run on localhost
+
+- MAPREDUCE-3672.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Anupam Seth (mr-am , mrv2)
+ Killed maps shouldn't be counted towards JobCounter.NUM_FAILED_MAPS
+
+- MAPREDUCE-3659.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
+ Host-based token support
+
+- MAPREDUCE-3650.
+ Blocker bug reported by Thomas Graves and fixed by Ravi Prakash (mrv2)
+ testGetTokensForHftpFS() fails
+
+- MAPREDUCE-3621.
+ Major bug reported by Thomas Graves and fixed by Ravi Prakash (mrv2)
+ TestDBJob and TestDataDrivenDBInputFormat ant tests fail
+
+- MAPREDUCE-3613.
+ Critical sub-task reported by Thomas Graves and fixed by Thomas Graves (mrv2)
+ web service calls header contains 2 content types
+
+- MAPREDUCE-3543.
+ Critical bug reported by Mahadev konar and fixed by Thomas Graves (mrv2)
+ Mavenize Gridmix.
+ Note that to apply this you should first run the script - ./MAPREDUCE-3543v3.sh svn, then apply the patch.
+
+If this is merged to more then trunk, the version inside of hadoop-tools/hadoop-gridmix/pom.xml will need to be udpated accordingly.
+- MAPREDUCE-3506.
+ Minor bug reported by Ratandeep Ratti and fixed by Jason Lowe (client , mrv2)
+ Calling getPriority on JobInfo after parsing a history log with JobHistoryParser throws a NullPointerException
+
+- MAPREDUCE-3493.
+ Minor bug reported by Ahmed Radwan and fixed by (mrv2)
+ Add the default mapreduce.shuffle.port property to mapred-default.xml
+
+- MAPREDUCE-3451.
+ Major new feature reported by Patrick Wendell and fixed by Patrick Wendell (mrv2 , scheduler)
+ Port Fair Scheduler to MR2
+
+- MAPREDUCE-3350.
+ Critical bug reported by Vinod Kumar Vavilapalli and fixed by Jonathan Eagles (mrv2 , webapps)
+ Per-app RM page should have the list of application-attempts like on the app JHS page
+
+- MAPREDUCE-3348.
+ Major bug reported by Devaraj K and fixed by Devaraj K (mrv2)
+ mapred job -status fails to give info even if the job is present in History
+ Fixed a bug in MR client to redirect to JobHistoryServer correctly when RM forgets the app.
+- MAPREDUCE-3289.
+ Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (mrv2 , nodemanager , performance)
+ Make use of fadvise in the NM's shuffle handler
+
+- MAPREDUCE-3082.
+ Major bug reported by Rajit Saha and fixed by John George (harchive)
+ archive command take wrong path for input file with current directory
+
+- MAPREDUCE-2786.
+ Minor improvement reported by Plamen Jeliazkov and fixed by Plamen Jeliazkov (benchmarks)
+ TestDFSIO should also test compression reading/writing from command-line.
+
+- MAPREDUCE-2739.
+ Minor bug reported by Ahmed Radwan and fixed by Bo Wang (mrv2)
+ MR-279: Update installation docs (remove YarnClientFactory)
+
+- MAPREDUCE-2374.
+ Major bug reported by Todd Lipcon and fixed by Andy Isaacson
+ "Text File Busy" errors launching MR tasks
+
+- MAPREDUCE-2289.
+ Major bug reported by Todd Lipcon and fixed by Ahmed Radwan (job submission)
+ Permissions race can make getStagingDir fail on local filesystem
+
+- MAPREDUCE-2220.
+ Minor bug reported by Rui KUBO and fixed by Rui KUBO (documentation)
+ Fix new API FileOutputFormat-related typos in mapred-default.xml
+
+- MAPREDUCE-987.
+ Minor new feature reported by Philip Zeyliger and fixed by Ahmed Radwan (build , test)
+ Exposing MiniDFS and MiniMR clusters as a single process command-line
+
+- HDFS-3972.
+ Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
+ Trash emptier fails in secure HA cluster
+
+- HDFS-3928.
+ Major bug reported by Eli Collins and fixed by Eli Collins (test)
+ MiniDFSCluster should reset the first ExitException on shutdown
+
+- HDFS-3902.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
+ TestDatanodeBlockScanner#testBlockCorruptionPolicy is broken
+
+- HDFS-3895.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
+ hadoop-client must include commons-cli
+
+- HDFS-3890.
+ Critical bug reported by Thomas Graves and fixed by Thomas Graves
+ filecontext mkdirs doesn't apply umask as expected
+
+- HDFS-3888.
+ Minor bug reported by Jing Zhao and fixed by Jing Zhao
+ BlockPlacementPolicyDefault code cleanup
+
+- HDFS-3887.
+ Trivial improvement reported by Jing Zhao and fixed by Jing Zhao (name-node)
+ Remove redundant chooseTarget methods in BlockPlacementPolicy.java
+
+- HDFS-3879.
+ Minor bug reported by Eli Collins and fixed by Eli Collins (name-node)
+ Fix findbugs warning in TransferFsImage on branch-2
+
+- HDFS-3873.
+ Major bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
+ Hftp assumes security is disabled if token fetch fails
+
+- HDFS-3871.
+ Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy (hdfs client)
+ Change NameNodeProxies to use HADOOP-8748
+
+- HDFS-3866.
+ Minor improvement reported by Ryan Hennig and fixed by Plamen Jeliazkov (build)
+ HttpFS POM should have property where to download tomcat from
+
+- HDFS-3864.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
+ NN does not update internal file mtime for OP_CLOSE when reading from the edit log
+
+- HDFS-3861.
+ Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
+ Deadlock in DFSClient
+
+- HDFS-3860.
+ Major bug reported by Jing Zhao and fixed by Jing Zhao
+ HeartbeatManager#Monitor may wrongly hold the writelock of namesystem
+
+- HDFS-3856.
+ Blocker bug reported by Thomas Graves and fixed by Eli Collins (test)
+ TestHDFSServerPorts failure is causing surefire fork failure
+
+- HDFS-3853.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
+ Port MiniDFSCluster enableManagedDfsDirsRedundancy option to branch-2
+
+- HDFS-3852.
+ Major bug reported by Aaron T. Myers and fixed by Daryn Sharp (hdfs client , security)
+ TestHftpDelegationToken is broken after HADOOP-8225
+
+- HDFS-3849.
+ Critical bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
+ When re-loading the FSImage, we should clear the existing genStamp and leases.
+
+- HDFS-3844.
+ Trivial improvement reported by Jing Zhao and fixed by Jing Zhao
+ Add @Override where necessary and remove unnecessary {@inheritdoc} and imports
+
+- HDFS-3841.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Port HDFS-3835 to branch-0.23
+
+- HDFS-3837.
+ Major bug reported by Eli Collins and fixed by Eli Collins (data-node)
+ Fix DataNode.recoverBlock findbugs warning
+
+- HDFS-3835.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node , security)
+ Long-lived 2NN cannot perform a checkpoint if security is enabled and the NN restarts with outstanding delegation tokens
+
+- HDFS-3833.
+ Major bug reported by Brandon Li and fixed by Brandon Li (test)
+ TestDFSShell fails on Windows due to file concurrent read write
+
+- HDFS-3832.
+ Major bug reported by Suresh Srinivas and fixed by Suresh Srinivas (data-node , name-node)
+ Remove protocol methods related to DistributedUpgrade
+
+- HDFS-3830.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ test_libhdfs_threaded: use forceNewInstance
+
+- HDFS-3819.
+ Minor improvement reported by Jing Zhao and fixed by Jing Zhao
+ Should check whether invalidate work percentage default value is not greater than 1.0f
+
+- HDFS-3816.
+ Major bug reported by Jing Zhao and fixed by Jing Zhao (name-node)
+ Invalidate work percentage default value should be 0.32f instead of 32
+
+- HDFS-3808.
+ Critical bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
+ fuse_dfs: postpone libhdfs intialization until after fork
+
+- HDFS-3803.
+ Minor bug reported by Andrew Purtell and fixed by (data-node)
+ BlockPoolSliceScanner new work period notice is very chatty at INFO level
+
+- HDFS-3802.
+ Trivial improvement reported by Jing Zhao and fixed by Jing Zhao
+ StartupOption.name in HdfsServerConstants should be final
+
+- HDFS-3796.
+ Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (test)
+ Speed up edit log tests by avoiding fsync()
+
+- HDFS-3794.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash (webhdfs)
+ WebHDFS Open used with Offset returns the original (and incorrect) Content Length in the HTTP Header.
+
+- HDFS-3790.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
+ test_fuse_dfs.c doesn't compile on centos 5
+
+- HDFS-3788.
+ Critical bug reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
+ distcp can't copy large files using webhdfs due to missing Content-Length header
+
+- HDFS-3765.
+ Major improvement reported by Vinay and fixed by Vinay (ha)
+ Namenode INITIALIZESHAREDEDITS should be able to initialize all shared storages
+
+- HDFS-3760.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (hdfs client)
+ primitiveCreate is a write, not a read
+
+- HDFS-3758.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
+ TestFuseDFS test failing
+
+- HDFS-3756.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ DelegationTokenFetcher creates 2 HTTP connections, the second one not properly configured
+
+- HDFS-3755.
+ Major bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
+ Creating an already-open-for-write file with overwrite=true fails
+
+- HDFS-3754.
+ Major bug reported by Eli Collins and fixed by Eli Collins (data-node)
+ BlockSender doesn't shutdown ReadaheadPool threads
+
+- HDFS-3738.
+ Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
+ TestDFSClientRetries#testFailuresArePerOperation sets incorrect timeout config
+
+- HDFS-3733.
+ Major bug reported by Andy Isaacson and fixed by Andy Isaacson (webhdfs)
+ Audit logs should include WebHDFS access
+
+- HDFS-3732.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
+ fuse_dfs: incorrect configuration value checked for connection expiry timer period
+
+- HDFS-3731.
+ Blocker bug reported by Suresh Srinivas and fixed by Kihwal Lee (data-node)
+ 2.0 release upgrade must handle blocks being written from 1.0
+
+- HDFS-3724.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ add InterfaceAudience annotations to HttpFS classes and making inner enum static
+
+- HDFS-3723.
+ Major improvement reported by E. Sammer and fixed by Jing Zhao (scripts , tools)
+ All commands should support meaningful --help
+
+- HDFS-3721.
+ Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (data-node , hdfs client)
+ hsync support broke wire compatibility
+
+- HDFS-3720.
+ Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ hdfs.h must get packaged
+
+- HDFS-3718.
+ Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (data-node)
+ Datanode won't shutdown because of runaway DataBlockScanner thread
+
+- HDFS-3715.
+ Major bug reported by Eli Collins and fixed by Andrew Wang (test)
+ Fix TestFileCreation#testFileCreationNamenodeRestart
+
+- HDFS-3711.
+ Major improvement reported by Andrew Wang and fixed by Andrew Wang
+ Manually convert remaining tests to JUnit4
+
+- HDFS-3710.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson (libhdfs)
+ libhdfs misuses O_RDONLY/WRONLY/RDWR
+
+- HDFS-3709.
+ Major test reported by Eli Collins and fixed by Eli Collins (test)
+ TestStartup tests still binding to the ephemeral port
+
+- HDFS-3707.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ TestFSInputChecker: improper use of skip
+
+- HDFS-3697.
+ Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (data-node , performance)
+ Enable fadvise readahead by default
+ The datanode now performs 4MB readahead by default when reading data from its disks, if the native libraries are present. This has been shown to improve performance in many workloads. The feature may be disabled by setting dfs.datanode.readahead.bytes to "0".
+- HDFS-3696.
+ Critical bug reported by Kihwal Lee and fixed by Tsz Wo (Nicholas), SZE
+ Create files with WebHdfsFileSystem goes OOM when file size is big
+
+- HDFS-3690.
+ Major bug reported by Eli Collins and fixed by Eli Collins
+ BlockPlacementPolicyDefault incorrectly casts LOG
+
+- HDFS-3688.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (data-node)
+ Namenode loses datanode hostname if datanode re-registers
+
+- HDFS-3683.
+ Minor bug reported by Todd Lipcon and fixed by Plamen Jeliazkov (name-node)
+ Edit log replay progress indicator shows >100% complete
+
+- HDFS-3679.
+ Minor bug reported by Conrad Meyer and fixed by Conrad Meyer (fuse-dfs)
+ fuse_dfs notrash option sets usetrash
+
+- HDFS-3675.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs: follow documented return codes
+
+- HDFS-3673.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ libhdfs: fix some compiler warnings
+
+- HDFS-3672.
+ Major improvement reported by Andrew Wang and fixed by Andrew Wang
+ Expose disk-location information for blocks to enable better scheduling
+
+- HDFS-3666.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins
+ Plumb more exception messages to terminate
+
+- HDFS-3665.
+ Major test reported by Eli Collins and fixed by Eli Collins (test)
+ Add a test for renaming across file systems via a symlink
+
+- HDFS-3664.
+ Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (test)
+ BlockManager race when stopping active services
+
+- HDFS-3663.
+ Major improvement reported by Eli Collins and fixed by Eli Collins (test)
+ MiniDFSCluster should capture the code path that led to the first ExitException
+
+- HDFS-3658.
+ Major bug reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE
+ TestDFSClientRetries#testNamenodeRestart failed
+
+- HDFS-3650.
+ Major improvement reported by Andrew Wang and fixed by Andrew Wang
+ Use MutableQuantiles to provide latency histograms for various operations
+
+- HDFS-3646.
+ Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
+ LeaseRenewer can hold reference to inactive DFSClient instances forever
+
+- HDFS-3641.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins
+ Move server Util time methods to common and use now instead of System#currentTimeMillis
+
+- HDFS-3637.
+ Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (data-node , hdfs client , security)
+ Add support for encrypting the DataTransferProtocol
+
+- HDFS-3634.
+ Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs)
+ Add self-contained, mavenized fuse_dfs test
+
+- HDFS-3633.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs: hdfsDelete should pass JNI_FALSE or JNI_TRUE
+
+- HDFS-3629.
+ Trivial bug reported by Brandon Li and fixed by Brandon Li (name-node)
+ fix the typo in the error message about inconsistent storage layout version
+
+- HDFS-3622.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Backport HDFS-3541 to branch-0.23
+
+- HDFS-3615.
+ Major bug reported by Eli Collins and fixed by Aaron T. Myers (security)
+ Two BlockTokenSecretManager findbugs warnings
+
+- HDFS-3613.
+ Trivial improvement reported by Harsh J and fixed by Andrew Wang (name-node)
+ GSet prints some INFO level values, which aren't really very useful to all
+
+- HDFS-3612.
+ Trivial improvement reported by Harsh J and fixed by Andy Isaacson (name-node)
+ Single namenode image directory config warning can be improved
+
+- HDFS-3611.
+ Trivial bug reported by Harsh J and fixed by Colin Patrick McCabe (name-node)
+ NameNode prints unnecessary WARNs about edit log normally skipping a few bytes
+
+- HDFS-3610.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ fuse_dfs: Provide a way to use the default (configured) NN URI
+
+- HDFS-3609.
+ Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs: don't force the URI to look like hdfs://hostname:port
+
+- HDFS-3608.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ fuse_dfs: detect changes in UID ticket cache
+
+- HDFS-3606.
+ Minor test reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs: create self-contained unit test
+
+- HDFS-3605.
+ Major bug reported by Brahma Reddy Battula and fixed by Todd Lipcon (ha , name-node)
+ Block mistakenly marked corrupt during edit log catchup phase of failover
+
+- HDFS-3604.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins
+ Add dfs.webhdfs.enabled to hdfs-default.xml
+
+- HDFS-3603.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (test)
+ Decouple TestHDFSTrash from TestTrash
+
+- HDFS-3597.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
+ SNN can fail to start on upgrade
+
+- HDFS-3591.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Backport HDFS-3357 to branch-0.23
+
+- HDFS-3583.
+ Major improvement reported by Eli Collins and fixed by Andrew Wang (test)
+ Convert remaining tests to Junit4
+
+- HDFS-3582.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
+ Hook daemon process exit for testing
+
+- HDFS-3581.
+ Major bug reported by Eli Collins and fixed by Eli Collins (name-node)
+ FSPermissionChecker#checkPermission sticky bit check missing range check
+
+- HDFS-3580.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
+ incompatible types; no instance(s) of type variable(s) V exist so that V conforms to boolean compiling HttpFSServer.java with OpenJDK
+
+- HDFS-3579.
+ Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs: fix exception handling
+
+- HDFS-3577.
+ Blocker bug reported by Alejandro Abdelnur and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
+ WebHdfsFileSystem can not read files larger than 24KB
+
+- HDFS-3575.
+ Minor bug reported by Brock Noland and fixed by Brock Noland
+ HttpFS does not log Exception Stacktraces
+
+- HDFS-3574.
+ Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
+ Fix small race and do some cleanup in GetImageServlet
+
+- HDFS-3572.
+ Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (name-node , security)
+ Cleanup code which inits SPNEGO in HttpServer
+
+- HDFS-3568.
+ Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ fuse_dfs: add support for security
+
+- HDFS-3559.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ DFSTestUtil: use Builder class to construct DFSTestUtil instances
+
+- HDFS-3555.
+ Major bug reported by Jeff Lord and fixed by Andy Isaacson (data-node , hdfs client)
+ idle client socket triggers DN ERROR log (should be INFO or DEBUG)
+
+- HDFS-3551.
+ Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
+ WebHDFS CREATE does not use client location for redirection
+
+- HDFS-3548.
+ Critical bug reported by Todd Lipcon and fixed by Colin Patrick McCabe (name-node)
+ NamenodeFsck.copyBlock fails to create a Block Reader
+
+- HDFS-3541.
+ Major bug reported by suja s and fixed by Vinay (data-node)
+ Deadlock between recovery, xceiver and packet responder
+
+- HDFS-3539.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ libhdfs code cleanups
+
+- HDFS-3537.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (fuse-dfs , libhdfs)
+ Move libhdfs and fuse-dfs source to native subdirectories
+
+- HDFS-3535.
+ Major new feature reported by Andy Isaacson and fixed by Andy Isaacson (name-node)
+ Audit logging should log denied accesses
+
+- HDFS-3531.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
+ EditLogFileOutputStream#preallocate should check for incomplete writes
+
+- HDFS-3524.
+ Major bug reported by Eli Collins and fixed by Brandon Li (test)
+ TestFileLengthOnClusterRestart failed due to error message change
+
+- HDFS-3522.
+ Major bug reported by Brandon Li and fixed by Brandon Li (name-node)
+ If NN is in safemode, it should throw SafeModeException when getBlockLocations has zero locations
+ getBlockLocations(), and hence open() for read, will now throw SafeModeException if the NameNode is still in safe mode and there are no replicas reported yet for one of the blocks in the file.
+- HDFS-3520.
+ Major improvement reported by Eli Collins and fixed by Eli Collins (name-node)
+ Add transfer rate logging to TransferFsImage
+
+- HDFS-3518.
+ Major bug reported by Bikas Saha and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
+ Provide API to check HDFS operational state
+ Add a utility method HdfsUtils.isHealthy(uri) for checking if the given HDFS is healthy.
+- HDFS-3517.
+ Minor bug reported by Eli Collins and fixed by Eli Collins (test)
+ TestStartup should bind ephemeral ports
+
+- HDFS-3516.
+ Major improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (hdfs client)
+ Check content-type in WebHdfsFileSystem
+
+- HDFS-3514.
+ Major improvement reported by Henry Robinson and fixed by Henry Robinson (test)
+ Add missing TestParallelLocalRead
+
+- HDFS-3513.
+ Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ HttpFS should cache filesystems
+
+- HDFS-3505.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ DirectoryScanner does not join all threads in shutdown
+
+- HDFS-3504.
+ Major improvement reported by Siddharth Seth and fixed by Tsz Wo (Nicholas), SZE
+ Configurable retry in DFSClient
+
+- HDFS-3502.
+ Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ Change INodeFile and INodeFileUnderConstruction to package private
+
+- HDFS-3501.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
+ Checkpointing with security enabled will stop working after ticket lifetime expires
+
+- HDFS-3491.
+ Major bug reported by Romain Rigaux and fixed by Alejandro Abdelnur
+ HttpFs does not set permissions correctly
+
+- HDFS-3490.
+ Minor bug reported by Todd Lipcon and fixed by Tsz Wo (Nicholas), SZE (webhdfs)
+ DN WebHDFS methods throw NPE if Namenode RPC address param not specified
+
+- HDFS-3487.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
+ offlineimageviewer should give byte offset information when it encounters an exception
+
+- HDFS-3486.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (security , tools)
+ offlineimageviewer can't read fsimage files that contain persistent delegation tokens
+
+- HDFS-3485.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
+ DataTransferThrottler will over-throttle when currentTimeMillis jumps
+
+- HDFS-3484.
+ Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
+ hdfs fsck doesn't work if NN HTTP address is set to 0.0.0.0 even if NN RPC address is configured
+
+- HDFS-3481.
+ Major improvement reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ Refactor HttpFS handling of JAX-RS query string parameters
+
+- HDFS-3480.
+ Major bug reported by Eli Collins and fixed by Vinay (build)
+ Multiple SLF4J binding warning
+
+- HDFS-3475.
+ Trivial improvement reported by Harsh J and fixed by Harsh J
+ Make the replication and invalidation rates configurable
+ This change adds two new configuration parameters.
+# {{dfs.namenode.invalidate.work.pct.per.iteration}} for controlling deletion rate of blocks.
+# {{dfs.namenode.replication.work.multiplier.per.iteration}} for controlling replication rate. This in turn allows controlling the time it takes for decommissioning.
+
+Please see hdfs-default.xml for detailed description.
+- HDFS-3474.
+ Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
+ Cleanup Exception handling in BookKeeper journal manager
+
+- HDFS-3469.
+ Minor bug reported by Vinay and fixed by Vinay (auto-failover)
+ start-dfs.sh will start zkfc, but stop-dfs.sh will not stop zkfc similarly.
+
+- HDFS-3468.
+ Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G
+ Make BKJM-ZK session timeout configurable.
+
+- HDFS-3466.
+ Major bug reported by Owen O'Malley and fixed by Owen O'Malley (name-node , security)
+ The SPNEGO filter for the NameNode should come out of the web keytab file
+
+- HDFS-3460.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ HttpFS proxyuser validation with Kerberos ON uses full principal name
+
+- HDFS-3454.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins (balancer)
+ Balancer unconditionally logs InterruptedException at INFO level on shutdown if security is enabled
+
+- HDFS-3452.
+ Blocker sub-task reported by suja s and fixed by Uma Maheswara Rao G
+ BKJM:Switch from standby to active fails and NN gets shut down due to delay in clearing of lock
+
+- HDFS-3446.
+ Major bug reported by Matthew Jacobs and fixed by Matthew Jacobs (name-node)
+ HostsFileReader silently ignores bad includes/excludes
+ HDFS no longer silently ignores missing or unreadable host files specified by dfs.hosts or dfs.hosts.exclude. In order to specify that no hosts should be included or excluded, administrators should either refrain from setting the relevant config properties, or create an empty file in order to represent an empty list.
+- HDFS-3444.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
+ hdfs groups command doesn't work with security enabled
+
+- HDFS-3442.
+ Minor bug reported by suja s and fixed by Andrew Wang
+ Incorrect count for Missing Replicas in FSCK report
+
+- HDFS-3441.
+ Major sub-task reported by suja s and fixed by Rakesh R
+ Race condition between rolling logs at active NN and purging at standby
+
+- HDFS-3440.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ should more effectively limit stream memory consumption when reading corrupt edit logs
+
+- HDFS-3438.
+ Major improvement reported by Todd Lipcon and fixed by Todd Lipcon (ha)
+ BootstrapStandby should not require a rollEdits on active node
+
+- HDFS-3436.
+ Major bug reported by Brahma Reddy Battula and fixed by Vinay (data-node)
+ adding new datanode to existing pipeline fails in case of Append/Recovery
+
+- HDFS-3433.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
+ GetImageServlet should allow administrative requestors when security is enabled
+
+- HDFS-3428.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ move DelegationTokenRenewer to common
+
+- HDFS-3423.
+ Major sub-task reported by Rakesh R and fixed by Ivan Kelly
+ BKJM: NN startup is failing, when tries to recoverUnfinalizedSegments() a bad inProgress_ ZNodes
+
+- HDFS-3422.
+ Minor bug reported by Todd Lipcon and fixed by Todd Lipcon (test)
+ TestStandbyIsHot timeouts too aggressive
+
+- HDFS-3419.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins
+ Cleanup LocatedBlock
+
+- HDFS-3417.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins (data-node)
+ Rename BalancerDatanode#getName to getDisplayName to be consistent with Datanode
+
+- HDFS-3416.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins (data-node)
+ Cleanup DatanodeID and DatanodeRegistration constructors used by testing
+
+- HDFS-3415.
+ Major bug reported by Brahma Reddy Battula and fixed by Brandon Li (name-node)
+ During NameNode starting up, it may pick wrong storage directory inspector when the layout versions of the storage directories are different
+
+- HDFS-3414.
+ Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (balancer)
+ Balancer does not find NameNode if rpc-address or servicerpc-address are not set in client configs
+
+- HDFS-3413.
+ Critical bug reported by Todd Lipcon and fixed by Aaron T. Myers (ha , test)
+ TestFailureToReadEdits timing out
+
+- HDFS-3408.
+ Minor sub-task reported by Rakesh R and fixed by Rakesh R (name-node)
+ BKJM : Namenode format fails, if there is no BK root
+
+- HDFS-3404.
+ Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers
+ Make putImage in GetImageServlet infer remote address to fetch from request
+
+- HDFS-3401.
+ Major improvement reported by Eli Collins and fixed by Eli Collins (data-node , test)
+ Cleanup DatanodeDescriptor creation in the tests
+
+- HDFS-3400.
+ Major improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (data-node , scripts)
+ DNs should be able start with jsvc even if security is disabled
+
+- HDFS-3398.
+ Minor bug reported by Brahma Reddy Battula and fixed by amith (hdfs client)
+ Client will not retry when primaryDN is down once it's just got pipeline
+
+- HDFS-3394.
+ Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ Do not use generic in INodeFile.getLastBlock()
+
+- HDFS-3391.
+ Critical bug reported by Arun C Murthy and fixed by Todd Lipcon
+ TestPipelinesFailover#testLeaseRecoveryAfterFailover is failing
+
+- HDFS-3390.
+ Minor improvement reported by Aaron T. Myers and fixed by Aaron T. Myers (hdfs client)
+ DFSAdmin should print full stack traces of errors when DEBUG logging is enabled
+
+- HDFS-3389.
+ Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
+ Document the BKJM usage in Namenode HA.
+
+- HDFS-3385.
+ Major bug reported by Brahma Reddy Battula and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ ClassCastException when trying to append a file
+
+- HDFS-3372.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tools)
+ offlineEditsViewer should be able to read a binary edits file with recovery mode
+
+- HDFS-3369.
+ Minor sub-task reported by John George and fixed by John George (name-node)
+ change variable names referring to inode in blockmanagement to more appropriate
+
+- HDFS-3368.
+ Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (name-node)
+ Missing blocks due to bad DataNodes coming up and down.
+
+- HDFS-3359.
+ Critical bug reported by Todd Lipcon and fixed by Todd Lipcon (hdfs client)
+ DFSClient.close should close cached sockets
+
+- HDFS-3341.
+ Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon
+ Change minimum RPC versions to 2.0.0-SNAPSHOT instead of 2.0.0
+
+- HDFS-3335.
+ Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ check for edit log corruption at the end of the log
+
+- HDFS-3334.
+ Major bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
+ ByteRangeInputStream leaks streams
+
+- HDFS-3331.
+ Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ setBalancerBandwidth do not checkSuperuserPrivilege
+
+- HDFS-3321.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash
+ Error message for insufficient data nodes to come out of safemode is wrong.
+
+- HDFS-3318.
+ Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
+ Hftp hangs on transfers >2GB
+
+- HDFS-3312.
+ Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
+ Hftp selects wrong token service
+
+- HDFS-3308.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (webhdfs)
+ hftp/webhdfs can't get tokens if authority has no port
+
+- HDFS-3306.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ fuse_dfs: don't lock release operations
+
+- HDFS-3291.
+ Major test reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ add test that covers HttpFS working w/ a non-HDFS Hadoop filesystem
+
+- HDFS-3276.
+ Minor improvement reported by Vinithra Varadharajan and fixed by Todd Lipcon (ha , name-node)
+ initializeSharedEdits should have a -nonInteractive flag
+
+- HDFS-3266.
+ Minor bug reported by Aaron T. Myers and fixed by madhukara phatak
+ DFSTestUtil#waitCorruptReplicas doesn't sleep between checks
+
+- HDFS-3258.
+ Major test reported by Eli Collins and fixed by Junping Du (test)
+ Test for HADOOP-8144 (pseudoSortByDistance in NetworkTopology for first rack local node)
+
+- HDFS-3243.
+ Major bug reported by Todd Lipcon and fixed by Henry Robinson (hdfs client , test)
+ TestParallelRead timing out on jenkins
+
+- HDFS-3235.
+ Minor bug reported by Henry Robinson and fixed by Henry Robinson
+ MiniDFSClusterManager doesn't correctly support -format option
+
+- HDFS-3230.
+ Minor improvement reported by Eli Collins and fixed by Eli Collins (test)
+ Cleanup DatanodeID creation in the tests
+
+- HDFS-3194.
+ Major bug reported by suja s and fixed by Andy Isaacson (data-node)
+ DataNode block scanner is running too frequently
+
+- HDFS-3190.
+ Minor sub-task reported by Todd Lipcon and fixed by Todd Lipcon (name-node)
+ Simple refactors in existing NN code to assist QuorumJournalManager extension
+
+- HDFS-3177.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (data-node , hdfs client)
+ Allow DFSClient to find out and use the CRC type being used for a file.
+
+- HDFS-3176.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (hdfs client)
+ JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
+
+- HDFS-3170.
+ Major improvement reported by Todd Lipcon and fixed by Matthew Jacobs (data-node)
+ Add more useful metrics for write latency
+
+- HDFS-3168.
+ Major sub-task reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ Clean up FSNamesystem and BlockManager
+
+- HDFS-3166.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (hdfs client)
+ Hftp connections do not have a timeout
+
+- HDFS-3157.
+ Major bug reported by J.Andreina and fixed by Ashish Singhi (name-node)
+ Error in deleting block is keep on coming from DN even after the block report and directory scanning has happened
+
+- HDFS-3150.
+ Major new feature reported by Eli Collins and fixed by Eli Collins (data-node , hdfs client)
+ Add option for clients to contact DNs via hostname
+
+- HDFS-3136.
+ Major bug reported by Jason Lowe and fixed by Jason Lowe (build)
+ Multiple SLF4J binding warning
+
+- HDFS-3134.
+ Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (name-node)
+ Harden edit log loader against malformed or malicious input
+
+- HDFS-3113.
+ Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur
+ httpfs does not support delegation tokens
+
+- HDFS-3110.
+ Major improvement reported by Henry Robinson and fixed by Henry Robinson (libhdfs , performance)
+ libhdfs implementation of direct read API
+ libhdfs is enhanced to read directly into user-supplied buffers when possible, reducing the number of memory copies.
+- HDFS-3067.
+ Major bug reported by Henry Robinson and fixed by Henry Robinson (hdfs client)
+ NPE in DFSInputStream.readBuffer if read is repeated on corrupted block
+
+- HDFS-3058.
+ Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
+ HA: Bring BookKeeperJournalManager up to date with HA changes
+
+- HDFS-3054.
+ Major bug reported by patrick white and fixed by Colin Patrick McCabe (tools)
+ distcp -skipcrccheck has no effect
+
+- HDFS-3048.
+ Major bug reported by Eli Collins and fixed by Andy Isaacson (name-node)
+ Small race in BlockManager#close
+
+- HDFS-3042.
+ Major new feature reported by Todd Lipcon and fixed by Todd Lipcon (auto-failover , ha)
+ Automatic failover support for NN HA
+
+- HDFS-3040.
+ Trivial improvement reported by Aaron T. Myers and fixed by madhukara phatak (test)
+ TestMulitipleNNDataBlockScanner is misspelled
+
+- HDFS-3037.
+ Minor bug reported by Aaron T. Myers and fixed by Aaron T. Myers (test)
+ TestMulitipleNNDataBlockScanner#testBlockScannerAfterRestart is racy
+
+- HDFS-3031.
+ Major bug reported by Stephen Chu and fixed by Todd Lipcon (ha)
+ HA: Error (failed to close file) when uploading large file + kill active NN + manual failover
+
+- HDFS-3002.
+ Trivial improvement reported by Suresh Srinivas and fixed by Suresh Srinivas (test)
+ TestNameNodeMetrics need not wait for metrics update with new metrics framework
+
+- HDFS-2988.
+ Minor improvement reported by Todd Lipcon and fixed by Miomir Boljanovic (name-node)
+ Improve error message when storage directory lock fails
+
+- HDFS-2982.
+ Critical bug reported by Todd Lipcon and fixed by Colin Patrick McCabe (name-node)
+ Startup performance suffers when there are many edit log segments
+
+- HDFS-2978.
+ Major new feature reported by Aaron T. Myers and fixed by Aaron T. Myers (name-node)
+ The NameNode should expose name dir statuses via JMX
+
+- HDFS-2966.
+ Minor bug reported by Steve Loughran and fixed by Steve Loughran (test)
+ TestNameNodeMetrics tests can fail under load
+
+- HDFS-2963.
+ Minor bug reported by J.Andreina and fixed by Andrew Wang
+ Console Output is confusing while executing metasave (dfsadmin command)
+
+- HDFS-2914.
+ Major bug reported by Hari Mankude and fixed by Vinay (ha , name-node)
+ HA: Standby should not enter safemode when resources are low
+
+- HDFS-2885.
+ Major improvement reported by Eli Collins and fixed by Tsz Wo (Nicholas), SZE (name-node)
+ Remove "federation" from the nameservice config options
+
+- HDFS-2834.
+ Major improvement reported by Henry Robinson and fixed by Henry Robinson (hdfs client , performance)
+ ByteBuffer-based read API for DFSInputStream
+
+- HDFS-2800.
+ Major bug reported by Aaron T. Myers and fixed by Todd Lipcon (ha , test)
+ HA: TestStandbyCheckpoints.testCheckpointCancellation is racy
+
+- HDFS-2797.
+ Major bug reported by Aaron T. Myers and fixed by Colin Patrick McCabe (ha , name-node)
+ Fix misuses of InputStream#skip in the edit log code
+
+- HDFS-2793.
+ Major new feature reported by Aaron T. Myers and fixed by Todd Lipcon (name-node)
+ Add an admin command to trigger an edit log roll
+ Introduced a new command, "hdfs dfsadmin -rollEdits" which requests that the active NameNode roll its edit log. This can be useful for administrators manually backing up log segments.
+- HDFS-2759.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (ha , name-node)
+ Pre-allocate HDFS edit log files after writing version number
+
+- HDFS-2757.
+ Major bug reported by Jean-Daniel Cryans and fixed by Jean-Daniel Cryans
+ Cannot read a local block that's being written to when using the local read short circuit
+
+- HDFS-2727.
+ Minor improvement reported by Sho Shimauchi and fixed by Colin Patrick McCabe (libhdfs)
+ libhdfs should get the default block size from the server
+ libhdfs now uses the server block size configuration rather than the deprecated dfs.block.size client configuration.
+- HDFS-2717.
+ Major sub-task reported by Ivan Kelly and fixed by Ivan Kelly
+ BookKeeper Journal output stream doesn't check addComplete rc
+
+- HDFS-2686.
+ Major improvement reported by Todd Lipcon and fixed by Suresh Srinivas (data-node , name-node)
+ Remove DistributedUpgrade related code
+ This jira removes functionality that has not been used/applicable since release 0.17. The incompatibility introduced by this change will not affect any HDFS users.
+- HDFS-2652.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp
Port token service changes from 205
- Need to merge the 205 token bug fixes and the feature to enable hostname-based tokens.
-
-- HADOOP-7810.
- Blocker bug reported by johnvijoe and fixed by johnvijoe
- move hadoop archive to core from tools
- "The HadoopArchieves classes are included in the $HADOOP_HOME/hadoop_tools.jar, but this file is not found in `hadoop classpath`.
A Pig script using HCatalog's dynamic partitioning with HAR enabled will therefore fail if a jar with HAR is not included in the pig call's '-cp' and '-Dpig.additional.jars' arguments."
I am not aware of any reason to not include hadoop-tools.jar in 'hadoop classpath'. Will attach a patch soon.
-
-- HADOOP-7811.
- Major bug reported by jeagles and fixed by jeagles (security, test)
- TestUserGroupInformation#testGetServerSideGroups test fails in chroot
- It is common when running in chroot to have root's group vector preserved when running as your self.
For example
# Enter chroot
$ sudo chroot /myroot
# still root
$ whoami
root
# switch to user preserving root's group vector
$ sudo -u user -P -s
# root's groups
$ groups root
a b c
# user's real groups
$ groups user
d e f
# user's effective groups
$ groups
a b c d e f
-------------------------------
-
-- HADOOP-7837.
- Major bug reported by stevel@apache.org and fixed by eli (conf)
- no NullAppender in the log4j config
- running sbin/start-dfs.sh gives me a telling off about no null appender -should one be in the log4j config file.
Full trace (failure expected, but full output not as expected)
{code}
./start-dfs.sh
log4j:ERROR Could not find value for key log4j.appender.NullAppender
log4j:ERROR Could not instantiate appender named "NullAppender".
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
cat: /Users/slo/J...
-
-- HADOOP-7843.
- Major bug reported by johnvijoe and fixed by johnvijoe
- compilation failing because workDir not initialized in RunJar.java
- Compilation is failing on 0.23 and trunk because workDir is not initialized in RunJar.java
-
-- HADOOP-7853.
- Blocker bug reported by daryn and fixed by daryn (security)
- multiple javax security configurations cause conflicts
- Both UGI and the SPNEGO KerberosAuthenticator set the global javax security configuration. SPNEGO stomps on UGI's security config which leads to kerberos/SASL authentication errors.
-
-- HADOOP-7854.
- Critical bug reported by daryn and fixed by daryn (security)
- UGI getCurrentUser is not synchronized
- Sporadic {{ConcurrentModificationExceptions}} are originating from {{UGI.getCurrentUser}} when it needs to create a new instance. The problem was specifically observed in a JT under heavy load when a post-job cleanup is accessing the UGI while a new job is being processed.
-
-- HADOOP-7858.
- Trivial improvement reported by tlipcon and fixed by tlipcon
- Drop some info logging to DEBUG level in IPC, metrics, and HTTP
- Our info level logs have gotten noisier and noisier over time, which is annoying both for users and when looking at unit tests. I'd like to drop a few of the less useful INFO level messages down to DEBUG.
-
-- HADOOP-7859.
- Major bug reported by eli and fixed by eli (fs)
- TestViewFsHdfs.testgetFileLinkStatus is failing an assert
- Probably introduced by HADOOP-7783. I'll fix it.
{noformat}
java.lang.AssertionError
at org.apache.hadoop.fs.FileContext.qualifySymlinkTarget(FileContext.java:1111)
at org.apache.hadoop.fs.FileContext.access$000(FileContext.java:170)
at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1142)
at org.apache.hadoop.fs.FileContext$15.next(FileContext.java:1137)
at org.apache.hadoop.fs.FileContext$FSLinkResolver.resolve(FileContext.java:2327)
at org.apache.hadoop.fs.FileContext.getF...
-
-- HADOOP-7864.
- Major bug reported by abayer and fixed by abayer (build)
- Building mvn site with Maven < 3.0.2 causes OOM errors
- If you try to run mvn site with Maven 3.0.0 (and possibly 3.0.1 - haven't actually tested that), you get hit with unavoidable OOM errors. Switching to Maven 3.0.2 or later fixes this. The enforcer should require 3.0.2 for builds.
-
-- HADOOP-7870.
- Major bug reported by jmhsieh and fixed by jmhsieh
- fix SequenceFile#createWriter with boolean createParent arg to respect createParent.
- After HBASE-6840, one set of calls to createNonRecursive(...) seems fishy - the new boolean createParent variable from the signature isn't used at all.
{code}
+ public static Writer
+ createWriter(FileSystem fs, Configuration conf, Path name,
+ Class keyClass, Class valClass, int bufferSize,
+ short replication, long blockSize, boolean createParent,
+ CompressionType compressionType, CompressionCodec codec,
+ Metadata meta...
-
-- HADOOP-7874.
- Major bug reported by tucu00 and fixed by tucu00 (build)
- native libs should be under lib/native/ dir
- Currently common and hdfs SO files end up under lib/ dir with all JARs, they should end up under lib/native.
In addition, the hadoop-config.sh script needs some cleanup when comes to native lib handling:
* it is using lib/native/${JAVA_PLATFORM} for the java.library.path, when it should use lib/native.
* it is looking for build/lib/native, this is from the old ant build, not applicable anymore.
* it is looking for the libhdfs.a and adding to the java.librar.path, this is not correct.
-
-- HADOOP-7877.
- Major task reported by szetszwo and fixed by szetszwo (documentation)
- Federation: update Balancer documentation
- Update Balancer documentation for the new balancing policy and CLI.
-
-- HADOOP-7878.
- Minor bug reported by stevel@apache.org and fixed by stevel@apache.org (util)
- Regression HADOOP-7777 switch changes break HDFS tests when the isSingleSwitch() predicate is used
- This doesn't show up until you apply the HDFS-2492 patch, but the attempt to make the {{StaticMapping}} topology clever by deciding if it is single rack or multi rack based on its rack->node mapping breaks the HDFS {{TestBlocksWithNotEnoughRacks}} test. Why? Because the racks go in after the switch topology is cached by the {{BlockManager}}, which assumes the system is always single-switch.
Fix: default to assuming multi-switch; remove the intelligence, add a setter for anyone who really wan...
-
-- HADOOP-7887.
- Critical bug reported by tucu00 and fixed by tucu00 (security)
- KerberosAuthenticatorHandler is not setting KerberosName name rules from configuration
- While the KerberosAuthenticatorHandler defines the name rules property, it does not set it in KerberosName.
-
-- HADOOP-7890.
- Trivial improvement reported by knoguchi and fixed by knoguchi (scripts)
- Redirect hadoop script's deprecation message to stderr
- $ hadoop dfs -ls
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
...
If we're still letting the command run, I think we should redirect the deprecation message to stderr in case users have a script taking the output from stdout.
-
-- HADOOP-7898.
- Minor bug reported by sureshms and fixed by sureshms (security)
- Fix javadoc warnings in AuthenticationToken.java
- Fix the following javadoc warning:
[WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java:33: warning - Tag @link: reference not found: HttpServletRequest
[WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationToken.java...
-
-- HADOOP-7902.
- Major bug reported by szetszwo and fixed by tucu00
- skipping name rules setting (if already set) should be done on UGI initialization only
- Both TestDelegationToken and TestOfflineEditsViewer are currently failing.
-
-- HADOOP-7907.
- Blocker bug reported by tucu00 and fixed by tucu00 (build)
- hadoop-tools JARs are not part of the distro
- After mavenizing streaming, the hadoop-streaming JAR is not part of the final tar.
-
-- HADOOP-7910.
- Minor improvement reported by sho.shimauchi and fixed by sho.shimauchi (conf)
- add configuration methods to handle human readable size values
- It's better to have a new configuration methods which handle human readable size values.
For example, see HDFS-1314.
-
-- HADOOP-7912.
- Major bug reported by revans2 and fixed by revans2 (build)
- test-patch should run eclipse:eclipse to verify that it does not break again
- Recently the eclipse:eclipse build was broken. If we are going to document this on the wiki and have many developers use it we should verify that it always works.
-
-- HADOOP-7914.
- Major bug reported by szetszwo and fixed by szetszwo (build)
- duplicate declaration of hadoop-hdfs test-jar
- [WARNING] Some problems were encountered while building the effective model for org.apache.hadoop:hadoop-common-project:pom:0.24.0-SNAPSHOT
[WARNING] 'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.hadoop:hadoop-hdfs:test-jar -> duplicate declaration of version ${project.version} @ org.apache.hadoop:hadoop-project:0.24.0-SNAPSHOT, /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-project/pom.xml, line 140, ...
-
-- HADOOP-7917.
- Major bug reported by tucu00 and fixed by tucu00 (build)
- compilation of protobuf files fails in windows/cygwin
- HADOOP-7899 & HDFS-2511 introduced compilation of proto files as part of the build.
Such compilation is failing in windows/cygwin
-
-- HADOOP-7919.
- Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (documentation)
- [Doc] Remove hadoop.logfile.* properties.
- The following only resides in core-default.xml and doesn't look like its used anywhere at all. At least a grep of the prop name and parts of it does not give me back anything at all.
These settings are now configurable via generic Log4J opts, via the shipped log4j.properties file in the distributions.
{code}
137 <!--- logging properties -->
138
139 <property>
140 <name>hadoop.logfile.size</name>
141 <value>10000000</value>
142 <description>The max size of each log file</description>
...
-
-- HADOOP-7933.
- Critical bug reported by sseth and fixed by sseth
- Viewfs changes for MAPREDUCE-3529
- ViewFs.getDelegationTokens returns a list of tokens for the associated namenodes. Credentials serializes these tokens using the service name for the actual namenodes. Effectively, tokens are not cached for viewfs (some more details in MR 3529). Affects any job which uses the TokenCache in tasks along with viewfs (some Pig jobs).
Talk to Jitendra about this, some options
1. Change Credentials.getAllTokens to return the key, instead of just a token list (associate the viewfs canonical name wit...
-
-- HADOOP-7934.
- Critical improvement reported by tucu00 and fixed by tucu00 (build)
- Normalize dependencies versions across all modules
- Move all dependencies versions to the dependencyManagement section in the hadoop-project POM
Move all plugin versions to the dependencyManagement section in the hadoop-project POM
-
-- HADOOP-7936.
- Major bug reported by eli and fixed by tucu00 (build)
- There's a Hoop README in the root dir of the tarball
- The Hoop README.txt is now in the root dir of the tarball.
{noformat}
hadoop-trunk1 $ tar xvzf hadoop-dist/target/hadoop-0.24.0-SNAPSHOT.tar.gz -C /tmp/
..
hadoop-trunk1 $ head -n3 /tmp/hadoop-0.24.0-SNAPSHOT/README.txt
-----------------------------------------------------------------------------
HttpFS - Hadoop HDFS over HTTP
{noformat}
-
-- HADOOP-7939.
- Major improvement reported by rvs and fixed by rvs (build, conf, documentation, scripts)
- Improve Hadoop subcomponent integration in Hadoop 0.23
- h1. Introduction
For the rest of this proposal it is assumed that the current set
of Hadoop subcomponents is:
* hadoop-common
* hadoop-hdfs
* hadoop-yarn
* hadoop-mapreduce
It must be noted that this is an open ended list, though. For example,
implementations of additional frameworks on top of yarn (e.g. MPI) would
also be considered a subcomponent.
h1. Problem statement
Currently there's an unfortunate coupling and hard-coding present at the
level of launcher scripts, configuration s...
-
-- HADOOP-7948.
- Minor bug reported by cim_michajlomatijkiw and fixed by cim_michajlomatijkiw (build)
- Shell scripts created by hadoop-dist/pom.xml to build tar do not properly propagate failure
- The run() function, as defined in dist-layout-stitching.sh and dist-tar-stitching, created in hadoop-dist/pom.xml, does not properly propagate the error code of a failing command. See the following:
{code}
...
"${@}" # call fails with non-zero exit code
if [ $? != 0 ]; then
echo
echo "Failed!"
echo
exit $? # $?=result of echo above, likely 0, thus exit with code 0
...
{code}
-
-- HADOOP-7949.
- Trivial bug reported by eli and fixed by eli (ipc)
- Updated maxIdleTime default in the code to match core-default.xml
- HADOOP-2909 intended to set the server max idle time for a connection to twice the client value. ("The server-side max idle time should be greater than the client-side max idle time, for example, twice of the client-side max idle time.") This way when a server times out a connection it's due a crashed client and not an inactive client so we don't close client connections with outstanding requests (by setting 2x the client value on the server side the client should time out the connection firs...
-
-- HADOOP-7964.
- Blocker bug reported by kihwal and fixed by daryn (security, util)
- Deadlock in class init.
- After HADOOP-7808, client-side commands hang occasionally. There are cyclic dependencies in NetUtils and SecurityUtil class initialization. Upon initial look at the stack trace, two threads deadlock when they hit the either of class init the same time.
-
-- HADOOP-7971.
- Blocker bug reported by tgraves and fixed by prashant_
- hadoop <job/queue/pipes> removed - should be added back, but deprecated
- The mapred subcommands (mradmin|jobtracker|tasktracker|pipes|job|queue)
were removed from the /bin/hadoop command. I believe for backwards compatibility at least some of these should have stayed along with the deprecated warnings.
-
-- HADOOP-7974.
- Major bug reported by eli and fixed by qwertymaniac (fs)
- TestViewFsTrash incorrectly determines the user's home directory
- HADOOP-7284 added a test called TestViewFsTrash which contains the following code to determine the user's home directory. It only works if the user's directory is one level deep, and breaks if the home directory is more than one level deep (eg user hudson, who's home dir might be /usr/lib/hudson instead of /home/hudson).
{code}
// create a link for home directory so that trash path works
// set up viewfs's home dir root to point to home dir root on target
// But home dir is diffe...
-
-- HADOOP-7975.
- Minor bug reported by qwertymaniac and fixed by qwertymaniac
- Add entry to XML defaults for new LZ4 codec
- HADOOP-7657 added in a new LZ4 codec, but failed to extend the io.compression.codecs list which MR/etc. use up to load codecs.
We should add an entry to the core-default XML for this new codec, just as we did with Snappy.
-
-- HADOOP-7981.
- Major bug reported by jeagles and fixed by jeagles (io)
- Improve documentation for org.apache.hadoop.io.compress.Decompressor.getRemaining
-
-- HADOOP-7982.
- Major bug reported by tlipcon and fixed by tlipcon (security)
- UserGroupInformation fails to login if thread's context classloader can't load HadoopLoginModule
- In a few hard-to-reproduce situations, we've seen a problem where the UGI login call causes a failure to login exception with the following cause:
Caused by: javax.security.auth.login.LoginException: unable to find
LoginModule class: org.apache.hadoop.security.UserGroupInformation
$HadoopLoginModule
After a bunch of debugging, I determined that this happens when the login occurs in a thread whose Context ClassLoader has been set to null.
-
-- HADOOP-7987.
- Major improvement reported by devaraj and fixed by jnp (security)
- Support setting the run-as user in unsecure mode
- Some applications need to be able to perform actions (such as launch MR jobs) from map or reduce tasks. In earlier unsecure versions of hadoop (20.x), it was possible to do this by setting user.name in the configuration. But in 20.205 and 1.0, when running in unsecure mode, this does not work. (In secure mode, you can do this using the kerberos credentials).
-
-- HADOOP-7988.
- Major bug reported by jnp and fixed by jnp
- Upper case in hostname part of the principals doesn't work with kerberos.
- Kerberos doesn't like upper case in the hostname part of the principals.
This issue has been seen in 23 as well as 1.0.
-
-- HADOOP-7993.
- Major bug reported by anupamseth and fixed by anupamseth (conf)
- Hadoop ignores old-style config options for enabling compressed output
- Hadoop seems to ignore the config options even though they are printed as deprecation warnings in the log: mapred.output.compress and
mapred.output.compression.codec
- settings that work on 0.20 but not on 0.23
mapred.output.compress=true
mapred.output.compression.codec=org.apache.hadoop.io.compress.BZip2Codec
- settings that work on 0.23
mapreduce.output.fileoutputformat.compress=true
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec
This breaks bac...
-
-- HADOOP-7997.
- Major bug reported by gchanan and fixed by gchanan (io)
- SequenceFile.createWriter(...createParent...) no longer works on existing file
- SequenceFile.createWriter no longer works on an existing file, because old version specified OVEWRITE by default and new version does not. This breaks some HBase tests.
Tested against trunk.
Patch with test to follow.
-
-- HADOOP-7998.
- Major bug reported by daryn and fixed by daryn (fs)
- CheckFileSystem does not correctly honor setVerifyChecksum
- Regardless of the verify checksum flag, {{ChecksumFileSystem#open}} will instantiate a {{ChecksumFSInputChecker}} instead of a normal stream.
-
-- HADOOP-7999.
- Critical bug reported by jlowe and fixed by jlowe (scripts)
- "hadoop archive" fails with ClassNotFoundException
- Running "hadoop archive" from a command prompt results in this error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/tools/HadoopArchives
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.tools.HadoopArchives
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
...
-
-- HADOOP-8000.
- Critical bug reported by arpitgupta and fixed by arpitgupta
- fetchdt command not available in bin/hadoop
- fetchdt command needs to be added to bin/hadoop to allow for backwards compatibility.
-
-- HADOOP-8001.
- Major bug reported by daryn and fixed by daryn (fs)
- ChecksumFileSystem's rename doesn't correctly handle checksum files
- Rename will move the src file and its crc *if present* to the destination. If the src file has no crc, but the destination already exists with a crc, then src will be associated with the old file's crc. Subsequent access to the file will fail with checksum errors.
-
-- HADOOP-8002.
- Major bug reported by arpitgupta and fixed by arpitgupta
- SecurityUtil acquired token message should be a debug rather than info
+ - HDFS-2619.
+ Major bug reported by Owen O'Malley and fixed by Owen O'Malley (build)
+ Remove my personal email address from the libhdfs build file.
-
-- HADOOP-8006.
- Major bug reported by umamaheswararao and fixed by daryn (fs)
- TestFSInputChecker is failing in trunk.
- Trunk build number 939 failed with TestFSInputChecker.
https://builds.apache.org/job/Hadoop-Hdfs-trunk/939/
junit.framework.AssertionFailedError: expected:<10> but was:<0>
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:283)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:130)
at junit.framework.Assert.assertEquals(Assert.java:136)
at org.apache.hadoop.hdfs.TestFSInputChecker.ch...
-
-- HADOOP-8009.
- Critical improvement reported by tucu00 and fixed by tucu00 (build)
- Create hadoop-client and hadoop-minicluster artifacts for downstream projects
- Using Hadoop from projects like Pig/Hive/Sqoop/Flume/Oozie or any in-house system that interacts with Hadoop is quite challenging for the following reasons:
* *Different versions of Hadoop produce different artifacts:* Before Hadoop 0.23 there was a single artifact hadoop-core, starting with Hadoop 0.23 there are several (common, hdfs, mapred*, yarn*)
* *There are no 'client' artifacts:* Current artifacts include all JARs needed to run the services, thus bringing into clients several JARs t...
-
-- HADOOP-8012.
- Minor bug reported by rvs and fixed by rvs (scripts)
- hadoop-daemon.sh and yarn-daemon.sh are trying to mkdir and chow log/pid dirs which can fail
- Here's what I see when using Hadoop in Bigtop:
{noformat}
$ sudo /sbin/service hadoop-hdfs-namenode start
Starting Hadoop namenode daemon (hadoop-namenode): chown: changing ownership of `/var/log/hadoop': Operation not permitted
starting namenode, logging to /var/log/hadoop/hadoop-hdfs-namenode-centos5.out
{noformat}
This is a cosmetic issue, but it would be nice to fix it.
-
-- HADOOP-8013.
- Major bug reported by daryn and fixed by daryn (fs)
- ViewFileSystem does not honor setVerifyChecksum
- {{ViewFileSystem#setVerifyChecksum}} is a no-op. It should call {{setVerifyChecksum}} on the mount points.
-
-- HADOOP-8015.
- Major improvement reported by daryn and fixed by daryn (fs)
- ChRootFileSystem should extend FilterFileSystem
- {{ChRootFileSystem}} simply extends {{FileSystem}}, and attempts to delegate some methods to the underlying mount point. It is essentially the same as {{FilterFileSystem}} but it mangles the paths to include the chroot path. Unfortunately {{ChRootFileSystem}} is not delegating some methods that should be delegated. Changing the inheritance will prevent a copy-n-paste of code for HADOOP-8013 and HADOOP-8014 into both {{ChRootFileSystem}} and {{FilterFileSystem}}.
-
-- HADOOP-8018.
- Major bug reported by mattf and fixed by jeagles (build, test)
- Hudson auto test for HDFS has started throwing javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
- Hudson automated testing has started failing with one javadoc warning message, consisting of
javadoc: warning - Error fetching URL: http://java.sun.com/javase/6/docs/api/package-list
This may be due to Oracle's decommissioning of the sun.com domain. If one tries to access it manually, it is redirected to
http://download.oracle.com/javase/6/docs/api/package-list
So it looks like a build script needs to be updated.
-
-- HADOOP-8027.
- Minor improvement reported by qwertymaniac and fixed by atm (metrics)
- Visiting /jmx on the daemon web interfaces may print unnecessary error in logs
- Logs that follow a {{/jmx}} servlet visit:
{code}
11/11/22 12:09:52 ERROR jmx.JMXJsonServlet: getting attribute UsageThreshold of java.lang:type=MemoryPool,name=Par Eden Space threw an exception
javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:856)
...
{code}
-
-- HDFS-69.
- Minor bug reported by raviphulari and fixed by qwertymaniac
- Improve dfsadmin command line help
- Enhance dfsadmin command line help informing "A quota of one forces a directory to remain empty"
-
-- HDFS-362.
- Major improvement reported by szetszwo and fixed by umamaheswararao (name-node)
- FSEditLog should not writes long and short as UTF8 and should not use ArrayWritable for writing non-array items
- In FSEditLog,
- long and short are first converted to String and are further converted to UTF8
- For some non-array items, it first create an ArrayWritable object to hold all the items and then writes the ArrayWritable object.
These result creating many intermediate objects which affects Namenode CPU performance and Namenode restart.
-
-- HDFS-442.
- Minor bug reported by rramya and fixed by qwertymaniac (test)
- dfsthroughput in test.jar throws NPE
- On running hadoop jar hadoop-test.jar dfsthroughput OR hadoop org.apache.hadoop.hdfs.BenchmarkThroughput, we get NullPointerException. Below is the stacktrace:
{noformat}
Exception in thread "main" java.lang.NullPointerException
at java.util.Hashtable.put(Hashtable.java:394)
at java.util.Properties.setProperty(Properties.java:143)
at java.lang.System.setProperty(System.java:731)
at org.apache.hadoop.hdfs.BenchmarkThroughput.run(BenchmarkThroughput.java:198)
...
-
-- HDFS-554.
- Minor improvement reported by stevel@apache.org and fixed by qwertymaniac (name-node)
- BlockInfo.ensureCapacity may get a speedup from System.arraycopy()
- BlockInfo.ensureCapacity() uses a for() loop to copy the old array data into the expanded array. {{System.arraycopy()}} is generally much faster for this, as it can do a bulk memory copy. There is also the typesafe Java6 {{Arrays.copyOf()}} to consider, though here it offers no tangible benefit.
-
-- HDFS-2178.
- Major improvement reported by tucu00 and fixed by tucu00
- HttpFS - a read/write Hadoop file system proxy
- We'd like to contribute Hoop to Hadoop HDFS as a replacement (an improvement) for HDFS Proxy.
Hoop provides access to all Hadoop Distributed File System (HDFS) operations (read and write) over HTTP/S.
The Hoop server component is a REST HTTP gateway to HDFS supporting all file system operations. It can be accessed using standard HTTP tools (i.e. curl and wget), HTTP libraries from different programing languages (i.e. Perl, Java Script) as well as using the Hoop client. The Hoop server compo...
-
-- HDFS-2335.
- Major improvement reported by eli and fixed by umamaheswararao (data-node, name-node)
- DataNodeCluster and NNStorage always pull fresh entropy
- Jira for giving DataNodeCluster and NNStorage the same treatment as HDFS-1835. They're not truly cryptographic uses as well. We should also factor this out to a utility method, seems like the three uses are slightly different, eg one uses DFSUtil.getRandom and the other creates a new Random object.
-
-- HDFS-2349.
- Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
- DN should log a WARN, not an INFO when it detects a corruption during block transfer
- Currently, in DataNode.java, we have:
{code}
LOG.info("Can't replicate block " + block
+ " because on-disk length " + onDiskLength
+ " is shorter than NameNode recorded length " + block.getNumBytes());
{code}
This log is better off as a WARN as it indicates (and also reports) a corruption.
-
-- HDFS-2397.
- Major improvement reported by tlipcon and fixed by eli (name-node)
- Undeprecate SecondaryNameNode
- I would like to consider un-deprecating the SecondaryNameNode for 0.23, and amending the documentation to indicate that it is still the most trust-worthy way to run checkpoints, and while CN/BN may have some advantages, they're not battle hardened as of yet. The test coverage for the 2NN is far superior to the CheckpointNode or BackupNode, and people have a lot more production experience. Indicating that it is deprecated before we have expanded test coverage of the CN/BN won't send the right ...
-
-- HDFS-2454.
- Minor improvement reported by umamaheswararao and fixed by qwertymaniac (data-node)
- Move maxXceiverCount check to before starting the thread in dataXceiver
- We can hoist the maxXceiverCount out of DataXceiverServer#run, there's no need to check each time we accept a connection, we can accept when we create a thread.
-
-- HDFS-2502.
- Minor improvement reported by eli and fixed by qwertymaniac (documentation)
- hdfs-default.xml should include dfs.name.dir.restore
+ - HDFS-2617.
+ Major improvement reported by Jakob Homan and fixed by Jakob Homan (security)
+ Replaced Kerberized SSL for image transfer and fsck with SPNEGO-based solution
+ Due to the requirement that KSSL use weak encryption types for Kerberos tickets, HTTP authentication to the NameNode will now use SPNEGO by default. This will require users of previous branch-1 releases with security enabled to modify their configurations and create new Kerberos principals in order to use SPNEGO. The old behavior of using KSSL can optionally be enabled by setting the configuration option "hadoop.security.use-weak-http-crypto" to "true".
+- HDFS-2421.
+ Major improvement reported by Hairong Kuang and fixed by Jing Zhao (name-node)
+ Improve the concurrency of SerialNumberMap in NameNode
-
-- HDFS-2511.
- Minor improvement reported by tlipcon and fixed by tucu00 (build)
- Add dev script to generate HDFS protobufs
- Would like to add a simple shell script to re-generate the protobuf code in HDFS -- just easier than remembering the right syntax.
-
-- HDFS-2533.
- Minor improvement reported by tlipcon and fixed by tlipcon (data-node, performance)
- Remove needless synchronization on FSDataSet.getBlockFile
- HDFS-1148 discusses lock contention issues in FSDataset. It provides a more comprehensive fix, converting it all to RWLocks, etc. This JIRA is for one very specific fix which gives a decent performance improvement for TestParallelRead: getBlockFile() currently holds the lock which is completely unnecessary.
-
-- HDFS-2536.
- Trivial improvement reported by atm and fixed by qwertymaniac (name-node)
- Remove unused imports
- Looks like it has 11 unused imports by my count.
-
-- HDFS-2541.
- Major bug reported by qwertymaniac and fixed by qwertymaniac (data-node)
- For a sufficiently large value of blocks, the DN Scanner may request a random number with a negative seed value.
- Running off 0.20-security, I noticed that one could get the following exception when scanners are used:
{code}
DataXceiver
java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:250)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.getNewBlockScanTime(DataBlockScanner.java:251)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.addBlock(DataBlockScanner.java:268)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(Da...
-
-- HDFS-2543.
- Major bug reported by bmahe and fixed by bmahe (scripts)
- HADOOP_PREFIX cannot be overriden
- hadoop-config.sh forces HADOOP_prefix to a specific value:
export HADOOP_PREFIX=`dirname "$this"`/..
It would be nice to make this overridable.
-
-- HDFS-2544.
- Major bug reported by bmahe and fixed by bmahe (scripts)
- Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
- It would be nice to be able to specify some other location for hadoop-config.sh
-
-- HDFS-2545.
- Major bug reported by szetszwo and fixed by szetszwo
- Webhdfs: Support multiple namenodes in federation
- DatanodeWebHdfsMethods only talks to the default namenode. It won't work if there are multiple namenodes in federation.
-
-- HDFS-2552.
- Major task reported by szetszwo and fixed by szetszwo (documentation)
- Add WebHdfs Forrest doc
+ - HDFS-2391.
+ Major improvement reported by Rajit Saha and fixed by Harsh J (balancer)
+ Newly set BalancerBandwidth value is not displayed anywhere
-
-- HDFS-2553.
- Critical bug reported by tlipcon and fixed by umamaheswararao (data-node)
- BlockPoolSliceScanner spinning in loop
- Playing with trunk, I managed to get a DataNode in a situation where the BlockPoolSliceScanner is spinning in the following loop, using 100% CPU:
at org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.isAlive(DataNode.java:820)
at org.apache.hadoop.hdfs.server.datanode.DataNode.isBPServiceAlive(DataNode.java:2962)
at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:625)
at org.apache.hadoop.hdfs.server.data...
-
-- HDFS-2560.
- Major improvement reported by tlipcon and fixed by tlipcon (data-node)
- Refactor BPOfferService to be a static inner class
- Currently BPOfferService is a non-static inner class of DataNode. For HA we are adding another inner class inside of this, which makes the scope very hard to understand when reading the code (and has resulted in subtle bugs like HDFS-2529 where a variable is referenced from the wrong scope. Making it a static inner class with a reference to the DN has two advantages: a) scope is now explicit, and b) enables unit testing of the BPOS against a mocked-out DN.
-
-- HDFS-2562.
- Minor improvement reported by tlipcon and fixed by tlipcon (data-node)
- Refactor DN configuration variables out of DataNode class
- Right now there are many member variables in DataNode.java which are just read from configuration when the DN is started. Similar to what we did with DFSClient, we should refactor them into a new DNConf class which can be passed around - the motivation is to remove the many references we have throughout the code that read package-protected members of DataNode and reduce the number of members in DataNode itself.
-
-- HDFS-2563.
- Major improvement reported by tlipcon and fixed by tlipcon (data-node)
- Some cleanup in BPOfferService
- BPOfferService is currently rather difficult to follow and not really commented. This JIRA is to clean up the code a bit, add javadocs/comments where necessary, and improve the formatting of the log messages.
-
-- HDFS-2566.
- Minor improvement reported by tlipcon and fixed by tlipcon (data-node)
- Move BPOfferService to be a non-inner class
- Rounding out the cleanup of BPOfferService, it would be good to move it to its own file, so it's no longer an inner class. DataNode.java is really large and hard to navigate. BPOfferService itself is ~700 lines, so seems like a large enough unit to merit its own file.
-
-- HDFS-2567.
- Major bug reported by qwertymaniac and fixed by qwertymaniac (name-node)
- When 0 DNs are available, show a proper error when trying to browse DFS via web UI
- Trace:
{code}
HTTP ERROR 500
Problem accessing /nn_browsedfscontent.jsp. Reason:
n must be positive
Caused by:
java.lang.IllegalArgumentException: n must be positive
at java.util.Random.nextInt(Random.java:250)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:556)
at org.apache.hadoop.net.NetworkTopology.chooseRandom(NetworkTopology.java:524)
at org.apache.hadoop.hdfs.server.namenode.NamenodeJspHelper.getRandomDatanode(NamenodeJspHelper.java:372)
at org....
-
-- HDFS-2568.
- Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
- Use a set to manage child sockets in XceiverServer
- Found while reading up for HDFS-2454, currently we maintain childSockets in a DataXceiverServer as a Map<Socket,Socket>. This can very well be a Set<Socket> data structure -- since the goal is easy removals.
-
-- HDFS-2570.
- Trivial improvement reported by eli and fixed by eli (documentation)
- Add descriptions for dfs.*.https.address in hdfs-default.xml
- Let's add descriptions for dfs.*.https.address in hdfs-default.xml.
-
-- HDFS-2572.
- Trivial improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
- Unnecessary double-check in DN#getHostName
- We do a double config.get unnecessarily inside DN#getHostName(...). Can be removed by this patch.
-
-- HDFS-2574.
- Trivial task reported by joecrobak and fixed by joecrobak (documentation)
- remove references to deprecated properties in hdfs-site.xml template and hdfs-default.xml
- Some examples: hadoop-hdfs/src/main/packages/templates/conf/hdfs-site.xml contains an entry for dfs.name.dir rather than dfs.namenode.name.dir and hdfs-default.xml references dfs.name.dir twice in <description> tags rather than using dfs.namenode.name.dir.
List of deprecated properties is here: http://hadoop.apache.org/common/docs/r0.23.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html
-
-- HDFS-2575.
- Minor bug reported by tlipcon and fixed by tlipcon (test)
- DFSTestUtil may create empty files
- DFSTestUtil creates files with random sizes, but there is no minimum size. So, sometimes, it can make a file with length 0. This will cause tests that use this functionality to fail - for example, TestListCorruptFileBlocks assumes that each of the created files has at least one block. We should add a minSize parameter to prevent this.
-
-- HDFS-2587.
- Major task reported by szetszwo and fixed by szetszwo (documentation)
- Add WebHDFS apt doc
- This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552).
-
-- HDFS-2588.
- Trivial bug reported by davevr and fixed by davevr (scripts)
- hdfs jsp pages missing DOCTYPE [post-split branches]
- Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch.
-
-- HDFS-2590.
- Major bug reported by szetszwo and fixed by szetszwo (documentation)
- Some links in WebHDFS forrest doc do not work
- Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default.
-
-- HDFS-2594.
- Critical bug reported by tucu00 and fixed by szetszwo
- webhdfs HTTP API should implement getDelegationTokens() instead getDelegationToken()
- The current API returns a single delegation token, that method from the FileSystem API is deprecated in favor of the one that returns a list of tokens. The HTTP API should implement the new/undeprecated signature getDelegationTokens().
-
-- HDFS-2596.
- Major bug reported by eli and fixed by eli (data-node, test)
- TestDirectoryScanner doesn't test parallel scans
- The code from HDFS-854 below doesn't run the test with parallel scanning. They probably intended "parallelism < 3".
{code}
+ public void testDirectoryScanner() throws Exception {
+ // Run the test with and without parallel scanning
+ for (int parallelism = 1; parallelism < 2; parallelism++) {
+ runTest(parallelism);
+ }
+ }
{code}
-
-- HDFS-2604.
- Minor improvement reported by szetszwo and fixed by szetszwo (data-node, documentation, name-node)
- Add a log message to show if WebHDFS is enabled
- WebHDFS can be enabled/disabled by the conf key {{dfs.webhdfs.enabled}}. Let's add a log message to show if it is enabled.
-
-- HDFS-2606.
- Critical bug reported by tucu00 and fixed by tucu00 (hdfs client)
- webhdfs client filesystem impl must set the content-type header for create/append
- Currently the content-type header is not being set and for some reason for append it is being set to the form encoded content type making jersey parameter parsing fail.
For this and to avoid any kind of proxy transcoding the content-type should be set to binary.
{code}
conn.setRequestProperty("Content-Type", "application/octet-stream");
{code}
-
-- HDFS-2614.
- Major bug reported by bmahe and fixed by tucu00 (build)
- hadoop dist tarball is missing hdfs headers
- It would be nice to provide hdfs header so one could easily write programs to be linked against that library and access HDFS
-
-- HDFS-2640.
- Major bug reported by tomwhite and fixed by tomwhite
- Javadoc generation hangs
- Typing 'mvn javadoc:javadoc' causes the build to hang.
-
-- HDFS-2646.
- Major bug reported by umamaheswararao and fixed by tucu00
- Hadoop HttpFS introduced 4 findbug warnings.
- https://builds.apache.org/job/PreCommit-HDFS-Build/1665//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
-
-- HDFS-2649.
- Major bug reported by jlowe and fixed by jlowe (build)
- eclipse:eclipse build fails for hadoop-hdfs-httpfs
- Building the eclipse:eclipse target fails in the hadoop-hdfs-httpfs project with this error:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-eclipse-plugin:2.8:eclipse (default-cli) on project hadoop-hdfs-httpfs: Request to merge when 'filtering' is not identical. Original=resource src/main/resources: output=target/classes, include=[httpfs.properties], exclude=[**/*.java], test=false, filtering=true, merging with=resource src/main/resources: output=target/classes, include=[], e...
-
-- HDFS-2653.
- Major improvement reported by eli and fixed by eli (data-node)
- DFSClient should cache whether addrs are non-local when short-circuiting is enabled
- Something Todd mentioned to me off-line.. currently DFSClient doesn't cache the fact that non-local reads are non-local, so if short-circuiting is enabled every time we create a block reader we'll go through the isLocalAddress code path. We should cache the fact that an addr is non-local as well.
-
-- HDFS-2654.
- Major improvement reported by eli and fixed by eli (data-node)
- Make BlockReaderLocal not extend RemoteBlockReader2
- The BlockReaderLocal code paths are easier to understand (especially true on branch-1 where BlockReaderLocal inherits code from BlockerReader and FSInputChecker) if the local and remote block reader implementations are independent, and they're not really sharing much code anyway. If for some reason they start to share significant code we can make the BlockReader interface an abstract class.
-
-- HDFS-2657.
- Major bug reported by eli and fixed by tucu00
- TestHttpFSServer and TestServerWebApp are failing on trunk
- >>> org.apache.hadoop.fs.http.server.TestHttpFSServer.instrumentation
>>> org.apache.hadoop.lib.servlet.TestServerWebApp.lifecycle
-
-- HDFS-2658.
- Major bug reported by eli and fixed by tucu00
- HttpFS introduced 70 javadoc warnings
- {noformat}
hadoop1 (trunk)$ grep warning javadoc.txt |grep -c httpfs
70
{noformat}
-
-- HDFS-2675.
- Trivial improvement reported by tlipcon and fixed by tlipcon (name-node)
- Reduce verbosity when double-closing edit logs
- Currently the edit logs log at WARN level when they're double-closed. But this happens in the normal flow of things, so we may as well reduce it to DEBUG to reduce log spam in unit tests, etc.
-
-- HDFS-2705.
- Major bug reported by tucu00 and fixed by tucu00
- HttpFS server should check that upload requests have correct content-type
- The append/create requests should require 'application/octet-stream' as content-type when uploading data. This is to prevent the default content-type form-encoded (used as default by some HTTP libraries) to be used or text based content-types to be used.
If the form-encoded content type is used, then Jersey tries to process the upload stream as parameters
If a test base content-type is used, HTTP proxies/gateways could do attempt some transcoding on the stream thus corrupting the data.
-
-- HDFS-2706.
- Major bug reported by szetszwo and fixed by szetszwo (name-node)
- Use configuration for blockInvalidateLimit if it is set
- HDFS-2191 accidentally removed the following code.
{code}
- this.blockInvalidateLimit = conf.getInt(
- DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_KEY, this.blockInvalidateLimit);
{code}
-
-- HDFS-2707.
- Major bug reported by tucu00 and fixed by tucu00 (security)
- HttpFS should read the hadoop-auth secret from a file instead inline from the configuration
- Similar to HADOOP-7621, the secret should be in a file other than the configuration file.
-
-- HDFS-2710.
- Critical bug reported by sseth and fixed by
- HDFS part of MAPREDUCE-3529, HADOOP-7933
- viewfs implementation of getDelegationTokens(String, Credentials)
-
-- HDFS-2722.
- Major bug reported by qwertymaniac and fixed by qwertymaniac (hdfs client)
- HttpFs shouldn't be using an int for block size
- {{./hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/FSOperations.java: blockSize = fs.getConf().getInt("dfs.block.size", 67108864);}}
Should instead be using dfs.blocksize and should instead be long.
I'll post a patch for this after HDFS-1314 is resolved -- which changes the internal behavior a bit (should be getLongBytes, and not just getLong, to gain formatting advantages).
-
-- HDFS-2726.
- Major improvement reported by bien and fixed by qwertymaniac
- "Exception in createBlockOutputStream" shouldn't delete exception stack trace
- I'm occasionally (1/5000 times) getting this error after upgrading everything to hadoop-0.18:
08/09/09 03:28:36 INFO dfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream
08/09/09 03:28:36 INFO dfs.DFSClient: Abandoning block blk_624229997631234952_8205908
DFSClient contains the logging code:
LOG.info("Exception in createBlockOutputStream " + ie);
This would be better written with ie as the second argument to LOG.info, so that the stac...
-
-- HDFS-2729.
- Minor improvement reported by qwertymaniac and fixed by qwertymaniac (name-node)
- Update BlockManager's comments regarding the invalid block set
- Looks like after HDFS-82 was covered at some point, the comments and logs still carry presence of two sets when there really is just one set.
This patch changes the logs and comments to be more accurate about that.
-
-- HDFS-2751.
- Major bug reported by tlipcon and fixed by tlipcon (data-node)
- Datanode drops OS cache behind reads even for short reads
- HDFS-2465 has some code which attempts to disable the "drop cache behind reads" functionality when the reads are <256KB (eg HBase random access). But this check was missing in the {{close()}} function, so it always drops cache behind reads regardless of the size of the read. This hurts HBase random read performance when this patch is enabled.
-
-- HDFS-2784.
- Major sub-task reported by daryn and fixed by kihwal (hdfs client, name-node, security)
- Update hftp and hdfs for host-based token support
- Need to port 205 token changes and update any new related code dealing with tokens in these filesystems.
-
-- HDFS-2785.
- Major sub-task reported by daryn and fixed by revans2 (name-node, security)
- Update webhdfs and httpfs for host-based token support
- Need to port 205 tokens into these filesystems. Will mainly involve ensuring code duplicated from hftp is updated accordingly.
-
-- HDFS-2788.
- Major improvement reported by eli and fixed by eli (data-node)
- HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
- HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never used. Perhaps was renamed to DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was written and the old one wasn't deleted.
-
-- HDFS-2790.
- Minor bug reported by arpitgupta and fixed by arpitgupta
- FSNamesystem.setTimes throws exception with wrong configuration name in the message
- the api throws this message when hdfs is not configured for accessTime
"Access time for hdfs is not configured. Please set dfs.support.accessTime configuration parameter."
The property name should be dfs.access.time.precision
-
-- HDFS-2791.
- Major bug reported by tlipcon and fixed by tlipcon (data-node, name-node)
- If block report races with closing of file, replica is incorrectly marked corrupt
- The following sequence of events results in a replica mistakenly marked corrupt:
1. Pipeline is open with 2 replicas
2. DN1 generates a block report but is slow in sending to the NN (eg some flaky network). It gets "stuck" right before the block report RPC.
3. Client closes the file.
4. DN2 is fast and sends blockReceived to the NN. NN marks the block as COMPLETE
5. DN1's block report proceeds, and includes the block in an RBW state.
6. (x) NN incorrectly marks the replica as corrupt, since i...
-
-- HDFS-2803.
- Minor improvement reported by jxiang and fixed by jxiang (name-node)
- Adding logging to LeaseRenewer for better lease expiration triage.
- It will be helpful to add some logging to LeaseRenewer when the daemon is terminated (Info level),
and when the lease is renewed (Debug level). Since lacking logging, it is hard to know
if a DFS client doesn't renew the lease because it hangs, or the lease renewer daemon is gone somehow.
-
-- HDFS-2810.
- Critical bug reported by tlipcon and fixed by tlipcon (hdfs client)
- Leases not properly getting renewed by clients
- We've been testing HBase on clusters running trunk and seen an issue where they seem to lose their HDFS leases after a couple of hours of runtime. We don't quite have enough data to understand what's happening, but the NN is expiring them, claiming the hard lease period has elapsed. The clients report no error until their output stream gets killed underneath them.
-
-- HDFS-2814.
- Minor improvement reported by hitesh and fixed by hitesh
- NamenodeMXBean does not account for svn revision in the version information
- Unlike the jobtracker where both the UI and jmx information report the version as "x.y.z, r<svn revision", in case of the namenode, the UI displays x.y.z and svn revision info but the jmx output only contains the x.y.z version.
-
-- HDFS-2816.
- Trivial bug reported by hitesh and fixed by hitesh
- Fix missing license header in hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
+ - HDFS-2330.
+ Major sub-task reported by Uma Maheswara Rao G and fixed by Uma Maheswara Rao G (name-node)
+ In NNStorage.java, IOExceptions of stream closures can mask root exceptions.
-
-- HDFS-2817.
- Minor improvement reported by tlipcon and fixed by tlipcon (test)
- Combine the two TestSafeMode test suites
- We currently have two tests by the same name. We should combine them. Also adding a new test for safemode extension, which wasn't previously covered.
-
-- HDFS-2818.
- Trivial bug reported by tlipcon and fixed by (name-node)
- dfshealth.jsp missing space between role and node name
- There seems to be a missing space in the titles of our webpages. EG: <title>Hadoop NameNodestyx01.sf.cloudera.com:8021</title>. It seems like the JSP compiler is doing something to the space which is in the .jsp. Probably a simple fix if you know something about JSP :)
-
-- HDFS-2822.
- Major bug reported by tlipcon and fixed by tlipcon (ha, name-node)
- processMisReplicatedBlock incorrectly identifies under-construction blocks as under-replicated
- When the NN processes mis-replicated blocks while exiting safemode, it considers under-construction blocks as under-replicated, inserting them into the neededReplicationsQueue. This makes them show up as corrupt in the metrics and UI momentarily, until they're pulled off the queue. At that point, it realizes that they aren't in fact under-replicated, correctly. This affects both the HA branch and trunk/23, best I can tell.
-
-- HDFS-2825.
- Minor improvement reported by tlipcon and fixed by tlipcon (name-node)
- Add test hook to turn off the writer preferring its local DN
- Currently, the default block placement policy always places the first replica in the pipeline on the local node if there is a valid DN running there. In some network designs, within-rack bandwidth is never constrained so this doesn't give much of an advantage. It would also be really useful to disable this for MiniDFSCluster tests, since currently if you start a multi-DN cluster and write with replication level 1, all of the replicas go to the same DN.
_[per discussion below, this was changed...
-
-- HDFS-2826.
- Minor improvement reported by tlipcon and fixed by tlipcon (name-node, test)
- Test case for HDFS-1476 (safemode can initialize repl queues before exiting)
- HDFS-1476 introduced a feature whereby SafeMode can trigger the initialization of replication queues before the safemode exit threshold has been reached. But, it didn't include a test for this new behavior. This JIRA is to contribute such a test
-
-- HDFS-2827.
- Major bug reported by umamaheswararao and fixed by umamaheswararao (name-node)
- Cannot save namespace after renaming a directory above a file with an open lease
- When i execute the following operations and wait for checkpoint to complete.
fs.mkdirs(new Path("/test1"));
FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
fs.rename(new Path("/test/"), new Path("/test1/"));
Check-pointing is failing with the following exception.
2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) - Unable to save image for E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
java.io.IOException: saveLease...
-
-- HDFS-2835.
- Major bug reported by revans2 and fixed by revans2 (tools)
- Fix org.apache.hadoop.hdfs.tools.GetConf$Command Findbug issue
- https://builds.apache.org/job/PreCommit-HDFS-Build/1804//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html shows a findbugs warning. It is unrelated to the patch being tested, and has shown up on a few other JIRAS as well.
-
-- HDFS-2836.
- Major bug reported by revans2 and fixed by revans2
- HttpFSServer still has 2 javadoc warnings in trunk
- {noformat}
[WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:241: warning - @param argument "override," is not a parameter name.
[WARNING] hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/HttpFSServer.java:450: warning - @param argument "override," is not a parameter name.
{noformat}
These are causing other patches to get a -1 in automated testing.
-
-- HDFS-2837.
- Major bug reported by revans2 and fixed by revans2
- mvn javadoc:javadoc not seeing LimitedPrivate class
- mvn javadoc:javadoc not seeing LimitedPrivate class
{noformat}
[WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in type 'org.apache.hadoop.classification.InterfaceAudience.LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found
[WARNING] org/apache/hadoop/fs/FileSystem.class(org/apache/hadoop/fs:FileSystem.class): warning: Cannot find annotation method 'value()' in typ...
-
-- HDFS-2840.
- Major bug reported by eli and fixed by tucu00 (test)
- TestHostnameFilter should work with localhost or localhost.localdomain
- TestHostnameFilter may currently fail with the following:
{noformat}
Error Message
null expected:<localhost[.localdomain]> but was:<localhost[]>
Stacktrace
junit.framework.ComparisonFailure: null expected:<localhost[.localdomain]> but was:<localhost[]>
at junit.framework.Assert.assertEquals(Assert.java:81)
at junit.framework.Assert.assertEquals(Assert.java:87)
at org.apache.hadoop.lib.servlet.TestHostnameFilter$1.doFilter(TestHostnameFilter.java:50)
at org.apache.hadoop.lib.servlet.Hos...
-
-- HDFS-2864.
- Major sub-task reported by szetszwo and fixed by szetszwo (data-node)
- Remove redundant methods and a constant from FSDataset
- - METADATA_VERSION is declared in both FSDataset and BlockMetadataHeader.
- In FSDataset, the methods findBlockFile(..), getBlockFile(..) and getFile(..) are very similar.
-
-- HDFS-2868.
- Minor improvement reported by qwertymaniac and fixed by qwertymaniac (data-node)
- Add number of active transfer threads to the DataNode status
- Presently, we do not provide any stats from the DN that specifically indicates the total number of active transfer threads (xceivers). Having such a metric can be very helpful as well, over plain num-ops(type) form of metrics, which already exist.
-
-- HDFS-2879.
- Major sub-task reported by szetszwo and fixed by szetszwo (data-node)
- Change FSDataset to package private
+ - HDFS-2285.
+ Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (name-node)
+ BackupNode should reject requests trying to modify namespace
-
-- HDFS-2889.
- Major bug reported by gchanan and fixed by gchanan (hdfs client)
- getNumCurrentReplicas is package private but should be public on 0.23 (see HDFS-2408)
- See https://issues.apache.org/jira/browse/HDFS-2408
HDFS-2408 was not committed to 0.23 (or trunk it looks like).
This is breaking HBase unit tests with "-Dhadoop.profile=23"
-
-- HDFS-2893.
- Minor bug reported by eli2 and fixed by eli2
- The start/stop scripts don't start/stop the 2NN when using the default configuration
- HDFS-1703 changed the behavior of the start/stop scripts so that the masters file is no longer used to indicate which hosts to start the 2NN on. The 2NN is now started, when using start-dfs.sh, on hosts only when dfs.namenode.secondary.http-address is configured with a non-wildcard IP. This means you can not start a NN using an http-address specified using a wildcard IP. We should allow a 2NN to be started with the default config, ie start-dfs.sh should start a NN, 2NN and DN. The packaging a...
-
-- MAPREDUCE-1744.
- Major bug reported by dking and fixed by dking
- DistributedCache creates its own FileSytem instance when adding a file/archive to the path
- According to the contract of {{UserGroupInformation.doAs()}} the only required operations within the {{doAs()}} block are the
creation of a {{JobClient}} or getting a {{FileSystem}} .
The {{DistributedCache.add(File/Archive)ToClasspath()}} methods create a {{FileSystem}} instance outside of the {{doAs()}} block,
this {{FileSystem}} instance is not in the scope of the proxy user but of the superuser and permissions may make the method
fail.
One option is to overload the methods above to rece...
-
-- MAPREDUCE-2450.
- Major bug reported by matei and fixed by rajesh.balamohan
- Calls from running tasks to TaskTracker methods sometimes fail and incur a 60s timeout
- I'm seeing some map tasks in my jobs take 1 minute to commit after they finish the map computation. On the map side, the output looks like this:
<code>
2009-03-02 21:30:54,384 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=MAP, sessionId= - already initialized
2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 800
2009-03-02 21:30:54,437 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 300
2009-03-02 21:30:55,493 I...
-
-- MAPREDUCE-3045.
- Minor bug reported by rramya and fixed by jeagles (jobhistoryserver, mrv2)
- Elapsed time filter on jobhistory server displays incorrect table entries
- The elapsed time filter on the jobhistory server filters incorrect information.
For e.g. on a cluster where the elapsed time of all the tasks is either 7 or 8sec, the filter displays non null table entries for 1sec or 3sec
-
-- MAPREDUCE-3121.
- Blocker bug reported by vinodkv and fixed by ravidotg (mrv2, nodemanager)
- DFIP aka 'NodeManager should handle Disk-Failures In Place'
- This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial.
-
-- MAPREDUCE-3147.
- Major improvement reported by raviprak and fixed by raviprak (mrv2)
- Handle leaf queues with the same name properly
- If there are two leaf queues with the same name, there is ambiguity while submitting jobs, displaying queue info. When such ambiguity exists, the system should ask for clarification / show disambiguated information.
-
-- MAPREDUCE-3169.
- Major improvement reported by tlipcon and fixed by ahmed.radwan (mrv1, mrv2, test)
- Create a new MiniMRCluster equivalent which only provides client APIs cross MR1 and MR2
- Many dependent projects like HBase, Hive, Pig, etc, depend on MiniMRCluster for writing tests. Many users do as well. MiniMRCluster, however, exposes MR implementation details like the existence of TaskTrackers, JobTrackers, etc, since it was used by MR1 for testing the server implementations as well.
This JIRA is to create a new interface which could be implemented either by MR1 or MR2 that exposes only the client-side portions of the MR framework. Ideally it would be "recompile-compatible"...
-
-- MAPREDUCE-3194.
- Major bug reported by sseth and fixed by jlowe (mrv2)
- "mapred mradmin" command is broken in mrv2
- $mapred mradmin
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/tools/MRAdmin
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.tools.MRAdmin
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launc...
-
-- MAPREDUCE-3238.
- Trivial improvement reported by tlipcon and fixed by tlipcon (mrv2)
- Small cleanup in SchedulerApp
- While reading this code, I did a little bit of cleanup:
- added some javadoc
- rather than using a Map<Priority, Integer> for keeping counts, switched to Guava's HashMultiset, which makes a simpler API.
-
-- MAPREDUCE-3243.
- Major bug reported by rramya and fixed by jeagles (contrib/streaming, mrv2)
- Invalid tracking URL for streaming jobs
- The tracking URL for streaming jobs currently display "http://N/A"
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill <jobID>
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: <jobID>
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is <host:port>
{noformat}
-
-- MAPREDUCE-3251.
- Critical task reported by anupamseth and fixed by anupamseth (mrv2)
- Network ACLs can prevent some clients to talk to MR ApplicationMaster
- In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN, AM can be launched on any port and the client will have to have ACL open to that port to talk to AM and get the job status. When the client is within the same grid network access to AM is not a problem. But some applications may have one installation per set of clusters and may launch jobs even across such sets (on job trackers in another set of clusters). For that to work only the JT port needs to be open c...
-
-- MAPREDUCE-3265.
- Blocker improvement reported by tlipcon and fixed by acmurthy (mrv2)
- Reduce log level on MR2 IPC construction, etc
- Currently MR's IPC logging is very verbose. For example, I see a lot of:
11/10/25 12:14:06 INFO ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
11/10/25 12:14:06 INFO mapred.ResourceMgrDelegate: Connecting to ResourceManager at c0309.hal.cloudera.com/172.29.81.91:40012
11/10/25 12:14:06 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
11/10/25 12:14:07 INFO mapred.ResourceMgrDelegate...
-
-- MAPREDUCE-3291.
- Blocker bug reported by rramya and fixed by revans2 (mrv2)
- App fail to launch due to delegation token not found in cache
- In secure mode, saw an app failure due to "org.apache.hadoop.security.token.SecretManager$InvalidToken: token (HDFS_DELEGATION_TOKEN token <id> for <user>) can't be found in cache" Exception in the next comment.
-
-- MAPREDUCE-3324.
- Critical bug reported by jeagles and fixed by jeagles (jobhistoryserver, mrv2, nodemanager)
- Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
- Nodemanager has no tools listed under tools UI.
Jobhistory server has no logs tool listed under tools UI.
-
-- MAPREDUCE-3326.
- Critical bug reported by tgraves and fixed by jlowe (mrv2)
- RM web UI scheduler link not as useful as should be
- The resource manager web ui page for scheduler doesn't have all the information about the configuration like the jobtracker page used to have. The things it seems to show you are the current queues - each queues used, set, and max percent and then what apps are running in that queue.
It doesn't list any of yarn.scheduler.capacity.maximum-applications, yarn.scheduler.capacity.maximum-am-resource-percent, yarn.scheduler.capacity.<queue-path>.user-limit-factor, yarn.scheduler.capacity.<queue...
-
-- MAPREDUCE-3327.
- Critical bug reported by tgraves and fixed by anupamseth (mrv2)
- RM web ui scheduler link doesn't show correct max value for queues
- Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity < 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%.
-
-- MAPREDUCE-3328.
- Critical bug reported by tgraves and fixed by raviprak (mrv2)
- mapred queue -list output inconsistent and missing child queues
- When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4.
- the child queues do not show up
- The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int:
Queue Name : default
Queue State : running
Scheduling Info : queueName: ...
-
-- MAPREDUCE-3329.
- Blocker bug reported by tgraves and fixed by acmurthy (mrv2)
- capacity schedule maximum-capacity allowed to be less then capacity
- When configuring the capacity scheduler capacity and maximum-capacity, it allows the maximum-capacity to be less then the capacity. I did not test to see what true limit is, I assume maximum capacity.
output from mapred queue -list where capacity = 10%, max capacity = 5%.
Queue Name : test2
Queue State : running
Scheduling Info : queueName: "test2", capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING,
-
-- MAPREDUCE-3331.
- Minor improvement reported by anupamseth and fixed by anupamseth (mrv2)
- Improvement to single node cluster setup documentation for 0.23
- This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html
-
-- MAPREDUCE-3336.
- Critical bug reported by tgraves and fixed by tgraves (mrv2)
- com.google.inject.internal.Preconditions not public api - shouldn't be using it
- com.google.inject.internal.Preconditions does not exist in guice 3.0 and from in guice 2.0 it was an internal api and shouldn't have been used. We should use com.google.common.base.Preconditions instead.
This is currently being used in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java.
-
-- MAPREDUCE-3341.
- Major improvement reported by anupamseth and fixed by anupamseth (mrv2)
- Enhance logging of initalized queue limit values
- Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized.
For example, this is what is currently shown in the RM log for an initialized queue:
# <datestamp> INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing
default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100,
userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=50000...
-
-- MAPREDUCE-3344.
- Major bug reported by brocknoland and fixed by brocknoland
- o.a.h.mapreduce.Reducer since 0.21 blindly casts to ReduceContext.ValueIterator
- 0.21 mapreduce.Reducer introduced a blind cast to ReduceContext.ValueIterator. There should an instanceof check around this block to ensure we don't throw a CastClassException:
{code}
// If a back up store is used, reset it
((ReduceContext.ValueIterator)
(context.getValues().iterator())).resetBackupStore();
{code}
-
-- MAPREDUCE-3346.
- Blocker bug reported by karams and fixed by amar_kamat (tools/rumen)
- Rumen LoggedTaskAttempt getHostName call returns hostname as null
- After MAPREDUCE-3035 and MAPREDUCE-3317
Now MRV2 job history contains hostName and rackName.
when rumen trace builder is ran on jobhistory, its generated trace contains hostname in form of
hostName : /raclname/hostname
But getHostName for LoggedTaskAttempt returns hostname as null
Seems that TraceBuilder is setting hostName properly but JobTraceReader is not able read it.
-
-- MAPREDUCE-3354.
- Blocker bug reported by vinodkv and fixed by jeagles (jobhistoryserver, mrv2)
- JobHistoryServer should be started by bin/mapred and not by bin/yarn
- JobHistoryServer belongs to mapreduce land.
-
-- MAPREDUCE-3366.
- Major bug reported by eyang and fixed by eyang (mrv2)
- Mapreduce component should use consistent directory structure layout as HDFS/common
- Directory structure for MRv2 layout looks like:
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/conf
/lib
/modules
{noformat}
The directory structure layout should be updated to reflect changes implemented in HADOOP-6255.
{noformat}
hadoop-mapreduce-0.23.0-SNAPSHOT/bin
/etc/hadoop
/lib
/libexec
...
-
-- MAPREDUCE-3369.
- Major improvement reported by ahmed.radwan and fixed by ahmed.radwan (mrv1, mrv2, test)
- Migrate MR1 tests to run on MR2 using the new interfaces introduced in MAPREDUCE-3169
- This ticket tracks the migration of MR1 tests (currently residing in "hadoop-mapreduce-project/src/test/") to run on MR2. The migration is using the new interfaces introduced in MAPREDUCE-3169.
-
-- MAPREDUCE-3370.
- Major bug reported by ahmed.radwan and fixed by ahmed.radwan (mrv2, test)
- MiniMRYarnCluster uses a hard coded path location for the MapReduce application jar
- MiniMRYarnCluster uses a hard coded relative path location for the MapReduce application jar. It is better to have this location as a system property so tests can pick the application jar regardless of their working directory.
-
-- MAPREDUCE-3371.
- Minor improvement reported by raviprak and fixed by raviprak (documentation, mrv2)
- Review and improve the yarn-api javadocs.
- Review and improve the yarn-api javadocs.
-
-- MAPREDUCE-3372.
- Major bug reported by bmahe and fixed by bmahe
- HADOOP_PREFIX cannot be overriden
- hadoop-config.sh forces HADOOP_prefix to a specific value:
export HADOOP_PREFIX=`dirname "$this"`/..
It would be nice to make this overridable.
-
-- MAPREDUCE-3373.
- Major bug reported by bmahe and fixed by bmahe
- Hadoop scripts unconditionally source "$bin"/../libexec/hadoop-config.sh.
- It would be nice to be able to specify some other location for hadoop-config.sh
-
-- MAPREDUCE-3376.
- Major bug reported by revans2 and fixed by subrotosanyal (mrv1, mrv2)
- Old mapred API combiner uses NULL reporter
- The OldCombinerRunner class inside Task.java uses a NULL Reporter. If the combiner code runs for an extended period of time, even with reporting progress as it should, the map task can timeout and be killed. It appears that the NewCombinerRunner class uses a valid reporter and as such is not impacted by this bug.
-
-- MAPREDUCE-3380.
- Blocker sub-task reported by tucu00 and fixed by mahadev (mr-am, mrv2)
- Token infrastructure for running clients which are not kerberos authenticated
- The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job.
What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code)
-
-- MAPREDUCE-3389.
- Critical bug reported by tucu00 and fixed by tucu00 (mrv2)
- MRApps loads the 'mrapp-generated-classpath' file with classpath from the build machine
- The 'mrapp-generated-classpath' file contains the classpath from where Hadoop was build. This classpath is not useful under any circumstances.
For example the content of the 'mrapp-generated-classpath' in my dev environment is:
/Users/tucu/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/tucu/.m2/repository/asm/asm/3.2/asm-3.2.jar:/Users/tucu/.m2/repository/com/cenqua/clover/clover/3.0.2/clover-3.0.2.jar:/Users/tucu/.m2/repository/com/google/guava/guava/r09/guava-r09.ja...
-
-- MAPREDUCE-3391.
- Minor bug reported by subrotosanyal and fixed by subrotosanyal (applicationmaster)
- Connecting to CM is logged as Connecting to RM
- In class *org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster*
{code}
private void connectToCM() {
String cmIpPortStr = container.getNodeId().getHost() + ":"
+ container.getNodeId().getPort();
InetSocketAddress cmAddress = NetUtils.createSocketAddr(cmIpPortStr);
LOG.info("Connecting to ResourceManager at " + cmIpPortStr);
this.cm = ((ContainerManager) rpc.getProxy(ContainerManager.class, cmAddress, conf));
}
{code}
-
-- MAPREDUCE-3408.
- Major bug reported by bmahe and fixed by bmahe (mrv2, nodemanager, resourcemanager)
- yarn-daemon.sh unconditionnaly sets yarn.root.logger
- yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any override from happening.
From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
> export YARN_ROOT_LOGGER="INFO,DRFA"
> export YARN_JHS_LOGGER="INFO,JSA"
and then yarn-daemon.sh will call "$YARN_HOME"/bin/yarn which does the following:
> YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
> YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
This has at leas...
-
-- MAPREDUCE-3411.
- Minor improvement reported by jeagles and fixed by jeagles (mrv2)
- Performance Upgrade for jQuery
- jQuery 1.6.4 is almost twice as fast as current version 1.4.4 on modern browsers on some operations. There are also many modern browser compatibility fixes
http://jsperf.com/jquery-15-unique-traversal/15
-
-- MAPREDUCE-3413.
- Minor bug reported by jeagles and fixed by jeagles (mrv2)
- RM web ui applications not sorted in any order by default
+ - HDFS-2025.
+ Minor bug reported by sravankorumilli and fixed by Ashish Singhi (data-node)
+ Go Back to File View link is not working in tail.jsp
-
-- MAPREDUCE-3422.
- Major bug reported by tomwhite and fixed by jeagles (mrv2)
- Counter display names are not being picked up
- When running a job I see "MAP_INPUT_RECORDS" rather than "Map input records" for the counter name. To fix this the resource bundle properties files need to be moved to the src/main/resources tree.
-
-- MAPREDUCE-3427.
- Blocker bug reported by tucu00 and fixed by hitesh (contrib/streaming, mrv2)
- streaming tests fail with MR2
- After Mavenizing streaming and getting its testcases to use the MiniMRCluster wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
Following is an assessment of those failures. Note that the testcases have been tweaked only to set the streaming JAR and yarn as the framework.
(If these issues are unrelated we should create sub-tasks for each one of them).
*TestStreamingCombiner*, fails because returned counters don't match assertion. However, counters printed in the test output indicate va...
-
-- MAPREDUCE-3433.
- Major sub-task reported by tomwhite and fixed by tomwhite (client, mrv2)
- Finding counters by legacy group name returns empty counters
- Attempting to find counters with a legacy group name (e.g. org.apache.hadoop.mapred.Task$Counter rather than the new org.apache.hadoop.mapreduce.TaskCounter) returns empty counters. This causes TestStreamingCombiner to fail when run with YARN.
-
-- MAPREDUCE-3434.
- Blocker bug reported by hitesh and fixed by hitesh (mrv2)
- Nightly build broken
- https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/901/
Results :
Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs)
testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs)
testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs)
Tests in error:
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs: Failed to Start org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.h...
-
-- MAPREDUCE-3436.
- Major bug reported by bmahe and fixed by ahmed.radwan (mrv2, webapps)
- JobHistory webapp address should use the host from the jobhistory address
- On the following page : http://<RESOURCE_MANAGER>:8088/cluster/apps
There are links to the history for each application. None of them can be reached since they all point to the ip 0.0.0.0. For instance:
http://0.0.0.0:8088/proxy/application_1321658790349_0002/jobhistory/job/job_1321658790349_2_2
Am I missing something?
[root@bigtop-fedora-15 ~]# jps
9968 ResourceManager
1495 NameNode
1645 DataNode
12935 Jps
11140 -- process information unavailable
5309 JobHistoryServer
10237 NodeManager
[r...
-
-- MAPREDUCE-3437.
- Blocker bug reported by jeagles and fixed by jeagles (build, mrv2)
- Branch 23 fails to build with Failure to find org.apache.hadoop:hadoop-project:pom:0.24.0-SNAPSHOT
- [INFO] Scanning for projects...
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.apache.hadoop:hadoop-mapreduce-examples:0.24.0-SNAPSHOT (/home/jeagles/hadoop/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM: Failure to find org.apache.hadoop:hadoop-project:pom:0.24.0-SNAPSHOT in http://stormwalk.champ.corp.yahoo.com:8081/nexus/content/groups/public was cached in the local repository,...
-
-- MAPREDUCE-3443.
- Blocker bug reported by mahadev and fixed by mahadev (mrv2)
- Oozie jobs are running as oozie user even though they create the jobclient as doAs.
- Oozie is having issues with job submission, since it does the following:
{code}
doAs(userwhosubmittedjob) {
jobclient = new JobClient(jobconf);
}
jobclient.submitjob()
{code}
In 0.20.2** this works because the JT proxy is created as soon as we call new JobClient(). But in 0.23 this is no longer true since the client has to talk to multiple servers (AM/RM/JHS). To keep this behavior we will have to store the ugi in new JobClient() and make sure all the calls are run with a doAs() inside t...
-
-- MAPREDUCE-3444.
- Blocker bug reported by hitesh and fixed by hitesh (mrv2)
- trunk/0.23 builds broken
- https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/208/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1310/
-
-- MAPREDUCE-3447.
- Blocker bug reported by tgraves and fixed by mahadev (mrv2)
- mapreduce examples not working
- Since the mavenization went in the mapreduce examples jar no longer works.
$ hadoop jar ./hadoop-0.23.0-SNAPSHOT/modules/hadoop-mapreduce-examples-0.23.0-SNAPSHOT.jar wordcount input output
Exception in thread "main" java.lang.ClassNotFoundException: wordcount
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.Class...
-
-- MAPREDUCE-3448.
- Minor bug reported by jeagles and fixed by jeagles (mrv2)
- TestCombineOutputCollector javac unchecked warning on mocked generics
- [javac] found : org.apache.hadoop.mapred.IFile.Writer
[javac] required: org.apache.hadoop.mapred.IFile.Writer<java.lang.String,java.lang.Integer>
[javac] Writer<String, Integer> mockWriter = mock(Writer.class);
[javac] ^
[javac] /home/jeagles/hadoop/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestCombineOutputCollector.java:125: warning: [unchecked] unchecked conversion
[javac] found : org.a...
-
-- MAPREDUCE-3450.
- Major bug reported by sseth and fixed by sseth (mr-am, mrv2)
- NM port info no longer available in JobHistory
- The NM RPC port used to be part of the hostname field in JobHistory. That seems to have gone missing. Required for the task log link on the history server.
-
-- MAPREDUCE-3452.
- Major bug reported by tgraves and fixed by jeagles (mrv2)
- fifoscheduler web ui page always shows 0% used for the queue
- When the fifo scheduler is configured to be on, go to the RM web ui page and click the scheduler link. Hover over the default queue to see the used%. It always shows used% as 0.0% even when jobs are running.
-
-- MAPREDUCE-3453.
- Major bug reported by tgraves and fixed by jeagles (mrv2)
- RM web ui application details page shows RM cluster about information
- Go to the RM Web ui page. Click on the Applications link, then click on a particular application. The applications details page inadvertently includes the RM about page information after the application details:
Cluster ID: 1321943597242
ResourceManager state: STARTED
ResourceManager started on: 22-Nov-2011 06:33:17
ResourceManager version: 0.23.0-SNAPSHOT from 1203458 by user source checksum 0c288fc0971ed28c970272a62f547eae on Tue Nov 22 06:31:09 UTC 2011
Hadoop version: 0.23.0-SNAPSH...
-
-- MAPREDUCE-3454.
- Major bug reported by amar_kamat and fixed by hitesh (contrib/gridmix)
- [Gridmix] TestDistCacheEmulation is broken
- TestDistCacheEmulation is broken as 'MapReduceTestUtil' no longer exists.
-
-- MAPREDUCE-3456.
- Blocker bug reported by eepayne and fixed by eepayne (mrv2)
- $HADOOP_PREFIX/bin/yarn should set defaults for $HADOOP_*_HOME
- If the $HADOOP_PREFIX/hadoop-dist/target/hadoop-0.23.0-SNAPSHOT.tar.gz tarball is used to distribute hadoop, all of the HADOOP components (HDFS, MAPRED, COMMON) are all under one directory. In this use case, HADOOP_PREFIX should be set and should point to the root directory for all components, and it should not be necessary to set HADOOP_HDFS_HOME, HADOOP_COMMON_HOME, and HADOOP_MAPRED_HOME. However, the $HADOOP_PREFIX/bin/yarn script requires these 3 to be set explicitly in the calling envir...
-
-- MAPREDUCE-3458.
- Major bug reported by acmurthy and fixed by devaraj.k (mrv2)
- Fix findbugs warnings in hadoop-examples
- I see 12 findbugs warnings in hadoop-examples:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1336//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-examples.html
-
-- MAPREDUCE-3460.
- Blocker bug reported by sseth and fixed by revans2 (mr-am, mrv2)
- MR AM can hang if containers are allocated on a node blacklisted by the AM
- When an AM is assigned a FAILED_MAP (priority = 5) container on a nodemanager which it has blacklisted - it tries to
find a corresponding container request.
This uses the hostname to find the matching container request - and can end up returning any of the ContainerRequests which may have requested a container on this node. This container request is cleaned to remove the bad node - and then added back to the RM 'ask' list.
The AM cleans the 'ask' list after each heartbeat - The RM Allocator i...
-
-- MAPREDUCE-3463.
- Blocker bug reported by karams and fixed by sseth (applicationmaster, mrv2)
- Second AM fails to recover properly when first AM is killed with java.lang.IllegalArgumentException causing lost job
- Set yarn.resourcemanager.am.max-retries=5 in yarn-site.xml. Started yarn 4 Node cluster.
First Ran Randowriter/Sort/Sort-validate successfully
Then again sort, when job was 50% complete
Login node running AppMaster, and killed AppMaster with kill -9
On Client side failed with following:
{code}
11/11/23 10:57:27 INFO mapreduce.Job: map 58% reduce 8%
11/11/23 10:57:27 INFO mapred.ClientServiceDelegate: Failed to contact AM/History for job job_1322040898409_0005 retrying..
11/11/23 10:57:28 INF...
-
-- MAPREDUCE-3464.
- Trivial bug reported by davevr and fixed by davevr
- mapreduce jsp pages missing DOCTYPE [post-split branches]
- Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch.
-
-- MAPREDUCE-3465.
- Minor bug reported by hitesh and fixed by hitesh (mrv2)
- org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin fails on 0.23
- Running org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.121 sec <<< FAILURE!
Tests in error:
testParsingProcStatAndCpuFile(org.apache.hadoop.yarn.util.TestLinuxResourceCalculatorPlugin): /homes/hortonhi/dev/hadoop-common/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/CPUINFO_943711651 (No such file or directory)
testParsingProcMemFile(org.apache.hadoop.yarn.util.TestLinuxResourceCalcu...
-
-- MAPREDUCE-3468.
- Major task reported by sseth and fixed by sseth
- Change version to 0.23.1 for ant builds on the 23 branch
- Maven version has been changed to 0.23.1-SNAPSHOT. The ant build files need to change as well.
-
-- MAPREDUCE-3477.
- Major bug reported by bmahe and fixed by jeagles (documentation, mrv2)
- Hadoop site documentation cannot be built anymore on trunk and branch-0.23
- Maven fails and here is the issue I get:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.0:site (default-site) on project hadoop-yarn-site: Error during page generation: Error parsing '/home/bruno/freesoftware/bigtop/build/hadoop/rpm/BUILD/apache-hadoop-common-e127450/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm': line [23] Unable to execute macro in the APT document: ParseException: expected SECTION2, found SECTION3 -> [...
-
-- MAPREDUCE-3478.
- Minor bug reported by abayer and fixed by tomwhite (mrv2)
- Cannot build against ZooKeeper 3.4.0
- I tried to see if one could build Hadoop 0.23.0 against ZooKeeper 3.4.0, rather than 3.3.1 (3.3.3 does work, fwiw) and hit compilation errors:
{quote}
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:testCompile (default-testCompile) on project hadoop-yarn-server-common: Compilation failure: Compilation failure:
[ERROR] /Volumes/EssEssDee/abayer/src/asf-git/hadoop-common/hadoop-...
-
-- MAPREDUCE-3479.
- Major bug reported by tomwhite and fixed by tomwhite (client)
- JobClient#getJob cannot find local jobs
- The problem is that JobClient#submitJob doesn't pass the Cluster object to Job for the submission process, which means that two Cluster objects and two LocalJobRunner objects are created. LocalJobRunner keeps an instance map of job IDs to Jobs, and when JobClient#getJob is called the LocalJobRunner with the unpopulated map is used which results in the job not being found.
-
-- MAPREDUCE-3485.
- Major sub-task reported by hitesh and fixed by ravidotg (mrv2)
- DISKS_FAILED -101 error code should be defined in same location as ABORTED_CONTAINER_EXIT_STATUS
- With MAPREDUCE-3121, it is defined in ContainerExecutor as part of yarn-nodemanager which would be a problem for client-side code if it needs to understand the exit code.
A short term fix would be to move it into YarnConfiguration where ABORTED_CONTAINER_EXIT_STATUS is defined. A longer term fix would be to find a more formal and extensible approach for new yarn framework error codes to be added and be easily accessible to client-side code or other AMs.
-
-- MAPREDUCE-3488.
- Blocker bug reported by mahadev and fixed by mahadev (mrv2)
- Streaming jobs are failing because the main class isnt set in the pom files.
- Streaming jobs are failing since the main MANIFEST file isnt being set in the pom files.
-
-- MAPREDUCE-3496.
- Major bug reported by jeagles and fixed by jeagles (mrv2)
- Yarn initializes ACL operations from capacity scheduler config in a non-deterministic order
- 'mapred queue -showacls' does not output put acls in a predictable manner. This is a regression from previous versions.
-
-- MAPREDUCE-3499.
- Blocker bug reported by tucu00 and fixed by johnvijoe (mrv2, test)
- New MiniMR does not setup proxyuser configuration correctly, thus tests using doAs do not work
- The new MiniMR implementation is not taking proxyuser settings.
Because of this, testcases using/testing doAs functionality fail.
This affects all Oozie testcases that use MiniMR.
-
-- MAPREDUCE-3500.
- Major bug reported by tucu00 and fixed by tucu00 (mrv2)
- MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH
- With HADOOP-7874 we are removing the arch from the java.library.path.
The LD_LIBRARY_PATH being set should not include the ARCH.
{code}
public static final String DEFAULT_MAPRED_ADMIN_USER_ENV =
"LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native/" + PlatformName.getPlatformName();
{code}
-
-- MAPREDUCE-3505.
- Major bug reported by bmahe and fixed by ahmed.radwan (mrv2)
- yarn APPLICATION_CLASSPATH needs to be overridable
- Right now MRApps sets the classpath to just being mrapp-generated-classpath, its content and a hardcoded list of directories.
If I understand correctly mrapp-generated-classpath is only there for testing and may change or disappear at any time
The list of hardcoded directories is defined in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java at line 92.
For convenience, here is its current content:
{noformat}
/**
* Clas...
-
-- MAPREDUCE-3510.
- Major bug reported by jeagles and fixed by jeagles (contrib/capacity-sched, mrv2)
- Capacity Scheduler inherited ACLs not displayed by mapred queue -showacls
- mapred queue -showacls does not show inherited acls
-
-- MAPREDUCE-3513.
- Trivial bug reported by mahadev and fixed by chaku88 (mrv2)
- Capacity Scheduler web UI has a spelling mistake for Memory.
- The web page for capacity scheduler has a column named "Memopry Total", a spelling mistake which needs to be fixed.
-
-- MAPREDUCE-3518.
- Critical bug reported by jeagles and fixed by jeagles (client, mrv2)
- mapred queue -info <queue> -showJobs throws NPE
- mapred queue -info default -showJobs
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:572)
at org.apache.hadoop.mapred.JobQueueClient.displayQueueInfo(JobQueueClient.java:190)
at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at o...
-
-- MAPREDUCE-3521.
- Minor bug reported by revans2 and fixed by revans2 (mrv2)
- Hadoop Streaming ignores unknown parameters
- The hadoop streaming command will ignore any command line arguments to it.
{code}
hadoop jar streaming.jar -input input -output output -mapper cat -reducer cat ThisIsABadArgument
{code}
Works just fine. This can mask issues where quotes were mistakenly missed like
{code}
hadoop jar streaming.jar -input input -output output -mapper xargs cat -reducer cat -archive someArchive.tgz
{code}
Streaming should fail if it encounters an unexpected command line parameter
-
-- MAPREDUCE-3522.
- Major bug reported by jeagles and fixed by jeagles (mrv2)
- Capacity Scheduler ACLs not inherited by default
- Hierarchical Queues do not inherit parent ACLs correctly by default. Instead, if no value is specified for submit or administer acls, then all access is granted.
-
-- MAPREDUCE-3527.
- Major bug reported by tomwhite and fixed by tomwhite
- Fix minor API incompatibilities between 1.0 and 0.23
- There are a few minor incompatibilities that were found in HADOOP-7738 and are straightforward to fix.
-
-- MAPREDUCE-3529.
- Critical bug reported by sseth and fixed by sseth (mrv2)
- TokenCache does not cache viewfs credentials correctly
- viewfs returns a list of delegation tokens for the actual namenodes. TokenCache caches these based on the actual service name - subsequent calls to TokenCache end up trying to get a new set of tokens.
Tasks which happen to access TokenCache fail when using viewfs - since they end up trying to get a new set of tokens even though the tokens are already available.
{noformat}
Error: java.io.IOException: Delegation Token can be issued only with kerberos or web authentication
at org.apach...
-
-- MAPREDUCE-3531.
- Blocker bug reported by karams and fixed by revans2 (mrv2, resourcemanager, scheduler)
- Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
- Filling this Jira a bit late
Started 350 cluster
sbummited large sleep job.
Foud that job was not running as RM has not allocated resouces to it.
{code}
2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
type NODE_UPDATE to the scheduler
java.lang.IllegalAr...
-
-- MAPREDUCE-3537.
- Blocker bug reported by acmurthy and fixed by acmurthy
- DefaultContainerExecutor has a race condn. with multiple concurrent containers
- DCE relies cwd before calling ContainerLocalizer.runLocalization. However, with multiple containers setting cwd on same localFS reference leads to race.
-
-- MAPREDUCE-3541.
- Blocker bug reported by raviprak and fixed by raviprak (mrv2)
- Fix broken TestJobQueueClient test
- Ant build complains
[javac] /hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestJobQueueClient.java>:80: printJobQueueInfo(org.apache.hadoop.mapred.JobQueueInfo,java.io.Writer,java.lang.String) in org.apache.hadoop.mapred.JobQueueClient cannot be applied to (org.apache.hadoop.mapred.JobQueueInfo,java.io.StringWriter)
[javac] client.printJobQueueInfo(root, writer);
-
-- MAPREDUCE-3542.
- Major bug reported by tomwhite and fixed by tomwhite
- Support "FileSystemCounter" legacy counter group name for compatibility
- The group name changed from "FileSystemCounter" to "org.apache.hadoop.mapreduce.FileSystemCounter", but we should support the old one for compatibility's sake. This came up in PIG-2347.
-
-- MAPREDUCE-3544.
- Major bug reported by tucu00 and fixed by tucu00 (build, tools/rumen)
- gridmix build is broken, requires hadoop-archives to be added as ivy dependency
- Having moved HAR/HadoopArchives to common/tools makes gridmix to fail as HadoopArchives is not in the mr1 classpath anymore.
hadoop-archives artifact should be added to gridmix dependencies
-
-- MAPREDUCE-3547.
- Critical sub-task reported by tgraves and fixed by tgraves (mrv2)
- finish unit tests for web services for RM and NM
- Write more unit tests for the web services added for rm and nm.
-
-- MAPREDUCE-3548.
- Critical sub-task reported by tgraves and fixed by tgraves (mrv2)
- write unit tests for web services for mapreduce app master and job history server
- write more unit tests for mapreduce application master and job history server web services added in MAPREDUCE-2863
-
-- MAPREDUCE-3553.
- Minor sub-task reported by tgraves and fixed by tgraves (mrv2)
- Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
- When the web services apis for rm, nm, app master, and job history server throw an exception - like bad request, not found, they always return the data in JSON format. It would be nice to return based on what they requested - xml or JSON.
-
-- MAPREDUCE-3557.
- Major bug reported by tucu00 and fixed by tucu00 (build)
- MR1 test fail to compile because of missing hadoop-archives dependency
- MAPREDUCE-3544 added hadoop-archives as dependency to gridmix and raid, but missed to add it to the main ivy.xml for the MR1 testcases thus the ant target 'compile-mapred-test' fails.
I was under the impression that this stuff was not used anymore but trunk is failing on that target.
-
-- MAPREDUCE-3560.
- Blocker bug reported by vinodkv and fixed by sseth (mrv2, resourcemanager, test)
- TestRMNodeTransitions is failing on trunk
- Apparently Jenkins is screwed up. It is happily blessing patches, even though tests are failing.
Link to logs: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1454//testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMNodeTransitions/testExpiredContainer/
-
-- MAPREDUCE-3563.
- Major bug reported by acmurthy and fixed by acmurthy (mrv2)
- LocalJobRunner doesn't handle Jobs using o.a.h.mapreduce.OutputCommitter
- LocalJobRunner doesn't handle Jobs using o.a.h.mapreduce.OutputCommitter, ran into this debugging PIG-2347.
-
-- MAPREDUCE-3566.
- Critical sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2)
- MR AM slows down due to repeatedly constructing ContainerLaunchContext
- The construction of the context is expensive, includes per-task trips to NameNode for obtaining the information about job.jar, job splits etc which is redundant across all tasks.
We should have a common job-level context and a task-specific context inheriting from the common job-level context.
-
-- MAPREDUCE-3567.
- Major sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2, performance)
- Extraneous JobConf objects in AM heap
- MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and TaskImpl which occupy non-trivial amount of heap.
While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing those extraneous objects helped in addressing OOM with 2GB AM heap size.
-
-- MAPREDUCE-3569.
- Critical sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2, performance)
- TaskAttemptListener holds a global lock for all task-updates
- This got added via MAPREDUCE-3274. We really don't need the lock if we just implement what I mentioned on that ticket [here|https://issues.apache.org/jira/browse/MAPREDUCE-3274?focusedCommentId=13137214&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13137214].
This has performance implications on MR AM with lots of tasks.
-
-- MAPREDUCE-3572.
- Critical sub-task reported by vinodkv and fixed by vinodkv (mr-am, mrv2, performance)
- MR AM's dispatcher is blocked by heartbeats to ResourceManager
- All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
The event processing should be in a separate thread.
-
-- MAPREDUCE-3579.
- Major bug reported by atm and fixed by atm (mrv2)
- ConverterUtils should not include a port in a path for a URL with no port
- In {{ConverterUtils#getPathFromYarnURL}}, it's incorrectly assumed that if a URL includes a valid host it must also include a valid port.
-
-- MAPREDUCE-3582.
- Major bug reported by ahmed.radwan and fixed by ahmed.radwan (mrv2, test)
- Move successfully passing MR1 tests to MR2 maven tree.
- This ticket will track moving mr1 tests that are passing successfully to mr2 maven tree.
-
-- MAPREDUCE-3588.
- Blocker bug reported by acmurthy and fixed by acmurthy
- bin/yarn broken after MAPREDUCE-3366
- bin/yarn broken after MAPREDUCE-3366, doesn't add yarn jars to classpath. As a result no servers can be started.
-
-- MAPREDUCE-3595.
- Major test reported by tomwhite and fixed by tomwhite (test)
- Add missing TestCounters#testCounterValue test from branch 1 to 0.23
+ - HDFS-1490.
+ Minor bug reported by Dmytro Molkov and fixed by Vinay (name-node)
+ TransferFSImage should timeout
-
-- MAPREDUCE-3596.
- Blocker bug reported by raviprak and fixed by vinodkv (applicationmaster, mrv2)
- Sort benchmark got hang after completion of 99% map phase
- Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after completion 99% map phase. There are some map tasks failed. Also it's not scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==============
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version: revision 1212681 by someone source checksum on Fri Dec 9 16:52:07 PST ...
-
-- MAPREDUCE-3604.
- Blocker bug reported by acmurthy and fixed by acmurthy (contrib/streaming)
- Streaming's check for local mode is broken
- Streaming isn't checking for mapreduce.framework.name as part of check for 'local' mode.
-
-- MAPREDUCE-3608.
- Major bug reported by mahadev and fixed by mahadev (mrv2)
- MAPREDUCE-3522 commit causes compilation to fail
- There are compilation errors after MAPREDUCE-3522 was committed. Some more changes were need to webapps to fix the compilation issue.
-
-- MAPREDUCE-3610.
- Minor improvement reported by sho.shimauchi and fixed by sho.shimauchi
- Some parts in MR use old property dfs.block.size
- Some parts in MR use old property dfs.block.size.
dfs.blocksize should be used instead.
-
-- MAPREDUCE-3615.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- mapred ant test failures
- The following mapred ant tests are failing. This started on December 22nd.
[junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs
[junit] Running org.apache.hadoop.mapred.TestMiniMRDFSSort
[junit] Running org.apache.hadoop.mapred.TestBadRecords
[junit] Running org.apache.hadoop.mapred.TestClusterMRNotification
[junit] Running org.apache.hadoop.mapred.TestDebugScript
[junit] Running org.apache.hadoop.mapred.TestJobCleanup
[junit] Running org.apac...
-
-- MAPREDUCE-3616.
- Major sub-task reported by vinodkv and fixed by vinodkv (mr-am, performance)
- Thread pool for launching containers in MR AM not expanding as expected
- Found this while running some benchmarks on 350 nodes. The thread pool stays at 60 for a long time and only expands to 350 towards the fag end of the job.
-
-- MAPREDUCE-3617.
- Major bug reported by jeagles and fixed by jeagles (mrv2)
- Remove yarn default values for resource manager and nodemanager principal
- Default values should be empty since no use can be made of them without correct values defined.
-
-- MAPREDUCE-3624.
- Major bug reported by mahadev and fixed by mahadev (mrv2)
- bin/yarn script adds jdk tools.jar to the classpath.
- Thanks to Roman for pointing it out. Looks like we have the following lines in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
-
-- MAPREDUCE-3625.
- Critical bug reported by acmurthy and fixed by jlowe (mrv2)
- CapacityScheduler web-ui display of queue's used capacity is broken
- The display of the queue's used capacity at runtime is broken because it display's 'used' relative to the queue's capacity and not the parent's capacity as shown in the above attachment.
The display should be relative to parent's capacity and not leaf queues as everything else in the display is relative to parent's capacity.
-
-- MAPREDUCE-3640.
- Blocker sub-task reported by sseth and fixed by acmurthy (mrv2)
- AMRecovery should pick completed task form partial JobHistory files
- Currently, if the JobHistory file has a partial record, AMRecovery will start from scratch. This will become more relevant after MAPREDUCE-3512.
-
-- MAPREDUCE-3645.
- Blocker bug reported by tgraves and fixed by tgraves (mrv1)
- TestJobHistory fails
- TestJobHistory fails.
>>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderOnHDFS
>>> org.apache.hadoop.mapred.TestJobHistory.testDoneFolderNotOnDefaultFileSystem
>>> org.apache.hadoop.mapred.TestJobHistory.testHistoryFolderOnHDFS
>>> org.apache.hadoop.mapred.TestJobHistory.testJobHistoryFile
It looks like this was introduced by MAPREDUCE-3349 and the issue is that the test expects the hostname to be in the format rackname/hostname, but with 3349 it split those apart into 2 diff...
-
-- MAPREDUCE-3646.
- Major bug reported by rramya and fixed by jeagles (client, mrv2)
- Remove redundant URL info from "mapred job" output
- The URL information to track the job is printed for all the "mapred job"mrv2 commands. This information is redundant and has to be removed.
E.g:
{noformat}
-bash-3.2$ mapred job -list
Total jobs:3
JobId State StartTime UserName Queue Priority Maps Reduces UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
12/01/09 22:20:15 INFO mapred.ClientServiceDelegate: The url to track the job: <RM host>:8088/proxy/<application ID 1>/
<job ID 1> RUNNI...
-
-- MAPREDUCE-3648.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- TestJobConf failing
- TestJobConf is failing:
testFindContainingJar
testFindContainingJarWithPlus
java.lang.ClassNotFoundException: ClassWithNoPackage
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Cla...
-
-- MAPREDUCE-3649.
- Blocker bug reported by mahadev and fixed by raviprak (mrv2)
- Job End notification gives an error on calling back.
- When calling job end notification for oozie the AM fails with the following trace:
{noformat}
2012-01-09 23:45:41,732 WARN [AsyncDispatcher event handler] org.mortbay.log: Job end notification to http://HOST:11000/oozie/v0/callback?id=0000000-120109234442311-oozie-oozi-W@mr-node&status=SUCCEEDED& failed
java.net.UnknownServiceException: no content-type
at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
at java.net.URLConnection.getContent(URLConnection.java:689)
at org.a...
-
-- MAPREDUCE-3651.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- TestQueueManagerRefresh fails
- The following tests fail:
org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshWithRemovedQueues
org.apache.hadoop.mapred.TestQueueManagerRefresh.testRefreshOfSchedulerProperties
It looks like its simply trying to remove one of the queues but the remove is failing.It looks like MAPREDUCE-3328. mapred queue -list output inconsistent and missing child queues - change the getChilren routine to do a new JobQueueInfo on each one when returning it which is making the remove routine fail s...
-
-- MAPREDUCE-3652.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails
- org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization fails.
This is testing the old jsp web interfaces. I think this test should just be removed.
Any objections?
-
-- MAPREDUCE-3657.
- Minor bug reported by jlowe and fixed by jlowe (build, mrv2)
- State machine visualize build fails
- Attempting to build the state machine graphs with {{mvn -Pvisualize compile}} fails for the resourcemanager and nodemanager projects. The build fails because org.apache.commons.logging.LogFactory isn't in the classpath.
-
-- MAPREDUCE-3664.
- Minor bug reported by praveensripati and fixed by brandonli (documentation)
- HDFS Federation Documentation has incorrect configuration example
- HDFS Federation documentation example (1) has the following
<property>
<name>dfs.namenode.rpc-address.ns1</name>
<value>hdfs://nn-host1:rpc-port</value>
</property>
dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there.
(1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html
-
-- MAPREDUCE-3669.
- Blocker bug reported by tgraves and fixed by mahadev (mrv2)
- Getting a lot of PriviledgedActionException / SaslException when running a job
- On a secure cluster, when running a job we are seeing a lot of PriviledgedActionException / SaslExceptions. The job runs fine, its just the jobclient can't connect to the AM to get the progress information.
Its in a very tight loop retrying while getting the exceptions.
snip of the client log is:
12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47
52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34...
-
-- MAPREDUCE-3679.
- Major improvement reported by mahadev and fixed by vinodkv (mrv2)
- AM logs and others should not automatically refresh after every 1 second.
- If you are looking through the logs for AM or containers, the page is automatically refreshed after 1 second or so which makes it problematic to search through the page or debug using the content on the page. We should not refresh the logs page. There should be a button to manually refresh if the user needs to.
-
-- MAPREDUCE-3681.
- Critical bug reported by tgraves and fixed by acmurthy (mrv2)
- capacity scheduler LeafQueues calculate used capacity wrong
- In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
In updateResource in LeafQueue:
setUsedCapacity(
usedResources.getMemory() / (clusterResource.getMemory() * capacity));
I think the clusterResource.getMemory() should be something like getParentsMemory().
-
-- MAPREDUCE-3683.
- Blocker bug reported by tgraves and fixed by acmurthy (mrv2)
- Capacity scheduler LeafQueues maximum capacity calculation issues
- In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
It also seems to only use the parents capacity in the leaf queues max capacity calculat...
-
-- MAPREDUCE-3684.
- Major bug reported by tomwhite and fixed by tomwhite (client)
- LocalDistributedCacheManager does not shut down its thread pool
- This was observed by running a Hive job in local mode. The job completed but the client process did not exit for 60 seconds.
-
-- MAPREDUCE-3689.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- RM web UI doesn't handle newline in job name
- a user submitted a mapreduce job with a newline (\n) in the job name. This caused the resource manager web ui to get a javascript exception when loading the application and scheduler pages and the pages were pretty well useless after that since they didn't load everything. Note that this only happens when the data is returned in the JS_ARRAY, which is when you get over 100 applications.
errors:
Uncaught SyntaxError: Unexpected token ILLEGAL
Uncaught ReferenceError: appsData is not defined
...
-
-- MAPREDUCE-3691.
- Critical bug reported by tgraves and fixed by tgraves (mrv2)
- webservices add support to compress response
- The web services currently don't support header 'Accept-Encoding: gzip'
Given that the responses have a lot of duplicate data like the property names in JSON or the tag names in XML, it should
compress very well, and would save on bandwidth and download time when fetching a potentially large response, like the
ones from ws/v1/cluster/apps and ws/v1/history/mapreduce/jobs
-
-- MAPREDUCE-3692.
- Blocker improvement reported by eli and fixed by eli (mrv2)
- yarn-resourcemanager out and log files can get big
- I'm seeing 8gb resourcemanager out files and big log files, seeing lots of repeated logs (eg every rpc call or event) looks like we're being too verbose in a couple of places.
-
-- MAPREDUCE-3693.
- Minor improvement reported by rvs and fixed by rvs (mrv2)
- Add admin env to mapred-default.xml
- I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't forward the value of -Djava.library.path= from the parent JVM to the child JVM. Thus if one wants to use native libraries for compression the only option seems to be to manually include relevant java.library.path settings into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
This seems to be a change in behavior compared to MR1 where TaskRunner.java used to do that:
{noformat}
String libraryPath = System.get...
-
-- MAPREDUCE-3696.
- Blocker bug reported by johnvijoe and fixed by johnvijoe (mrv2)
- MR job via oozie does not work on hadoop 23
- NM throws an error on submitting an MR job via oozie on the latest Hadoop 23.
*Courtesy: Mona Chitnis (ooize)
-
-- MAPREDUCE-3697.
- Blocker bug reported by johnvijoe and fixed by mahadev (mrv2)
- Hadoop Counters API limits Oozie's working across different hadoop versions
- Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in
hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its
parent class AbstractCounters implements it. This is different from hadoop20X.
As a result, Oozie compiled with either hadoop version does not work with the
other version.
A specific scenario, Oozie compiled with .23 and run against 205, does not
update job status owing to a Counters API exception.
Will explicit re-compilation against th...
-
-- MAPREDUCE-3698.
- Blocker sub-task reported by sseth and fixed by mahadev (mrv2)
- Client cannot talk to the history server in secure mode
- {noformat}
12/01/19 02:56:22 ERROR security.UserGroupInformation: PriviledgedActionException as:XXX@XXX(auth:KERBEROS) cause:java.io.IOException: Failed to specify server's Kerberos principal name
12/01/19 02:56:22 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
{noformat}
-
-- MAPREDUCE-3701.
- Major bug reported by mahadev and fixed by mahadev (mrv2)
- Delete HadoopYarnRPC from 0.23 branch.
- HadoopYarnRPC file exists in 0.23 (should have been removed with the new HadoopYarnProtoRPC). Trunk does not have this issue.
-
-- MAPREDUCE-3702.
- Critical bug reported by tgraves and fixed by tgraves (mrv2)
- internal server error trying access application master via proxy with filter enabled
- I had a hadoop.http.filter.initializers in place to do user authentication, but was purposely trying to let it bypass authentication on certain pages. One of those was the proxy and the application master main page. When I then tried to go to the application master through the proxy it throws an internal server error:
Problem accessing /mapreduce. Reason:
INTERNAL_SERVER_ERROR
Caused by:
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFi...
-
-- MAPREDUCE-3705.
- Blocker bug reported by tgraves and fixed by tgraves (mrv2)
- ant build fails on 0.23 branch
- running the ant build in mapreduce on the latest 23 branch fails. Looks like the ivy properties file still has 0.24.0 and then the gridmix dependencies need to have rumen as dependency.
The gridmix errors look like:
[javac] /home/tgraves/anttest/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java:249: cannot find symbol
[javac] symbol : class JobStoryProducer
[javac] location: class org.apache.hadoop.mapred.gridmix...
-
-- MAPREDUCE-3708.
- Major bug reported by kam_iitkgp and fixed by kamesh (mrv2)
- Metrics: Incorrect Apps Submitted Count
- Submitted an application with the following configuration
{code:xml}
<property>
<name>yarn.resourcemanager.am.max-retries</name>
<value>2</value>
</property>
{code}
In the above case, application had failed first time. So AM attempted the same application again.
While attempting the same application, *Apps Submitted* counter also has been incremented.
-
-- MAPREDUCE-3709.
- Major bug reported by eli and fixed by hitesh (mrv2, test)
- TestDistributedShell is failing
- TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23.
-
-- MAPREDUCE-3712.
- Blocker bug reported by raviprak and fixed by mahadev (mrv2)
- The mapreduce tar does not contain the hadoop-mapreduce-client-jobclient-tests.jar.
- Working MRv1 tests were moved into the maven build as part of MAPREDUCE-3582. Some classes like MRBench, SleepJob, FailJob which are essential for QE got moved to jobclient-tests.jar. However the tar.gz file does not contain this jar.
-
-- MAPREDUCE-3717.
- Blocker bug reported by mahadev and fixed by mahadev (mrv2)
- JobClient test jar has missing files to run all the test programs.
- Looks like MAPREDUCE-3582 forgot to move couple of files from the ant builds. The current test jar from jobclient does not work.
-
-- MAPREDUCE-3718.
- Major sub-task reported by vinodkv and fixed by hitesh (mrv2, performance)
- Default AM heartbeat interval should be one second
- Helps in improving app performance. RM should be able to handle this, as the heartbeats aren't really costly.
-
-- MAPREDUCE-3721.
- Blocker bug reported by sseth and fixed by sseth (mrv2)
- Race in shuffle can cause it to hang
- If all current {{Fetcher}}s complete while an in-memory merge is in progress - shuffle could hang.
Specifically - if the memory freed by an in-memory merge does not bring {{MergeManager.usedMemory}} below {{MergeManager.memoryLimit}} and all current Fetchers complete before the in-memory merge completes, another in-memory merge will not be triggered - and shuffle will hang. (All new fetchers are asked to WAIT).
-
-- MAPREDUCE-3723.
- Major bug reported by kam_iitkgp and fixed by kamesh (mrv2, test, webapps)
- TestAMWebServicesJobs & TestHSWebServicesJobs incorrectly asserting tests
- While testing a patch for one of the MR issues, I found TestAMWebServicesJobs & TestHSWebServicesJobs incorrectly asserting tests.
Moreover tests may fail if
{noformat}
index of counterGroups > #counters in a particular counterGroup
{noformat}
{code:title=TestAMWebServicesJobs.java|borderStyle=solid}
for (int j = 0; j < counters.length(); j++) {
JSONObject counter = counters.getJSONObject(i);
{code}
where is *i* is index of outer loop. It should be *j* instead of *i*.
-
-- MAPREDUCE-3727.
- Critical bug reported by tucu00 and fixed by tucu00 (security)
- jobtoken location property in jobconf refers to wrong jobtoken file
- Oozie launcher job (for MR/Pig/Hive/Sqoop action) reads the location of the jobtoken file from the *HADOOP_TOKEN_FILE_LOCATION* ENV var and seeds it as the *mapreduce.job.credentials.binary* property in the jobconf that will be used to launch the real (MR/Pig/Hive/Sqoop) job.
The MR/Pig/Hive/Sqoop submission code (via Hadoop job submission) uses correctly the injected *mapreduce.job.credentials.binary* property to load the credentials and submit their MR jobs.
The problem is that the *mapre...
-
-- MAPREDUCE-3733.
- Major bug reported by mahadev and fixed by mahadev
- Add Apache License Header to hadoop-distcp/pom.xml
- Looks like I missed the Apache Headers in the review. Adding it now.
-
-- MAPREDUCE-3735.
- Blocker bug reported by mahadev and fixed by mahadev (mrv2)
- Add distcp jar to the distribution (tar)
- Distcp jar isnt getting added to the tarball as of now. We need to add it along with archives/streaming and others.
-
-- MAPREDUCE-3737.
- Critical bug reported by revans2 and fixed by revans2 (mrv2)
- The Web Application Proxy's is not documented very well
- The Web Application Proxy is a security feature, but there is no documentation for what it does, why it does it, and more importantly what attacks it is known not protect against. This is so that anyone addopting Hadoop can know exactly what they potential security issues they may encounter.
-
-- MAPREDUCE-3742.
- Blocker bug reported by jlowe and fixed by jlowe (mrv2)
- "yarn logs" command fails with ClassNotFoundException
- Executing "yarn logs" at a shell prompt fails with this error:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogDumper
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogDumper
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(U...
-
-- MAPREDUCE-3744.
- Blocker bug reported by jlowe and fixed by jlowe (mrv2)
- Unable to retrieve application logs via "yarn logs" or "mapred job -logs"
- Trying to retrieve application logs via the "yarn logs" shell command results in an error similar to this:
Exception in thread "main" java.io.FileNotFoundException: File /tmp/logs/application_1327694122989_0001 does not exist.
at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:226)
at org.apache.hadoop.fs.Hdfs$DirListingIterator.<init>(Hdfs.java:217)
at org.apache.hadoop.fs.Hdfs$2.<init>(Hdfs.java:192)
at org.apache.hadoop.fs.Hdfs.listStatusIterator(Hdfs.java:192)
at org.a...
-
-- MAPREDUCE-3747.
- Major bug reported by rramya and fixed by acmurthy (mrv2)
- Memory Total is not refreshed until an app is launched
- Memory Total on the RM UI is not refreshed until an application is launched. This is a problem when the cluster is started for the first time or when there are any lost/decommissioned NMs.
When the cluster is started for the first time, Active Nodes is > 0 but the Memory Total=0. Also when there are any lost/decommissioned nodes, Memory Total has wrong value.
This is a useful tool for cluster admins and has to be updated correctly without having the need to submit an app each time.
-
-- MAPREDUCE-3748.
- Minor bug reported by rramya and fixed by rramya (mrv2)
- Move CS related nodeUpdate log messages to DEBUG
- Currently, the RM has nodeUpdate logs per NM per second such as the following:
2012-01-27 21:51:32,429 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <nodemanager1>:<port1> clusterResources: memory: 57344
2012-01-27 21:51:32,510 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: nodeUpdate: <nodemanager2>:<port2> clusterResources: memory: 57344
2012-01-27 21:51:33,094 INFO org.apache.hadoop.yarn.server...
-
-- MAPREDUCE-3749.
- Blocker bug reported by tomwhite and fixed by tomwhite (mrv2)
- ConcurrentModificationException in counter groups
- Iterating over a counter's groups while adding more groups will cause a ConcurrentModificationException.
This was found while running Hive unit tests against a recent 0.23 version.
-
-- MAPREDUCE-3756.
- Major improvement reported by acmurthy and fixed by hitesh (mrv2)
- Make single shuffle limit configurable
- Make single shuffle limit configurable, currently it's hard-coded.
-
-- MAPREDUCE-3759.
- Major bug reported by rramya and fixed by vinodkv (mrv2)
- ClassCastException thrown in -list-active-trackers when there are a few unhealthy nodes
- When there are a few blacklisted nodes in the cluster, "bin/mapred job -list-active-trackers" throws "java.lang.ClassCastException: org.apache.hadoop.yarn.server.resourcemanager.resource.Resources$1 cannot be cast to org.apache.hadoop.yarn.api.records.impl.pb.ResourcePBImpl"
-
-- MAPREDUCE-3762.
- Critical bug reported by mahadev and fixed by mahadev (mrv2)
- Resource Manager fails to come up with default capacity scheduler configs.
- Thanks to [~harip] for pointing out the issue. This is the stack trace for bringing up RM with default CS configs:
{code}
java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01 used in call to setMaxCapacity for queue default
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:28)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:21...
-
-- MAPREDUCE-3764.
- Critical bug reported by sseth and fixed by acmurthy (mrv2)
- AllocatedGB etc metrics incorrect if min-allocation-mb isn't a multiple of 1GB
- MutableGaugeInt incremented as {{allocatedGB.incr(res.getMemory() / GB * containers);}}
Setting yarn.scheduler.capacity.minimum-allocation-mb to 1536 - each increment is counted as 1GB.
Trying to analyze the metrics - looks like the cluster is never over 67-68% utilized, depending on high ram requests.
-
-- MAPREDUCE-3765.
- Minor bug reported by hitesh and fixed by hitesh (mrv2)
- FifoScheduler does not respect yarn.scheduler.fifo.minimum-allocation-mb setting
- FifoScheduler uses default min 1 GB regardless of the configuration value set for minimum memory allocation.
-
-- MAPREDUCE-3771.
- Major improvement reported by acmurthy and fixed by acmurthy
- Port MAPREDUCE-1735 to trunk/0.23
- Per discussion in general@, we should port MAPREDUCE-1735 to 0.23 & trunk to 'undeprecate' old mapred api:
http://s.apache.org/undeprecate-mapred-apis
-
-- MAPREDUCE-3775.
- Minor bug reported by hitesh and fixed by hitesh (mrv2)
- Change MiniYarnCluster to escape special chars in testname
- When using MiniYarnCluster with the testname set to a nested classname, the "$" within the class name creates issues with the container launch scripts as they try to expand the $... within the paths/variables in use.
-
-- MAPREDUCE-3780.
- Blocker bug reported by rramya and fixed by hitesh (mrv2)
- RM assigns containers to killed applications
- RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation.
-
-- MAPREDUCE-3791.
- Major bug reported by rvs and fixed by mahadev (documentation, mrv2)
- can't build site in hadoop-yarn-server-common
- Here's how to reproduce:
{noformat}
$ mvn site site:stage -DskipTests -DskipTest -DskipITs
....
main:
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [49.017s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [5.152s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [4.973s]
[INFO] Apache Hadoop Project Dist POM ..................
-
-- MAPREDUCE-3794.
- Major bug reported by tomwhite and fixed by tomwhite (mrv2)
- Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
- The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x.
-
-- MAPREDUCE-3795.
- Major bug reported by vinodkv and fixed by vinodkv (mrv2)
- "job -status" command line output is malformed
- Misses new lines after numMaps and numReduces. Caused by MAPREDUCE-3720.
-
-- MAPREDUCE-3803.
- Major test reported by raviprak and fixed by raviprak (build)
- HDFS-2864 broke ant compilation
- compile:
[echo] contrib: raid
[javac] <somePath>/hadoop-mapreduce-project/src/contrib/build-contrib.xml:194: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 28 source files to <somepath>/hadoop-mapreduce-project/build/contrib/raid/classes
[javac] <somepath>/hadoop-mapreduce-project/src/contrib/raid/src/java/org/apache/hadoop/hdfs/server/datanode/RaidBlockSender.java:111: cannot find symbol
...
-
-- MAPREDUCE-3809.
- Blocker sub-task reported by sseth and fixed by sseth (mrv2)
- Tasks may take upto 3 seconds to exit after completion
- Task.TaskReporter.stopCommunicationThread can end up waiting for a thread.sleep(3000) before stopping the thread.
-
-- MAPREDUCE-3810.
- Blocker sub-task reported by vinodkv and fixed by vinodkv (mrv2, performance)
- MR AM's ContainerAllocator is assigning the allocated containers very slowly
- This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM->RM heartbeat cycle.
-
-- MAPREDUCE-3811.
- Critical task reported by sseth and fixed by sseth (mrv2)
- Make the Client-AM IPC retry count configurable
+ - HDFS-1249.
+ Minor bug reported by matsusaka kentaro and fixed by Colin Patrick McCabe (fuse-dfs)
+ with fuse-dfs, chown which only has owner (or only group) argument fails with Input/output error.
-
-- MAPREDUCE-3813.
- Major sub-task reported by vinodkv and fixed by vinodkv (mrv2, performance)
- RackResolver should maintain a cache to avoid repetitive lookups.
- With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance.
-
-- MAPREDUCE-3814.
- Major bug reported by acmurthy and fixed by acmurthy (mrv1, mrv2)
- MR1 compile fails
- $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal
BUILD FAILED
/grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir "/grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar" does not exist!
-
-- MAPREDUCE-3817.
- Major bug reported by arpitgupta and fixed by arpitgupta (mrv2)
- bin/mapred command cannot run distcp and archive jobs
+ - HDFS-1153.
+ Minor bug reported by Ravi Phulari and fixed by Ravi Phulari (data-node)
+ dfsnodelist.jsp should handle invalid input parameters
-
-- MAPREDUCE-3826.
- Major bug reported by arpitgupta and fixed by jeagles (mrv2)
- RM UI when loaded throws a message stating Data Tables warning and then the column sorting stops working
+ - HDFS-1013.
+ Minor improvement reported by Todd Lipcon and fixed by Eugene Koontz
+ Miscellaneous improvements to HTML markup for web UIs
+
+- HDFS-799.
+ Major improvement reported by Christian Kunz and fixed by Colin Patrick McCabe
+ libhdfs must call DetachCurrentThread when a thread is destroyed
+
+- HDFS-766.
+ Minor bug reported by Ravi Phulari and fixed by Jon Zuanich
+ Error message not clear for set space quota out of boundary values.
+
+- HDFS-744.
+ Major new feature reported by Hairong Kuang and fixed by Lars Hofhansl (data-node , hdfs client)
+ Support hsync in HDFS
+
+- HDFS-711.
+ Major bug reported by freestyler and fixed by Colin Patrick McCabe (documentation)
+ hdfsUtime does not handle atime = 0 or mtime = 0 correctly
+
+- HDFS-470.
+ Minor bug reported by Pete Wyckoff and fixed by Colin Patrick McCabe
+ libhdfs should handle 0-length reads from FSInputStream correctly
+
+- HADOOP-8801.
+ Major bug reported by Eli Collins and fixed by Eli Collins
+ ExitUtil#terminate should capture the exception stack trace
+
+- HADOOP-8794.
+ Major bug reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
+ Modifiy bin/hadoop to point to HADOOP_YARN_HOME
+
+- HADOOP-8781.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (scripts)
+ hadoop-config.sh should add JAVA_LIBRARY_PATH to LD_LIBRARY_PATH
+
+- HADOOP-8775.
+ Major bug reported by Sandy Ryza and fixed by Sandy Ryza
+ MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete
+
+- HADOOP-8770.
+ Blocker bug reported by Todd Lipcon and fixed by Eli Collins (trash)
+ NN should not RPC to self to find trash defaults (causes deadlock)
+
+- HADOOP-8766.
+ Major bug reported by Eli Collins and fixed by Colin Patrick McCabe (test)
+ FileContextMainOperationsBaseTest should randomize the root dir
+
+- HADOOP-8764.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (build)
+ CMake: HADOOP-8737 broke ARM build
+
+- HADOOP-8754.
+ Minor improvement reported by Brandon Li and fixed by Brandon Li (ipc)
+ Deprecate all the RPC.getServer() variants
+
+- HADOOP-8749.
+ Major bug reported by Ahmed Radwan and fixed by Ahmed Radwan (conf)
+ HADOOP-8031 changed the way in which relative xincludes are handled in Configuration.
+
+- HADOOP-8748.
+ Minor improvement reported by Arun C Murthy and fixed by Arun C Murthy (io)
+ Move dfsclient retry to a util class
+
+- HADOOP-8747.
+ Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
+ Syntax error on cmake version 2.6 patch 2 in JNIFlags.cmake
+
+- HADOOP-8738.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
+ junit JAR is showing up in the distro
+
+- HADOOP-8737.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
+ cmake: always use JAVA_HOME to find libjvm.so, jni.h, jni_md.h
+
+- HADOOP-8727.
+ Major bug reported by Harsh J and fixed by Harsh J (conf)
+ Gracefully deprecate dfs.umaskmode in 2.x onwards
+
+- HADOOP-8726.
+ Major bug reported by Benoy Antony and fixed by Daryn Sharp (security)
+ The Secrets in Credentials are not available to MR tasks
+
+- HADOOP-8725.
+ Blocker bug reported by Daryn Sharp and fixed by Daryn Sharp (security)
+ MR is broken when security is off
+
+- HADOOP-8722.
+ Minor bug reported by Eli Collins and fixed by Colin Patrick McCabe (documentation)
+ Update BUILDING.txt with latest snappy info
+
+- HADOOP-8721.
+ Critical bug reported by suja s and fixed by Vinay (auto-failover , ha)
+ ZKFC should not retry 45 times when attempting a graceful fence during a failover
+
+- HADOOP-8720.
+ Trivial bug reported by Vlad Rozov and fixed by Vlad Rozov (test)
+ TestLocalFileSystem should use test root subdirectory
+
+- HADOOP-8710.
+ Major improvement reported by Eli Collins and fixed by Eli Collins (fs)
+ Remove ability for users to easily run the trash emptier
+ The trash emptier may no longer be run using "hadoop org.apache.hadoop.fs.Trash". The trash emptier runs on the NameNode (if configured). Old trash checkpoints may be deleted using "hadoop fs -expunge".
+- HADOOP-8709.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (fs)
+ globStatus changed behavior from 0.20/1.x
+
+- HADOOP-8703.
+ Major bug reported by Dave Thompson and fixed by Dave Thompson
+ distcpV2: turn CRC checking off for 0 byte size
+ distcp skips CRC on 0 byte files.
+- HADOOP-8700.
+ Minor improvement reported by Tsz Wo (Nicholas), SZE and fixed by Tsz Wo (Nicholas), SZE (util)
+ Move the checksum type constants to an enum
+
+- HADOOP-8699.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (test)
+ some common testcases create core-site.xml in test-classes making other testcases to fail
+
+- HADOOP-8697.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
+ TestWritableName fails intermittently with JDK7
+
+- HADOOP-8695.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
+ TestPathData fails intermittently with JDK7
+
+- HADOOP-8693.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
+ TestSecurityUtil fails intermittently with JDK7
+
+- HADOOP-8692.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
+ TestLocalDirAllocator fails intermittently with JDK7
+
+- HADOOP-8689.
+ Major improvement reported by Eli Collins and fixed by Eli Collins (fs)
+ Make trash a server side configuration option
+ If fs.trash.interval is configured on the server then the client's value for this configuration is ignored.
+- HADOOP-8686.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
+ Fix warnings in native code
+
+- HADOOP-8660.
+ Major bug reported by Eli Collins and fixed by Alejandro Abdelnur
+ TestPseudoAuthenticator failing with NPE
+
+- HADOOP-8659.
+ Major bug reported by Trevor Robinson and fixed by Colin Patrick McCabe (native)
+ Native libraries must build with soft-float ABI for Oracle JVM on ARM
+
+- HADOOP-8655.
+ Major bug reported by Arun A K and fixed by (util)
+ In TextInputFormat, while specifying textinputformat.record.delimiter the character/character sequences in data file similar to starting character/starting character sequence in delimiter were found missing in certain cases in the Map Output
+
+- HADOOP-8654.
+ Major bug reported by Gelesh and fixed by (util)
+ TextInputFormat delimiter bug:- Input Text portion ends with & Delimiter starts with same char/char sequence
+
+- HADOOP-8648.
+ Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ libhadoop: native CRC32 validation crashes when io.bytes.per.checksum=1
+
+- HADOOP-8644.
+ Critical new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ AuthenticatedURL should be able to use SSLFactory
+
+- HADOOP-8637.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ FilterFileSystem#setWriteChecksum is broken
+
+- HADOOP-8635.
+ Critical improvement reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ Cannot cancel paths registered deleteOnExit
+
+- HADOOP-8634.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ Ensure FileSystem#close doesn't squawk for deleteOnExit paths
+
+- HADOOP-8633.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ Interrupted FsShell copies may leave tmp files
+
+- HADOOP-8632.
+ Major bug reported by Costin Leau and fixed by Costin Leau (conf)
+ Configuration leaking class-loaders
+
+- HADOOP-8627.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ FS deleteOnExit may delete the wrong path
+
+- HADOOP-8626.
+ Major bug reported by Jonathan Natkins and fixed by Jonathan Natkins (security)
+ Typo in default setting for hadoop.security.group.mapping.ldap.search.filter.user
+
+- HADOOP-8624.
+ Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (ipc)
+ ProtobufRpcEngine should log all RPCs if TRACE logging is enabled
+
+- HADOOP-8623.
+ Minor improvement reported by Steven Willis and fixed by Steven Willis (scripts)
+ hadoop jar command should respect HADOOP_OPTS
+
+- HADOOP-8620.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
+ Add -Drequire.fuse and -Drequire.snappy
+
+- HADOOP-8614.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ IOUtils#skipFully hangs forever on EOF
+
+- HADOOP-8613.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp
+ AbstractDelegationTokenIdentifier#getUser() should set token auth type
+
+- HADOOP-8611.
+ Major bug reported by Kihwal Lee and fixed by Robert Parker (security)
+ Allow fall-back to the shell-based implementation when JNI-based users-group mapping fails
+
+- HADOOP-8609.
+ Major improvement reported by Todd Lipcon and fixed by Jon Zuanich
+ IPC server logs a useless message when shutting down socket
+
+- HADOOP-8606.
+ Major bug reported by Daryn Sharp and fixed by Daryn Sharp (fs)
+ FileSystem.get may return the wrong filesystem
+
+- HADOOP-8599.
+ Major bug reported by Andrey Klochkov and fixed by Andrey Klochkov (fs)
+ Non empty response from FileSystem.getFileBlockLocations when asking for data beyond the end of file
+
+- HADOOP-8587.
+ Minor bug reported by Eli Collins and fixed by Eli Collins (fs)
+ HarFileSystem access of harMetaCache isn't threadsafe
+
+- HADOOP-8586.
+ Major bug reported by Eli Collins and fixed by Eli Collins
+ Fixup a bunch of SPNEGO misspellings
+
+- HADOOP-8585.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Fix initialization circularity between UserGroupInformation and HadoopConfiguration
+
+- HADOOP-8581.
+ Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ add support for HTTPS to the web UIs
+
+- HADOOP-8573.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (conf)
+ Configuration tries to read from an inputstream resource multiple times.
+
+- HADOOP-8566.
+ Major bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (io)
+ AvroReflectSerializer.accept(Class) throws a NPE if the class has no package (primitive types and arrays)
+
+- HADOOP-8563.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (build)
+ don't package hadoop-pipes examples/bin
+
+- HADOOP-8551.
+ Major bug reported by Robert Joseph Evans and fixed by John George (fs)
+ fs -mkdir creates parent directories without the -p option
+ FsShell's "mkdir" no longer implicitly creates all non-existent parent directories. The command adopts the posix compliant behavior of requiring the "-p" flag to auto-create parent directories.
+- HADOOP-8550.
+ Major bug reported by Robert Joseph Evans and fixed by John George (fs)
+ hadoop fs -touchz automatically created parent directories
+
+- HADOOP-8547.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Package hadoop-pipes examples/bin directory (again)
+
+- HADOOP-8543.
+ Major bug reported by Radim Kolar and fixed by Radim Kolar (build)
+ Invalid pom.xml files on 0.23 branch
+
+- HADOOP-8541.
+ Major improvement reported by Andrew Wang and fixed by Andrew Wang (metrics)
+ Better high-percentile latency metrics
+
+- HADOOP-8538.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (native)
+ CMake builds fail on ARM
+
+- HADOOP-8537.
+ Major bug reported by Todd Lipcon and fixed by Todd Lipcon (io)
+ Two TFile tests failing recently
+
+- HADOOP-8535.
+ Major improvement reported by Jonathan Eagles and fixed by Jonathan Eagles (build)
+ Cut hadoop build times in half (upgrade maven-compiler-plugin to 2.5.1)
+
+- HADOOP-8533.
+ Major improvement reported by Suresh Srinivas and fixed by Brandon Li (ipc)
+ Remove Parallel Call in IPC
+ Merged the change to branch-2
+- HADOOP-8531.
+ Trivial improvement reported by Harsh J and fixed by madhukara phatak (io)
+ SequenceFile Writer can throw out a better error if a serializer or deserializer isn't available
+
+- HADOOP-8525.
+ Trivial improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Provide Improved Traceability for Configuration
+
+- HADOOP-8524.
+ Trivial improvement reported by Harsh J and fixed by Harsh J (conf)
+ Allow users to get source of a Configuration parameter
+
+- HADOOP-8512.
+ Minor bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ AuthenticatedURL should reset the Token when the server returns other than OK on authentication
+
+- HADOOP-8509.
+ Minor bug reported by Matteo Bertozzi and fixed by Alejandro Abdelnur (util)
+ JarFinder duplicate entry: META-INF/MANIFEST.MF exception
+
+- HADOOP-8507.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Avoid OOM while deserializing DelegationTokenIdentifer
+
+- HADOOP-8501.
+ Major bug reported by Radim Kolar and fixed by Radim Kolar (benchmarks)
+ Gridmix fails to compile on OpenJDK7u4
+
+- HADOOP-8499.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Lower min.user.id to 500 for the tests
+
+- HADOOP-8495.
+ Critical bug reported by Jason Lowe and fixed by Jason Lowe (build)
+ Update Netty to avoid leaking file descriptors during shuffle
+
+- HADOOP-8488.
+ Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ test-patch.sh gives +1 even if the native build fails.
+
+- HADOOP-8485.
+ Minor bug reported by Eli Collins and fixed by Eli Collins (documentation)
+ Don't hardcode "Apache Hadoop 0.23" in the docs
+
+- HADOOP-8481.
+ Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (documentation)
+ update BUILDING.txt to talk about cmake rather than autotools
+
+- HADOOP-8480.
+ Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ The native build should honor -DskipTests
+
+- HADOOP-8466.
+ Major bug reported by Bruno Mahé and fixed by Bruno Mahé (build)
+ hadoop-client POM incorrectly excludes avro
+
+- HADOOP-8465.
+ Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ hadoop-auth should support ephemeral authentication
+
+- HADOOP-8463.
+ Major improvement reported by Eli Collins and fixed by madhukara phatak (security)
+ hadoop.security.auth_to_local needs a key definition and doc
+
+- HADOOP-8460.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans (documentation)
+ Document proper setting of HADOOP_PID_DIR and HADOOP_SECURE_DN_PID_DIR
+
+- HADOOP-8458.
+ Major new feature reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (security)
+ Add management hook to AuthenticationHandler to enable delegation token operations support
+
+- HADOOP-8452.
+ Minor bug reported by Andy Isaacson and fixed by Andy Isaacson
+ DN logs backtrace when running under jsvc and /jmx is loaded
+
+- HADOOP-8450.
+ Trivial bug reported by Colin Patrick McCabe and fixed by Eli Collins (test)
+ Remove src/test/system
+
+- HADOOP-8449.
+ Minor bug reported by Joey Echeverria and fixed by Harsh J
+ hadoop fs -text fails with compressed sequence files with the codec file extension
+
+- HADOOP-8444.
+ Major bug reported by Mariappan Asokan and fixed by madhukara phatak (fs , test)
+ Fix the tests FSMainOperationsBaseTest.java and F ileContextMainOperationsBaseTest.java to avoid potential test failure
+
+- HADOOP-8438.
+ Major bug reported by Devaraj K and fixed by Devaraj K
+ hadoop-validate-setup.sh refers to examples jar file which doesn't exist
+
+- HADOOP-8433.
+ Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (scripts)
+ Don't set HADOOP_LOG_DIR in hadoop-env.sh
+
+- HADOOP-8431.
+ Major bug reported by Eli Collins and fixed by Sandy Ryza
+ Running distcp wo args throws IllegalArgumentException
+
+- HADOOP-8423.
+ Major bug reported by Jason B and fixed by Todd Lipcon (io)
+ MapFile.Reader.get() crashes jvm or throws EOFException on Snappy or LZO block-compressed data
+
+- HADOOP-8422.
+ Minor bug reported by Eli Collins and fixed by Eli Collins (fs)
+ Deprecate FileSystem#getDefault* and getServerDefault methods that don't take a Path argument
+
+- HADOOP-8408.
+ Major bug reported by Aaron T. Myers and fixed by Aaron T. Myers (viewfs)
+ MR doesn't work with a non-default ViewFS mount table and security enabled
+
+- HADOOP-8406.
+ Major bug reported by Todd Lipcon and fixed by Todd Lipcon (io)
+ CompressionCodecFactory.CODEC_PROVIDERS iteration is thread-unsafe
+
+- HADOOP-8403.
+ Major task reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (build)
+ bump up POMs version to 2.0.1-SNAPSHOT
+
+- HADOOP-8400.
+ Major bug reported by Eli Collins and fixed by Alejandro Abdelnur (security)
+ All commands warn "Kerberos krb5 configuration not found" when security is not enabled
+
+- HADOOP-8393.
+ Major bug reported by Patrick Hunt and fixed by Patrick Hunt (scripts)
+ hadoop-config.sh missing variable exports, causes Yarn jobs to fail with ClassNotFoundException MRAppMaster
+
+- HADOOP-8390.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (test)
+ TestFileSystemCanonicalization fails with JDK7
+
+- HADOOP-8373.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (ipc)
+ Port RPC.getServerAddress to 0.23
+
+- HADOOP-8372.
+ Major bug reported by Junping Du and fixed by Junping Du (io , util)
+ normalizeHostName() in NetUtils is not working properly in resolving a hostname start with numeric character
+
+- HADOOP-8370.
+ Major bug reported by Trevor Robinson and fixed by Trevor Robinson (native)
+ Native build failure: javah: class file for org.apache.hadoop.classification.InterfaceAudience not found
+
+- HADOOP-8368.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Use CMake rather than autotools to build native code
+
+- HADOOP-8367.
+ Major improvement reported by Sanjay Radia and fixed by Sanjay Radia
+ Improve documentation of declaringClassProtocolName in rpc headers
+
+- HADOOP-8362.
+ Trivial improvement reported by Todd Lipcon and fixed by madhukara phatak (conf)
+ Improve exception message when Configuration.set() is called with a null key or value
+
+- HADOOP-8361.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ Avoid out-of-memory problems when deserializing strings
+
+- HADOOP-8358.
+ Trivial improvement reported by Harsh J and fixed by Harsh J (conf)
+ Config-related WARN for dfs.web.ugi can be avoided.
+
+- HADOOP-8342.
+ Major bug reported by Randy Clayton and fixed by Alejandro Abdelnur (fs)
+ HDFS command fails with exception following merge of HADOOP-8325
+
+- HADOOP-8341.
+ Major bug reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Fix or filter findbugs issues in hadoop-tools
+
+- HADOOP-8340.
+ Minor improvement reported by Todd Lipcon and fixed by Todd Lipcon (util)
+ SNAPSHOT build versions should compare as less than their eventual final release
+
+- HADOOP-8335.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (util)
+ Improve Configuration's address handling
+
+- HADOOP-8334.
+ Major bug reported by Daryn Sharp and fixed by Daryn Sharp
+ HttpServer sometimes returns incorrect port
+
+- HADOOP-8330.
+ Minor bug reported by John George and fixed by John George (test)
+ TestSequenceFile.testCreateUsesFsArg() is broken
+
+- HADOOP-8329.
+ Major bug reported by Kumar Ravi and fixed by Eli Collins (build)
+ Build fails with Java 7
+
+- HADOOP-8328.
+ Major bug reported by Tom White and fixed by Tom White (fs)
+ Duplicate FileSystem Statistics object for 'file' scheme
+
+- HADOOP-8327.
+ Major bug reported by Dave Thompson and fixed by Dave Thompson
+ distcpv2 and distcpv1 jars should not coexist
+ Resolve sporadic distcp issue due to having two DistCp classes (v1 & v2) in the classpath.
+- HADOOP-8325.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (fs)
+ Add a ShutdownHookManager to be used by different components instead of the JVM shutdownhook
+
+- HADOOP-8323.
+ Critical improvement reported by Harsh J and fixed by Harsh J (io)
+ Revert HADOOP-7940 and improve javadocs and test for Text.clear()
+
+- HADOOP-8317.
+ Major bug reported by Radim Kolar and fixed by (build)
+ Update maven-assembly-plugin to 2.3 - fix build on FreeBSD
+
+- HADOOP-8316.
+ Major bug reported by Eli Collins and fixed by Eli Collins (conf)
+ Audit logging should be disabled by default
+
+- HADOOP-8305.
+ Major bug reported by John George and fixed by John George (viewfs)
+ distcp over viewfs is broken
+
+- HADOOP-8288.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash
+ Remove references of mapred.child.ulimit etc. since they are not being used any more
+
+- HADOOP-8287.
+ Major bug reported by Eli Collins and fixed by Eli Collins (conf)
+ etc/hadoop is missing hadoop-env.sh
+
+- HADOOP-8286.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (conf)
+ Simplify getting a socket address from conf
+
+- HADOOP-8283.
+ Major test reported by Daryn Sharp and fixed by Daryn Sharp (test)
+ Allow tests to control token service value
+
+- HADOOP-8278.
+ Major improvement reported by Tom White and fixed by Tom White (build)
+ Make sure components declare correct set of dependencies
+
+- HADOOP-8268.
+ Major bug reported by Radim Kolar and fixed by Radim Kolar (build)
+ A few pom.xml across Hadoop project may fail XML validation
+
+- HADOOP-8244.
+ Major improvement reported by Henry Robinson and fixed by Henry Robinson
+ Improve comments on ByteBufferReadable.read
+
+- HADOOP-8242.
+ Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
+ AbstractDelegationTokenIdentifier: add getter methods for owner and realuser
+
+- HADOOP-8240.
+ Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (fs)
+ Allow users to specify a checksum type on create()
+
+- HADOOP-8239.
+ Major improvement reported by Kihwal Lee and fixed by Kihwal Lee (fs)
+ Extend MD5MD5CRC32FileChecksum to show the actual checksum type being used
+
+- HADOOP-8227.
+ Blocker improvement reported by Robert Joseph Evans and fixed by Robert Joseph Evans
+ Allow RPC to limit ephemeral port range.
+
+- HADOOP-8225.
+ Blocker bug reported by Mithun Radhakrishnan and fixed by Daryn Sharp (security)
+ DistCp fails when invoked by Oozie
+
+- HADOOP-8224.
+ Major improvement reported by Eli Collins and fixed by Tomohiko Kinebuchi (conf)
+ Don't hardcode hdfs.audit.logger in the scripts
+
+- HADOOP-8197.
+ Critical bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (conf)
+ Configuration logs WARNs on every use of a deprecated key
+
+- HADOOP-8180.
+ Major bug reported by Ravi Prakash and fixed by Ravi Prakash
+ Remove hsqldb since its not needed from pom.xml
+
+- HADOOP-8179.
+ Minor bug reported by Steve Loughran and fixed by Daryn Sharp (fs)
+ risk of NPE in CopyCommands processArguments()
+
+- HADOOP-8172.
+ Critical bug reported by Robert Joseph Evans and fixed by Anupam Seth (conf)
+ Configuration no longer sets all keys in a deprecated key list.
+
+- HADOOP-8168.
+ Major bug reported by Eugene Koontz and fixed by Eugene Koontz (fs)
+ empty-string owners or groups causes {{MissingFormatWidthException}} in o.a.h.fs.shell.Ls.ProcessPath()
+
+- HADOOP-8167.
+ Blocker bug reported by Alejandro Abdelnur and fixed by Alejandro Abdelnur (conf)
+ Configuration deprecation logic breaks backwards compatibility
+
+- HADOOP-8144.
+ Minor bug reported by Junping Du and fixed by Junping Du (io)
+ pseudoSortByDistance in NetworkTopology doesn't work properly if no local node and first node is local rack node
+
+- HADOOP-8135.
+ Major new feature reported by Henry Robinson and fixed by Henry Robinson (fs)
+ Add ByteBufferReadable interface to FSDataInputStream
+
+- HADOOP-8129.
+ Major bug reported by Ravi Prakash and fixed by Ahmed Radwan (fs , test)
+ ViewFileSystemTestSetup setupForViewFileSystem is erring when the user's home directory is somewhere other than /home (eg. /User) etc.
+
+- HADOOP-8110.
+ Major bug reported by Tsz Wo (Nicholas), SZE and fixed by Jason Lowe (fs)
+ TestViewFsTrash occasionally fails
+
+- HADOOP-8104.
+ Major bug reported by Colin Patrick McCabe and fixed by Alejandro Abdelnur
+ Inconsistent Jackson versions
+
+- HADOOP-8088.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (security)
+ User-group mapping cache incorrectly does negative caching on transient failures
+
+- HADOOP-8075.
+ Major improvement reported by Eli Collins and fixed by Hızır Sefa İrken (native)
+ Lower native-hadoop library log from info to debug
+
+- HADOOP-8060.
+ Major bug reported by Kihwal Lee and fixed by Kihwal Lee (fs , util)
+ Add a capability to discover and set checksum types per file.
+
+- HADOOP-8031.
+ Major bug reported by Elias Ross and fixed by Elias Ross (conf)
+ Configuration class fails to find embedded .jar resources; should use URL.openStream()
+
+- HADOOP-8014.
+ Major bug reported by Daryn Sharp and fixed by John George (fs)
+ ViewFileSystem does not correctly implement getDefaultBlockSize, getDefaultReplication, getContentSummary
+
+- HADOOP-8005.
+ Major bug reported by Joe Crobak and fixed by Jason Lowe (scripts)
+ Multiple SLF4J binding message in .out file for all daemons
+
+- HADOOP-7967.
+ Critical bug reported by Daryn Sharp and fixed by Daryn Sharp (fs , security)
+ Need generalized multi-token filesystem support
+
+- HADOOP-7868.
+ Major bug reported by James Page and fixed by Trevor Robinson (native)
+ Hadoop native fails to compile when default linker option is -Wl,--as-needed
+
+- HADOOP-7818.
+ Minor bug reported by Eli Collins and fixed by madhukara phatak (util)
+ DiskChecker#checkDir should fail if the directory is not executable
+
+- HADOOP-7754.
+ Major sub-task reported by Todd Lipcon and fixed by Todd Lipcon (native , performance)
+ Expose file descriptors from Hadoop-wrapped local FileSystems
+
+- HADOOP-7703.
+ Major bug reported by Devaraj K and fixed by Devaraj K
+ WebAppContext should also be stopped and cleared
+ Improved excpetion handling of shutting down web server. (Devaraj K via Eric Yang)
+- HADOOP-7510.
+ Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (security)
+ Tokens should use original hostname provided instead of ip
+
+- HADOOP-6963.
+ Critical bug reported by Owen O'Malley and fixed by Ravi Prakash (fs)
+ Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links
+
+- HADOOP-6802.
+ Major improvement reported by Erik Steffl and fixed by Sho Shimauchi (conf , fs)
+ Remove FS_CLIENT_BUFFER_DIR_KEY = "fs.client.buffer.dir" from CommonConfigurationKeys.java (not used, deprecated)
+
+- HADOOP-3886.
+ Minor bug reported by brien colwell and fixed by Jingguo Yao (documentation)
+ Error in javadoc of Reporter, Mapper and Progressable
+
+- HADOOP-3450.
+ Minor improvement reported by Ari Rabkin and fixed by Sho Shimauchi (fs)
+ Add tests to Local Directory Allocator for asserting their URI-returning capability
-
-- MAPREDUCE-3833.
- Major bug reported by jlowe and fixed by jlowe (mrv2)
- Capacity scheduler queue refresh doesn't recompute queue capacities properly
- Refreshing the capacity scheduler configuration (e.g.: via yarn rmadmin -refreshQueues) can fail to compute the proper absolute capacity for leaf queues.
-
-
-
-
-
-Changes since Hadoop 0.22
-
+
+
+Hadoop 2.0.1-alpha Release Notes
+
+
+
+Hadoop 2.0.1-alpha Release Notes
+These release notes include new developer and user-facing incompatibilities, features, and major improvements.
+
+Changes since Hadoop 2.0.0-alpha
-- HADOOP-7778.
- Major bug reported by tomwhite and fixed by tomwhite
- FindBugs warning in Token.getKind()
- From https://builds.apache.org/job/PreCommit-HADOOP-Build/330//artifact/trunk/hadoop-common-project/patchprocess/newPatchFindbugsWarningshadoop-common.html
bq. org.apache.hadoop.security.token.Token.getKind() is unsynchronized, org.apache.hadoop.security.token.Token.setKind(Text) is synchronized
Looks like this was introduced by MAPREDUCE-2764.
-
-- HADOOP-7772.
- Trivial improvement reported by stevel@apache.org and fixed by stevel@apache.org
- javadoc the topology classes
- To help people understand and make changes to the Topology classes, their javadocs could be rounded off.
-
-- HADOOP-7771.
- Blocker bug reported by johnvijoe and fixed by johnvijoe
- NPE when running hdfs dfs -copyToLocal, -get etc
- NPE when running hdfs dfs -copyToLocal if the destination directory does not exist. The behavior in branch-0.20-security is to create the directory and copy/get the contents from source.
-
-- HADOOP-7770.
- Blocker bug reported by raviprak and fixed by raviprak (fs)
- ViewFS getFileChecksum throws FileNotFoundException for files in /tmp and /user
- Thanks to Rohini Palaniswamy for discovering this bug. To quote
bq. When doing getFileChecksum for path /user/hadoopqa/somefile, it is trying to fetch checksum for /user/user/hadoopqa/somefile. If /tmp/file, it is trying /tmp/tmp/file. Works fine for other FS operations.
-
-- HADOOP-7766.
- Major bug reported by jnp and fixed by jnp
- The auth to local mappings are not being respected, with webhdfs and security enabled.
- KerberosAuthenticationHandler reloads the KerberosName statically and overrides the auth to local mappings.
-
-- HADOOP-7764.
- Blocker bug reported by jeagles and fixed by jeagles
- Allow both ACL list and global path spec filters to HttpServer
- HttpServer allows setting global path spec filters in one constructor and ACL list in another constructor. Having both set in HttpServer is not user settable either by public API or constructor.
-
-- HADOOP-7763.
- Major improvement reported by tomwhite and fixed by tomwhite (documentation)
- Add top-level navigation to APT docs
- We need navigation menus for the APT docs that have been written so far.
-
-- HADOOP-7753.
- Major sub-task reported by tlipcon and fixed by tlipcon (io, native, performance)
- Support fadvise and sync_data_range in NativeIO, add ReadaheadPool class
- This JIRA adds JNI wrappers for sync_data_range and posix_fadvise. It also implements a ReadaheadPool class for future use from HDFS and MapReduce.
-
-- HADOOP-7749.
- Minor improvement reported by tlipcon and fixed by tlipcon (util)
- Add NetUtils call which provides more help in exception messages
- In setting up MR2, I accidentally had a bad configuration value specified for one of the IP configs. I was getting a NumberFormatException parsing this config, but no indication as to what config value was being fetched. This JIRA is to add an API to NetUtils.createSocketAddr which takes the configuration name, so that any exceptions thrown will point back to where the user needs to fix it.
-
-- HADOOP-7745.
- Major bug reported by raviprak and fixed by raviprak
- I switched variable names in HADOOP-7509
- As Aaron pointed out on https://issues.apache.org/jira/browse/HADOOP-7509?focusedCommentId=13126725&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13126725 I stupidly swapped CommonConfigurationKeys.HADOOP_SECURITY_AUTHENTICATION with CommonConfigurationKeys.HADOOP_SECURITY_AUTHORIZATION.
-
-- HADOOP-7744.
- Major bug reported by jeagles and fixed by jeagles (test)
- Incorrect exit code for hadoop-core-test tests when exception thrown
- Please see MAPREDUCE-3179 for a full description.
-
-- HADOOP-7743.
- Major improvement reported by tucu00 and fixed by tucu00 (build)
- Add Maven profile to create a full source tarball
- Currently we are building binary distributions only.
We should also build a full source distribution from where Hadoop can be built.
-
-- HADOOP-7740.
- Minor bug reported by arpitgupta and fixed by arpitgupta (conf)
- security audit logger is not on by default, fix the log4j properties to enable the logger
- Fixed security audit logger configuration. (Arpit Gupta via Eric Yang)
-
-- HADOOP-7737.
- Major improvement reported by tucu00 and fixed by tucu00 (build)
- normalize hadoop-mapreduce & hadoop-dist dist/tar build with common/hdfs
- Normalize the build fo hadoop-mapreduce and hadoop-dist with hadoop-common and hadoop-hdfs making the -Pdist and -Dtar maven options to be consistent.
* -Pdist should create the layout
* -Dtar should create the TAR
-
-- HADOOP-7728.
- Major bug reported by rramya and fixed by rramya (conf)
- hadoop-setup-conf.sh should be modified to enable task memory manager
- Enable task memory management to be configurable via hadoop config setup script.
-
-- HADOOP-7724.
- Major bug reported by gkesavan and fixed by arpitgupta
- hadoop-setup-conf.sh should put proxy user info into the core-site.xml
- Fixed hadoop-setup-conf.sh to put proxy user in core-site.xml. (Arpit Gupta via Eric Yang)
-
-- HADOOP-7721.
- Major bug reported by arpitgupta and fixed by jnp
- dfs.web.authentication.kerberos.principal expects the full hostname and does not replace _HOST with the hostname
+ - HADOOP-8552.
+ Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (conf , security)
+ Conflict: Same security.log.file for multiple users.
-
-- HADOOP-7720.
- Major improvement reported by arpitgupta and fixed by arpitgupta (conf)
- improve the hadoop-setup-conf.sh to read in the hbase user and setup the configs
- Added parameter for HBase user to setup config script. (Arpit Gupta via Eric Yang)
-
-- HADOOP-7709.
- Major improvement reported by jeagles and fixed by jeagles
- Running a set of methods in a Single Test Class
- Instead of running every test method in a class, limit to specific testing methods as describe in the link below.
http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html
Upgrade to the latest version of maven-surefire-plugin that has this feature.
-
-- HADOOP-7708.
- Critical bug reported by arpitgupta and fixed by eyang (conf)
- config generator does not update the properties file if on exists already
- Fixed hadoop-setup-conf.sh to handle config file consistently. (Eric Yang)
-
-- HADOOP-7707.
- Major improvement reported by arpitgupta and fixed by arpitgupta (conf)
- improve config generator to allow users to specify proxy user, turn append on or off, turn webhdfs on or off
- Added toggle for dfs.support.append, webhdfs and hadoop proxy user to setup config script. (Arpit Gupta via Eric Yang)
-
-- HADOOP-7705.
- Minor new feature reported by stevel@apache.org and fixed by stevel@apache.org (util)
- Add a log4j back end that can push out JSON data, one per line
- If we had a back end for Log4j that pushed out log events in single line JSON content, we'd have something that is fairly straightforward to machine parse. If: it may be harder to do than expected. Once working HADOOP-6244 could use it.
-
-- HADOOP-7691.
- Major bug reported by gkesavan and fixed by eyang
- hadoop deb pkg should take a diff group id
- Fixed conflict uid for install packages. (Eric Yang)
-
-- HADOOP-7684.
- Major bug reported by eyang and fixed by eyang (scripts)
- jobhistory server and secondarynamenode should have init.d script
- Added init.d script for jobhistory server and secondary namenode. (Eric Yang)
-
-- HADOOP-7681.
- Minor bug reported by arpitgupta and fixed by arpitgupta (conf)
- log4j.properties is missing properties for security audit and hdfs audit should be changed to info
- (Arpit Gupta via Eric Yang)
-
-- HADOOP-7671.
- Major bug reported by raviprak and fixed by raviprak
- Add license headers to hadoop-common/src/main/packages/templates/conf/
- hadoop-common/src/main/packages/templates/conf/ not in the exclude list for apache-rat plugin . This causes 10 release audit warnings for missing license headers (in the properties and xml files like hdfs-site.xml)
-
-- HADOOP-7668.
- Minor improvement reported by sureshms and fixed by stevel@apache.org (util)
- Add a NetUtils method that can tell if an InetAddress belongs to local host
- closing again
-
-- HADOOP-7664.
- Minor improvement reported by raviprak and fixed by raviprak (conf)
- o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.
- o.a.h.conf.Configuration complains of overriding final parameter even if the value with which its attempting to override is the same.
-
-- HADOOP-7663.
- Major bug reported by mayank_bansal and fixed by mayank_bansal (test)
- TestHDFSTrash failing on 22
- Seems to have started failing recently in many commit builds as well as the last two nightly builds of 22:
https://builds.apache.org/hudson/job/Hadoop-Hdfs-22-branch/51/testReport/org.apache.hadoop.hdfs/TestHDFSTrash/testTrashEmptier/
https://issues.apache.org/jira/browse/HDFS-1967
-
-- HADOOP-7662.
- Major bug reported by tgraves and fixed by tgraves
- logs servlet should use pathspec of /*
- The logs servlet in HttpServer should use a pathspec of /* instead of /.
logContext.addServlet(AdminAuthorizedServlet.class, "/*");
In making the changes for the yarn webapps (MAPREDUCE-2999), I registered a webapp to use "/". This blocked the /logs servlet from working. because both had a pathSpec of "/" and the guice filter seemed to take precendence. Changing the pathspec of the logs servlet to /* fixes the issue.
-
-- HADOOP-7658.
- Major bug reported by gkesavan and fixed by eyang
- to fix hadoop config template
- hadoop rpm config template by default sets the HADOOP_SECURE_DN_USER, HADOOP_SECURE_DN_LOG_DIR & HADOOP_SECURE_DN_PID_DIR
the above values should only be set for secured deployment ;
# On secure datanodes, user to run the datanode as after dropping privileges
export HADOOP_SECURE_DN_USER=${HADOOP_HDFS_USER}
# Where log files are stored. $HADOOP_HOME/logs by default.
export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
# Where log files are stored in the secure data environment.
export HADOOP_SE...
-
-- HADOOP-7655.
- Major improvement reported by arpitgupta and fixed by arpitgupta
- provide a small validation script that smoke tests the installed cluster
- Committed to trunk and v23, since code reviewed by Eric.
-
-- HADOOP-7642.
- Major improvement reported by tucu00 and fixed by tomwhite (build)
- create hadoop-dist module where TAR stitching would happen
- Instead having a post build script that stitches common&hdfs&mmr, this should be done as part of the build when running 'mvn package -Pdist -Dtar'
-
-- HADOOP-7639.
- Major bug reported by tgraves and fixed by tgraves
- yarn ui not properly filtered in HttpServer
- Currently httpserver only has .html", ".jsp as user facing urls when you add a filter. For the new web framework in yarn, the pages no longer have the *.html or *.jsp and thus they are not properly being filtered. The yarn ui just uses paths - for it would be serve:port/yarn/*
-
-- HADOOP-7637.
- Major bug reported by eyang and fixed by eyang (build)
- Fair scheduler configuration file is not bundled in RPM
- 205 build of tar is fine, but rpm failed with:
{noformat}
[rpm] Processing files: hadoop-0.20.205.0-1
[rpm] warning: File listed twice: /usr/libexec
[rpm] warning: File listed twice: /usr/libexec/hadoop-config.sh
[rpm] warning: File listed twice: /usr/libexec/jsvc.i386
[rpm] Checking for unpackaged file(s): /usr/lib/rpm/check-files /tmp/hadoop_package_build_hortonfo/BUILD
[rpm] error: Installed (but unpackaged) file(s) found:
[rpm] /etc/hadoop/fai...
-
-- HADOOP-7633.
- Major bug reported by arpitgupta and fixed by eyang (conf)
- log4j.properties should be added to the hadoop conf on deploy
- currently the log4j properties are not present in the hadoop conf dir. We should add them so that log rotation happens appropriately and also define other logs that hadoop can generate for example the audit and the auth logs as well as the mapred summary logs etc.
-
-- HADOOP-7631.
- Major bug reported by rramya and fixed by eyang (conf)
- In mapred-site.xml, stream.tmpdir is mapped to ${mapred.temp.dir} which is undeclared.
- Streaming jobs seem to fail with the following exception:
{noformat}
Exception in thread "main" java.io.IOException: No such file or directory
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.checkAndCreate(File.java:1704)
at java.io.File.createTempFile(File.java:1792)
at org.apache.hadoop.streaming.StreamJob.packageJobJar(StreamJob.java:603)
at org.apache.hadoop.streaming.StreamJob.setJobConf(StreamJob.java:798)
a...
-
-- HADOOP-7630.
- Major bug reported by arpitgupta and fixed by eyang (conf)
- hadoop-metrics2.properties should have a property *.period set to a default value foe metrics
- currently the hadoop-metrics2.properties file does not have a value set for *.period
This property is useful for metrics to determine when the property will refresh. We should set it to default of 60
-
-- HADOOP-7629.
- Major bug reported by phunt and fixed by tlipcon
- regression with MAPREDUCE-2289 - setPermission passed immutable FsPermission (rpc failure)
- MAPREDUCE-2289 introduced the following change:
{noformat}
+ fs.setPermission(stagingArea, JOB_DIR_PERMISSION);
{noformat}
JOB_DIR_PERMISSION is an immutable FsPermission which cannot be used in RPC calls, it results in the following exception:
{noformat}
2011-09-08 16:31:45,187 WARN org.apache.hadoop.ipc.Server: Unable to read call parameters for client 127.0.0.1
java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.fs.permission.FsPermission$2.<init>()
...
-
-- HADOOP-7627.
- Minor improvement reported by tlipcon and fixed by tlipcon (metrics, test)
- Improve MetricsAsserts to give more understandable output on failure
- In developing a test case that uses MetricsAsserts, I had two issues:
1) the error output in the case that an assertion failed does not currently give any information as to the _actual_ value of the metric
2) there is no way to retrieve the metric variable (eg to assert that the sum of a metric over all DNs is equal to some value)
This JIRA is to improve this test class to fix the above issues.
-
-- HADOOP-7626.
- Major bug reported by eyang and fixed by eyang (scripts)
- Allow overwrite of HADOOP_CLASSPATH and HADOOP_OPTS
- Quote email from Ashutosh Chauhan:
bq. There is a bug in hadoop-env.sh which prevents hcatalog server to start in secure settings. Instead of adding classpath, it overrides them. I was not able to verify where the bug belongs to, in HMS or in hadoop scripts. Looks like hadoop-env.sh is generated from hadoop-env.sh.template in installation process by HMS. Hand crafted patch follows:
bq. - export HADOOP_CLASSPATH=$f
bq. +export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:$f
bq. -export HADOOP_OPTS=...
-
-- HADOOP-7612.
- Major improvement reported by tomwhite and fixed by tomwhite (build)
- Change test-patch to run tests for all nested modules
- HADOOP-7561 changed the behaviour of test-patch to run tests for changed modules, however this was assuming a flat structure. Given the nested maven hierarchy we should always run all the common tests for any common change, all the HDFS tests for any HDFS change, and all the MapReduce tests for any MapReduce change.
In addition, we should do a top-level build to test compilation after any change.
-
-- HADOOP-7610.
- Major bug reported by eyang and fixed by eyang (scripts)
- /etc/profile.d does not exist on Debian
- As part of post installation script, there is a symlink created in /etc/profile.d/hadoop-env.sh to source /etc/hadoop/hadoop-env.sh. Therefore, users do not need to configure HADOOP_* environment. Unfortunately, /etc/profile.d only exists in Ubuntu. [Section 9.9 of the Debian Policy|http://www.debian.org/doc/debian-policy/ch-opersys.html#s9.9] states:
{quote}
A program must not depend on environment variables to get reasonable defaults. (That's because these environment variables would ha...
-
-- HADOOP-7608.
- Major bug reported by tucu00 and fixed by tucu00 (io)
- SnappyCodec check for Hadoop native lib is wrong
- Currently SnappyCodec is doing:
{code}
public static boolean isNativeSnappyLoaded(Configuration conf) {
return LoadSnappy.isLoaded() && conf.getBoolean(
CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_KEY,
CommonConfigurationKeys.IO_NATIVE_LIB_AVAILABLE_DEFAULT);
}
{code}
But the conf check is wrong as it defaults to true. Instead it should use *NativeCodeLoader.isNativeCodeLoaded()*
-
-- HADOOP-7606.
- Major bug reported by atm and fixed by tucu00 (test)
- Upgrade Jackson to version 1.7.1 to match the version required by Jersey
- As of 2 days ago, 13 tests started failing, all with errors in Avro-related tests.
-
-- HADOOP-7604.
- Critical bug reported by mahadev and fixed by mahadev
- Hadoop Auth examples pom in 0.23 point to 0.24 versions.
- hadoop-auth-examples/pom.xml has references to 0.24 in the 0.23 branch.
-
-- HADOOP-7603.
- Major bug reported by eyang and fixed by eyang
- Set default hdfs, mapred uid, and hadoop group gid for RPM packages
- Set hdfs uid, mapred uid, and hadoop gid to fixed numbers (201, 202, and 123, respectively).
-
-- HADOOP-7599.
- Major bug reported by eyang and fixed by eyang (scripts)
- Improve hadoop setup conf script to setup secure Hadoop cluster
- Setting up a secure Hadoop cluster requires a lot of manual setup. The motivation of this jira is to provide setup scripts to automate setup secure Hadoop cluster.
-
-- HADOOP-7598.
- Major bug reported by revans2 and fixed by revans2 (build)
- smart-apply-patch.sh does not handle patching from a sub directory correctly.
- smart-apply-patch.sh does not apply valid patches from trunk, or from git like it was designed to do in some situations.
-
-- HADOOP-7595.
- Major improvement reported by tucu00 and fixed by tucu00 (build)
- Upgrade dependency to Avro 1.5.3
- Avro 1.5.3 depends on Snappy-Java 1.5.3 which enables the use of its SO file from the java.library.path
-
-- HADOOP-7594.
- Major new feature reported by szetszwo and fixed by szetszwo
- Support HTTP REST in HttpServer
- Provide an API in HttpServer for supporting HTTP REST.
This is a part of HDFS-2284.
-
-- HADOOP-7593.
- Major bug reported by szetszwo and fixed by umamaheswararao (test)
- AssertionError in TestHttpServer.testMaxThreads()
- TestHttpServer passed but there were AssertionError in the output.
{noformat}
11/08/30 03:35:56 INFO http.TestHttpServer: HTTP server started: http://localhost:52974/
Exception in thread "pool-1-thread-61" java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.hadoop.http.TestHttpServer$1.run(TestHttpServer.java:164)
at java.util.concurrent.ThreadPoolExecutor$Worker.ru...
-
-- HADOOP-7589.
- Major bug reported by revans2 and fixed by revans2 (build)
- Prefer mvn test -DskipTests over mvn compile in test-patch.sh
- I got a failure running test-patch with a clean .m2 directory.
To quote Alejandro:
{quote}
The reason for this failure is because of how Maven reactor/dependency
resolution works (IMO a bug).
Maven reactor/dependency resolution is smart enough to create the classpath
using the classes from all modules being built.
However, this smartness falls short just a bit. The dependencies are
resolved using the deepest maven phase used by current mvn invocation. If
you are doing 'mvn compile' you don...
-
-- HADOOP-7580.
- Major bug reported by sseth and fixed by sseth
- Add a version of getLocalPathForWrite to LocalDirAllocator which doesn't create dirs
- Required in MR where directories are created by ContainerExecutor (mrv2) / TaskController (0.20) as a specific user.
-
-- HADOOP-7579.
- Major task reported by tucu00 and fixed by tucu00 (security)
- Rename package names from alfredo to auth
+
+