20 lines
1.1 KiB
Markdown
20 lines
1.1 KiB
Markdown
|
Resource Estimator Service
|
||
|
|
==========================
|
||
|
|
|
||
|
|
Resource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.
|
||
|
|
|
||
|
|
## Current Status
|
||
|
|
|
||
|
|
* Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.
|
||
|
|
* In-memory store for parsed history resource skyline and estimation.
|
||
|
|
* A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.
|
||
|
|
* Provides REST interface to parse logs, query history store and estimations.
|
||
|
|
|
||
|
|
## Upcoming features
|
||
|
|
|
||
|
|
* UI to query history and edit and save estimations.
|
||
|
|
* Persisent store implementation for store (either DB based or distributed key-value like HBase).
|
||
|
|
* Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.
|
||
|
|
|
||
|
|
Refer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.
|