fjy
52cdb20f10
add better messaging and error handling
2013-12-13 15:01:07 -08:00
Gian Merlino
e63c69dd57
TaskStorage: Return recently complete tasks in reverse chronological order
2013-12-13 12:27:45 -08:00
Gian Merlino
6c993d87bf
Indexing service API and GUI improvements!
...
- New APIs: waitingTasks, completeTasks, task payload
- GUI for the above, and for task logs + status
2013-12-13 11:38:18 -08:00
Gian Merlino
f36a5b677c
TaskLifecycleTest: Add test for noop task
2013-12-13 07:48:28 -08:00
Gian Merlino
3b053a66ff
TaskLifecycleTest: Add test for never-ready task
2013-12-13 07:48:27 -08:00
Gian Merlino
863012c384
TaskQueue: Exception during isReady does not warrant an alert.
2013-12-13 07:48:27 -08:00
Gian Merlino
6227963af9
TaskQueue: Copy task list before management loop.
2013-12-13 07:48:27 -08:00
Gian Merlino
70c153592f
CliPeon: Fix local mode
2013-12-12 14:22:57 -08:00
Gian Merlino
370e2f855a
TaskSerdeTest: Fix IndexTask test by including an actual firehoseFactory
2013-12-12 13:58:44 -08:00
Gian Merlino
169f149cf9
TaskLifecycleTest: Fix broken setUp and broken assumptions.
2013-12-12 13:51:13 -08:00
Gian Merlino
ba757b1e5a
IndexTask: Actually make and publish segments for the correct intervals.
2013-12-12 13:50:53 -08:00
Gian Merlino
be25d51a2c
RemoteTaskRunner: Fix issues leading to failing tests
2013-12-12 13:49:49 -08:00
Gian Merlino
c60158a21a
RemoteTaskRunner: Remove task from pendingTaskPayloads on shutdown if needed
2013-12-12 10:59:16 -08:00
Gian Merlino
0129ea99cf
RemoteTaskRunner changes to make bootstrapping actually work.
...
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
completeTasks as appropriate.
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
2013-12-12 10:44:46 -08:00
Gian Merlino
d92b88718c
OverlordResource: Fix comment
2013-12-12 08:46:24 -08:00
Gian Merlino
b6a52610bc
IndexTask: Call plumber.startJob()
2013-12-12 08:46:10 -08:00
Gian Merlino
db9b515e71
IndexTask: Remove unnecessary args to determinePartitions.
2013-12-12 08:46:00 -08:00
Gian Merlino
f4a09d4ee3
TaskAction: Add JsonSubType for LockTryAcquireAction
2013-12-12 08:45:23 -08:00
Gian Merlino
b17dc6f744
Task interval, isReady hygiene
2013-12-11 22:42:20 -08:00
Gian Merlino
05e24bd85c
RemoteTaskRunner: Fix typo
2013-12-11 22:38:04 -08:00
Gian Merlino
bed263efa5
VersionConverterTask: Less goofy import for Preconditions
2013-12-11 22:37:55 -08:00
Gian Merlino
53d90efe30
TaskQueueConfig: Copyright header
2013-12-11 22:37:40 -08:00
Gian Merlino
0adda97776
AbstractFixedIntervalTask: Copyright header
2013-12-11 22:37:28 -08:00
Gian Merlino
c4b8c8bc6f
Rework indexing service internals to hopefully be more reliable.
...
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.
Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).
Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
2013-12-11 15:05:16 -08:00
fjy
96f679f31c
clean up for merge
2013-12-10 17:51:13 -08:00
Gian Merlino
f3cfd1d781
Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service
2013-12-10 17:39:43 -08:00
Gian Merlino
47c1c8cab2
TaskStorage: Rename getRunningTasks -> getActiveTasks
2013-12-10 17:39:42 -08:00
fjy
303f6ff334
fix worker config setup problems
2013-12-09 18:25:29 -08:00
nishantmonu51
2186bd6cb7
Minor fixes and documentation changes
2013-12-09 19:07:48 +05:30
fjy
5d7173ac98
fix autoscaling termination duration bug
2013-11-15 19:21:25 -08:00
fjy
346cf0e04c
fix out of order in urls for hadoop classpath
2013-11-14 18:18:39 -08:00
fjy
6b41681424
fix ordering of urls, ugh, need to write more tests
2013-11-14 15:06:07 -08:00
fjy
23ebca6d32
more hadoop dependency hell, getting the right urls to hadoop is hard
2013-11-14 14:29:42 -08:00
fjy
c51eed060f
actually use a class loader pull out dependencies from extension modules
2013-11-14 11:52:21 -08:00
fjy
fcf0c6ce06
change the classpath ordering for batch processing and prepare for next release
2013-11-14 10:52:51 -08:00
fjy
64b93bf448
fix broken autoscaling and prepare for next release
2013-11-08 11:29:42 -08:00
fjy
a049b42674
fix an issue with task tables not getting created automatically and prepare for next release
2013-11-07 18:01:35 -08:00
fjy
7f85a126ac
fix broken event receiving firehose
2013-11-07 16:59:01 -08:00
fjy
621133d6f2
removing dead code
2013-11-07 16:53:34 -08:00
fjy
084c90aa19
cleanup to prepare for next release
2013-11-07 15:55:51 -08:00
fjy
bad1a7e9f8
fix according to code review
2013-11-07 15:52:34 -08:00
fjy
aeb411a3a3
fix according to code review and fix broken examples
2013-11-07 15:42:48 -08:00
fjy
e2e10fae1f
clean up code
2013-11-07 15:03:14 -08:00
fjy
913ff3a082
clean up code
2013-11-07 15:00:11 -08:00
fjy
6b573c76f1
more fixes
2013-11-07 14:40:45 -08:00
Gian Merlino
8660db93fc
RemoteTaskRunner: Run taskComplete after a task times out
2013-11-04 10:56:26 -08:00
Gian Merlino
186bbd1cb6
WorkerTaskMonitor: Add log message
2013-11-04 10:25:16 -08:00
Gian Merlino
781673a8f8
ForkingTaskRunner: Fix pass-down of nodeType
2013-10-23 14:13:05 -07:00
Gian Merlino
b68b3526e8
IndexingServiceFirehoseModule: Add header
2013-10-18 13:27:40 -07:00
Gian Merlino
2e9c46867f
Fixes for indexing service.
...
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
2013-10-18 11:14:33 -07:00
