Commit Graph

2992 Commits

Author SHA1 Message Date
javanna c185c1339a remove grok and join usage from IngestClientIT 2016-01-07 11:30:02 +01:00
javanna e036b5896d move registerFilter up 2016-01-07 11:29:43 +01:00
javanna c711570fd3 remove needless changes (missing static imports) 2016-01-06 19:22:28 +01:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna da3f460bd1 remove ProcessorFactoryProvider 2016-01-06 17:31:40 +01:00
javanna e923545269 add test for ProcessorsRegistry 2016-01-06 17:31:40 +01:00
javanna 931d321d58 prevent registering processor factories with same name as an existing one 2016-01-06 17:31:40 +01:00
javanna f651f5a531 remove MapBinder guice binding for processors, use ProcessorsRegistry instead 2016-01-06 17:31:40 +01:00
javanna 8251a50667 make ProcessorFactoryProvider extend BiFunction 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna fa4dbdaea1 create ProcessorsModule during Node creation rather than as part of IngestPlugin initialization
If we createProcessorsModule as part of the plugin, other plugins will not be able to register their own processors.
2016-01-06 17:31:39 +01:00
javanna 456351de2e move common test classes to test module 2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Yannick Welsch de6dfe15a7 Add PathHierarchy type back to path_hierarchy tokenizer for backward compatibility with 1.x
Closes #15785
2016-01-06 14:37:33 +01:00
Yannick Welsch a6ec1434d6 [TEST] Reduce log level in NodeVersionAllocationDeciderTests 2016-01-06 14:35:47 +01:00
Simon Willnauer 8a90c8085d Merge pull request #15788 from s1monw/dont_delete_tlog_file
Never delete translog-N.tlog file when creation fails
2016-01-06 14:31:22 +01:00
Simon Willnauer 5c833750d7 apply feedback from @bleskes 2016-01-06 14:19:58 +01:00
Simon Willnauer 12b93e72f0 Never delete translog-N.tlog file when creation fails
We today delete the translog-N.tlog file if any subsequent operation fails
but we might actually be in a good state if for instance the creation of the writer
failes after we sucessfully baked the new translog generation into the checkpoint. In this situation
we used to delete the translog-N.tlog file and failed on the next recovery of the translog with a
NoSuchFileException | FileNotFoundException just like in https://discuss.elastic.co/t/cannot-recover-index-because-of-missing-tanslog-files/38336

This commit changes the behavior and cleans up that limbo state on recovery if we already have a generation+1 file written but not baked into
the checkpoint we remove that file but only if the previous ckp file has already been renamed otherwise we know we can't be in this state.
2016-01-06 13:10:21 +01:00
Simon Willnauer 56329d0f53 Never call a listerner under lock in InternalEngine
We has a postIndex|DeleteUnderLock listener callback to load percolator
queries which is entirely private to the index shard in the meanwhile. Yet,
it still calls an external callback while holding an indexing lock which is scary
since we have no control over how long the operation could possibly take.

This commit decouples the percolator registry entirely from the ShardIndexingService
by pessimistically fetching percolator documents from the the engine using realtime get.
Even in situations where the same document is changed concurrently we will eventually end up
in the correct state without loosing an update. This also moves the index throtteling stats directly into
the engine to entirely remove the need for the dependency between InternalEngine and ShardIndexingService.
2016-01-06 11:38:34 +01:00
Yannick Welsch 55cc88e1ae Fix version-based allocation decider to prevent peer recovery from node with older version
Relocating a non-primary shard from one node to another is actually done by recovering from the active
primary shard in the cluster, and not the node that we are logically relocating from.

Closes #15775
2016-01-06 10:07:39 +01:00
Adrien Grand 7e3ccf2ee3 Merge pull request #15746 from jpountz/fix/missing_terms_agg
Make `missing` on terms aggs work with all execution modes.
2016-01-06 09:32:39 +01:00
Jason Tedor d032dabed5 Merge pull request #15777 from jasontedor/safer-cluster-state-task-notifications
Safe cluster state task notifications
2016-01-05 16:56:24 -05:00
Jason Tedor 05c46c9d35 Safe cluster state task notifications
This commit addresses an issue where a cluster state task listener
throwing an exception could prevent other listeners from being notified,
and could prevent the executor from receiving notifications that a new
cluster state was published. Additionally, this commit also addresses a
similar issue for executors handling cluster state publication
notifications.
2016-01-05 16:44:59 -05:00
Igor Motov 9ca4386fba Lower logging level for registering/unregistering tasks with the task manager
At the debug level the task manager is too talkative.
2016-01-05 15:15:02 -05:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Simon Willnauer dff30ece05 Double check if stream must be flush to allow tests to make better assumptions of what is visible and what isn't after tragic events 2016-01-05 17:58:20 +01:00
Simon Willnauer ea6718d878 Add pending review from @bleskes on #15771 2016-01-05 16:34:26 +01:00
Simon Willnauer 5272c98136 format finally block 2016-01-05 15:50:10 +01:00
Simon Willnauer 469a179ab1 remove stale comments 2016-01-05 15:48:35 +01:00
Simon Willnauer 2370ace922 assert that tragic even exception is never null 2016-01-05 15:45:41 +01:00
Simon Willnauer f69502dd04 Simplify TranslogWriter to always write to a stream
We used to write into an in-memory buffer and if necessary also allow reading
from the memory buffer if the some translog locations that are not flushed to
the channel need to be read. This commit hides all writing behind a buffered output
stream and if ncecessary flushes all buffered data to the channel for reading. This allows
for several simplifcations like reusing javas build in BufferedOutputStream and removes the
need for read write locks on the translog writer. All thread safety is now achived using
the synchronized primitive.
2016-01-05 15:22:56 +01:00
Simon Willnauer 107859f347 Merge pull request #15766 from s1monw/add_more_io_exceptions
Add serialization support for more important IOExceptions
2016-01-05 12:00:35 +01:00
Simon Willnauer 3a41dfe1dd Add serialization support for more important IOExceptions
Several IOExceptions are always wrapped in an NotSerializableWrapper which is
annoying to read. These exceptions are important to get right across the network
and we should support the important ones that indicate problems on the Filesystem.

This commit also adds general support for IOException to preserve the parent type
across the network if no specific type is serializable.
2016-01-05 10:30:26 +01:00
Adrien Grand 1fcf9958b2 Fix GeoPointFieldMapperTests expectations. 2016-01-05 10:26:39 +01:00
Simon Willnauer bf7e2c333a Merge pull request #14632 from s1monw/indexing_stats_javadocs_
Add javadocs to IndexingStats.Stats
2016-01-05 09:37:42 +01:00
Simon Willnauer d4de8dbcfe Merge pull request #15762 from s1monw/issues/15754
Close recovered translog readers if createWriter fails
2016-01-05 08:23:11 +01:00
Simon Willnauer f6020a6ff2 Close recovered translog readers if createWriter fails
If we fail to create a writer all recovered translog readers are not
closed today which causes all open files to leak.

Closes #15754
2016-01-04 23:40:20 +01:00
Jason Tedor 06851b7224 Merge pull request #15736 from jasontedor/shard-state-action-cluster-state-refactoring
Make cluster state external to o.e.c.a.s.ShardStateAction
2016-01-04 16:22:12 -05:00
Jason Tedor 8a793b6564 Rename Shard(.*)ClusterStateHandler to Shard$1ClusterStateTaskExecutor 2016-01-04 16:13:39 -05:00
Nicholas Knize ff1aed80cc Revert GeoPointField.stored default back to false
As a default in V2, the GeoPointField.stored option was set to true. Since this consumes disk space with no positive benefit the default stored option is being reverted back to false.
2016-01-04 11:11:02 -06:00
Jason Tedor af523c4236 Fix order of logging parameters in ShardStateAction 2016-01-04 11:29:00 -05:00
Jason Tedor 7591f2047a Modify nested classes in ShardStateAction to be static 2016-01-04 11:17:18 -05:00
Jason Tedor fdb0c909ec Inline Shard(.*)TransportHandler#handleShard\1OnMaster methods 2016-01-04 11:10:52 -05:00
Jason Tedor 754bd66b63 Always log shard ID in ShardStateAction
This commit restores logging the ShardRouting#shardId at the front of
the log messages in ShardStateAction. The reason for this is so that
shard-level log messages have the format "[component][node][shard]
message".
2016-01-04 11:10:41 -05:00
Jason Tedor 1a131cf917 Inline ShardStateAction#innerShardFailed method 2016-01-04 11:10:23 -05:00
Adrien Grand c934f859c7 Make `missing` on terms aggs work with all execution modes.
There are two bugs:
 - the 'global_ordinals_low_cardinality' mode requires a fielddata-based impl so
   that it can extract the segment to global ordinal mapping
 - the 'global_ordinals_hash' mode abusively casts to the values source to a
   fielddata-based impl while it is not needed

Closes #14882
2016-01-04 11:13:17 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Jason Tedor a70f76f763 Make cluster state external to o.e.c.a.s.ShardStateAction
This commit modifies the handling of cluster states in
o.e.c.a.s.ShardStateAction so that all necessary state is obtained
externally to the ShardStateAction#shardFailed and
ShardStateAction#shardStarted methods. This refactoring permits the
removal of the ClusterService field from ShardStateAction.
2016-01-03 19:15:49 -05:00