mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-10 06:55:32 +00:00
9d55c45b5a
The work to switch file upload over to treating delimited files like semi-structured text and using the ingest pipeline for CSV parsing makes the multi-line start pattern used for delimited files much more critical than it used to be. Previously it was always based on the time field, even if that was towards the end of the columns, and no multi-line pattern was created if no timestamp was detected. This change improves the multi-line start pattern by: 1. Never creating a multi-line pattern if the sample contained only single line records. This improves the import efficiency in a common case. 2. Choosing the leftmost field that has a well-defined pattern, whether that be the time field or a boolean/numeric field. This reduces the risk of a field with newlines occurring earlier, and also means the algorithm doesn't automatically fail for data without a timestamp.