Andrei Balici 19a336e8d3 Add max_token_length setting to the CharGroupTokenizer ()
Adds `max_token_length` option to the CharGroupTokenizer.
Updates documentation as well to reflect the changes.

Closes 
2020-05-20 14:28:40 +02:00
..