Add oov_token argument to Tokenizer doc arguments list
@@ -81,7 +81,8 @@ keras.preprocessing.text.Tokenizer(num_words=None,
filters='!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\t\n',
lower=True,
split=" ",
- char_level=False)
+ char_level=False,
+ oov_token=None)
```
Class for vectorizing texts, or/and turning texts into sequences (=list of word indexes, where the word of rank i in the dataset (starting at 1) has index i).
@@ -89,6 +90,7 @@ Class for vectorizing texts, or/and turning texts into sequences (=list of word
- __Arguments__: Same as `text_to_word_sequence` above.
- __num_words__: None or int. Maximum number of words to work with (if set, tokenization will be restricted to the top num_words most common words in the dataset).
- __char_level__: if True, every character will be treated as a token.
+ - __oov_token__: None or str. If given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls.
- __Methods__:
