Update tokenization 101 and add illustration
@@ -94,9 +94,10 @@ p
include _spacy-101/_tokenization
+infobox
- | To learn more about how spaCy's tokenizer and its rules work in detail,
- | how to #[strong customise] it and how to #[strong add your own tokenizer]
- | to a processing pipeline, see the usage guide on
+ | To learn more about how spaCy's tokenization rules work in detail,
+ | how to #[strong customise and replace] the default tokenizer and how to
+ | #[strong add language-specific data], see the usage guides on
+ | #[+a("/docs/usage/adding-languages") adding languages] and
| #[+a("/docs/usage/customizing-tokenizer") customising the tokenizer].
+h(3, "annotations-pos-deps") Part-of-speech tags and dependencies
