Notes for Box.
@@ -38,6 +38,22 @@ possible future extension.
1. There are currently some fields unique to json_pickle, but they are just
indexing meta which do not affect the semantics.
+1. Supporting multi-doc processing in the pipeline.
+ 1. Adding another layer on top of Pack, called Box
+ 1. A data box is a container of multiple packs
+ 1. The box can have links and groups to support cross-document references
+ - e.g. Link can be used to create alignment (for MT or summarization)
+ - e.g. Group can be used to represent cross-document coreference
+ 1. Box should contain named tuple of Packs
+ - We can add a "source" pack and a "target" pack, but also refer them
+ via indices
+ 1. The pipeline should be able to handle box in the same way as packs.
+ - Creates extra complexity in batching.
+ - We can start by only allowing box-level batching (i.e.
+ can not create batches of half-boxes)
+ 1. Now, which multi-doc examples should we include?
+
+
***Readability Concerns***
1. The core of the format is simple, but it gets large with all
the meta data. They might look daunting to people.
