Updating README and adding references
@@ -24,7 +24,7 @@ When you execute ```./create_toy_data.sh``` script, it will create the following
* Target: reversed source sequence
-The task is to learn to reverse sequences. You can test single or multi-gpu training.
+The task is to learn to reverse randomly generated sequences. You can test single or multi-gpu training.
**Single GPU:**
```
@@ -35,18 +35,22 @@ $ python run.py --config=example_configs/toy_data_config.json --mode=train --log
**Multi-GPU:**
-We follow *data parallel* approach for training. Each GPU receives a full copy of the model and its own mini-batch.
+We follow *data parallel* approach for training. Each GPU receives a full copy of the model and its own mini-batch of data.
To start multi-GPU training, set ```"num_gpus"``` parameter in the model config accordingly.
-The ```"batch_size"``` parameter specifies batch size *per GPU*. Hence, global (or algorithmic) batch size will be equal ```"num_gpus"*"batch_size"```
+The ```"batch_size"``` parameter specifies batch size *per GPU*. Hence, global (or algorithmic) batch size will be equal to ```"num_gpus" * "batch_size"```
+
+*example_configs/toy_data_config_2GPUs.json* shows how to use the multi-gpu case
```
$ python run.py --config=example_configs/toy_data_config_2GPUs.json --mode=train --logdir=ModelAndLogFolder
```
+This will begin the training process and save the checkpoints to the ModelAndLogFolder directory
+
## Example inference
-Once you have trained the model and saved it in ``ModelAndLogFolder``, run:
+Once the model has been trained and saved in ``ModelAndLogFolder``, run:
```
$ python run.py --config=example_configs/toy_data_config_2GPUs.json --mode=infer --logdir=ModelAndLogFolder --inference_out=pred.txt
```
@@ -55,7 +59,7 @@ $ python run.py --config=example_configs/toy_data_config_2GPUs.json --mode=infer
```
$ ./multi-bleu.perl test/toy_data/test/target.txt < pred.txt
```
-If you just used provided configs, your BLUE score should be > 98 for both single and 2 gpu runs.
+If you just used the provided configs, your BLUE score should be > 98 for both single and 2 gpu runs of the toy task.
# Training German to English translation
@@ -76,6 +80,8 @@ python run.py --config_file=example_configs/nmt.json --logdir=nmt --checkpoint_f
python run.py --config_file=example_configs/nmt.json --logdir=nmt --mode=infer --inference_out=wmt_pred.txt
```
+Before we calculate the BLEU score, we must remove the BPE segmentations from the translated outputs.
+
Clean ```wmt_pred.txt``` and ```newstest2015.tok.bpe.32000.en``` from BPE segmentation
## Cleaning BPE segmentation
@@ -105,3 +111,7 @@ Contributions are welcome!
* [Tf-seq2seq](https://github.com/google/seq2seq)
* [Moses](http://www.statmt.org/moses/)
+## References
+* [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)
+* [Massive Exploration of Neural Machine Translation Architectures](https://arxiv.org/abs/1703.03906)
+* [Effective approaches to attention-based neural machine translation](https://arxiv.org/abs/1508.04025)
