Setting Global sync to 50 in BMUF
Summary:
In all our final settings, we are using global_sync = 50 and we get comparable results with DDP and caffe2.
Setting the default global-sync-iter = 50
and users can just define --use-bmuf to enable it for training.
@@ -47,7 +47,7 @@ class FairseqBMUF(FairseqOptimizer):
)
parser.add_argument(
"--global-sync-iter",
- default=10,
+ default=50,
type=int,
help="Iteration for syncing global model",
)
