validation accuracy during training is different than validate.py

[AmnonGeifman]

First thanks for the great repo!
I'm training mobilenetv2-140 and it seems that I got an accuracy of 76.35 during the training (see line from summary.csv below):

456 | 2.249339835984370 | 1.0110450189209000 | 76.3519998852539 | 92.94199993652340

The problem is when I use the validation.py script I'm not getting the same accuracy (I'm getting 76.0).
Do you have an idea why this gap is happening?

Thanks

[rwightman]

@AmnonGeifman If you are doing distributed training without sync-bn there can be a fairly noticeable gap between train and post-train validation. One way to reduce the gap is to train with one of the --dist-bn modes to synchronize just the batch norm stats between distributed training nodes before eval. It has less overhead than using sync-bn.

[AmnonGeifman]

Thanks very much! It probably explains the gap.