Repository navigation
Grad become NaN #53
Description
Activity
If you aren't seeing NaNs with larger batch sizes, I would recommend keeping the batch size high (2048 if you want to mimic our experiment) and set
device_train_microbatch_sizeto the largest value before an OOM, in your case sounds like 12.device_train_microbatch_sizeis related to gradient accumulation where the amount of gradient accumulation is equal tobatch_size // device_train_microbatch_size. This does multiple forward passes with a smaller set of samples, then a single backward pass whenbatch_sizenumber of samples have been processed.This is mathematically equivalent to training on a large batch size at once as long as the network does not have batch norm layers. Let me know if this works!
Extra: you can set
device_train_microbatch_sizeto 'auto' and composer will decrease the microbatch size until it fits into memory. But this is an experimental feature, so it may not work out of the box for your use-caseHello, did you run on 3090 (24GB) device?
No, we have used A100s (40GB/80GB) and H100s (80GB)
When training on my local machine (3090 24Gb) with batch size 12, grad value become NaN after few steps
But I don't meet this when training on Google Cloud A100 40Gb with bs 20. Why? How I fix that?