Issue with distributed batches over multiple GPUs

[ClaasBeger]

Hello,

I am currently implementing a large model employing SpikingJelly MultiStepLIFNodes with cupy backend, and am trying to distribute the training process over multiple GPUs using torch.nn.DataParallel. Unfortunately, the training seems to be stuck, and does not produce any results whatsoever (no exception is thrown either). The program runs normally without the LIFs. I have found there was a similar issue in the past related to cupy/cupy#6569 where the cupy device was always set to cuda:0, which has been marked as fixed in spikingjelly version 10. Are there still any known issue when using the cupy backend with DataParallel, or is it expected to work fine?
Thank you in advance!

[fangwei123456]

Are there still any known issue when using the cupy backend with DataParallel, or is it expected to work fine?

I have not found any other issue to cause DDP training problems. You can try to use the latest version of SpikingJelly and check if it works well.

[ClaasBeger]

Does it only work with Distributed Data Parallel, or is DataParallel (https://pytorch.org/docs/stable/generated/torch.nn.DataParallel.html) also supported ?

[fangwei123456]

I have not tested DataParallel. It works well in Distributed Data Parallel.

[ClaasBeger]

Alright, then I might try to transition to distributed data parallel in the future. Thanks for the help👍