PGI User Forum

[alfvenwave]

Hi.

I am having a problem with GPUs being left in a hung state. The code I am running deploys multiple GPUs within an openMP parallel loop. The code works fine for a single thread, however when running multiple cards I sometimes get a crash due to memory problems. This would be fine, except that often one or more cards get left in a hung state. Rebooting the machine fixes this, however the machine is part of a cluster so this is not a very satisfactory solution. Is there some way of restarting graphics cards from the command line if they end up in a jammed state?

Rob.

[dholt]

Hi Rob,

If you're in Linux, you can try running (as root):

[alfvenwave]

Great - thanks for this. I have managed to persuade our system administrator to give me root access for now. Bit annoying that they hang so often - is this common?

Rob.

[mkcolg]

While I don't know if it's common for others, my device hangs periodically as well. Like you, it occurs more often when I'm running code that hits a device limit or encounters some other error.

- Mat

[alfvenwave]

Hi Mat. Bit annoying, but now I have root access I can reboot as is necessary. My next thread shows why I keep hitting problems....

Thanks,

Rob.