I devised my solution after reading about it here, related to the H320:
Fixed H320 Always connected prob
I think it's firmware related as I managed to reflash the firmware and
the USB Connect/Disconnect loop went away, but not this time unfortunately. It seems to be triggered by transferring recordings to my PC.
Actually, I have no real idea why it happens or why this fixes it. I RMA'd it to iRiver last year and will do so again before the warranty runs out.