It’s not actually a power switch, but rather a button labeled “Shut down” that primarily used to ensure properly complete any recordings that may still be in progress.
Ultimately, you should unplug the power cord, and the device will restart when you plug it back in.