Mouse is more low level then keyboard in Windows 10
I have been using VM in my daily work for as long as I can remember. Before Multiple Desktops was first class citizen in Windows 10, I used to use VirtualWin. The problem with combining VirualWin and VM, was that I had to go out of VM in order to switch desktop.
I don't know if you want to call it luck or chance, as I tried to switch desktop nothing happened. I took my mouse out of VM, the normal way to put Windows host in focus. For some reason my keyboard was still stuck in VM. I had to force de-select the VM. In order to be able to switch desktop.
Solution: Stop using VirtualWin.