I checked with process explorer to check GPU load to see if something was up:

As you can see, the system GPU memory sometimes just spikes to nothing. I'm not entirely sure, but I think there's a correlation(couldn't time it exactly). And that the GPU just doesn't even try to draw for a moment, causing a stutter. Also, you can see that my GPU isn't on 100% load at any time, making the drop to 30FPS strange.
Also, when going back to menu, it's even more clear that GPU load just drops when the memory spikes happen:
