I have ridden with a couple of the ball bearings missing and it was ok, just put the headset back in properly and put some grease on and it should be okay depending on how good the headset is.
But yes it probably will make some difference, but i didn't notice much.