Change the damage model slightly: have an "absorption threshold" for each object, which is the number of damage points removed from a "hit" before any hit points are removed (this is what many RPGs do).
Use it so that:
- a car doesn't take any damage in a small bump, but still does if you ram trees etc.
- your factory buildings and walls can also survive undamaged, so long as it's a small bump
- ditto for a tree; you can't just keep "pushing" them with a car to knock them down now
- tanks and walls could be completely invulnerable to small biters/spitters; not sure about this
- give rocks 1HP and a high absorption threshold so that they either get destroyed or are left undamaged