Because you're thinking too small. You are imagining a hub with couple of cargo bays that block the access to the hub.
Imagine a platform hub, 100 cargo bays to the west and 100 cargo bays to the east.
Allowing inserters to access bays would mean you just allowed to replace 800 tile-long belts with bays, increasing throughput tenfolds while decreasing latency from over 100 seconds to under a second.
Or similar setup on a planet. You've just made all belts, trains, and bots obsolete.
That would destroy the puzzle & challenge of logistics entirely.