This seems very weird to me, as a programmer. I'd assume that pointers/indicies are used to cache bottom most slots for removing, and first non full slot for filling. Frequently (not always) containers contain the same thing in each slot so that should be sufficient for most cases. Likely the pointers will point to the same slot in these cases and only update when a slot empties or fills, generally just to the previous/next slot.In: every slot checked with the first to see if there's room, stopping at the limit if there is one.
Out: Every slot (limit doesn't matter, you can manually put stuff in the red/limited slots) checked to see if there's something that can be taken out.
Chests which have different things in it might suffer from sort of cache miss, though some kind of hashtable could help theoretically. Also, if you disturb a container by manually removing something, then the in pointers may become less optimal and some searching will be required.
I can understand with filter inserters there might be an issue, but otherwise it seems to me chests shouldn't suffer from the above.