Keep in mind that actors are working even in between takes by staying in character and preparing for future scenes. So even though they may look like they are doing nothing and are open for a conversation, they need to have their space on set.

It's considered normal if you feel like you are being moved around like cattle. You are there to complete the background of the scene, not to be discovered. Occasionally they pull an extra to say a line, but don't count on it, it is very rare.