With a hotfix that is now live on all realms, we have addressed this issue. Now, once players are in combat with the boss, the queue should no longer look for additional players.