Yeah, the added stress and time is real. It's supposed to be organized but just creates a crowd of people all standing near the gate way too early for their group. Then you have to shuffle past everyone in the wrong rows when you finally do board. I've found it's faster when airlines just do back to front without all the group numbers.
Consider how back-to-front boarding works in real life. When everyone from the back enters first, they all stop to stash bags while people in front rows are still getting settled, blocking the aisle. That creates a bottleneck that actually slows the whole process down. Group numbers aim to break up the crowd so not everyone is fighting for space at the same time. It's messy, but removing groups might make the shuffle worse because everyone crams in at once. I guess airlines keep it because some order, even if annoying, beats total chaos lol.
The hidden problem is how these groups create a feeling of unfairness. People see others boarding first for unclear reasons and it just makes everyone annoyed before they even sit down. That social friction actually slows everything down more than any order could help.