I have heard in the past that there is a slight delay with the Contact - Groups field not being updated immediately. Although I think it could be down to the size of your database as well. But I would say within 5 minutes it should be updated.
I have recently been doing a sync development work as well. My approach of populating the assigned Tags is to extract them from the Contact table Groups field. It would take too long pulling them from the ContactGroupAssign table. Also, that table does not have any Last Updated column making it difficult to track what tags have been updated, plus nor can see what Tags have been removed. The Contact table is the best way to track updated data.
The only caveat from the Contact - Groups field is that if you need to track the Date the Tag was assigned, then you will have to read the ContactGroupAssign table.