The Jesus Itinerary

[John V]

Aside from the fact that it's the logical conclusion barring any evidence in the story to the contrary, there's also the description of his forced relocation to Nazareth to avoid Herod's son (2:22). The narrative description in 2:23 of how he came to dwell in Nazareth is also consistent with the idea that he was coming to live there for the first time, not returning to his old home town. That's how I read it and I have three good reasons to do so.

There are simple solutions to these. That you disparage them as fan-fic is your choice, but many reasonable people understand that the gospels do not claim to be exhaustive accounts.