ALLOPHONES are AURAL ILLUSIONS Allophones are different sounds that map to the same perception. This is a very strong aural illusion. Most native speakers are quite surprised to discover this. See this interesting research paper on tonal allophones in Vietnamese.

A linguistic description of the modern Hanoi dialect. A description of the Vietnamese sound system for non-native speakers from the Vietnamese grammar handbook, Cẩm Nang Ngữ Pháp Tiếng Việt do GS Trần Ngọc Dụng biên soạn. Northern Vietnamese Intonation