No, in such a case, you can keep your village, but the new owner can banish you from their land. If that happens, you will have to get access to a new town-slot in order to continue playing. You will not lose any progress in your village.