The thing with past relationships is that they give the worms to people so that the worms can learn new experiences and pass them on to the next generation. If the worms are basically just living the same generation over and over again it defeats the purpose (in their eyes). I think the whole banishment thing was a bit harsh of a punishment, though.