alliances can not be removed.
you cant remove something when it comes from the society naturally. even a bird does a pact with a rhino.
setting strict alliance making/leaving rules instead, would help.
Its a non agression pact, not an alliance.
If alliances are removed, people will make NAPs, but friendly fire will be there.