False, it takes 2 people less than 5 minutes to break down a door, kill the guards, and vanish into the night. This was a common tactic which lead to it not being viable to mass up for every castle ping. By the time most guilds get a scout out to a castle and verify that its under attack by a group you're looking at 15~25 minutes to mass for castle defense which is very short notice to bring up any sizable zerg. This means that either small groups take your castle guerrilla warfare style and there aren't fights or a big zerg rolls up on your castle and you will not have adequate time to mass to fight them so there will also be no fight.This is false and misleading - people may not have defended castles previously, but if they didn't it was because they didn't care enough to defend them. There is plenty of time to react to a castle ping, from the time the first door is knocked down it's a minimum of 30 minutes before the castle lord is spawned.
If two noobs with demo hammers want to troll, that's their perogative and it's not different for territory owners who get pings at all hours except that two people can actually kill mages and take 12 hours worth of siphoned energy generation from you in less than 5 minutes. What you're asking for already exists in the fact that if a group actually wants to take a castle, they have to commit for at least 30 minutes.
Even if a group brings 10-20 with the intent to actually capture a castle - there's no reason they can't bail on the objective. Running away from a disadvantageous fight to live and fight another battle is a valid tactic.
AO Quick Reference Guide