Hi friends, You can destroy a castle using Your Longbowmens. For this first go to 
Impiral age research yeomen and the attack a castle with lots of Longbowmens.Since 
they have range 12 and castle can have maximum range 11 so it won't be able to attack. 
But don't forget to guard your longbowmen using Knights.