The castle and village were founded by Wizo, a Flemish settler who was granted the land by Henry I of England after he had wrested control from the previous Welsh owners.
The castle was captured by the Welsh on several occasions but on each occasion it was retaken. It was abandoned during the thirteenth century when the owner moved to nearby Picton Castle and is one of the best examples of its type in Wales.
The castle is situated opposite St Mary Magdalene Church and there are approximately 75 steps leading up to it. It is in the care of Cadw and is a Grade I listed building.