Ashlar Stone

This would be the ideal construction method for a "real"castle, pure stone. There aren't many of us that have the knowledge, time, or money to purchase this stone and build with it. While it wouldn't be hard to use, you'd have a lot of mistakes and it would probably look pretty rough if you didn't have the help of a stonemason to keep you pointed in the right direction. There are several options available to those of us that are less able to make use of this method. There are ashlar stone veneers that look very realistic and cam be applied easily to concrete block or ICF structure. The building with blocks or ICF will eliminate most code difficulties because they are known, safe construction methods (to the inspectors, anyway), and you'll have less difficulty with inspections should you choose to build in an area where they are necessary. Building the structure rapidly to get some living space and temporarily ignoring the exterior finish allows you to move in and keep initial costs down, you can add the veneer over time. I looked around for sites offering veneers and was surprised to see such a variety of finishes available. Everything from pebble to block type finishes can be found, and the best part is, modern veneers don't have to have that cheap, monochromatic, and repetitive pattern that really make it look fake.

The point is affordability and DIY-ability. Take a look at veneers from Boulder Creek Stone. They look great.