One brick at a time...

To build a structure that fits in the range of one might call a castle on a human budget (in other words, no "rich uncle" or winning lottery ticket), there's just no way to build it all at once. Your lender would laugh you out the door, perfect credit or not. That leaves one option: build in sections. Build what you can, when you can. Start with a small structure that may be the core or a building that is part of the layout you want to have in the end. Most of the best castles I've found online were built over a decade or more and started small. There's no reason you can't build a modest structure to live in, say... just a box shape with a high roof that you can have a large open living space below and use the high roof for bedrooms or whatever. Keep it simple, just as an old building would have been; open beams, white stucco or stone interior, stone or wide rough plank flooring, not a lot of extra walls or cabinetry. Not a lot of finish work, period. A box is completely buildable for 50K not including land or permits, even using ICF. Get that done and start building equity. Now that you're on the property, you can build the rest as you can, not so much rush anymore.