The earlies records of this building date back to 1174, when it was erected as a German feudal castle. The future Emperor Frederick II of Swabia spent several years of his childhood here in the care of Conrad of Urslingen, and was baptised in Assisi at the age of three in 1197. The following year, in Conrad's absence, the people of Assisi rebelled and destroyed the castle. The castle remained a ruin until 1367, when Cardinal Albornoz rebuilt the fortress using the western section of outside walls and part of the interior fortifications. In 1458 Jacopo Piccinino, then lord of Assisi, erected the twelve-sided tower and the long curtain wall connecting the castle to the city. In 1478 Pope Sixtus IV restored the castle's keep, while between 1535 and 1538 Pope Paul III built the round tower near the main gate.