The village Panenské Břežany was first referenced as the possession of the Benedictine monastery of St. George at Prague Castle in 1233. The Benedictine nuns built the castle as a summer residence at the turn of the 17th and 18th century. The Chapel of St. Anne was built near the castle between 1705-1707. The chapel is the early work of baroque architect Jan Santini Aichel.

When the emperor Josef II dissolved the Benedictine monastery of St. George, the domain was sold. Mathias Friedrich von Riese-Stallburg bought it in 1828. Mathias began to construct a new castle at Panenské Břežany – the Lower Castle. Shortly after the death of Adolf, Matthias´son, the manor was taken over by the Economic and Credit Bank of Bohemia.

Ferdinad Bloch-Bauer bought the estate including the Upper and Lower Castles in Panenské Břežany in 1909. He was born into a Jewish family, that had been running a business in the sugarindustry. Ferdinand rose to became a “Sugar Magnate” during his career. He married Adele Bauer, the youngest daughter of the Viennese banker Moritz Bauer, in 1899. His wife Adele created a famous salon in Vienna, where the high society would meet. Adele Bloch-Bauer was a patron and close friend of Gustav Klimt. Klimt´s portrait of Adele, known as “The Lady in Gold”, is one of the most famous and beautiful paintings of the Modern Period.

Ferdinand and Adele rebuilt the Lower Castle in Panenské Břežany into a luxurious summer residence. They entertained their family and many distinguished friends. Ferdinand loved to spend his spare time deer-hunting.

The Upper Castle at in Panenské Břežany  fell into the hands of Friedrich Gerstel, the furniture producer, after 1921. The Gerstel family was a prominent Jewish family from Prague.  Friedrich Gerstel reconstructed the Upper Castle and used it as a showroom for his luxury products. He also built a pseudo-Baroque garden and a unique irrigation system there.

A significant change in the Bloch-Bauer and Gerstel families lives occurred in the year 1938. Ferdinand Bloch-Bauer moved to the Lower Castle at in Panenské Břežany for some time after the “Anschluss” of Austria.  He spent the rest of his life in Zurich, where he died on November 13, 1945. Friedrich Gerstel remained in Czechoslovakia until March 1939, then he immigrated to the United States.

The property of Ferdinand Bloch-Bauer and Friedrich Gerstel was Aryanized shortly after the establishment of the Protectorate of Bohemia and Moravia. The Upper and Lower Castles at in Panenské Břežany were secured by the German secret police.

The Upper Castle was turned into the residence of the State Secretary of the Protector and then the State Minister for Bohemia and Moravia, Karl Hermann Frank, as of 1942. He and his family resided there until the liberation of Czechoslovakia. The Lower Castle became the residence of the Deputy “Reichsprotector” Reinhard Heydrich with his wife and children.

Lina Heydrich lived in the castle even after the assassination of her husband, up until the end of World War II. Lina Heydrich administered the estate of in Panenské Břežany, therefore one of the smallest branch concentration camps was established during World War II at in Panenské Břežany. It was run by the Flossenbürg concentration camp. The prisoners worked for Lina Heydrich on reconstructing the Lower Castle and garden.

The Upper Castle at in Panenské Břežany became state property after the liberation of Czechoslovakia. It was used as a home for the elderly as of 1957.