Saji-jo Castle is a medieval castle located on a hill in the north of Kosaji area in Koka Town, overlooking the Yasu-gawa River. The castle is said to have belonged to lord Saji. When Toyotomi Hideyoshi tried to confiscate his territory in 1585, Saji resisted by retreating to his castle, however, the castle was destroyed after the attack from Hori Hidemasa and Nakamura Kazuuji.
Koka warriors of equal power built their castles of the same size and shape during the Warring States period, however, Saji-jo Castle being larger in size with different characteristics compared to other castles in the Koka region is thought to be from several Koka warriors working together for its construction.