The first record of Sveti Stefan is as the capital of the independent Paštrovići community from the 12th century, which became a Venetian protectorate in 1423. The Paštrovići accepted Venetian rule out of fear of Turkish attacks. They accepted Venetian rule on the condition that they would not pay any tribute to Venice and would stop pirating Venetian ships.

According to a legend passed down by Stjepan Mitrov Ljubiša, the town's defensive walls were allegedly funded after the Paštrovići looted Turkish ships anchored in front of Jazz during the Siege of Kotor in 1539. However, the town was soon destroyed during the Fourth Ottoman-Venetian War. Following the continuous pleas of the Paštrovići, the Republic of Venice agreed to rebuild the town in the 16th century. In the 15th century, the fortified village was built for defense against the Turks and became a sanctuary for Adriatic pirates.