Legends speak of a severe penance Bhadra, the son of Meru, did in this sacred place to have darshan of Lord Vishnu. The Lord appeared before him and promised to stay at this place during Rama Avataar. Hence the place is known as Bhadragiri or Bhadrachalam. When Sri Rama left Ayodhya on a 14-year vanvaas (exile), he along with Sita and Lakshmana crossed the Ganges and reached this area, then known as Dhandakaaranya. They preferred to stay in a place called Parnashala (32 km away) for its serene atmosphere. References are found to these events in the epic Ramayana. The Lord is believed to have informed a woman by name Dhammakka in her dream that certain idols are lying at a particular spot on the hill. The devotee promptly picked up the idols (of Sri Rama, Sita and Lakshmana) and installed them in a place which is the site of the present shrine.
In later years, Ramadas, an ardent Rama devotee who was Gopanna by birth, renovated the temple. There is an interesting incident associated with this development. Gopanna was serving the then Sultan Tanashah as his Tahasildar. Being a Rama bhakta, he utilised revenue collections, to the tune of Rs 6 lakh, to renovate the temple. The Sultan came to know of this and got Gopanna arrested. He was languishing in jail for many years, spending his time singing the glory of Sri Rama. The Lord decided to show mercy on His staunch devotee. Rama and Lakshmana one day appeared in the guise of merchants in the Sultans court and returned to the treasury all the money in gold coins. The Sultan, realising the greatness of Gopanna, released him from jail. He also offered him and the temple valuable gifts. Even today, tourists and pilgrims visiting the famous Golconda Fort can see the cell where Gopanna was kept under arrest.
The temple is also associated with Saint Kabirdas. Kabir was the foster son of a Muslim family and a staunch devotee of Sri Rama. Once he came to the shrine to worship the Lord. But the priest prevented him from entering the shrine. To the shock of the priest, the images in the sanctum disappeared. Ramadas happened to be in the temple at that time. He intervened and convinced the priest to let Kabir into the shrine. As Kabir came into the temple, the images miraculously reappeared in the sanctum and Kabir had darshan of Lord Rama. This devotee later came to be known as Kabirdas. As time passed, Ramadas became a follower of Kabirdas.