They have mostly been pushed back into the Old Town, where they are hiding among local residents.
Human Rights Watch says the Badush prison was used by IS militants to execute as many as 600 people and house abducted Yazidi women.
IS gunmen forced up to 670 prisoners to kneel down in four rows and executed them.
Sunni and Christian inmates who had been separated from the Shia were taken to another location.
The jihadists are under mounting pressure from twin US-backed ground offensives targeting Mosul and their other main stronghold, Raqa in Syria. The road to Tal Afar then continues to the Syrian border.
"The shelling began and we fled - wherever we found somewhere safe, we'd settle there", said Abu Hammoud, an elderly man who left his home near the pumping station. They linked up with Kurdish fighters to encircle IS fighters.
In mid-February, Iraqi forces - backed by a USA -led air coalition - began fresh operations aimed at purging Daesh terrorists from western Mosul.
Displacement due to ongoing fighting in western Mosul continues to rise rapidly as thousands of Iraqi people are fleeing the north Iraqi city, Farhan Haq, the deputy United Nations spokesman, told reporters here Wednesday.
IS militants have deployed snipers and used vehicle bombs to defend their last major Iraqi stronghold and slow down the Iraqi troops' advance.
Thousands of Iraqi civilians continued to flee western Mosul amid a major military offensive to drive Daesh militants from the northern city.
Iraqi government forces that have been engaged in five months of bloody street to street fighting to liberate Mosul are now within sight of the city's grand mosque, where Baghdadi ostentatious declared himself so-called Caliph of the Islamic State in 2014.
Military officials had said that Rapid Response troops, an elite interior ministry division, on Tuesday recaptured the provincial government headquarters, the central bank branch and the museum where militants filmed themselves destroying priceless statues in 2015.