Can confirm what others have said - Cary Street pool is much nicer and bigger than the Franklin Street pool. I was a morning swimmer at VCU (also had no affiliation, just lived nearby) so can't speak to evening crowds but you'd probably be ok on the later side. The one benefit of the Y is that you can go to nicer pools at other branches (north Richmond Y's pool is great and just refurbished) but Cary Street is the best in the downtown area. With community membership at VCU you can also access the MCV pool which is much less busy but has crappy hours.