I lived near Penn Station for years, and had zero issues when walking. Not that anyone can guarantee your safety, but I never hesitate to make that walk, especially Charles and St Paul. Calvert is too quiet at night, but lovely by day.