The problem is more of a cost/economic one than a construction one - there are already plenty of freight rail lines that go through nearly all of the US’s varied topography.

Making it competitive with the established airline industry and actually getting the infrastructure built without descending into a California High-Speed Rail money pit is the problem.

Cross country travel is a “solved” problem with extensive aircraft infrastructure. A system that is slower and requires additional infrastructure is a hard sell without some very obvious and immediate benefits.