Sand is just a particular grain size for a material, but for a rock having been broken into sand sized particles reflects that it has experienced a combination of weathering and erosion. No rocks (at the surface) are immune from weathering or erosion, but some minerals that make up rocks are less stable at surface conditions and so will not persist as sediment (sand sized or otherwise) for too long, though when thinking about geologic time, a short time can still be hundreds of thousands to millions of years. Minerals that easily dissolve in water or weak acids (e.g., halite, calcite, etc) or generally tend to react easily at surface conditions and form other minerals (e.g., olivines, pyroxenes, etc) are all components of rock that you wouldn't expect to make up large components of your average sand. But there are definitely exceptions, e.g., sand formed from the weathering of basalts can be predominantly olivine and pyroxene. It's more that all told, these are not particularly stable minerals at the surface so these don't make up much of sand on a global scale.