From the point of a photographer trying to recreate the colors an object as it was seen in the lighting that was there the grey card works just like you say. I think we actually agree like you say, but just the see the problem from different perspectives. I work as a theatrical lighting designer so my go to light source in my mind is anything but a source with a broad and continous spectrum. Anyways, colours are wierd and fantastic and this conversation really made me think this through a lot so thanks a lot for that :)

The method you describes will of course work well for most practical applications, but if we are using it to test the actual colour of on object we have to make som assumptions. Firstly the light has to be a) a broad spectrum source and b) continuous in its spectrum. If an object reflect a very specific wavelength of, let's say red, you can only know this if that exact wavelength is in the light we are under. Finding a source with a broad spectrum is easy. This is what we usually call white light, it can be weighed differently across the spectrum to have different colour temperatures and different tints, but they are all sources with broad spectrums. Finding a source with a continuous spectrum is harder. Sunlight and incandescent light is continuous, but florescent tubes and CFL's are certainly not. LED's vary to say the least. Some are close to continuous, but most are far from and often appears as a few bars or blobs on a spectral chart. To probably evaluate the colour we also need to be sure that the light reflections of the grey reference material is also continuous across the spectrum and we need to know exactly how the image sensor captures different wavelengths. This sensor needs to be seeing continuous across the spectrum as well.

This would work, but only if you don't consider the lighting. The perceived colour of an object is determined by three factors: the source(s) of light, the reflection of light by the object itself and how we then "see" or measure the resulting reflection of light (The colour). The issue is that two objects that appear to have the exact same colour under one light, might look radically different under another. A quick test of this that you can do yourself is to bring some different black fabrics, that look identical in daylight, to a bar/restaurant with heating lamps. Under the light from the very infrared heavy heating lamp you will probably notice that some of the fabrics now look dark red, while others still look black. This is because of different dyes with very different reflectance in the near infrared part of the spectrum. It's a fun little experiment that makes you start thinking of colour in a little different way.