I'm sure you're right. If every human had exactly same distribution of rods and cones, of identical sensitivity, and with exactly the same frequency response curve, for humans only, then yes. But only for humans.

Imagine you have a surface which appears to be exactly the same colour to any number of these perfect human observers.

It could either consist of a magic pigment which absorbed all light frequencies apart from one specific frequency for that colour. In that case, it would excite the red, green, and blue cones at a specific level, depending on how the specific residual reflected frequency interacts with with frequency response curves of the RGB cones.

Or it could consist of a large number of red, green and blue dots of various reflectivities (or sizes: I don't think it matters), each of which absorbed all light frequencies apart from the specific frequencies of red, green or blue which trigger the R G or B cones, most likely at the centre and most sensitive part of the their frequency response graph.

These two surfaces are illuminated by white light (e.g. light from the sun or a tungsten lamp which emits a full range of frequencies within the visible spectrum).

Subjectively, the two surfaces look identical, but the frequencies being reflected from either of them are clearly extremely different. If the eye worked like the ear, the first surface would sound like a single pure note (as from a celestial ultra-high-frequency flute) while the other would sound like a chord, and probably a not-very-harmonious one at that. But the eye is basically an ear with only 3 cilia in it (and a more sensitive general "loudness" detector to represent the rods, but since there aren't really any in the fovea which is the part of the retina where your colour vision is most acute, we can ignore this).

To anyone (or any species) who doesn't have the exact standard setup of RGB cones I described above, the two surfaces will almost certainly look very different.

So, any objective assessment, if it can work, can *only* work for a specific physiology. And then there are all the other points raised in the thread about surface reflectivity, specular effects and so on, which make it even more difficult to do so.