>Why is python so widely used in AI when it’s a really inefficient language under the hood? Wouldn’t Rust be better to optimize models? Or do you just need that optimization at the infrastructure level while the models are so high level it doesn’t matter?

You make calls to a high level framework, usually pytorch, that have the effect of creating a pipeline. "Take this shape of input, inference it through this architecture using this activation function, calculate the error, backprop using this optimizer".

The python calls can be translated to a graph. I usually see these in *.onnx files though there are several other representations. These describe how the data will flow.

In the python code, you form the object, then call a function to actually inference it a step.

So internally it's taking that graph, creating a GPU kernel that is modified for the shapes of your data, compiling it, and then running it on the target GPU. (or on the project i work on, it compiles it for what is a TPU).

The compile step is slow, using a compiler that is likely C++. The loading step is slow. But once it's all up and running, you get essentially the same performance as if all the code were in C/C++, but all the code you need to touch to do AI work is in Python.

Python is just scripting for whatever talks to the metal…