thanks ill check it out

I'll try this solution if this works, will get back to you.

what cloud works best that have like a gpu option?

im using numpy

my problem was during the faiss indexing of the embeddings, this is where it crashes

load straight into colab

like as a test, i downloaded 1 gb dataset from kaggle into colab directly