vitalyc t1_jdo3iji wrote on March 25, 2023 at 9:25 PM

Reply to comment by Bewaretheicespiders in Goodbye Google. Welcome AI. by OmegaConstant

So how are people running LLMs locally on laptops and phones? It seems the training costs are unimaginable but you can optimize the models to run on consumer hardware.

Bewaretheicespiders t1_jdokdmg wrote on March 25, 2023 at 11:33 PM

They arent running GPT4 locally, it sends the request through an API.

GPT3 has 175 billion parameters, at float16 thats 326 gigabyte just for the parameters. That would fill most phone's storage, not to mention the 12 gig of ram the most expensive phones have.

Then GPT4 is many times that...

vitalyc t1_jdov7q8 wrote on March 26, 2023 at 12:55 AM

They are running a stripped down version of facebook's LLM. https://github.com/antimatter15/alpaca.cpp is one of the first projects demonstrating it.

Bewaretheicespiders t1_jdoyplc wrote on March 26, 2023 at 1:23 AM

Its 4gig of weights. There is no comparison with GPT-4.