Viewing a single comment thread. View all comments

Cryptizard t1_j36zy17 wrote

I think you are really, really, underestimating how hard AI is, or this is some anti-intellectual BS. There is so much background knowledge you need to be able to contribute to state-of-the-art AI, if you were capable of it you would already have a job doing that. It is not something a random Reddit user can just decide one day they are going to do.

Moreover, as others have said, it costs millions of dollars to train these things. To suggest, "hey guys we should just like, build an AGI" is insulting to the people that work on it in academia and in industry. You may as well build a space ship in your back yard and colonize Mars.

As soon as you tried to make a comparison to a movie I knew you were going way off the rails.

34

footurist t1_j37hsvn wrote

It's the underestimation. The thing is, for some reason AGI seems like an approachable problem on first sight. There's something about it that makes you think there has to be some simple, yet surprisingly undiscovered way of building it.

But if and when you actually try to build something, no matter how naive or small, you very quickly recognize the incredible hidden complexity.

I've tried it too, I admit. You go from "I think it's doable" to "hell no, this isn't ever gonna work" in a couple of hours, lol.

7

Scarlet_pot2 OP t1_j398xn5 wrote

The expectations should be tampered. the foundations of AGI aren't going to be made in a couple of hours, but just as "guess the next word" was found out and led to LLMs, I'm sure there are many of simple small discoveries waiting to be found. And many diverse groups trying different things and sharing their results could lead to some of those. It may not be you that builds the million dollar model, but you could make the first simple program that shows promise and ends up being the base idea for large models a few years down the line, by someone.

2

footurist t1_j399rxd wrote

Between the lines I read the assumption that "guess the next word" is definitely agreed upon as being part of or precessor of future AGI, when that's actually highly unclear. Right now they're standing in front of the brick wall of lack of actual reasoning and therefore highly inconsistent emulated reasoning. And it's not clear that's susceptible to a fix or workaround. It could actually be a fundamental limitation of the architecture.

1

Scarlet_pot2 OP t1_j39b08w wrote

IMO, guess the next word isn't going to lead to AGI alone, but it most likely will play a part. Let's assume "guess the next word" fills the part of the brain for prediction down the line for when AGI is developed. Maybe a small group develops the first thing that will later on fit another part of the brain, like how to make memory work. or how to develop reasoning. or any other parts.

The goal should be to make discoveries that could lead to parts of AGI when extrapolated out.. and at least some of those can be found by small groups trying new approaches. John Carmack said that all the code of AGI would be able to fit on a USB. the goal should be to find parts of that code.

1

visarga t1_j39x1lv wrote

The code, yes, but the dataset will be the entire internet and loads of generated data. We have the people, what is necessary is to give them access to compute.

1

Mental-Swordfish7129 t1_j3bbbpn wrote

>I've tried it too, I admit. You go from "I think it's doable" to "hell no, this isn't ever gonna work" in a couple of hours, lol.

I've been at it for around 12 years in my little free time and I've made fairly steady progress excluding a few setbacks. I think I must have gotten very lucky many times. I know that when I look at my approach back then, that I was wayyy off and very ignorant and ridiculous.

2

Xist3nce t1_j397edp wrote

As a developer who tinkered with the move to AI, after looking over what I’d have to learn as a prerequisite I said fuck it. It’s way more than it ever seems from the outside, even if you have skills to transfer.

3

harrier_gr7_ftw t1_j37iwr5 wrote

He went FR after the first paragraph but the algorithms are well known.... it just gets expensive because you need to buy the data for training on, and like you say, the computing time.

Everyone is/was surprised that transformers give better results the more data you give them but this is literally OpenAI's raison d'etre; make the next better GPT by feeding in more data. Sadly most of us can't afford a 1000TB RAID setup to store the Common Crawl and tons of scanned books on, as well as a load of A100 Nvidia TPUs. :-(

AGI is another thing of course which will need a lot more research.

2

Scarlet_pot2 OP t1_j397wnk wrote

Yes making large models is expensive, but coming across the next discovery like "guess the next word" isn't. That small discovery led to all the LLMs we post about today, and it was made from a small group of people. The goal shouldn't be to train million dollar massive models. The goal should be to find new novel approaches.

A small group could make the next discovery like guess the next word, and I'm sure there are many discoveries to be made. building massive models from it may happen years later, from the creators or a better funded group.

1

Cryptizard t1_j3998va wrote

Who exactly are you crediting with inventing this guess the next word approach?

2

Scarlet_pot2 OP t1_j39cjh8 wrote

it was a small group of engineers at google. Not highly funded. They were trying to make something for google translate when they figured out they can make a program that guesses the next word.

1

visarga t1_j39xs2x wrote

No, this concept is older, it predates Google. Hinton was working on it in 1986 and Schmidhuber in 1990s. By the way, "next token prediction" is not necessarily state of the art. The UL2 paper showed it is better to use a mix of masked spans.

If you follow the new papers, there are a thousand ideas floating around. How to make models learn better, how to make them smaller, how to teach the network to compose separate skills, why training on code improves reasoning skills, how to generate problem solutions as training data... we just don't know which are going to matter down the line. It takes a lot of time to try them out.

Here's a weird new idea: StitchNet: Composing Neural Networks from Pre-Trained Fragments. (link) People try anything and everything.

Or this one: Massive Language Models Can Be Accurately Pruned in One-Shot. (link) - maybe it means we will be able to run GPT-3 size models on a gaming desktop instead of a $150,000 computer

2

Cryptizard t1_j39dcvq wrote

I can’t find any evidence of this happening.

1

Scarlet_pot2 OP t1_j39g574 wrote

https://en.wikipedia.org/wiki/Word2vec

"Word2vec is a technique for natural language processing (NLP) published in 2013 (Google). The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text."

This was the first "guess the next word" model.

https://towardsdatascience.com/attention-is-all-you-need-discovering-the-transformer-paper-73e5ff5e0634

This next link is the "Attention is all you need" paper that describes how to build a transformer model for the first time.

These two discoveries didn't take millions or billions in funding. Made by small groups of passionate people, and their work led to the LLMs of today. We need to find new methods that would be similarly disruptive when extrapolated out.. and the more people we have working on it, the better chance we have of finding things like these. IMO these are parts of the future AGI, or at least important steps towards it. It doesn't take ungodly amounts to make the important innovations like these

1

Cryptizard t1_j39gpo3 wrote

They all have PhDs in AI though…

2

Scarlet_pot2 OP t1_j39hw2h wrote

Lets say there's a group of passionate PhDs self funded, over time they have a chance of 20% of finding a innovation or discovery in AI.

now let's say there is another group of intermediate and beginners, self funded, over time they have a 2% chance of making a discovery in AI.

But for the second example, there is 10 of those teams. All the teams mentioned are trying different things. If the end goal is advancement towards AGI, they all should be encouraged to keep trying and sharing right?

1

Cryptizard t1_j39jqjy wrote

I am claiming, though, that amateurs and enthusiasts are incapable of contributing to state-of-the-art AI. There is too much accumulated knowledge. If it was a low, but possible, chance to just make AGI from first principles it would have already happened sometime in the last 50 years that people were working on it. If, however, it is like every other field of science, you need to build the next thing with at least deep understanding of the previous thing.

Your examples might not have had a lot of money, but they all certainly were experts in AI and knew what they were doing.

2