FirstOrderCat

FirstOrderCat t1_iyw8tuu wrote

Reply to comment by Ambiwlans in bit of a call back ;) by GeneralZain

Here is recent paper, they improved previous SOTA in GSM8K by 2%: 78->80: https://arxiv.org/pdf/2211.12588v3.pdf

​

>Llms are basically doa waiting on gpt4 in a few months now anyways unless they offer something really novel.

why are you so confident? Current gpt is very far from doing any useful work, it can't replace programmer, lawyer, accounter, the is a huge space for improvement before they reach some AGI and replace knowledge workers.

2

FirstOrderCat t1_iybm4dy wrote

>First in a MMO like setting, then the real world.

nop, this transition is very hard because of following reason:

current wave of AI can approximate giant datasets, that's something it is doing very well. So, all your examples is: they throw terrabites of data on neural network, and it learns patterns. But this kind AI can't generalize and do abstract thinking which means it can't learn from very few examples.

Meaning yes, they can ask AI to play MMO 100 millions times and it will learn from its own mistakes, but you would need to do the same 100 million times in real world, which is not very feasible.

Another issue is that: MMO has much smaller level of freedom than real world, which makes MMO is not a good benchmark.

1

FirstOrderCat t1_iybg5rh wrote

> it can understand and search pretty dang well from voice alone.

there is component where some model translates your voice to text, but searching part contains tons of human hand-crafted code.

So, current language models are good for some narrow tasks (translation is the main one), but still not on the level of abstract thinking human posses. My bet they won't be able, unless some large advancement.

1

FirstOrderCat t1_itc6pne wrote

Reply to comment by Spoffort in U-PaLM 540B by xutw21

It looks like they had point of diminishing return somewhere at 0.5*1e25 FLOPS.

After that model trains much slower. They could continue training farther, and say they "saved" another 20M TPU hours.

1

FirstOrderCat t1_itahazn wrote

Reply to comment by AsthmaBeyondBorders in U-PaLM 540B by xutw21

>This model had up to 21% gains in some benchmarks, as you can see there are many benchmarks

Meaning they received less than 2 points in many others..

> it is about a different model which can be as good and better than the previous ones while cheaper to train.

Model is the same, they changed training procedure.

> You seem to know a lot about Google's internal decisions and strategies as of today

This is public information.

2

FirstOrderCat t1_itaeypy wrote

Reply to comment by AsthmaBeyondBorders in U-PaLM 540B by xutw21

this race maybe over.

On the graph guy is proud of getting 2 points in some synthetic benchmark, while spending 4 millions TPUv4 hours = $12M.

At the same time we hear that Google cuts expenses and considering layoffs, and LLM part of Google Research will be the first in the line, because they don't provide much value in Ads/Search business.

1

FirstOrderCat t1_itad0zs wrote

Reply to comment by AsthmaBeyondBorders in U-PaLM 540B by xutw21

>The problem is you don't know what emergent skills are yet to be found because we didn't scale enough.

Yes, and you don't know if such skills will be found and we hit the wall or not yet.

1

FirstOrderCat t1_itacdp4 wrote

Reply to comment by AsthmaBeyondBorders in U-PaLM 540B by xutw21

>A wall is when we can't improve the results of the last LLMs.

The wall is a lack of break through innovations.

Latest "advances" are:

- build Nx larger model

- tweak prompt with some extra variation

- fine-tune on another dataset, potentially leaking benchmark data to training data

- receive marginal improvement in benchmarks irrelevant to any practical task

- call your new model with some epic-cringe name: path-to-mind, surface-of-intelligence, eye-of-wisdom

But none of these "advances" somehow can replace humans on real tasks, with exception to style-transfer of images and translation.

−5

FirstOrderCat t1_ita7yel wrote

Reply to comment by AsthmaBeyondBorders in U-PaLM 540B by xutw21

> form but don't forget LLMs are behind stable diffusion, dreamfusion, dreambooth, etc.

But its not discussed AGI, it is more stochastic parroting, or style transferring.

0

FirstOrderCat t1_it9oqhg wrote

Reply to comment by CommentBot01 in U-PaLM 540B by xutw21

> Currently deep learning and LLM are very successful and not even close to its limit.

to me it is opposite, companies already invested enormous resources, but LLM can solve some simplistic limited scope tasks, and no much AGI-like real applications have been demonstrated.

2