I'm surprised it hasn't happened yet.

Yeah ripping off Google's open code and then keeping theirs closed. Not a good precedent for OpenAI. Still, I have a feeling open source will beat closed eventually.

[deleted]

I mean fuck scientific and societal progress right.

[removed]

I don't see how unless open source figures out a way to distribute training across machines which afaik is incredibly inefficient/impossible right now. It seems that most progress is due to testing out ideas on $100mil worth of hardware iteratively. Oh and also having massive data on a scale that open source will never have access to.

"Open"AI seems anything but.

Yeah like I said in another post, under capitalism it's likely that some company seeks complete monopoly of the labor market before we can all have access to the benefits of AGI. There's no good reason to release your model if it's much better than the current competition if you're a company.

I think this hasn't happened yet because they don't have AGI yet, they'll likely keep it open to the public in case anyone figures out how to advance the research and release it as an open source project so they can copy again.

Distributed training will get better

Ah, yes, because thus far Google/DeepMind have released all of their advanced models such as LaMDA, PaLM, Imagen, Parti, Chinchilla, Gopher, Flamingo, Sparrow, etc. etc.

>[...] a shift towards secrecy and aggressive competition could significantly hinder the pace of innovation.

Or, you know, competition might lead to transforming (no pun intended) these research artifacts into useful products? Google's Code Red sounds like good news to me as an end-user.

What a nonsense article that seems written with the sole intent to shoehorn an ex-Googler's new startup into a post.

>I have a feeling open source will beat closed eventually.

no ways that happens. If the open source is so good, just fork it, and add your company's secret sauce.

> Co-founder of Neeva

Ok, so direct competition for search is commenting on Google. Maybe they want to imply they also have a language model that is special and closed, and worthy of receiving investments.

I don't believe what he says, there are no signs of that happening. On the contrary, it would seem the head of the pack is just 6-12 months ahead. Everything trickles down pretty quickly. There are still many roadblocks to AGI and no lab is within striking distance.

We already have nice language models, now we need something else - validation systems. So we can use our language models without worrying they would catastrophically hallucinate or miss a trivial thing. We want to keep the useful 90% and drop the bad 10%. It is possible to integrate web search, knowledge bases and python code execution into the model to keep it from messing up. This is what I see ahead, not the end of open research.

Google is likely the best positioned for dataset creation.

Text
>google search/cache/AMP links/youtube subtitles

Image
>Image search thumbnails/reCAPTCHA/youtube

Video
>Youtube

They can afford to give away research because I doubt many can match them on shear dataset scale alone.

I highly doubt this validation route would go nearly as smooth as the path hereto. I mean the very root cause for GPT messing up so often and in such strange ways is that there's no real reasoning there, only surprisingly well working emulation of reasoning.

However, for validation this emulated reasoning won't nearly cut it. So you end up where you started : finding architectures that can actually reason, which of course nobody knows...

If you were thinking about something like trying to match its responses to similar "actual" search results and then validating via comparison to that : What mechanism to use? Because this seems to require actual reasoning aswell.

This is an interesting point. Do you think the dataset that google has is high quality enough to actually train ai? In particular, search queries etc aren’t mapped to specific answers to be useful for supervised learning. Maybe I’m missing something?

Open source code could still win, if the secret sauce lies in a massive closed source dataset.

Could better, larger datasets be solution to the hallucination problem? Ref chinchilla for example- but maybe even an order of magnitude bigger than that?

LLMs where it's a statistical likelihood for next token prediction benefit from more data.

That along with the truism

"You always find things in the last place you look"

can be very powerful tools.

There will be some correlation between search term and result otherwise search would be pointless. That on a large enough scale can sift signal from noise, not only in terms of search results but in delta between individual search terms.

There are models which require your new iteration to also be open source.

Elon open sources Tesla and SpaceX tech and yet they're doing better than their competitors

Elon open-sources Tesla and SpaceX tech yet those companies are doing better than others. "Patents are for the weak", he said. I just wish he would slap some sense into Sam Altman like stop being weak Sam. They both co-founded the company

In the wiki for openAI says gpt started when a researcher who isn't even an openAI contributor, a guy named Alec Radford posted a paper to the openAI forums. If the wiki info is correct it sounds like open discussion about the project is what got them there in the first place because it doesn't look like he was even an employee.

You mean this guy? https://openai.com/blog/authors/alec/

Yeah that's the name credited on the wiki.

do you know the link or where to find that wiki?

https://en.wikipedia.org/wiki/OpenAI

There you go. It's under the gpt section in the middle.

Tesla on self driving is far behind waymo and other competitors

SpaceX has no real competitors as the sector is basically kept afloat by nasa contracts and government subsidies

thanks bud

Progress for who?

The shift will have winners and losers, just like any other competition.

It wasn’t a lack of effort on Bezo’s side to win those contracts.

Will it get better faster than the closed systems improve?

Google doesn't release most of their models open source, it just releases some. OpenAI does the same, releases some projects but keeps the bigger ones closed source.

Imagine developing a power that will eventually be used against you. There are.. nations that do not care about patents or I.P.

I would be bummed if you put billions into research and development just to see someone else taking a run with it.

Eventually yes it will become way faster than “closed systems”, because it will be in the cloud on the best machines. Cloud hosting services are clearly incentivized to make distributed training for open source communities affordable and accessible.

The cloud? That is owned and operated by the companies developing the closed systems?

The best machines? While competing against companies with billions of dollars of dedicated funding?

You need to reevaluate your logic.

Surely there’s some trade off between qualitative vs quantitative data?

Eg. 50 billion high quality QA pairs may beat 500B random google queries as training data.

There are approaches to combine multiple stages of language modelling and retrieval. Demonstrate Search Predict: Composing retrieval and language models for knowledge intensive NLP.

This paper is very interesting. They don't create or fine-tune new models. Instead they create sophisticated pipelines of language models and retrieval models. They even publish a new library and show this way of working with LMs.

Practically, by combining retrieval with language modelling it is possible to verify against references. The ability to freely combine these transformations opens up the path to consistency verification. A LM could check itself for contradictions.