It's unlikely that the main problem is the RHLF data and not the base model.

🤷

Sometimes models just come out crap. Like BLOOM which has almost the same number of parameters as GPT3, but is absolute garbage in any practical use case. Like a kid from two smart parents that turns out dumb. Just blind chance.

Or they could be wrong. 🤷

It's not about copyright

https://www.reddit.com/r/MachineLearning/comments/11v4h5z/-/jct0s11

They are. It's less to do with copyright and more to do with the fact that you signed the T&C before using their system ( and then broke ). It's simmilar to the LinkedIn data scraping case where the court ruled that it wasn't illegal for them to scrape ( nor did it violate copyright ) but they still got in trouble ( and had to settle ) because of violating the T&C.

One way around this is to have two parties, one generating and publishing the dataset ( doesn't violate T&C ) and another independant party ( who didn't sign the T&C ) fine-tuning a model on the dataset.

Just publish the diff. between the original model and the finetuned model. That's what a lot of people are doing to avoid any license issues.

• There's a already a couple high quality instruction datasets/compilations like FLAN that I think should also be mixed in.

• Be sure to check the generated dataset for issues. Might require some cleanup like the original did.

He's using the actual API.

I've heard from some experienced testers that the 33B model is shockingly bad compared to even the 13B one. Despite what the benchmarks say. That we should either use the 65B one ( very good apparently ) or stick to 13B/7B. Not because of any technical reason but random luck/chance involved with training these models and the resultant quality.

I wonder if there's any truth to it. If you've tested it yourself, I'd love to hear what you thought.

You can see some benchmarks here

https://github.com/qwopqwop200/GPTQ-for-LLaMa

No

Yes

He's talking about the human preference data used for RHLF fine-tuning ( which is what makes ChatGPT from GPT3 ). It's not really that massive.

Nvidia RTX voice

Looks so dated..

Fair enough. I was speaking from a practical perspective, considering the types of questions that people typically ask search engines, not benchmarks.

It's not just better, wrong information from these models is pretty rare, unless the source it is retrieving from is also false. The LM basically just acts as a summary tool.

I don't think it needs to be 100% resolved for it to be a viable replacement for a search engine.

Retrieval augmented models ( whether via architecture or prompt ) don't have that issue.

Even GPT3 API based services like perplexity.ai that retrieval augment using just the prompt don't spew wrong information all that much.

Depends on which Amish faction/group we're talking about. It's not so much that they can't use technology, but more that they can't be connected to the outside/grid. So some groups even use solar panels and battery powered electrical equipment. They also have fridges that run on gas.

Looks like perplexity.ai , but better ( maybe? )

Doesn't feel very GPT4. Seems like simple retrieval augmentation.

It's not really the colours. It's the layout. Feels dated and blog spammy. There's some fundamental rules of modern responsive website design it breaks such as not putting borders on the side. Especially thick ones.

I don't think it's something you'll be able to fix in a second. Take some time with it.

It's fine. I think the main issue is I'm on mobile right now. Desktop website is better.

Good. The format is perfect. All that needs work is the visual design of the site, and automating this to work on a large scale ( I don't know how much of that is automated ).

>find exact duplicates of images which were not in the training data, you'd have a point

The process isn't exactly the same, but isn't this how all the diffusion based editing techniques work?

From paper

>Our attack extracts images from Stable Diffu- sion most often when they have been duplicated at least k = 100 times

for the 100 number. The 10 is supposed to be the number of epochs, but I don't think it was trained on that many epochs. More like 5 or so ( you can look at the model card ; it's not easy to give an exact number ).

They also manually annotated the top 1000 results, adding only 13 more images. The number you're replying to counted those.