In the long run, I think this is something that will be solved with more specialized architectures for running neural networks. TPUs and Tensor Cores are great first steps, but the Von Neumann architecture is holding us back.

Tensor Cores are very fast. But since the Von Neumann architecture has separate compute and memory connected by a bus, the entire network has to travel through the memory bus for every step of training or inference. The overwhelming majority of time is spent waiting on this:

>200 cycles (global memory) + 34 cycles (shared memory) + 1 cycle (Tensor Core) = 235 cycles.

A specialized architecture that physically implements neurons on silicon would no longer have this bottleneck. Since each neuron would be directly connected to the memory it needs (weights, data from previous layer) the entire network could run in parallel regardless of size. You could do inference as fast as you could shovel data through the network.

There's some mutually assured destruction going on here. Microsoft/OpenAI also own patents that cover Google's products. If Google sued them over Transformers, they would sue right back for something else.

https://www.aeaweb.org/research/a-truce-in-the-patent-wars

They use the open LAION 50B dataset, everybody knows what's in there.

Still, some preprocessing and deduplication would have been a good idea just for output quality.

>Right now basically all progress is with large models,

You mean all progress... in machine learning. A lot of scientific fields necessarily must make do with a smaller number of data points.

You can't test a new drug on a million people, especially in early phase trials. Even outside of medicine, you may have very few samples if you're studying a rare phenomena.

Statistics gives you tools to make limited conclusions from small samples, and also measure how meaningful those conclusions actually are.

Stats is tremendously useful, especially when your dataset is small by ML standards. Basically every scientific paper relies on statistics to tell you whether or not their result is meaningful.

ML is great when you have millions of data points, but when you only have a hundred it's not going to help you.

Agreed, financial markets have built-in protections against this kind of analysis. If it works, everyone else would do it, and the more people do it the less any of them benefit from it.

The only way to beat the market consistently is to have a source of information nobody else has access to, or at least hasn't discovered yet.

While those are on the same topic, they're very different papers. The Anthropic paper spends most of its time going on about safety/bias/toxicity, while the Google paper is focused on more useful things like the technical abilities of the models.

>What is the standard modeling approach to these kinds of problems?

The standard approach is reinforcement learning. It works, but it's not very sample-efficient and takes many iterations to train.

LLMs are probably so good at this because of their strong meta-learning abilities; during the process of pretraining they not only learn the task but also learn good strategies for learning new tasks.

This has some really interesting implications. Pretraining seems to drastically improve sample efficiency even if the pretraining was on a very different task. Maybe we could pretrain on a very large amount of synthetic, generated data before doing our real training on our finitely-sized real datasets.

SNN-ANN conversion is kludge - not only do you have to train an ANN first, it means your SNN is incapable of learning anything new.

Surrogate gradients are better! But they're still non-local and require backwards passes, which means you're missing out on the massive parallelization you could achieve with local learning rules on the right hardware.

Local learning is the dream, and would have benefits for ANNs too: you could train a single giant model distributed across an entire datacenter or even multiple datacenters over the internet. Quadrillion-parameter models would be technically feasible - I don't know what happens at that scale, but I'd sure love to find out.

So far nobody's figured out a good way to train them.

You can't easily do backprop, but you wouldn't want to anyway - the goal of SNNs is to run on ultra-low-power analog computers. For this you need local learning, where neurons can learn by communicating only with adjacent neurons. There's some ideas (forward-forward learning, predictive coding, etc) but so far nothing is as good as backprop.

There's a bit of a chicken-and-egg problem too. Without a good way to train SNNs, there's little interest in the specialized hardware - and without the hardware, there's little interest in good ways to train them. You can emulate them on regular computers but that removes all their benefits.

That's a no from me dawg.

I'll wait my turn, I'm not installing their app.

Meh, I think the safety concerns are overblown. It's really more of bad PR for Microsoft than an actual threat.

You can already find out how to make drugs, build a bomb, etc from the internet. The Anarchist Cookbook has been well-known for decades and you can find a pdf with a simple google search.

Call me when logistic regression can generate a realistic detailed digital painting by greg rutkowski.

All the computation is happening on the GPU. Python is just making a bunch of calls to the GPU drivers.

Researchers spend a lot of time making neural networks as fast as possible. If switching to another language would have given a substantial speed boost, they would have done it already.

The exception Google Images got is pretty narrow and only applies to their role as a search engine. Fair use is complex, depends on a lot of case law, and involves balancing several factors.

One of the factors is "whether your use deprives the copyright owner of income or undermines a new or potential market for the copyrighted work." Google Image thumbnails clearly don't compete with the original work, but generative AI arguably does - the fact that it could automate art production is one of the coolest things about it.

That said, this is only one of several factors, so it's not a slam dunk for Getty either. The most important factor is how much you borrow from the original work. AI image generators borrow only abstract concepts like style, while Google was reproducing thumbnails of entire works.

Anybody who thinks they know how the courts will rule on this is lying to themselves.

> Also, LAION gathered the images via crowdsourcing. I participated.

I don't think the data collection methodology is really relevant. However the dataset was gathered, there are certainly ways to use it that would violate copyright. You couldn't print it in a book for example.

The important question is if training a generative AI on copyrighted data is a violation of copyright. US copyright law doesn't address this because AI didn't exist when it was written. It will be up to the courts to decide how this new application interacts with the law.

>the only thing I have seen is cheating on homeworks and exams, faking legal documents, and serving as a dungeon master for D&D. The last one is kind of cool, but the first two are illegal.

Well that's just cherry-picking. LLMs could do very socially-good things like act as an oracle for all internet knowledge or automate millions of jobs. (assuming they can get the accuracy issues worked out - which there are tons of researchers trying to do, some of whom are even on this sub)

By far the most promising use is allowing computers to understand and express complex ideas in plain english. We're already seeing uses of this, for example text-to-image generators use a language model to understand prompts and guide the generation process. Or how Github Copilit can turn instructions from english into implementations in code.

I expect we'll see them applied to many more applications in the years to come, especially once desktop computers get fast enough to run them locally.

>starts playing by the same rules as everyone else in the industry.

Everyone else in the industry is also training on copyrighted data, because there is no source of uncopyrighted data big enough to train these models.

Also, your brain is updating its weights based on the copyrighted data in my comment right now, and that doesn't violate my copyright. Why should AI be any different?

OpenAI is doing a good thing. They've found a new and awesome way to use data from the open web, and they deserve their reward.

Getty's business model is outdated now, and the legal system shouldn't protect old industries from new inventions. Why search for a stock image that sorta kinda looks like what you want, when you could generate one that matches your exact specifications for free?

> Besides, it's not clear to me whether these AI tools be used to benefit humanity as a whole

Of course they benefit humanity as a whole.

• Language models allow computers to understand complex ideas expressed in plain english.
• Automating art production will make custom art/comics/movies cheap and readily available.
• ChatGPT-style AIs (if they can fix hallucination/accuracy problems) give you an oracle with all the knowledge of the internet.
• They're getting less hype right now, but there's big advances in computer vision (CNNs/Vision Transformers) that are revolutionizing robotics and image processing.

>I really hope this case sets ome decent precedents about how AI developers can use data they did not create.

You didn't create the data you used to train your brain, much of which was copyrighted. I see no reason why we should put that restriction on people trying to create artificial brains.

Getty is just the test case for the question of copyright and AI.

If you can't train models on copyrighted data this means that they can't learn information from the web outside of specific openly-licensed websites like Wikipedia. This would sharply limit their usefulness. It also seems distinctly unfair, since copyright is only supposed to protect the specific arrangement of words or pixels, not the information they contain or the artistic style they're in.

The big tech companies can afford to license content from Getty, but us little guys can't. If they win it will effectively kill open-source AI.

Huh, I haven't used it in years but it's great to see them still chugging along!

Very high.

The standard of evidence required for aliens is also very high. It's not enough to have no known natural explanation; there's lots of natural phenomena we don't know about yet. It must affirmatively and unavoidably be artificial even after many follow-up observations.

> they make their money on the sale of the franchise.

What does this mean? Are they planning to sell the business someday, and they will make their money back when they do so?

If the business is a net loss except when you sell it to someone else, why is anyone willing to buy it? EDIT: I guess people were willing to buy NFTs.

Those are overhyped. They haven't automated the actual cooking - just replaced the cashier with a touchscreen. Fully automated restaurants remain a dream.

Of course, that's not the same as the actual profit margin of the restaurant - that's just what McDonald's corporate makes from the franchise vs the costs of what they provide to the franchise.