Can a knife cut better than a human hand? After all, it was made by human hands... Yes, AI can be designed and trained to outperform humans at any task that we can frame as a ML task. The big advances have come from clever ways to frame tasks in a way that ML can work on it.

I would argue that there is a significant difference between how a knife works and how ML works. You do not have to train a knife how to slide bread.

Besides, it looks to me that ML can outperform humans just because it utilises the fact that modern day computers can do zylions of computations per second. Of course, the sheer speed of computation is not enough and this is why we need smart algorithms as well. But those algorithms benefit from the fact that they have super power hardware available, often not only during training phase but also during normal operation.

>Can a knife cut better than a human hand? After all, it was made by human hands...

Facts and logic

From a very theoretical point of view, we can imagine a knowledge "A" useful for a task A, a knowledge B useful for a task B. That's how humans would apply their knowledge. But we could teach a model to learn knowledge A (or A+B) and apply it to task B, and it would eventually perform better.

Humans don't have all the knowledge and don't apply everything they could know to every tasks perfectly. Models also aren't perfect but they could do more combinations and perform better on certain tasks because of that.

But I can take another exemple. Here is a task: "a human says N images contain dogs and M images contain cats, the model must reproduce this behavior". Would a perfect model designed to exactly reproduce the human be able to outperform a human on this task? No. The human would make mistakes, and the model would reproduce these mistakes. But we don't design or train our models to exactly reproduce what a human did, that would be a risk of overfitting, we use regularizations so that even by reproducing what humans did a model can do better and not reproduce some mistakes.

>But we don't design or train our models to exactly reproduce what a human did, that would be a risk of overfitting, so even by reproducing humans a model can do better and not reproduce some mistakes.

Can you please elaborate on this? Let's say your train data contains 10% of errors. Can you train a model that it would be more than 90% accurate? If yes, why?

Edit: My guess would be that the model during the training phase, can "find out" what are features typical for cats provided that the training set is "good enough". So even if the set contains some errors, they will not impact significantly a prediction the model can give.

ML can be use to train a model to perform a specific task very well. We humans have a way more broad intelligence.

Imagine if we could use 100% of our brain power to perform one task 24/7, that was trained all its life to perform that one task, we could outperform an AI easily.

I can give another example. Input / Output: 1.7/0, 2/0, 2.2/1 ,3.5/0 ,4/0 ,5/0 ,8/0 ,9.6/0 ,11/1, 13/1, 14/1, 16/1, 18/1, 20/1. There is an error in this dataset: 2.2/1. But you can train a model on this set to predict 2.2/0 (a small / regularized model would do that) . You could also train a model to predict 1 for 2.2, but it would probably be overfitting. The same idea applies to any concept in input and any concept in output.

My question was slighly different. My understanding is that one of major factors that impact your quality of your model predictions is your training set. But since your training set could be inaccurare (in other words, made by humans), how this fact can impact quality of learning and then quality of predictions.

Of course, as u/IntelArtiGen wrote, models can avoid reproducing errors made by humans (I guess because they are able to learn specific features during a teaching phase when your training set is good enough). But I wonder what this good enough means exactly (in other words, how inevitable errors made by humans when preparing it impact an entire learning process and what kind of errors are acceptable) and how an entire training process can be described mathematically. Of course, I have seen many explanation using gradient descent as an example, but none of them incorporated the fact that a training set (or loss function) was imperfect.

I think this is quite an important and fundamental question. Of course the answer will depend on the task. But in theory, what deep learning is doing is maximum likelihood. That is, minimize the average error. Doing "average" on the "whole task" is superhuman most of the time.

I think the context in which the task is performed, and what is meant by "outperform", is important. If given enough time and energy, a person could probably find all the dogs in a dataset of images. But could they find 200k dogs in a dataset of millions of images overnight? Probably not. In this sense, machines far outperform humans who are limited by attention span and ability to parralelize.

Yes, ML can outperform humans in certain tasks.

1. Quantity can sometimes make a very big difference - if you could sit down and train a human on the same amount of data, the human might be on par with ML...but that's often not possible

2. Training data is not always generated by humans.

3. Given the same data, there are connections or perspectives that humans have not followed or considered.

[removed]

Supervised learning isn't the only game in town, and human demonstrations aren't the only kind of data we can collect. For example we can record human preferences over model outputs and then use this data to fine tune models using reinforcement learning (e.g. https://arxiv.org/abs/2203.02155). Even though I'm not a musician, I can still make a meaningful judgement about whether one piece of music is better than another. By analogy, we can use human preferences to train models that are capable of superhuman performance.

It absolutely can, but it depends on the task.

When it comes to questions of human estimation, there is the Wisdom of the Crowd effect where individual people can either over- or underestimate a value, but the average of many predictions often is extremely accurate. You can either train a system directly on the average of many human predictions or you only have a single prediction for each sample and during training the errors tend to average out.

And even for tasks where there is a definite ground truth available that doesn't depend on human estimation it can happen. For example in many medical applications the diagnosis is provided by the lab, but physicians often need to make a decision whether it is worth it to take a biopsy and send to the lab. In this case they often are taught heuristics: A few features by which they decide whether to have a sample analyzed.

If you take samples and use the groundtruth provided by the lab, it is not inconceivable that a deep learning based classifier can discover additional features that are not used by physicians that lead to better predictions. Obviously, it won't get better than the lab results.

As you migth have guessed, I work in the medical field. And we have done the evaluation for one of our products: It is not able to outperform the top-of-the-line experts in the field. But it is easily able to outperform the average physician.

Yes, in general. As long as the human is accurate most of the time, the ML model will take the errors as outliers and outperform the human. The AI is generally finding trends in the data and then using those trends to predict future points. As long as the human tagging is accurate enough to get the true trend across, the AI can outperform them. It's important to understand that an AI model will perform DIFFERENTLY than a human, for better or worse, since it uses different processes. Some mistakes that a human makes, the AI would not make, but there are mistakes a human might not make that the AI will make.

Yes, I believe there are 2 factors playing a role here:

1. Models could potentially correct some errors of the human labeler using their generalization power, provided that the model is not overfitted.

2. You should differentiate between outperforming a human and outperforming humans. Labels usually represent the collective knowledge of a number of people not just one.

I think Karpathy said this in a tweet ? Or was it LeCun ? Someone talked about it though ...

Think of it more like this: the total intelligence/capability of all humans greatly exceeds the total intelligence/capability of any individual human. When we train ai models, we are imbuing them with capabilities derived from many humans. But once produced, they are easily copied so we could quickly have a population of them, where each individual ai is equivalent to many humans (and thus greater than any individual human) but maybe less capable than all humans. Eventually the population of enough such ais is more capable than the population of humans.

And then, if the ais can train new generations from prior ais, this pattern could repeat and explode.

Yep, you are right, I was not clear enough. What I meant was that AI would to a task "significantly better" (whatever this means exactly). For instance, if humans can find 90% of dogs in a dataset, that AI would be able to find 99.999% of dogs.

[removed]

Are you asking if it can outperform a single human or a collective of humans?

Can a machine learning model determine the make and model of a Toyota better than I can? Yes. Is there a human on the planet that can better determine the male and model? Yes.

What kinds of tasks can (or can’t) be framed as an ML task?

When I put some thoughts in my question (yes, I know, I should have done it before I posted it), I realised that I was interested in how training in general and training set imperfections impact a model performance. For instance, if a training set is 90% accurate, then how and why the model which used that data for training can be more than 90% accurate. And what kind of errors in the training set the model can correct?

That’s not an easy question to answer because the 90% that are correct may be super easy to fit, and those 10% errors may just be unfittable and will just keep the loss high without impacting the model. On the other hand, since models tend to be very over-parameterized that 10% could very well be “fit” and have an outsized impact on the model. It could also be the case that the model ends up with 10% variance on its accuracy.

I’ve never seen a definitive theoretical answer since deep learning models are over parameterized and have seen models replicate the error in the training data, especially when it came to keypoint prediction. When I assessed the error in the training data, I showed the team that the model has the same degree of error. I was arguing for cleaner training data. I got told no and to come up with a magic solution to fix the problem. I quit 🤣

Generally, it is a good idea to split your data into a training, validation, and testing set. Something like 80/10/10 or 80/20 depending on how much data you're feeding a neural network (NN).

So, 80% of the data, randomly selected, would be used to train an NN, and with, say, every epoch or batch if using batch normalization, it would validate against what it has "learn."

Once you're happy with said model performance, then you can use the test data set to see how well your model performs to "new" data in the sense that the 10% you set aside for testing was never introduced to the model during training.

Of course, there are many, many other methods to minimize loss, performance, etc. But, even if your network was "perfect," if the person building it didn't spend the time to "clean" the data, then no matter what it will always have some higher degree of error.

Or something like that. I'm just a fledgling when it comes to deep learning.

I guess "errors" in the dataset could be equivalent to introducing noise (like random perturbations with mean 0) or a bias (perturbation with non 0 expectation). I guess those would be the two main kind of innacuracies found in data.

Bias has been the plague of some language models which were based on internet forum data. The training data was biased towards certain opinions, and the model just spat them out. This is has caused the creators of those models to shut them down. I don't know how could one do to correct bias, since this is not at all my expertise.

Learning techniques resistant to noise (often called robust) are an active field of research, and some methods actually perform really well.

We should have a more rigorous definition of "outperform". What are we comparing? Your question touches on the idea of internal versus external validity - if the data is fundamentally flawed, there is performance ceiling if it doesn't reflect the use case of the ML algorithm developed using it. It may be internally valid (the ML model is trained correctly), but has poor external validity (the ML model doesn't apply to the task it was trained for).

AI will surpass humans in all domains where it can generate problem solving data. AlphaZero did it. Trained in self-play and beat humans. No imitation, no human data at all.

What we need is to set up challenges, problems, tasks or games for the language model to play at. And test when it does well, and add those solutions to the training set. It will be a loop of self improvement by problem solving. The learning signal is provided by validation, so it doesn't depend on our data or manual work. It can even generate its own challenges.

More recently AlphaTensor found a better way to do matrix multiplication. Humans tried their hand for decades at this task, and in the end the AI surpassed all of us. Why? Massive search + verification + learning = a "smart brute forcing" approach.

It’s a really important question. It surely can for certain tasks. I think the general reason why models trained in a supervised setting can outperform us, even though we are the ones that tell them all the right answers, is because we don’t tell them how to get to those answers. When models learn to find the answers via some novel method/logic that is better than ours it can use that logic to outperform us.

My only rebuttal to this theory is a knife is physical matter made by a human thinking. I feel this is an over simplification of OP’s question and personally have this same concern as OP. The question is how can immaterial mechanical thought be any better than immaterial organically produced thought when created by the latter?

Edit: Sorry, meant to reply to above post but not sure it matters..

sometimes it is possible, like AlphaZero for example.

Yes eg a medical image classifier can outperform the average doctor. This can be verified by having 10 doctors review each image in the test set to establish a true-consensus.

Even when ML is trained with noisy labels, there are techniques to obtain robust models whose accuracy is overall better than the noise-level in each label. One good opensource library for this: https://github.com/cleanlab/cleanlab/

Another way to get ML that outperforms individual data labelers is to have multiple annotators label your data. Crowdlab is an effective method to analyze such data: https://cleanlab.ai/blog/multiannotator/

So there are a few things to consider here. What exactly does it mean to "best humans"? In some cases, we can train on data from professional or highly skilled humans, and use that to outperform the average human.

In other situations, we can train an ML model to be a sort-of average of a lot of humans, getting rid of a lot of their individual errors and thus performing better than the average.

In other situations, we may have access to data that is not directly generated by humans, or it's made by humans with access to more data than what the model has access to. For example, to train a vision based ML system for autonomous driving, you could use ground truth data obtained by other means (e.g. LIDAR).

Another sort-of similar situation is training an ML model to predict the stock market. Here, you can use actual stock market trends as the ground truth, rather than the guesses of humans.

So all in all, there are many situations where ML models aren't limited to human capabilities.

Google search, AlphaZero… don’t we have examples of AI all around us that are better than humans?

Also isn’t this the same as asking “if a human learns from other humans can they never be smarter than the people around them”?

Well, the AI is already beating humans at the example you gave, the best accuracies on imagenet are now higher than humans.

But there are ways your data can be changed that can easily make AI superhuman. You can classify a full resolution image of a dog then compress and shrink it down before training the AI. A lot fewer humans could then see the dog in a tiny low-res image, but an AI could get it correct more just as often.

There are also AI that can improve themselves more than the human given data. The AlphaGo project started off with human Go matches as training data, and evolved into tabula-rasa training by self play. By the end, the AI beats the best human.

One of the most interesting things about ML to me is that it can exploit large amount of data (even not entirely correct) to reach super-human level.

Suppose you trained a classification model on a large amount of images. Some of these images will be probably labelled wrong by humans (e.g. because are blurred, the object is not entirely visible, undecision between two classes, etc.). Other similar images will be labelled right, perhaps by another human. But since the model saw many many images it can average its understanding of the images. Keep in mind that this is not anymore true if you have biased training data, i.e. the error is always done in the same way.

If kings eat food harvested by slaves, could they win a 1-1 fight against one?

I take a similar thought process with all automation efforts that use data. The effort doesn't need to improve the state of execution, just perform it at the same level of effectiveness as its human counterpart. By simply doing that 24/7, it inherently outperforms humans.

https://www.nature.com/articles/s41746-019-0112-2

TL;DR AI is better at detecting and scoring prostate cancer than expert pathologists themselves.

I think the best way to answer your question is to ask you to be more precise about what, exactly, you mean by "outperform".

There's some limited sense in which your reasoning works as you seem to have envisioned. A generative model like GPT or GANs is typically built at least partly to produce output that's indistinguishable from what is produced by a human, using some kind of autoregressive data set or adversarial objective. By definition, it cannot do better at that goal, because a human has a 100% success rate, by definition, at producing something indistinguishable from what is produced by a human.

But there are limitations to this reasoning:

1. Producing any arbitrary human-like output is not actually the goal. People don't evaluate generative models on how human-like they are, but rather on how useful their results are. There are lots of ways their results can be more useful even if they aren't quite as "human-like". In fact, the motivation for trying to keep the results human-like is mainly that allowing a generative model too much freedom to generate samples that are very different from its training set decreases accuracy, not that it's a goal in its own right.
2. That's not all of machine learning anyway. Another very common task is, for example, Netflix predicting what movies you will want to watch to build their recommendations. Humans are involved in producing that data, but it's not learning from data about what other humans predicted users would watch. It's learning directly from observed data about what humans really did watch. Such a system isn't aiming to emulate humans at all. Some machine learning is even trained on data that's not generated by humans at all, but rather the objective it's training to optimize is either directly observed and measured, or directly computed.
3. Even in cases where a supervised model is learning to predict human labeling, which is where your reasoning best applies, the quantity of data can overcome human accuracy. Imagine this simpler scenario: I am learning to predict which President is on a U.S. bill, given the denomination amount. This is an extremely simple function to learn, of course, but let's say I only have access to data with a rather poor accuracy rate of 60%, with errors occurring uniformly. Well, with enough of that data, I can still learn to be 100% accurate, simply by noting which answer is the most common for each input! That's only a theoretical argument, and in a realistic ML context it's very difficult to get better-than-human performance on a supervised human-labeled task like this. But it's not impossible.
4. And, of course, if you look at more than just accuracy, ML can be "better" than humans in many ways. They can be cheaper, faster, more easily accessible, more deterministic, etc.

OP, you should watch the machine learning street talk YouTube channel and watch whatever videos interest you. It is an incredible resource from top minds all around the world in the field. Only caveat is i would say don’t start with the Noam chomsky episode. It is a great episode, but a bad first episode to introduce yourself to the series because so much exposition goes into explaining the extreme unfortunate technical issues they faced in fixing a messed up recording.

I’ll tell you what i think you are asking then answer. Fwiw i am NOT an ML engineer. I think you are asking “can machine learning models with current technological limits outperform humans in any given task considering the data is possibly compromised by human error? What about in the future?”

Firstly, self supervised learning approaches are apparently beginning to perform better than human guided models in many tasks. I’m referring to isolated cognitive tasks which can operate within a VERY limited amount of specified parameters. Take diffusion image generation. There are perhaps 60 to 80 settings you tweak. before you include token processing (processing the prompt. Smth like 4 words is more or less equivalent to a token) . My point here is that these are GAN driven, and in a way, while the data is not cleaned, the really poorly fitting data will be statistically reflected in output probably to the extent that it is an outlier. So low quality things will be output less because within the context of the model, the model may “understand” this is not a desired output.

Second, while your question is basic on the surface, it is actually a major subject of debate still. My personal opinion is that there are major structural components and AI architectures that still need to be created, if not to imitate human conscious thinking, executive function, attention, strategizing, and possibly desire and more nuanced reward systems, and they must all be integrated in such a way that when they train themselves on data that they are able to discover best practices for multi-step, multimodal, and various cognitive approaches before it is as intuitive as talking to another person and explaining what you want and getting the desired result.

While transformers (an ai architecture invented in 2017) are powerful and appear to learn data and it’s semantical importance, and can lead to better performance than humans in many tasks, there is still something “dumb” about these models, namely that they are highly specialized. This is why the datasets are absolutely ginormous, yet if you put them outside of their speciality they have no clue what the hell is going on really.

There are actually some interesting counterpoints to this. For instance, for google imagen, a diffusion image generator, i believe it has one of the largest parameter datasets for image generation. What is particularly interesting though is that even though the model is trained on images, because it has seen so much text in images, has learned to spell spontaneously. Therefore you can ask for an image of a guy holding up a sign that says “a computer wrote this sign” and it will create images of each letter in order to appear as the words requested.

While that is incredibly interesting, datasets will be eventually approaching the size of the internet itself and only able to do simple tasks like this. As a tiny human, i didn’t need to absorb the entire world too learn about its generalities. Ultimately i think that is the answer you are looking for.

I personally believe that consciousness has to do with integrated multimodal information processing and that intelligence is simply a brain having ample raw data stored in memory, but in extremely efficient and compressed ways structured from general to specific. It is less like the information is stored there as discrete facts, and more like the information is stored in configurations of layers of processing, so that multiple different concepts can be represented within these spaces at any given time.

One very strong support to this idea is considering what human attention actually is. I think attention is less an accessing of raw data and “focusing” on that data than it is a holistic reconstruction of a concept from multiple constituent components. This is especially why it would be nearly impossible for a person to think in great depth about two very different and unrelated concepts simultaneously. However This is also why metaphors and analogies appear so powerful as explanatory devices for humans, because a metaphor takes two simple but seemingly unrelated concepts and makes them fit together in a way that makes sense within the given context of the metaphor. We understand that a metaphor is not literal though, which is the only reason they work, and is why even high functioning autistic people may have difficulty understanding them, because their “attention” in sensory processing and therefore concept representation is not focused enough and gets processed more as a broad and heavy multimodal concept that is hard to parse because it is taken as literal data.

My point though is that current machine learning models, while they have layers of processing, still are behind in general and broad intelligence because they do not have multiple integrated data types and models consulting with one another to form more generalized notions of concepts. They only have these concepts in particular forms of data by themselves which in turn makes them error prone no matter the data.

I don’t think it is bad data that is the problem as much as missing context to allow the model to understand what is and isn’t bad data

>There are also AI that can improve themselves more than the human given data. The AlphaGo project started off with human Go matches as training data, and evolved into tabula-rasa training by self play. By the end, the AI beats the best human.

Neither AlphaGo Zero or AlphaZero were trained with supervised learning. They were both trained with reinforcement learning (and MCTS, so it's not purely RL, but it's more like RL + planning). It's then not surprising that it can beat humans - its "ground truth" doesn't come from humans anyway.

Let me just post this as an example: https://www.youtube.com/watch?v=kBFMsY5ZP0o

A model trained to detect humans through walls using bouncing wifi signals, based on teacher-student training.

Even though all the original data has been generated by humans, I feel pretty confident in claiming it probably outperforms us at estimating body position from wifi signals.

This obviously transfers to other domains as well- there's certain types of features our brains are wired to pick up(vision, sound, proprioception), some types it is capable of learning(written language, math), and plenty it isn't though to be fair we're pretty bad at most of our learnable tasks- we read and calculate ridiculously slowly, and our ability to break that work down into chunks and memorize intermediate results is the only thing that saves us.

Compared to that, a DNN is a tabula rasa ready to learn how to extract arbitrary features independent of how our cognition of them works, without having to jump through repurposing brain networks to decipher non-native stimuli the way our brains do.

Not all AI is ML.

In closed systems, AI can emulate how the brain is forward-looking.

For example in a game like chess or go, there isnt data to be dissected, rather there is sheer force of compuation.

The monte carlo tree algorithim can simulate games based on the specific state of a board. Humans can do that however to a much lower degree and less deep.

For some tasks it seems hard to even define the question. What would it even mean to have superhuman performance at art?

You should look into some work by Jaron Lanier about this, he sometimes argues that AI is just the theft and encapsulation of human abilities.

Yes, but I'm less sure about language models at the really high level (eg arriving at novel solutions to hard problems through LLMs).

Most ML in practice isn't about doing better than a person, it's about doing it faster and cheaper. Could a human who studied my viewing habits curate better Netflix recommendations for me? Obviously, but Netflix can't afford to do that for everyone and it would take forever.

There's also ML that's not based on data generated by humans. I know we're in the era of LLMs, but that's not all there is

> Wisdom of the Crowd

Something I recently saw mentioned by Ajeya Cotra is to query the LLM by re entering the previous output and asking if its correct, repeat this multiple times, take an average the answers provides a higher level of accuracy than just taking the first answer. (something that sounds weird to me)

Well ok, if viewed from the vantage point that the models are very good at doing certain things and people have not worked out how to correctly prompt/fine tune yet, it's not that weird. It's more that the base level outputs are shockingly good and then someone introduces more secret sauce and makes them even better. The problem with this is there is no saying what the limit to the models that already exist are.

Depends on how your labels, or more generally the target data distribution, is generated. If it’s generated by human subjectivity yea I would agree. However it’s not hard to think of situations where the labels are the output of a physical process or, in the case of prediction, a future event. In these cases the label is not ambiguous and thus not subject to human interpretation. You also have RL systems that play games against each other and reach super human performance that way. Read about AlphaGo or AlphaStar for example.

we can measure art's performance by metrics such as views, likes, sales. If an AI art bot rises the ranks on art platforms we can say they're better right? Especially because they can churn out art all the time. If the art is subpar and the metrics are lacking it can be used as an example of bad art for a next iteration.

>Of course, in certain areas, like speed of computation or accuracy, computers will also be better than humans

I'm very confused what else do you expect it to do ? That's all ML is ; a formalized, structured way of categorizing data. The only reason it's better than human is because it's so much faster that it can't possibly be performed by an human.

>can it truly outperform humans?

Again I'm confused, because if speed isn't the concern, at least accuracy indeniably is. And ML already outperforms humans accuracy-wise.

Wrong analogy.

The calculators are already outperforming humans. I bet you can't compute 567 times 891 in 1 second. They can.

Adding to the chorus, but I would have to say yes. Like you, I have hardly any background in ML, so I'm just going off of what our ML team presented to my team (since we're the ones on the front lines with customers)...

In the real estate world - I think there is honestly just too much data for a human to go through. Our company was estimating how long we think it would take a house to sell, and were training the ML model based on local historical data. Going through all of that data, for what I think was the past 30 years, and then making an estimation for every property? No way a human could keep up. The accuracy compared to a human estimation I can't really speak to, but with how quickly the real estate world moves, speed is the number one here.

I'd be interested to hear other takes on specific industries. Are there any that you guys work in that you feel humans>ML?

I think that actually measures how good it is at getting popular on social media, which is not the same task as making good art.

There's also some backlash against AI art right now, so this might favor models that can't be distinguished from human art rather than models that are better than human art.

Once you have a model that performs as well as a human in some domain, you can then use reinforcement learning to get it to perform better. Of course doing this is easier in some places than others.

What does "truly outperform humans" mean? It sounds so broad, some kind of philosophical question, like "How many angels can dance on the head of a pin?". What are you asking? Can a machine truly outperform a human at climbing, hammering, flying, calculating, sorting, or drawing accurate conclusions in a limited domain given a certain input? Of course, obviously. Can it truly outperform a human at falling in love, tasting an apple, or getting drunk on wine? No.

Humans have always been augmented by their tools. That is one of the fundamental characteristics of being human. At the tasks they're designed for, artificial tools vastly increase the performance of humans, and allows them to outperform what they could do without it. Humans have enhanced their cognitive abilities with all kinds of calculating and "thinking" machines, for millennia. A human with a clay tablet and a reed can far outperform other humans at remembering things. But what is a clay tablet - or a PC, or anything - without humans? Nothing, as far as humans are concerned.

It’s not really fair to have a dog labelled as a Japanese spaniel (that is one) and let a a deep neural network train on a bunch of images of Japanese spaniels for a week, then have me try to identify the dog when I’ve never heard of or seen a Japanese spaniel before or heard or read about them so I guess papillon, then you tell me the CNN is “superior”.

If you consolidated all dog classes into “dog” humans wouldn’t get a single one wrong. Also, if you took an intelligent person and let them study and train on these classes with flashcards for as many training iterations as the CNN has during training, I imagine the human would perform at least comparably if not better than the CNN but that usually is not how the test is performed.

Due to the amount of knowledge that exists and humans limited time and speed to learn there is an upper cap at how much performance a human can achive, yet a computer isnt limited by this in the same way.

Imagine a medical analisys taking into consideration all data from multiple exams and checking every medical study in existance is something humans could never do, simply because the amount of data the exams bring in is far too much and there is far more medical knowleadge already discovered than a human could learn. Hence a machine with all that knowledge would outperform any human doctor. (Obvious problem is building such a machine, but we have already studies of ML doing increadible analysis based on x-ray scans)

One risk is that the human knowleadge migh end reaching a cap since there isnt a incentive to have humans push the limits given that the models are already better than them. (One of the big problems that will likely be caused by image and text generating models, so we will actually have soon the results in a decade or two with how they influenced new artists, teachers and students)

If ML is based on human data, can it outperform humans? Yes. HINT: scale.

{Search, Recommendation, Ads} systems are something we all use today. These ML systems greatly outperform humans. Most of these successful ML systems rely on human-generated data to train. For example, Google looks at what users clicked to train their relevance models. Facebook uses which ads get the most dwell time to learn what types of ads to show next time. Reddit uses user upvote/downvote data and user clicks to learn which posts to boost.

Peter Norvig who ran search quality at Google stated that getting above 80% recall for these systems was quite good [reference: https://qr.ae/pryCgm]. The average human performance on most of these tasks is around 90%. Most of these systems are outperforming humans even though they are not getting high enough recall on individual samples.

Why?

Since these things operate at scale, not every suggestion has to be perfect. The user can ignore the bad suggestions. Furthermore, in the more advanced versions of these systems (i.e. personalized versions of these systems), one could improve recall by simply learning about the user.

Most ML systems that have a defined goal and scale will be able to outperform humans. They will outperform humans even if they use human-generated data. This will only be true if they perform at scale.

A more interesting version of your question would be can we build a single system that can outperform all humans in all tasks? This is the AGI question.

I think with an ability to understand, it would be able to outpace humans. In a sense outperform, so I'd say yes. I find the research being done (I think by Nvidia) on recovering color and shape data from sparse images really interesting. It goes beyond what I'd imagine possible.

There is a youtube channel, TwoMinutePapers. Highly recommend it

Besides the computational speed advantage mentioned by OP and other human-related aspects such as attention span / human fatigue, I think there's a bit more nuance in the answer depending on what "generated by humans" means.

​

Take a binary image classification example, where the task is to classify whether an image is a dog or not a dog. Here, the labels are "generated by a human" by looking at the same input that the algorithm receives. In this case, assuming all labels are correct, I would argue that machine learning is unable to achieve a higher accuracy than the human labeler ignoring the previous factors mentioned, since the "correctness" of the classification was defined by the human itself. If the human was in peak condition and without time constraints, the human labeler should be able to achieve 100% accuracy all the time.

​

On the other hand, suppose the task is to perform time-series prediction, where the label is obtained from the future. Even though the dataset was collected by a human, the "generation process" is not from a human labeling the data but rather by some other process. In this case, machine learning has the potential to outperform humans.

It will outperform sdome humans. I.e: some shitty lawyer who cant go beyond the surface will be surp assed. Those whounderstand subtelties wont have to worry

Id say an ML model can only hope to be as good as its training data.

If the person generating that data just looks through tons of pictures and marks them as apples or not then the model will probably be about as accurate. But if the person choses pictures they’re certain have apples in them or not from some other method (like taking pictures of apples and things that definitely arent apples) then the model may do much better than people can, since its training data is perfect.

Yes. First of all the ai can learn from all humans not just one. And the end result could be an ai with the knowledge and skills of 4 billion people. Which would ofcourse outperform anyone on the planet simply because no person can do it all...

But then ai has also been used already to advance mathematics. Solving previously unsolved mathematical problems.

Here's an example:

https://www.sciencealert.com/ai-is-discovering-patterns-in-pure-mathematics-that-have-never-been-seen-before/amp

It already does. There are many complex tasks which can't even be performed by humans. (e.g. time-series forecasting). The relationships between variables is in a lot of cases so complex that humans simply can't find them on their own. But, we can use ML as a black-box approach to perform these tasks.

Where ML completely blows humans out of the water is when it has some kind of continuous learning scheme. Well known examples are GANs and active learning. Another major technique right now is to use reinforcement learning schemes to improve pre-trained models. By creating another model that gives a preference between two outcomes (original model and finetuned model), a reward can be computed to further improve the model. This is why ChatGPT is so good. Definitely look up Reinforcement Learning from Human Feedback to learn more about this.

I've worked on projects where our ML model significantly outperformed the bulk of our operators. For structured data I've even had simple random forest models that we preferred to our operators, just because the model provided much more consistent decisions.

I think that no-one properly answered your questions. All of the answers above talk about the quantity and how ML model can find milion dogs on milion images per night. But this is obvious and you also mention it in the post. ML models can outperform us in the speed of computation or accuracy as you say, but they can not outperform us in a truly general case as you say. At least not now.

Supervised learning can never do better than humans since, the best ML solution will have the same accuracy as the humans.
We would need something like universal problem solver, which is able to solve more and more complex problems over time and learn from it. But it seems that not so many people are working on this general AI.

>The system's neural networks were initially bootstrapped from human
gameplay expertise. AlphaGo was initially trained to mimic human play by
attempting to match the moves of expert players from recorded
historical games, using a database of around 30 million moves.[21]
Once it had reached a certain degree of proficiency, it was trained
further by being set to play large numbers of games against other
instances of itself, using reinforcement learning to improve its play.

https://en.wikipedia.org/wiki/AlphaGo#Algorithm

Machines doesn’t seem to think rationally and logically as we humans do. I’m not exactly sure about the exact research in this direction, but maybe it is possible to stimulate this rational/logical thinking by set of rules/discrete maths etc? (Like I’ve heard there are many theorem solvers which exists) But again it should be coded by humans right? it’s like expecting machines to be conscious of themselves, which at the current stage they aren’t.

The zero in alpha zero means it starts with no human knowledge. They figures out that this approach is eventually stronger than the base alpha go strategy.

The hand can be used as a blade

But that doesn’t work on a tomato

You make good points, but why do we humans think art is good?

Some of the current art like bananas or toilets could really have been the idea of a bot. Haha, put a banana against a wall and add some im14andthisisdeep level text about existentialism.

I did not look into it tbf, but it's kinda silly a banana on a wall could pay for a handful of students education.

The same for a toilet.. It's just a toilet.

But it's so dumb a human could have prompted an old AI to come up with something and execute it.

have been the idea of a bot. Haha

But the person you responded to didn't talk about the zero variant. Maybe I misread the point of your post?

Good question.

Unfortunately, I have no clue what makes "good" art either. This is a pretty old problem that may not be solvable.

Absolutely. Because it's never just the input from humans- presented with an image and a label for it given by a user, the model is not limited to learning only the relationships that the human used to generate the label- the image is right there, after all. So when all goes well, the model can learn relationships in the data that humans are unable to because the human labels are used to guide learning on the source material.

Additionally, there are many ways to allow a model to treat labels as 100% true (i.e. the word of God) but allow for some incorrect labels. In which case, it's entirely possible for the model to do better than the human(s) did even on the same data.

I think you make an important point. In the end it comes down to the distribution of the data, which always consists of signal and noise. The signal will always be similar, however the noise is random as people can be wrong in all sorts of ways.

The person two comments up was talking about the zero version. Thread is about how AI can surpass humans and the point is they already can if they have a way to improve without human data

Of course something trained on humanS can possibly outperform a human. Is it a mystery to you that a tug of war between humanS on one side and a human on the other will have a known outcome?

Less facetious example. People have created 'averaged' faces (using machine learning) across all men and women and also across men and women of certain races. Is it a surprise that these 'averaged' faces are more attractive than the average person?

Still does not make sense to me as the person before was specifically talking about vanilla. But no point in arguing about any of that i guess.

Makes sense.

An interesting question related to OPs: could there be a ML solution that humans /can't/ understand?

Not /don't/ understand...but I mean given enough time and study a given solution both outperforms humans and is relatively easy to verify but we cannot understand the underlying model at all.

My current belief is that no model is truly beyond human reasoning. But I've seen some results that make me wonder.

Are you trolling me or something? YOU are the person I responded to. YOU brought up the vanilla version, in a response to someone else who was talking about the zero version. The zero version is most relevant here because it learns from scratch, without human knowledge.

>There are also AI that can improve themselves more than the human given data. The AlphaGo project started off with human Go matches as training data, and evolved into tabula-rasa training by self play. By the end, the AI beats the best human.

https://www.reddit.com/r/MachineLearning/comments/103694n/comment/j2ycihi/?utm_source=share&utm_medium=web2x&context=3

​

>YOU brought up the vanilla version, in a response to someone else who was talking about the zero version.

... who responded to someone who talked about the vanilla version. In my first response to you, I did not realize you were not actually the person I responded to in the first place. Apparently you have not read what they responded to, which seems to be the reason you're missing the context.

I assume they must be laughing if they see us still talking about this.

Yes, it can, because it can gather it's own data.

I think it comes down to someone besides the creator saying something is art. In before bots become Art Influencers

If a house is built out of parts of trees, can that house ever stand taller than those trees used to be?

I wonder this about too. Like if there was some alien technology that we cant understand even if we were taught.

Doesn't even have to be alien. Just the familiar seen from a wholly different perspective.

Like whatever these models are doing here: https://scitechdaily.com/artificial-intelligence-discovers-alternative-physics/

If the human errors are random then yes, if there is a consistent pattern that allows the model to learn the "correct" errors then that can cause problems

Sounds like a god of the gaps argument. Machines may not be better at humans for any given task today, but it seems to be moving pretty quickly in that direction. With that said, it seems humans become a lot smarter with the aid of machines and vice verse.

That is the million dollar question. A lot of clever people seem to be finding new ways all the time. I think that, at this point, it is safe to say that any task that has sufficient relevant data probably can be modeled and subject to ML. I might not be able to figure out how, but I am sure someone could.