victotronics
victotronics t1_ire6ha5 wrote
Reply to comment by emotionalfool123 in [R] Discovering Faster Matrix Multiplication Algorithms With Reinforcement Learning by EducationalCicada
Behavior under roundoff. Floating point numbers are not actually mathematical numbers so all algorithms are inexact. You want them to be not too inexact: small perturbations should give only small errors. The fact that STrassen (and other algorithms) sometimes subtract quantities means that you can have numerical cancellation.
victotronics t1_ir9fc44 wrote
Reply to comment by Ulfgardleo in [R] Discovering Faster Matrix Multiplication Algorithms With Reinforcement Learning by EducationalCicada
Yes yes and yes. Can half a dozen authors really be that ignorant that they don't know about all the work that's been done after Strassen? And how did this pass review?
To add to 2: numerical stability of Strassen is doubtful too.
victotronics t1_ir9f2a0 wrote
Reply to [R] Discovering Faster Matrix Multiplication Algorithms With Reinforcement Learning by EducationalCicada
Ok, I'm no expert but
> improves on Strassen’s two-level algorithm for the first time, to our knowledge, since its discovery 50 years ago
looks very suspicious. There has been *tons* of work on improving Strassen. It would be mind-blowing if they didn't know about that research.
Then: Strassen and its further developments are theoretical curiosities. Numerically they suffer from grave instabilities.
This stuff should really be posted in r/math.
victotronics t1_ix5g254 wrote
Reply to [D] Why do we train language models with next word prediction instead of some kind of reinforcement learning-like setup? by blazejd
Children pick up on rules and then extrapolate them. "He bringed this to me". I don't think an AI will generate that since it has no general rules that it tries to apply to a special case.