Submitted by enryu42 t3_122ppu0 in MachineLearning
https://medium.com/@enryu9000/gpt4-and-coding-problems-8fbf04fa8134
Apparently it cannot solve coding problems which require any amount of thinking. LeetCode examples were most likely data leakage.
Such drastic gap between MMLU performance and end-to-end coding is somewhat surprising. <sarcasm>Looks like AGI is not here yet.</sarcasm> Thoughts?
liqui_date_me t1_jdr8516 wrote
This comment about GPT-4’s limited abilities in solving arithmetic was particularly interesting: https://www.reddit.com/r/singularity/comments/122ilav/why_is_maths_so_hard_for_llms/jdqsh5c/?utm_source=share&utm_medium=ios_app&utm_name=iossmf&context=3
Controversial take: GPT-4 is probably good for anything that needs lots of boilerplate code or text, like ingesting a book and writing an essay, or drafting rental contracts. There’s a lot of value in making that area of the economy more efficient for sure.
But for some of the more creative stuff it’s probably not as powerful and might actually hinder productivity. It still makes mistakes and programmers are going to have to go and fix those mistake’s retroactively.