Neurons don't mean shit in isolation; what matters is neural connections (AKA synapses). When you're a tiny baby, first you make a ton of synapses, and then you get rid of a huge number of unused ones and make the most useful synapses efficient and well-connected; this "efficient, well-connected" structure is why you're smarter than a two year old (as opposed to more sensible or knowledgeable).

Think about your brain like a computer: a laptop model having a few percent more space to install a big CPU is okay, but the quality of the code it runs will have vastly more say in how fast the computer actually is.