The threshold is that there's less than X% chance this occurred due to random events unless we mucked something up

Replication is critical for showing that something weird actually is happening and there isn't a quirk in the setup.

If you're testing say a sweet electrically powered EmDrive that could be used on spaceships and your test measures something well above the noise floor of the system then the drive works! Right? Mmmm but what if those big ol' power cables happen to be interacting with the Earth's magnetic field? Whoops! Drive is garbage, and the setup had a parameter that wasn't accounted for!

Or maybe you measure some neutrinos traveling faster than the speed of light! But you forgot to account for the time to sync the clock on the surface with the one underground resulting in all of your time measurements on just one end being offset

Just because you can do something once doesn't mean you did what you intended to do. There are plenty of experiments that have produced the desired result because that's what people wanted to happen but were unintentionally setup incorrectly or had quirks which made it look like the desired result was real even though it was something unrelated