OP is trying to make a point, but doesn’t know how to properly convey that point. First his title references “search index” but then switches to “search engines” and further tries to dumb the process down to “hyperlink crawling”. These are specific terms with different uses. That’s why everyone is jumping down his throat instantly.

OP is also taking A.I. composition at its current face value and not looking even a few years ahead. Even at its current state, A.I. can be hyper focused to generate vastly different results. OP just doesn’t understand how to use it properly.

Example, if you wanted to bake cookies in 1980, you would generally open a cookbook (maybe you had 2 or 3 and maybe a box of index cards). You would be presented with the same recipe and create the same cookies, because you limited your recipes to only your kitchen. With the internet, now you have access to millions of cookie recipes. OP is stuck thinking that A.I. is only using 1 recipe. Just as with most internet searching, you need to ask properly or else be given canned results.

“Give me chocolate chip cookie recipe from Betty Crocker 1985, substitute sugar for Truvia”. Bam…A.I. just showed you the tip of the iceberg. Google will show you separate results for the individual keywords and make you go down the rabbit hole. A.I. is basically asking someone else to Google search a bunch of stuff for you and compile a proper response…in seconds.