You are correct at an extremely high level that the differences in fracture stress and strain between materials (and indeed even between samples of materials) are related to the strength of the atomic bonds within the materials and the crystalline structure and microstructure.

However, that is not an accurate description of what happens to a metal under strain. First, it entirely ignores the possibility (and actual behavior) of elastic strain. Second, even confining ourselves to plastic strain, it's a fundamentally incorrect description of what happens.

It is true that the most common mechanism by which plastic strain occurs is through the movement of dislocations, defects in the crystal structure of the material. But it's not generally true that the strain is taken up by the destruction of dislocations. In fact, it's the exact opposite. When just enough shear stress is applied to begin shifting atoms relative to each other within the crystal structure, existing dislocation patterns within the material begin producing additional dislocations under the applied shear stress. These dislocations entail, by definition, the local shifting of one plane of the crystal relative to another. So when dislocations are generated under shear stress, they accommodate that shear stress by allowing the material to shift in a local way. It is the fact that dislocation generation allows local relief of stress which explains why dislocations are preferentially generated. In order for the crystal to deform along an entire slip plane, all of the atoms must move at once. A dislocation entails the movement of a much smaller number of atoms, on the order of hundreds to thousands.

But the dislocations themselves impose a stress field around them which impedes the movement of other dislocations. So in order for these dislocation sources to produce additional dislocations, they must be subjected to higher stresses. This explains the phenomenon of work hardening, which is present in every metal. If you stress a metal adequately to deform it plastically, additional plastic deformation requires you to exceed that stress in the future, absent any intervening processes which allow the dislocations to heal.

If it were the destruction of dislocations that was responsible for plastic information, metals would actually get softer as they were worked. This is because dislocations disappearing reduces the amount of stress required for those Frank Read sources to generate new ones. And there will always be dislocations present in a crystal at a temperature greater than 0K because it is entropically favored.

If we ignore this effect, and just concentrate on what happens once you have a single crystal without dislocations, under your theory, materials would get far stronger than we can make them at a macro scale today. If there really are no dislocations in a crystal, the amount of stress required to plastically deform the crystal is the amount of stress required to move an entire layer of atoms at once. The amount of stress required to do this is only about an order of magnitude less than the actual stiffness of the material. For a generic steel, this would imply that the fracture stress would be about 20 GPa. But we can't make steels that are stronger than about 1/10 of that, no matter how much we work them.