Other than the physics (chemical storage vs. electric-field storage) one huge difference is they have different charge-vs-voltage curves.

The voltage across a capacitor is proportional to how much charge has flowed through it (it's the integral of the current). This simple mathematical relationship is vital for all sorts of analogue circuitry (and all circuitry is analogue eventually).

A battery, on the other hand, has a relatively constant voltage for most of its lifetime. A common 1.5v alkaline cell will only gradually droop to 1v or so before suddenly dropping off. NiMH batteries are very flat around 1.2v for most of their discharge. The voltage is determined by their chemistry, and is strongly affected by all sorts of real-world stuff like temperature and diffusion rates. For rechargeable batteries the relationship between state-of-charge and terminal voltage can be quite complex.

Of course real-world capacitors do have a lot of non-ideal behavior, like leakage, ESR, dielectric absorption, and microphonics, but even with all that they're a lot closer to a simple "voltage × capacitance = summed charge" relationship than a battery is.