Not the first to wonder. Vincent Courtillot (of fame by his work on the volcanic aspect of the Cretaceous extinction) at the Institut de Physique du Globe de Paris has done with colleagues a very readable overview paper.

ScienceDirect - Earth and Planetary Science Letters : Are there connections between the Earth's magnetic field and climate?

The most intriguing feature may be the recently proposed archeomagnetic jerks, i.e. fairly abrupt ( 100 yr long) geomagnetic field variations found at irregular intervals over the past few millennia, using the archeological record from Europe to the Middle East. These seem to correlate with significant climatic events in the eastern North Atlantic region. A proposed mechanism involves variations in the geometry of the geomagnetic field (f.i. tilt of the dipole to lower latitudes), resulting in enhanced cosmic-ray induced nucleation of clouds. No forcing factor, be it changes in CO2 concentration in the atmosphere or changes in cosmic ray flux modulated by solar activity and geomagnetism, or possibly other factors, can at present be neglected or shown to be the overwhelming single driver of climate change in past centuries. Intensive data acquisition is required to further probe indications that the Earth's and Sun's magnetic fields may have significant bearing on climate change at certain time scales.