But the actual, math-laden practice of physics? Taking measurements and plugging them into equations? Bleah.
Yet, the really advanced stuff cannot really be properly understood any other way, as it is so beyond the normal realm of perception or experience. Quantum stuff is just strange. I can listen to explanations of it, but I do so in a manner that I might listen to a creation myth - I can see sub-atomic particles no better than I can see gods, and although one most definitely has a very important and profound influence on my very existence and on the existence of all matter, those actions are as completely invisible and imperceptible to the unaided eye as are the acts of mythological entities.
Now, if I were the sort of person who really got math, I could actually come to understand advanced physics on a real level, or at least as well as anyone out there does. But I'm not, and I'm hardly alone. It will always be something that I just have to take on faith.
Whether this is the cause or the effect of my disinterest I cannot say, but in any case, the problems of physics are not the ones that really interest me. I feel just fine taking the workings of the physical world for granted. What interests me is society, and how social groups work. Given that physical reality is, at the level I can percieve, pretty much the same for me as it is for anyone else on Earth, it's more the background against which we work than the substance of the problem that interests me.
If I may offer a suggestion, the next time you're in a bookstore see if they have a copy of Morris Kline's Mathematics for the Non-Mathematician, thumb through it, and see if floats your boat. It's a history of math from a Humanities POV. It was written as a textbook for college students who had primary interests other than math/techie/science. At 500 pages it's a bit of a slog but there's no reason not to skip around to find the bits one finds interesting.