Mathematical physics is best described as the intersection between mathematics and physics. This intersection is quite large but the compliment is non-empty. Here are two quotes, one of which talks exclusively about physics while the other talks exclusively about mathematics:

• We never experiment with just one electron or atom or (small) molecule. In thought-experiments we sometimes assume that we do; this invariably entails ridiculous consequences.

  Erwin Schrodinger
  1952

• [after proving Euler's formula in a lecture] ... we cannot understand it, and we don't know what it means. But we have proved it, and therefore we know it is the truth.

  Quoted in E Kasner and J Newman Mathematics and the Imagination (New York 1940)

Goals

The goal of a mathematician is to be correct. They do this using proof. A proof consists of three parts, the assumptions, the logical arrows, and the destination. Once the mathematician has provided the proof, they may state "P implies Q" and give themselves a round of applause because they have finally said something which they are certain of is "correct".

• Logic is invincible because in order to combat logic it is necessary to use logic.

  Pierre Boutroux

The goal of a physicist is to understand the universe. Currently, their main method of accomplishing this is by using mathematics. Physicists are not always interested in saying things which are correct, especially if they do not believe it will help them gain a deeper understanding of the universe.

• It all works because Avogadro's number is closer to infinity than to 10.

  Ralph Baierlein

A mathematical physicist is someone whose goals happen to align with both that of the physicist and the mathematician.

On the Process

My research in mathematical physics is best described as "motivated by physics, performed using mathematics". Mathematical physics is about rigorously answering questions about the universe, which first requires one to formulate the question in a mathematical framework. Formulating the question in the framework of mathematics is considered pure physics. After one solves the problem it becomes a job for the physicist to analyze the results and see what it says about the universe. What defines the mathematical vs theoretical physicist is how they go about the middle step, solving the purely mathematical problem.

• In a world in which the price of calculation continues to decrease rapidly, but the price of theorem proving continues to hold steady or increase, elementary economics indicates that we ought to spend a larger and larger fraction of our time on calculation.

  John Tukey

Most american mathematical physicists are found in math departments. This is not because mathematical physicists do not participate in the first and third steps, they too indulge in pure physics. But the middle step is what occupies most of their time and so they are mathematicians. In addition, Mathematical physicists often find themselves trying to clean up the arguments made by physicists.

• [Dirac] required a function which was zero everywhere, except at a single point, where it was discontinuous and behaved like an infinitely high, infinitely narrow spike of unit area. Mathematicians were quick to point out that, strictly speaking, there is no function which has these properties. But Dirac supposed there was, and proceeded to use it so successfully that a new branch of mathematics was developed in order to justify its use.

  B.R Krusse

Ontology

A statement is defined by its assumptions as much as its contents. The ontology of any physical theory is what determines the axioms that a mathematical physicist uses in their analysis. All statements about the universe should follow from these axioms. If the ontology is unclear then the physical theory will be too.

• Try to get your understanding of the physics to the same level of understanding that you have for the mathematics. If you can't do that, then it's probably a problem with the physics and not you.

  Sheldon Goldstein

Here are some examples of physical theories which have clear ontologies.

Newtonian mechanics is a classical theory of physics in which point particles interact via forces. We study Newtonian mechanics by translating this ontology into a purely mathematical framework. The positions of particles are vector quantities which evolve according to second order differential equations.

Maxwell-Lorentz Electrodynamics is a relativistic theory in which point particles act as singularities in electromagnetic fields which are defined over Minkowski space-time. I have purposefully excluded the fact that the electromagnetic field governs the motion of its singularities because this law, which is described via the Lorentz force, is not actually well-defined. The ontology of this theory is clear, but it is not consistent.

General Relativity is a relativistic theory in which energy acts as a source for the curvature of space-time. The ontology of this theory is clear, and is (ish) consistent for energy of nice "regularity". Like vacuum.

Physics as a Foundation

None of these physical theories describe the universe we live in. Regardless of how well they happen to predict the outcomes of experiments, it is important not to forget this. Some people argue that this is perfectly fine, in fact most physicists are not interested in the holy "theory of everything". Some go a step further, and claim that they aren't even interested in having a physical theory which has a clear ontology, so long as it is able to correctly predict the outcomes of their desired experiments.

• There will always be voices (even from otherwise reasonable people) insisting that it's simply not the role and purpose of physics to provide a coherent and objective description of nature.

  Detleff Durr and Dustin Lazarovici

This is common sentiment among those who study quantum mechanics. Luckily, the history of quantum mechanics also provides us with the perfect counterargument against this line of reasoning. The experiments which we so desperately want to predict the results of are a part of the universe they exist in. If we don't have a coherent description of this universe, we cannot possibly have a coherent description of the experiments which live in it. Confusion regarding the definitions of experiments will inevitably arise, as is evidenced by the measurement problem of quantum mechanics.

• Are billions upon billions of acts of observer-participancy the foundation of everything? The very fact that we can ask such a strange question shows how uncertain we are about the deeper foundations of the quantum and its ultimate foundation.

  John Wheeler

What if the universe is fundamentally inconsistent?

Then I wouldn't want to study it. At the same time, I suppose I would want to study it. Which statement is true? Both.
If you see a problem with this then you likely agree with my belief that an inconsistent universe isn't worth studying.

• Why go on? And you only go on if the game is worth the gamble. Now the universe has been going on for an incredible long time. And so really, a satisfactory theory of the universe has to be one that's worth betting on... If you make a theory of the universe which isn't worth betting on, why bother? Just [quit].

  Alan Watts