We run into this 2 or 3 times a week in our office. Best thing I have found it do straight up admit that you understand that you would not consider it surgery in a typical sense, however it does not go by whether it is surgery or not. The CPT book is broken up into many section like radiology, labs, e/m. There is also a section for surgery. Within this section the codes that are used to treat all fractures are located. To make it easy they normally put the codes in order of treatment without manipulation, treatment with manipulation, and typically ORIF of the fracture. The explanation of benefits received by the patient is not stating that the patient had surgery, rather instead of giving the code to the patient, the "surgery" that the EOB shows is just the section of the CPT book to which the code is located. That will usually stop the patient from thinking that they were charged for a surgery that they did not have.
From a cost justification standpoint it really depends on what your practice uses for a fee schedule. If you use something similar to FairHealth, depending on your zipcode, they are most likely going to be upset no matter what. Sometimes the best thing to say is that you do not determine the fee schedule. I have never seen a patient go to one of our doctor's and ask why they charge so much for this. They will ask billing staff yes, but never the doctor.
The toughest I have found is when the patient states "the patient just examined me, he didn't do anything" like you stated in your post. I just try explain to the patient that this is the code that is billed in order for the doctor to make a determination that it is in the best interest to not perform a "real surgery" or manipulation in order for the fracture to heal properly. If you are in network I would inform them that it is their insurance company, not you, that is determining that this is the going rate for this code. Just like we are not suppose to over code we are also not suppose to under code.
I have just come to accept that these phone calls are usually 10 - 20 minutes of torture. These calls always feel like a lose/lose whenever I get off the phone. Hope that helps a little bit though.