Per the CPT E/M Guidelines, Time is considered to be the controlling factor for the code selection of an Office or Outpatient Evaluation and Management code when counseling and/or coordination of care dominates (more than 50%) the face-to-face portion of the encounter. This face-to-face time (or intraservice time) includes time spent with the patient and/or family performing such tasks as obtaining a history, performing an examination, and counseling the patient.
Your question does have some merit. Here's an excerpt from the Evaluation and Management Guidelines from CPT: "It should be recognized that the specific times expressed in the visit code descriptors are averages and, therefore, represent a range of times that may be higher or lower depending on actual clinical circumstances."
Because CPT identifies the times as averages, that represent time ranges and not specific benchmarks, it is understood that there can be some variability. You could have an established patient that takes 20 minutes, and Time dominates the encounter, and you can use your discretion as to whether that should be coded as a level 3 (which requires 15 minutes) or a level 4 (which requires 20 minutes.) In terms of rounding up or rounding down, it is permissible - but be consistent. If you as a coder or auditor are conservative and you generally round down unless the time is extremely close, that's fine. Just have a defined policy for your office, include it in the compliance policy, and make sure everyone is on the same page. That's the most important thing.