When in a Teams Meeting, Please click the “More Actions” button shown here:
Then click “Show Device Settings” in the menu that pops up:
When your headphones are plugged in, change the “Speaker” setting to “Headphones” instead of “Speakers”: