Heat olive oil in a cast iron skillet over medium heat until sizzling but not smoking. Stir in garlic. Add chanterelles and butter. Reduce heat to medium-low and cook, stirring frequently, until most of the excess liquid evaporates, about 10 minutes. Season with salt and pepper.
Increase heat to high and carefully pour port into the skillet; stir until reduced, about 2 minutes. Serve steaming hot.