Preheat oven to 350 degrees F (175 degrees C). Lightly grease an 8 inch square baking dish.
Place garlic in small ovenproof bowl, and drizzle with olive oil. Roast for 30 minutes, or until very soft. Cool and peel the garlic, and reserve the oil.
Boil potatoes and yams in a large pot of salted water until tender, about 20 minutes. Drain, reserving 1 cup liquid.
Place potatoes and yams in a large bowl with milk, butter, rosemary, garlic, and reserved olive oil. Mash to desired consistency, adding reserved cooking liquid as needed. Mix in 1/4 cup cheese. Season with salt and pepper to taste. Transfer to the prepared baking dish. Sprinkle with remaining cheese.
Bake until heated through and golden on top, about 45 minutes.
