Roast whole yams in preheated oven until heated through but not quite tender, 35 to 40 minutes. Allow to cool.
Reduce oven heat to 350 degrees F (175 degrees C).
Melt butter in a large pot over medium heat. Stir brown sugar, orange juice, and nutmeg with the melted butter to dissolve the sugar completely; bring to a boil, reduce heat to low, and cook until the liquid thickens into a syrup, 10 to 15 minutes.
Peel cooled yams. Cut yams into bite-size chunks; arrange in a single layer in a 13x9-inch baking dish. Drizzle syrup over the yams.
Bake in the preheated oven, twice opening the oven to spoon syrup over the yams, until the chunks are fork-tender, about 35 minutes. Arrange marshmallows over the yams and continue baking until the marshmallows are puffy and browned, about 10 minutes more.
