Preheat oven to 350 degrees F (175 degrees C). Line 24 muffin cups with paper liners.
Beat cake mix, 1 1/4 cups eggnog, eggs, 1/2 teaspoon nutmeg, and 1/2 teaspoon vanilla extract in a bowl using an electric mixer on low until batter is blended. Increase speed to medium and beat for 2 more minutes. Spoon batter into the prepared muffin cups.
Bake in the preheated oven until a toothpick inserted in the center of a cupcake comes out clean, about 18 to 20 minutes. Transfer cupcakes to a wire rack to cool completely.
Beat butter and cream cheese together in a bowl using an electric mixer until smooth and creamy. Gradually beat confectioners' sugar, alternating with small amounts of 1/4 cup eggnog and ending with confectioners' sugar, into butter mixture using an electric mixer on low speed until icing is smooth and fluffy. Beat 1/2 teaspoon vanilla extract and 1/2 teaspoon nutmeg into icing until incorporated; spread over cooled cupcakes.
