Preheat oven to 350 degrees F (175 degrees C). Line 12 muffin cups with paper liners.
Whisk flour, sugar, cocoa powder, baking soda, baking powder, and salt together in a large bowl; stir in warm water, eggs, milk, butter, sour cream, and vanilla extract until batter is smooth. Fill each muffin cup halfway with batter.
Bake in the preheated oven until a toothpick inserted in the center of a cupcake comes out clean, 15 to 18 minutes. Cool cupcakes for about 5 minutes.
Turn on the oven's broiler.
Place 1 marshmallow vertically in the center of each cupcake.
Place cupcakes back into oven and broil, watching the whole time, until marshmallows are golden brown and slightly puffed, 10 to 30 seconds. Press the marshmallows into the cupcakes to cover the whole top using the back of a spoon.
