Preheat oven to 350 degrees F (175 degrees C). Grease 9x13-inch baking dish.
Whisk eggs, milk, brown sugar, butter, vanilla extract, ground cinnamon, and ground nutmeg together in a bowl. Stir bread cubes and bananas into egg mixture until coated; pour into prepared baking dish.
Bake in preheated oven until a knife inserted in center comes out clean, 35 to 40 minutes.