Preheat an oven to 350 degrees F (175 degrees C). Grease a 9-inch square baking pan.
Mix 1/2 cup flour, brown sugar, 1/4 cup butter, and cinnamon together in a bowl until the mixture is the consistency of wet sand. Set the topping aside.
Beat 1/2 cup butter and sugar with an electric mixer in a large mixing bowl until light and fluffy. The mixture should be noticeably lighter in color. Add the egg and mix well. Stir in the vanilla extract. In a separate bowl, combine 2 cups flour, baking powder, baking soda, salt, and nutmeg.
Mix the flour mixture into the creamed butter mixture alternately with the buttermilk, stirring just to combine. After the last of the flour mixture is incorporated, gently fold in the diced mango and raspberries. Spread the batter into the prepared pan (the batter will be thick). Sprinkle with crumb topping.
Bake in the preheated oven until a toothpick inserted into the center comes out clean, about 50 minutes. Cool on a wire rack.
