Preheat oven to 350 degrees F (175 degrees C). Grease and flour a 9x5-inch loaf pan.
Mix brown sugar and cinnamon together in a bowl.
Beat white sugar and butter together in a bowl using an electric mixer until smooth and creamy. Beat in eggs, 1 at a time, until incorporated; add vanilla extract.
Combine flour and baking powder together in a bowl; stir into creamed butter mixture. Mix milk into batter until smooth. Pour half the batter into the prepared loaf pan; add half the apples and half the brown sugar mixture. Lightly pat apple mixture into batter. Pour the remaining batter over apple layer; top with remaining apples and brown sugar mixture. Lightly pat apples into batter; swirl brown sugar mixture through apples using a finger or spoon.
Bake in the preheated oven until a toothpick inserted in the center of the loaf comes out clean, 30 to 40 minutes.
