Preheat oven to 350 degrees F (175 degrees C). Butter a 12-inch oven-proof skillet.
Beat 1/2 cup butter and white sugar together in a large bowl using an electric mixer until light and fluffy. Add eggs, 1 at a time, beating well after each addition. Beat in vanilla extract.
Whisk 2 cups flour, baking powder, cinnamon, ginger, and salt together in a separate bowl. Add to the butter mixture alternately with 1 1/2 cups buttermilk, beating well after each addition. Pour batter into the prepared skillet.
Mix brown sugar and 1/2 cup flour together in a small bowl. Cut in cold butter with 2 knives or a pastry blender until mixture is crumbly. Stir in pecans and oats; sprinkle over the batter in the skillet. Top with apples.
Bake in the preheated oven until apples are golden brown, 60 to 70 minutes. Let cool in pan or a wire rack.
Combine caramels and 1 tablespoon buttermilk in a microwave-safe bowl. Microwave until caramels are melted, 30 seconds at a time, stirring after each melting, for 3 to 6 minutes total. Stir until smooth; drizzle over the cake. Let stand until set, about 20 minutes.
