Preheat oven to 425 degrees F (220 degrees C). Line a 9-inch pie plate with pastry.
Whisk sugar, flour, cinnamon, and nutmeg together in a small bowl; sprinkle 1/4 cup of sugar mixture over pastry in pie plate.
Stir apples, peaches, pears, plums, and ginger together in a bowl; sprinkle with lemon juice and stir to distribute. Place fruit mixture in pie plate; sprinkle with remaining sugar mixture. Cover with pastry, pinch to seal top and bottom crusts together, and cut a few slits in top pastry with a sharp knife. Alternatively, cover with a lattice crust.
Bake in the preheated oven for 15 minutes; reduce temperature to 350 degrees F (180 degrees C) and continue baking until crust is golden and filling has thickened, about 35 minutes. Cool completely before serving.
