Melt 1/4 cup butter or margarine in an 8-inch square pan. Sprinkle with brown sugar and nutmeg. Arrange peach halves, cut side down, in pan.
In a large bowl, cream the butter and sugar until light and fluffy. Beat in egg. Stir together flour, baking powder and salt. Add flour mixture to creamed mixture alternately with milk, beating well after each addition. Spread batter over peaches.
Bake in preheated oven until lightly browned on top, 35 to 40 minutes. Remove cake from oven, and let stand in pan for 5 minutes; invert onto serving platter.