Preheat the oven to 400 degrees F (200 degrees C).
Mix oats, flour, baking powder, and salt together in a bowl. Fold in butter. Mix in milk until dough comes together in a ball.
Divide dough into thirds. Roll out very thinly on a floured surface using a Swedish or notched rolling pin. Cut into 2-inch pieces and place on a baking sheet.
Bake crackers until tops are just very slightly golden, 10 to 15 minutes. Cool on a wire rack.