Place bacon in a large skillet and cook over medium-high heat, turning occasionally, until evenly browned, about 10 minutes. Drain bacon slices on paper towels.
Preheat oven to 350 degrees F (175 degrees C).
Scatter about 1/3 of the butter pats onto the bottom of a 9x13-inch baking dish and arrange 6 slices of bread in a single layer, overlapping as necessary, on top of the butter. Top bread slices with bacon slices.
Sprinkle 1 1/2 teaspoons cinnamon, 1 1/2 teaspoons white sugar, and 1 1/2 teaspoons brown sugar, and nutmeg over bacon slices. Scatter 1/3 of the butter pats over bacon and top with remaining bread slices.
Whisk milk, eggs, remaining cinnamon, remaining white sugar, remaining brown sugar, and vanilla extract together in a bowl. Pour over bread mixture into baking dish. Scatter remaining butter pats over the top.
Bake in the preheated oven until golden brown and set, 35 to 45 minutes. Cool for 10 minutes before serving.
