Lightly grease a 9x13 inch baking dish. In a medium bowl, whisk together the Alfredo sauce and milk. Spread 1/4 cup of the sauce in the bottom of the baking dish. Then layer 1/3 of the potatoes over the sauce in the dish. Sprinkle with 1 tablespoon of Parmesan cheese and salt and pepper to taste.
In a separate medium bowl, combine the ham, broccoli and 1 1/2 cups of the Swiss cheese. Mix well and spread 1/3 of this mixture over the potatoes in the baking dish. Then top with another layer of potatoes, followed by the ham mixture, finally topping all with the remaining Swiss cheese and Parmesan cheese. Pour the remaining Alfredo sauce over all.
Cover and bake at 400 degrees F (200 degrees C) for 45 minutes, then uncover and bake at 350 degrees F (175 degree C) for additional 25 minutes or until potatoes are tender. Let stand 10 to 15 minutes before serving.