Put graham cracker crumbs, brown sugar, white sugar, butter and milk into a saucepan and bring it to a boil. Boil for 5 minutes.
Put a layer of buttery round crackers in a 9x13 inch pan and pour 1/2 of the graham cracker crumb mixture over it. Put another layer of crackers and pour the remaining mixture. Add last row of crackers.
To Make Topping: Melt over low heat the butterscotch chips, chocolate chips, and peanut butter. Melt at low heat and spread over the top.
