Beat milk and instant pudding mix together with a whisk or electric mixer until pudding mix is completely dissolved, about 2 minutes. Refrigerate until slightly set, at least 5 minutes.
Stir 1/2 of the pudding and peanut butter together in a bowl until peanut butter is evenly distributed.
Melt chocolate chips in a microwave-safe glass or ceramic bowl in 30-second intervals, stirring after each melting, for 1 to 3 minutes (depending on your microwave). Stir remaining 1/2 of the pudding and melted chocolate together in a bowl.
Spray 7 ice-pop molds with cooking spray. Pour crushed graham crackers into molds and shake to coat all surfaces in the mold. Drop 1 spoonful peanut butter-pudding mixture into each mold; top with 1 spoonful chocolate pudding. Repeat layering until molds are full. Top with ice-pop top or stick and freeze until frozen, at least 4 hours.