In a saucepan over medium heat, combine brown sugar, white sugar and milk. Bring to a boil. Stir in butter, pecans and vanilla. Heat, without stirring, to between 234 and 240 degrees F (112 to 116 degrees C), or until a small amount of syrup dropped into cold water forms a soft ball that flattens when removed from the water and placed on a flat surface. Remove from heat and let cool 5 minutes.
Beat until thickened, then pour immediately onto prepared surface and let rest until firm and completely cool before cutting.