A feature that was tricky in AE CS5 is much easier in CS6. Select the Rectangle tool and draw the rectangle mask inside the frame.
Click and hold on the Pen tool to reveal the Mask Feather Tool or press G twice. Click on a corner and drag out to create a feathered mask.
Click on any corner to reset that area of the mask to a hard edge or click and hold on the frame and drag out to add extra control points.