The most difficult objects to mask are those with soft, undefined edges. This is a common issue in photos snapped using a wide aperture (or too close to the object), resulting in an increasingly soft focus as the object extends out of the lens’ focal range. Accommodating this variance in focus can be a challenge. To help get your masks just right, I’ll show you two advanced masking techniques for Adobe Photoshop CS4 and CS5.
Prepare your mask
You’ll need to do a bit of prep work before executing the steps below. Open a photo and create a mask using the techniques outlined in an Introduction to Masking Parts and Advanced Masking: Vector Masks. In areas where the object veers out of focus, make sure the mask’s edge runs comfortably inside the object’s blurry edge.
Once you’ve completed the basic mask, you’ll have to arrange the document’s layers so that you can correctly follow the steps. The topmost layer should contain the object photo and the initial mask, and below it, a color fill layer that will help you see the mask. You must convert any vector masks to raster masks before continuing. Control-click the mask thumbnail and choose Rasterize Mask from the contextual menu.
Technique 1: Use the Blur Tool
This technique can be used to selectively soften the edges of your mask. It’s an effective way to deal with objects that have multiple levels of blurring (i.e., a watering can whose spout and handle extend out of focus in opposite directions).
Choose the Blur Tool.
In the Options Bar, click the Brush Preset picker and set the brush size to 100 pixels with a hardness of 0 percent. This means that the blur is applied most densely at the center of the brush and barely at all at the edge, so that the transition between blurred and non-blurred areas of the mask is smooth.
In the Options Bar, set the Strength to 100 percent.
Go to Layers (Windows -> Layers) and click the layer mask thumbnail, which you’ll see to the right of the layer thumbnail. This allows you to make edits directly to the mask without having to enter Quick Mask mode or switch to Channels.
Use the Blur Tool to paint over the edges of the object that require a softer focus. You’ll notice that the mask’s edges get blurrier with each stroke. Apply blur more generously as you move away from the focal point, and in a few minutes, you’ll have a reasonably accurate mask.
Technique 2: Use Lens Blur
This technique can approximate the natural blur of the camera, making it a better choice for objects with more pronounced blur. It’s best used on objects with simple shapes (i.e., a chocolate bar) that slip out of focus in only one direction.
Double-click the Quick Mask Mode icon in the tool panel to call up the Quick Mask Options and ensure that Color Indicates: Selected Areas is chosen. Click OK to continue.
Enter Quick Mask Mode (Hit Q on the keyboard).
Choose the Gradient Tool.
In the Options Bar, click the Gradient picker and then choose the gradient in the top left (Foreground to Background). Ensure that the current foreground color is white and the current background color is black.
Using the Gradient Tool, click the canvas where the photo is most in focus, and then drag the cursor to where it’s least in focus. As soon as you let go, a red gradient will appear over the canvas.
Return to Standard Mode (Hit Q on the keyboard).
Go to Layers (Windows -> Layers) and click the layer mask thumbnail, which you’ll see to the right of the layer thumbnail. Once again, this allows you to make edits directly to the mask without having to enter Quick Mask mode or flip over to Channels.
Choose Filters -> Blur -> Lens Blur. On the right of the filter interface, you’ll see a smattering of options. The only one you should concern yourself with is Radius (under Iris). As you drag the slider from left to right, you’ll see the mask get progressively blurrier along the gradient you just added. When you think you’ve approximated the actual focal dropoff in the photo, click OK. The results are immediate; if you don’t like what you see, back up and try again.
Finishing up
After applying either of these techniques, you might find that the object has a hint of an aura when overlaid on darker colors. This aura can be curtailed quite easily in Adobe Photoshop CS5.
Go to Layers (Windows -> Layers) and click the layer mask thumbnail.
Choose Select -> Refine Edge. A window offering refinement options will appear, and you’ll need to tweak two details. First, enter a figure of -2 percent in Shift Edge.
Check the option to Decontaminate Colors and adjust the slider to 75 percent. All other fields should remain 0. Click OK to lock in the changes. (Note that this action creates a duplicate layer because it modifies the original photo.) The mask should be nearly flawless now; to finalize it, just control-click the layer mask thumbnail and choose Apply Mask.
Here again, the more you apply the Blur Tool, the softer the mask’s edges.
[ Chris McVeigh, a frequent contributor to Macworld, is an author, illustrator, and toy photographer in Halifax, Nova Scotia.]