Master Photoshop's Masking Tools

Expert Techniques to Simplify Making Selections

by David Blatner, Macworld.com

Anyone who's ever painted a room or a house knows that masking tape is an essential tool for determining what gets paint on it and what doesn't: put tape on a windowpane and paint the frame, pull the tape off and the glass is paint free. Adobe Photoshop also comes bundled with a roll of masking tape–digital masking tape, of course–hidden in three different features: selections, alpha channels, and masks.

Digital masking tape is better than real masking tape–you just need to learn how to manage and manipulate it efficiently. Someone could write an entire book about the hundreds of tips and techniques for making selections (which determine where the masking tape is placed), but it all really comes down to one thing: getting a good transition between what's selected and what isn't.

To borrow a phrase from the folks at Ultimatte: "There are no edges . . . only transitions." A clumsy transition results in edge spill: colors from the background spill over into the masked part of the image, making the image look artificial and out of place when it's dropped onto a different background (see "Choke and Spread Edges").

In this article I discuss techniques that help you get good selections quickly and then build selections to composite images together. The techniques are also appropriate for other uses, such as making a selection before applying a filter or using painting tools. In some cases, commercial plug-ins (mentioned below) may do the job better and faster, but with a little practice you should be able to make expert selections with relative ease.

Principles of Masking

If you remember only one thing from this article, it should be this: selections, alpha channels, and masks are all exactly the same thing in Photoshop's eyes. Whenever you make a selection in Photoshop, the program "sees" that selection as a gray-scale image in which the selected parts are white and the nonselected parts are black. The black part is the digital masking tape, covering the parts of the image that aren't selected; the white part is "uncovered" and ready for painting.

The powerful thing about digital masking tape is its relationship with the color gray: you can also partially select pixels in your image, and these areas are represented by shades of gray. The paler the gray (that is, the closer to white) an area is, the more pixels are selected. Generally, the edges of a selection's mask are slightly gray (or antialiased) to create a gentler transition between white and black areas.

Quick Mask There are several ways to see a visual representation of your selection. One is Quick Mask mode, which you can switch to by clicking on the Quick Mask button at the bottom of the Tool palette or by pressing the Q key. In Quick Mask mode, the masking tape is colored (usually red, to remind us old-timers of rubylith), and the selected parts are transparent instead of white. Because Photoshop's regular "marching ants" selection marquee isn't very precise, I tend to use Quick Mask mode for everything but the most basic rectangular selections.

You can edit your selection by editing the Quick Mask–paint in it, use filters, and so forth. Don't be afraid to custom-paint your selections instead of using the standard selection tools. Just switch to Quick Mask mode, paint in areas with the various brush tools, and press Q again to leave Quick Mask mode and see the marching ants around the area you painted.

Channels and Layer Masks Another way to get a visual representation of your selection is to save it as an alpha channel. Don't be scared by the term alpha; it just refers to a gray-scale channel–in addition to the regular CMYK or RGB channels (see "Extract from a Channel")–that stores the mask information. You can save a selection as an alpha channel by choosing Save Selection from the Select menu or by clicking on the Save Selection As Channel button at the bottom of the Channels palette. Whereas Quick Masks disappear when you deselect your selection, you can save alpha channels with your document and reload them later as selections.

You can also save a selection as a layer mask (see "Make Layer Masks"). When you choose Add Layer Mask from the Layer menu, you're telling Photoshop to add masking tape to that layer so that specified parts of it are obscured. The mask you create is then attached only to that layer. This way, you needn't delete parts of a layer–layer masks are non-destructive, so if you change your mind, the original layer is still intact.

Selection Tips and Tricks

Even if you know your way around Photoshop's selection tools, there are always new ways to use them more efficiently. Here are a few helpful hints.

Marquee Positioning If you're expanding a rectangular or oval selection marquee and you realize that you didn't position its origin in exactly the right place, don't fret: with the mouse button still held down, press the spacebar. Now you can move the entire selection and reposition its starting point. Release the spacebar and then continue expanding the selection as before.

Selecting Hidden Parts In Photoshop 5 you can select pieces of your image that lie outside the image boundary–--hidden by the gray canvas that surrounds the document–by option-dragging with the lasso tool.

Lasso Toggling If you want to draw a straight line while using the lasso tool, hold down the option key, release the mouse button, and draw the line. To draw freely again, press the mouse button and release the option key.

Magic-Wand Trickery People typically use the magic-wand tool to select parts of an image, but sometimes it's easier to deselect using the magic wand. Drag a rectangular marquee over a big chunk of your image, hold down the option key, and click on the magic wand to deselect parts you don't want.

Color Quick Change It's often helpful to change the Quick Mask's red color to something that stands out better–especially if your image contains red. To do this, double-click on the Quick Mask icon at the bottom of the Tool palette, and alter its color and opacity in the Quick Mask Options dialog box. I generally set the opacity to 70 or 80 percent.

One-Step Loading Here's a shortcut to load a channel as a selection: command-click on the channel in the Channels palette. You can also load a layer's transparency mask (the parts of the layer that are transparent) by command-clicking on the layer in the Layers palette.

Should You Buy a Plug-In?

If after trying the selection tools and techniques described in this article you feel you're spending too much time making a good selection, you might want to consider buying a commercial plug-in that facilitates selections, such as the $300 Mask Pro from Extensis (800/796-9798, http://www.'extensis.com), $100 MagicMask from Chroma Graphics (888/824-7662, http://www.chromagraphics.com ), and $495 KnockOut from Ultimatte (818/993-8007, http://www.ultimatte.com ). Each plug-in has its strengths and weaknesses, so make sure the one you pick will do the job for you. Time is valuable, so a plug-in may pay for itself quickly.

See the story (""Turn Snapshots into Art"") to learn how to transform selections made with the standard marquee tool.

But try to be patient–making a good selection takes time, and the more detailed the area, the longer it'll take. With a little practice you should be able to make expert selections with relative ease, and put Photoshop's powerful digital masking tape to work for you. DAVID BLATNER ( david@moo.com ) is a coauthor of Real World Photoshop 5 (Peachpit Press, 1998) and a frequent speaker at seminars and conferences.

June 1999 page: 107

Choke and Spread Edges

Never expect a tool to make a good selection by itself. The magic wand and Photoshop 5s magnetic-lasso tools are good cases in point: both make a perfectly adequate first pass at creating a selection, but then its your turn to fine-tune it. Because the biggest problem with selections is usually edge spill, you need a good technique to choke or spread the selectionmake it uniformly smaller or larger. While the Contract and Expand options in the Select menus Modify submenu work fairly well, I find theyre usually too blunt an instrument. Heres another option you can use instead.

1 At first glance, using the basic magic wand to draw around the lightbulb gives me a reasonable selection (A). But when I switch to Quick Mask mode and zoom in, I can see where edge spill will occur (B). (Quick Mask mode is one of the best ways to detect edge spill.)	A	B
2 While in Quick Mask mode, I run the Gaussian Blur filter with a radius of 1 or 2 pixels. The edge spill immediately worsens, but its one step back in order to go forward.
3 Now I open the Levels dialog box (command-L) and drag the gray midpoint slider to the left and the right, making sure the Preview check box is turned on so I can see how the Quick Mask chokes or spreads as I adjust this slider. Dragging to the right chokes and dragging to the left spreads. I drag about three-quarters of the way across the slider, stopping when the edge spill is gone. Then I exit Quick Mask mode and the selection is complete, sans edge spill.

Extract from a Channel

One of the most powerful benefits of selections and channels being equal in Photoshop is that you can use the information in a channel to make a superior selection. There are many variants of the following technique, including running a High Pass or Median filter on your alpha channel. Keep an open mind and experiment.

1 Trying to select the wires and steel structures in this complex image using only the standard selection tools would be a nightmare.	2 Instead, I look at each of the images channels by pressing command-1 for red (at left), command-2 for blue (center), and command-3 for green (right). I find the channel with the best contrast between the background and the objects Im trying to select. Here, the red channel is the winner.
3 I duplicate the red channel by selecting Duplicate Channel from the Channels palettes pop-up menu. Then I manipulate the Levels on the duplicate until the parts I want masked are fully black and the parts I want selected are fully white; I make sure to have a soft gray transition between the two to prevent the result from appearing choppy. (Instead of using Levels, you can use painting tools, filters, Curves, or any other Photoshop feature to achieve the mask.) When finished, I invert the channel (command-I) to make the background black.
4 Finally, to load the selection I command-click on this channel in the Channels palette (A). For greatest flexibility I convert the background layer to a regular layer by double-clicking on it, and then choose Add Layer Mask from the Layer menu. This way, the background becomes transparent (B). I can then drag the entire layer over to the composited file for final adjustments.

Make Layer Masks

Compositing images is more art than science, and it involves many painstaking tweaks to get right. If you dont use layer masks when compositing, youre going to need aspirin by the end of the day, because each tweak destroys more and more of your image. Layer masks give you (and/or your art director) many opportunities to change your (or his or her) mind without degrading image quality.

1 Here I've dropped an image of a city map over the background picture and set the opacity to 28 percent. However, the map covers the frying pan. But instead of trying to cut out the frying pan and put it on a layer over the map, its easier to use a layer mask in the map layer.
2 I select the frying pan and save it as a layer mask by clicking on the Layer Mask button at the bottom of the Layers palette. Then I add a gradient to the layer mask and use the Filter menus Fade command after running the gradient to adjust the mode and opacity of the effect.
3 With the layer mask turned on, that portion of the map is invisible. To move the map image without moving the layer mask, I can click on the link icon between the two thumbnails in the Layers menu, which deactivates the link. To turn off the layer mask without deleting it, I can shift-click on the layer mask.