Adobe Creative Suite programs let you play with many transparency effects but there's no guarantee any of them will print correctly. The answer is transparency 'flattening'. Alistair Dabbs explains what flattening does, how to decide whether to flatten or not, and suggests some workarounds.

What is it about transparency that needs 'flattening'? To understand this a little better, cast your mind back to the early days of Adobe PostScript as a method of defining computerized typography and describing objects on a printed page. With PostScript, everything is a solid object: if two objects are located in the same position, the object in front obscures the one behind it. No allowance was made for see-through objects.

Adobe's competitors, unburdened by the legacy of PostScript, have let you work with transparent objects for many years. The only tricky bit was how to deal with transparency when exporting the artwork to PostScript - saving in EPS format, for example - or when printing to a PostScript printer. The solution for programs such as Macromedia FreeHand, CorelDraw and Deneba Canvas was simple: convert those overlapping transparent objects to pixels. In other words, rasterize the vectors to bitmaps.

Actually, it's more complicated than that. For a start, you wouldn't want to rasterize the entire graphic, but just the areas where transparency is taking place. Second, a vector graphic is resolution-independent, but rasterized graphics have a physical size measured in pixels. So do you make those rasterized areas 72dpi, 300dpi or what? If the graphic gets resized to a billboard poster, what resolution do you choose then?

To make things even more complicated, Adobe introduced the concept of 'flattening' not as an on/off switch as you might expect but as a sliding scale. This means you can decide how many vectors are rasterized to bitmap as opposed to broken up into smaller vector intersections, Pathfinder-style. Since then, the feature has been rolled into InDesign and Acrobat too. Now everyone has gone mad for transparency - only later to go stark raving mad when they see the artwork in print. It seems transparency is a minefield of sliders, resolutions and file formats, all potentially leading to disastrous and wholly unexpected results.

Flattening explained
Let's put an end to the nightmare by working out what transparency flattening is up to in Adobe Creative Suite products. Whether you're running InDesign CS or Illustrator CS, choose Edit > Transparency Flattener Presets, and then click New. This opens the Transparency Flattener Preset Options window as shown here:

Don't panic. All you are doing is configuring a bunch of settings which tell InDesign/Illustrator how to deal with transparency when printing and exporting. This saves you having to enter all this information every time in the Print or Export dialogue windows later on. First, look at the Raster/Vector Balance slider. When the slider is all the way to the left (0%), every single instance of transparency on the page or in the artwork will be rasterized to a bitmap. Everything else will be left alone as vector objects, no worries (this is not entirely true, by the way, but I don't want to upset you just yet). When the slider is dragged all the way to the right (100%), all instances of transparency, no matter how complex, will be broken into smaller intersecting vector objects and no rasterization will take place (this also is not completely true, but see later).

You can choose to position the slider somewhere in between those two extremes. For example, at 50% the flattener engine will maintain simple transparent objects as vectors while only rasterizing areas that are extremely complex. Although it sounds like a sensible compromise, this is precisely where problems set in. You can end up with a jigsaw-like effect where you can see the join between abutting vector and bitmap areas. Worse, you can't see this join on-screen; you have to wait for the final printout. A graphic that incorporates text can look pretty weird this way, so it's often best to tick the 'Convert All Text To Outlines' option and ensure the 'Line Art and Text Resolution' value matches your intended output device: 300dpi for a laser printer, for example, or 1200dpi for an imagesetter or platemaker. The 'Gradient and Mesh Resolution' is unhelpfully labelled: it actually means 'any bitmap halftone, whether a gradient, mesh fill, or anything overlapping a bitmap image'. This should be treated like any bitmap image, typically 150dpi for laser printing and somewhere between 300dpi and 450dpi for an imagesetter.

If these values seem familiar, they are. They're the same settings as you'd enter for image compression downsampling when exporting to Adobe PDF.

Myths exploded
Now all you have to do is create several different flattener presets, or use the three standard ones already built into InDesign and Illustrator, and choose them when using the Print dialog window or Export dialog window. In both cases you'll find the Transparency Flattener Preset pop-up selector in the Advanced section of the window.

Sometimes, however, the flattener options are greyed out. This will happen if you are printing to a non-PostScript printer (in which case everything is flattened regardless) or when exporting to PDF 1.4 or 1.5. So what's the deal with PDF 1.4 and 1.5?

Although PostScript does not natively support transparency, PDF 1.4 and 1.5 do. The good news is that this means you can play with transparency all you like, export to PDF and you'll never have to gamble with flattening again. The bad news is that only a handful of Adobe RIPs currently support PDF 1.4 and 1.5 natively for transparency, and it will be years before they go mainstream in printers, imagesetters and workflow systems. So you're stuck with flattening for a while yet.

Another myth to explode is the idea that flattening applies only to transparent vector graphics. In practice, it also affects soft drop-shadows, edge feathering and alpha-channel cut-outs. Unfortunately, this means text overlapped by a drop-shadow or too close to a feathered edge can end up being jigsawed into bitmap and vector segments. Part of the problem is that Adobe's flattening methods apply in strict rectangular blocks, so there may be an overlap in the flattening process even when you can't see any overlap between the object edges. For example, an alpha-channel cut-out image placed over a coloured vector background fill can cause a rectangular area around the image to be rasterized. If you have run text around the irregular edge of the image, you're back to the jigsaw nightmare of rasterized text butting up against real text.

Here's a demonstration of the problem. In the following example from InDesign CS, we have applied a drop shadow to an image with a text wrap.

If we open the Flattener Preview palette from the Window menu and select Transparent Objects from the Highlight pop-up, you can now see for yourself the full horror of the area which will be rasterized by transparency flattening if the Raster/Vector Balance isn't set to 100%. The rectangular area around the drop shadow will be turned into a bitmap, text included.

To make your hair stand on end, try rotating the image. The flattener still maintains its rectangular shape but because the rotated drop shadow is now diamond-shaped, the flattened area is bigger than ever and covers even more text. Argh!

The only workaround for this problem is to use InDesign's nested frames feature. Here we drew an empty frame with no stroke or fill, then use the Paste Into command to nest the image and its drop shadow inside it. We then rotated the outer frame. The flattener still keeps to a rectangular area but ignores the rotation of the outer frame. Phew!

Further help
Just so you know, there are occasions where setting the Raster/Vector Balance slider to 100% or outputting through a native PDF 1.5 RIP will still allow some rasterization to take place. They all refer to areas that can't be described as simple colours or gradients. For example, transparent gradient-filled text over an image will be converted to vector text strokes with rasterized fills. Drop-shadows over text (rather than under it) will also cause the whole area to be rasterized. Also be warned that Acrobat 6.0 does not show transparency over spot colours on-screen, even though it will print them correctly. Don't for heaven's sake tick the Simulate Overprint option when creating the PDF because although it will then appear correctly on-screen in Acrobat, it will print as process separations instead of spot colours. But that's another story...

For more information on transparency flattening, read pages 2 to 4 in the document 'Illustrator CS Printing.pdf' found on your Illustrator CS installation CD or Creative Suite Standard Resources Disc in the Adobe Technical Info folder. The previous version of the software, Illustrator 10, came with a friendly Flattening Guide that's worth reading - even though the flattening interface has changed somewhat since. If you don't have a copy, you can still download it as a full colour PDF.