When there is something that processes pixels, I usually get curious at one point or another (after all, I used to design compositing systems for a living).
So when a long lost moment of time unexpectedly appeared at my door step, I finally got to play with SVG filter effects. And as I kind of like interactive tweaking more than editing XML files, I ended up writing a simple editor for SVG effects that runs in a browser in the process.
This post contains:
- An overview of SVG filters
- A walkthrough of an example filter
- A discussion of the limitations of the spec
- A comparison with image effects in Flash and Silverlight
- And finally an appendix that describes the editor app in a few words
Let’s get started
SVG Filter Effects: An Overview
SVG filter effects are a mechanism to apply image processing effects to vector graphics defined in SVG. In the beta version of Firefox 4, filter effects can apparently also be applied to HTML elements.
Filter effects? Why? Image processing effects allow for visuals effects that simply can’t be created with vector graphics alone. They are a welcome and necessary ingredient for the visual vocabulary of any modern presentation platform. Making image effects applicable to an HTML element dramatically raises the scope for SVG filters and makes them an interesting venue for exploration.
Platforms such as Flash and Silverlight have offered image processing effects for a while, but the underlying models are quite a bit different. In this article, I’ll first look a bit at SVG filter effects for themselves, and then I’ll do a quick comparison with image processing in Silverlight and Flash.
SVG filter effects are made up from a set of primitive image processors, such as offset, blur, color matrix and so on. These primitive processors can be wired together into a graph that renders a more complex effect.
Filters also need something to process, and therefore your filter graphs have access to the pixels of the element they are applied to and to a mask for that element (basically a silhouette of the element). This mask (referred to as source alpha) is really useful for many effects. The pictures below show the source pixels and the source mask rendered on a grey background.
Optionally – browsers are not required to implement this – SVG filters can also access the pixels of the background behind the element the filter is applied to, and a mask for the background. Access to background pixels might be relatively expensive, performance-wise.
Finally, SVG effects can also have access to pseudo images filled with the stroke or fill color of the element the effect is applied to.
The effect primitives defined in the SVG spec include:
- Some source effects that generate images to be used in the filter graph. One of these lets you simply feed an image from a URL as an additional input into the graph, another generates a noise or turbulence image (based on a perlin noise function), and finally a simple flood fill. The picture below shows a turbulence picture.
- Then there are a few image processing primitives that take an input image and modify it:
- ColorMatrix: a simple color processor that can do saturation, hue rotation, or color matrix transforms.
- ComponentTransfer: A channel processor that lets you apply a transfer function to each channel. This can be used for example to process contrast and brightness.
- ConvolveMatrix: A generic convolution matrix
- DiffuseLighting: Simulates a lighting effect on the input image, using the alpha channel as a bump map. The lighting is calculated using the diffuse portion of the Phong lighting model from 3D computer graphics.
- GaussianBlur: A blur filter with separate blur amount in x and y
- Morphology: filter for “fattening and thinning” (erosion and dilation)
- Offset: a simple 2D translate filter with separate x and y control
- SpecularLighting: Simulates a lighting effect on the input image, using the alpha channel as a bump map. The lighting is calculated using the specular portion of the Phong lighting model from 3D computer graphics. The picture below shows the source mask from above with an applied specular lighting effect.
- Tile: Creates a tile pattern from the input in x and y direction
- ColorMatrix: a simple color processor that can do saturation, hue rotation, or color matrix transforms.
- Finally, there are a handful of effects that require at least two input images to do something useful:
- Blend implements simple blend modes (only a very small subset of those made popular by Photoshop)
- Composite, as the name suggests, composites the two input images together using Porter-Duff compositing functions
- Displacement: distorts the first input image using values from the second. The picture below shows a displacement applied to the source graphic, distorted with the turbulence image from above.
Walkthrough: An Example Filter
The topology of the filter graph looks like this:
When applied to the SVG graphic shown above, the result looks like this:
The effect creates an embossed, lit look of the source graphics, shifts its color from red to pink, adds a drop shadow and distorts the outcome for both drop shadow and foreground pixels in slightly different ways. Pretty or not, that’s quite a lot for an effect to do.
If you look at the topology graph for the effect, at the very right you can see an feMerge primitive. Think of this as a basic multi-layer compositor that in this case puts two layer on top of each other, just as Photoshop would.
The first layer is the drop shadow of the input graphics, generated by processing the source alpha of the underlying SVG. If you look at the topology chart, you can see the source alpha connected to an feGaussianBlur effect, which yields this result:
The output of this blur is then wired to an feOffset effect that basically translates the output by a few pixel. The following screenshots show the blur and the offset blur side by side:
Next, the blur is piped into an feDisplacement effect that is driven by a turbulence primitive, set up to produce fractal noise:
The result of this is shown in the following image – this is basically the finished drop shadow layer.
Next comes the foreground layer, which is a little bit more complex. The effect we are going for is an emboss with lighting, plus some entirely gratuitous displacement on top, just because we can.
To get the lit emboss effect, we use an feSpecularLighting primitive, driven by the source alpha image (remember, the simple flat black outline of the SVG graphic shown above). The result of this is here:
If we overlay this embossed shape onto the original source graphics, we get the following result (there is an extra color shift in here, we get to that later):
That is not really what we were looking for, because the emboss effect spills outside of the edges of the original source graphics. We really would like to clip to the original source outline.
We can do that by simply piping the output of the specular lighting effect into an feComposite node first, with the second input set to the source alpha. We set the composite operation to “in”, which causes the composite node to only output pixels that are within the outline defined by the alpha channel:
The relevant portion of the effects graph is this:
The blurred alpha is piped into the specular lighting effect to get the embossed version, and the result is then clipped to the alpha using the compositor set to an “in” operation.
Next, we take the result of this and composite it over the source graphics. Well, not quite the source graphics: We first pipe it through an feColorMatrix set to Hue Rotation to shift the color of the source graphic from red to pink:
And then, after the second feComposite node (this time set up to do a simple “over” composite:
The sub graph that generates this result is this:
Finally, we send this result through another displacement filter driven by the same fractal noise used for the shadow, just set to a higher amount of distortion:
And this concludes the foreground layer. The two layers care then composited together using feMerge (which is really just a multi-input version of a simple “over” composite):
Looking at the set of primitives, the selection prescribed in the SVG spec seems a little bit curious: The two lighting effects are by far the most complex effects (even offering three types of light sources – point, distant and spot) with lots of detailed parameters. Apparently the creation of glass effects with drop shadows was very much what the creators of the SVG filter spec had in mind.
For a more useful repertoire of effects, I’d rather have a few more basic primitives, for example:
- directional and radial blurs
- channel arithmetic, wiring and threshold/gain processing
- other distortions (warp, twist…)
- pattern generators (for example for wipe patterns) and gradients.
Some “specials” such as lens flares or other camera effects would be nice too.
Finally, given the importance of the use case: A motion blur primitive tied in to animation of the underlying source element would be most welcome.
Also, it is fairly limiting that the only available pixel space transform is just a simple offset in x and y. There is no support for rotation or scale, to not even speak of 2.5D (perspective)transforms.
Transforms are of course available on the SVG scene graph (or CSS level for HTML), but that does not really help within an effects graph, where I want to be able to transform a branch of the graph alone, not the entire input to the graph. The drop shadow in the screen shot below demonstrates the use of this – only a part of the graph (in this case, the blurred alpha) is transformed by an offset. A global transform applied to the source element upstream of the effect can not provide this effect.
Therefore, the effects set for SVG effects should also include what in compositing systems is often known as “DVE” (for “Digital Video Effect”) primitive, to position, rotate and scale (and perhaps even deform) the input image in space.
This highlights a general design problem that the SVG environment shares with other current user interface frameworks: The typical scene graph model used in UX toolkits is designed with a very simple compositing model in mind, where elements are arranged in a strict tree, and image effects, where they exist, are added as an afterthought. This way, image processing expressions, even with the topological flexibility of SVG filters, are only able to “locally” modify the pixels of a sub-tree of the scene graph, rather than allowing for richer compositing topologies globally.
One immediate side effect of this is that now functions such as transforms have to be introduced twice, once as part of the scene graph functionality, and once as part of the image processing primitives set.
Image Effects in Silverlight and Flash
Flash and Silverlight also provide the ability to apply image processing effects to primitives in the scene. The overall model for application of effects is similar (effects are applied to an element in the scene graph, and are effectively local modifiers to the rendering of individual elements).
The effects themselves, in both Flash and Silverlight, are pixel shaders in the GPU sense of the word. There are big differences in how the effects are written, and how they can be combined.
In Silverlight, pixel effects are written in the same HLSL shader language that is used to write pixel shaders anywhere else in the DirectX universe, including for games. Silverlight then takes care rendering these effects on the target platform, even if the target platform is a Mac that does not support DirectX. This is possible because Silverlight effects can be rendered in software. Silverlight shaders are currently limited to Shader Model 2.0, which includes fairly stringent shader size restrictions. The Silverlight shader model only supports single pass effects, but effects with multiple inputs are possible.
In Silverlight, only one shader is allowed per element.
In Flash, pixel shaders are implemented in a custom language called PixelBender. This is a special programming language similar to the HLSL or OpenGL shading languages, made for writing platform independent pixels shaders. PixelBender effects can also be written for After Effects, but Flash only supports a subset of the capabilities. Shaders are also evaluated in software, as far as I know.
In Flash, you can apply more than one effect to an element.
In comparison to SVG filter effects, both Flash and Silverlight let you write complex custom effects using a shader programming language. There is a lot of power and flexibility here. SVG, on the other hand, lets you create filters from a set of basic primitives that can be combined in flexible ways. Given the limitations of the primitives set, this is an overall less flexible approach, but it is easier to use for non-programmers. Of course, in principle it is possible to provide the same type of graph-based editing to program a shader – in fact this approach is used in the latest version of 3D Studio Max, using Mental Images’ Mental Mill shader editing and compilation technology.
From a performance point of view, I would expect a custom-written shader in principle to be able to perform better that a filter graph. Today, given that the shaders in Flash and Silverlight are executed in software, that might not matter too much.
From today’s point of view, SVG filter effects are mostly handicapped by the limited set of primitives, as compared to shader-based effects.
For now, both the shader-based and the primitives-based image effects technologies still have many shortcomings. In a better world, we’d have the ability to run pixel effects as shaders on the GPU (for performance reasons), without size limits, and with tools support for visual shader construction, filter-graph style
SVG Filter Summary
The SVG filter effects spec allows for flexible and powerful topologies of effects primitives. Unfortunately, the choice of primitives described in the spec is quite limiting, and is overly skewed towards glass-style emboss effects. The set of effects primitives is not extensible. I think that the idea to apply SVG filters to HTML is very exciting, and I hope it becomes a widely available reality. For general use, the SVG community should urgently plan to extend the set of filter primitives, ideally with an extensible model.
Another issue that remains to be seen is how complex filter topologies actually perform in real world applications, across different browsers and hardware platforms. Even with hardware acceleration, there can be performance challenges with this approach. The spec does not provide any APIs for performance adaptation, so that it could turn out to be difficult or impossible to create scalable applications that work well across browser implementations and on different hardware platform.
Finally, it would be quite important to have SVG filter effects animate-able using CSS animation.
Appendix: A Simple Editor for SVG Effects
Next, I decided to create property editors for each filter effect, and to change the SVG DOM live as the properties for each filter are adjusted.
The property editor is created dynamically – whenever you select a filter from the filter list, the right property editor is created and displayed. Here, as an example, the editor for the feSpecularLighting primitive.
Next, I decided it would be good to be able to see the filter topology. So I built a simple graph viewer, where the nodes are created as HTML, and the connections are created as SVG in an underlying SVG object tag. The graph viewer currently has no auto layout, so there is some manual dragging around of nodes required to make sense of the topology.
Finally, I found it useful to add a second preview that would let me see the result of the sub graph up to a selected node in the tree. This is helpful because subsequent steps of the graph sometimes obscure the intermediate result – it is a great tool for understanding a graph and for debugging.
In this screenshot, the final result is on top, and the intermediate preview, showing the result of the feSpecularLighting effect, is at the bottom.
Next step is to change the graph viewer into a graph editor. But that will have to wait until I find another unexpected dose of time again