electric beach

Christian Schormann

Thursday, February 17, 2011

Experimenting with SVG Filter Effects

When there is something that processes pixels, I usually get curious at one point or another (after all, I used to design compositing systems for a living).

So when a long lost moment of time unexpectedly appeared at my door step, I finally got to play with SVG filter effects. And as I kind of like interactive tweaking more than editing XML files, I ended up writing a simple editor for SVG effects that runs in a browser in the process.

Source proc1 proc2

Editor

This post contains:

  • An overview of SVG filters
  • A walkthrough of an example filter
  • A discussion of the limitations of the spec
  • A comparison with image effects in Flash and Silverlight
  • And finally an appendix that describes the editor app in a few words

Let’s get started :)

SVG Filter Effects: An Overview

SVG filter effects are a mechanism to apply image processing effects to vector graphics defined in SVG. In the beta version of Firefox 4, filter effects can apparently also be applied to HTML elements.

Filter effects? Why? Image processing effects allow for visuals effects that simply can’t be created with vector graphics alone. They are a welcome and necessary ingredient for the visual vocabulary of any modern presentation platform. Making image effects applicable to an HTML element dramatically raises the scope for SVG filters and makes them an interesting venue for exploration.

Platforms such as Flash and Silverlight have offered image processing effects for a while, but the underlying models are quite a bit different. In this article, I’ll first look a bit at SVG filter effects for themselves, and then I’ll do a quick comparison with image processing in Silverlight and Flash.

SVG Effects

SVG filter effects are made up from a set of primitive image processors, such as offset, blur, color matrix and so on. These primitive processors can be wired together into a graph that renders a more complex effect.

Filters also need something to process, and therefore your filter graphs have access to the pixels of the element they are applied to and to a mask for that element (basically a silhouette of the element). This mask (referred to as source alpha) is really useful for many effects. The pictures below show the source pixels and the source mask rendered on a grey background.

Source sourceAlpha

Optionally – browsers are not required to implement this – SVG filters can also access the pixels of the background behind the element the filter is applied to, and a mask for the background. Access to background pixels might be relatively expensive, performance-wise.

Finally, SVG effects can also have access to pseudo images filled with the stroke or fill color of the element the effect is applied to.

The effect primitives defined in the SVG spec include:

  • Some source effects that generate images to be used in the filter graph. One of these lets you simply feed an image from a URL as an additional input into the graph, another generates a noise or turbulence image (based on a perlin noise function), and finally a simple flood fill. The picture below shows a turbulence picture.
    turbulence
  • Then there are a few image processing primitives that take an input image and modify it:
    • ColorMatrix: a simple color processor that can do saturation, hue rotation, or color matrix transforms.
      hueRotate
    • ComponentTransfer: A channel processor that lets you apply a transfer function to each channel. This can be used for example to process contrast and brightness.
    • ConvolveMatrix: A generic convolution matrix
    • DiffuseLighting: Simulates a lighting effect on the input image, using the alpha channel as a bump map. The lighting is calculated using the diffuse portion of the Phong lighting model from 3D computer graphics.
    • GaussianBlur: A blur filter with separate blur amount in x and y
      blur
    • Morphology: filter for “fattening and thinning” (erosion and dilation)
    • Offset: a simple 2D translate filter with separate x and y control
    • SpecularLighting: Simulates a lighting effect on the input image, using the alpha channel as a bump map. The lighting is calculated using the specular portion of the Phong lighting model from 3D computer graphics. The picture below shows the source mask from above with an applied specular lighting effect.
      specularLight
    • Tile: Creates a tile pattern from the input in x and y direction
  • Finally, there are a handful of effects that require at least two input images to do something useful:
    • Blend implements simple blend modes (only a very small subset of those made popular by Photoshop)
    • Composite, as the name suggests, composites the two input images together using Porter-Duff compositing functions
    • Displacement: distorts the first input image using values from the second. The picture below shows a displacement applied to the source graphic, distorted with the turbulence image from above.
      displacement

Walkthrough: An Example Filter

Let’s look at a slightly modified example filter from the SVG spec. In the example, the filter is applied to the same source graphic that was shown above, with the same source alpha:Source sourceAlpha

The topology of the filter graph looks like this:

Topology

When applied to the SVG graphic shown above, the result looks like this:

proc3 

The effect creates an embossed, lit look of the source graphics, shifts its color from red to pink, adds a drop shadow and distorts the outcome for both drop shadow and foreground pixels in slightly different ways. Pretty or not, that’s quite a lot for an effect to do. 

If you look at the topology graph for the effect, at the very right you can see an feMerge primitive. Think of this as a basic multi-layer compositor that in this case puts two layer on top of each other, just as Photoshop would.

The first layer is the drop shadow of the input graphics, generated by processing the source alpha of the underlying SVG. If you look at the topology chart, you can see the source alpha connected to an feGaussianBlur effect, which yields this result:

blurWT

The output of this blur is then wired to an feOffset effect that basically translates the output by a few pixel. The following screenshots show the blur and the offset blur side by side:

blurWT offset

Next, the blur is piped into an feDisplacement effect that is driven by a turbulence primitive, set up to produce fractal noise:

fractalNoise

The result of this is shown in the following image – this is basically the finished drop shadow layer.

blurWTDisp 

Next comes the foreground layer, which is a little bit more complex. The effect we are going for is an emboss with lighting, plus some entirely gratuitous displacement on top, just because we can.

To get the lit emboss effect, we use an feSpecularLighting primitive, driven by the source alpha image (remember, the simple flat black outline of the SVG graphic shown above). The result of this is here:

specularLight2

If we overlay this embossed shape onto the original source graphics, we get the following result (there is an extra color shift in here, we get to that later):

specPlusGraphics

That is not really what we were looking for, because the emboss effect spills outside of the edges of the original source graphics. We really would like to clip to the original source outline.

We can do that by simply piping the output of the specular lighting effect into an feComposite node first, with the second input set to the source alpha. We set the composite operation to “in”, which causes the composite node to only output pixels that are within the outline defined by the alpha channel:

clippedSpecular  

The relevant portion of the effects graph is this:

subGraph1

The blurred alpha is piped into the specular lighting effect to get the embossed version, and the result is then clipped to the alpha using the compositor set to an “in” operation.

Next, we take the result of this and composite it over the source graphics. Well, not quite the source graphics: We first pipe it through an feColorMatrix set to Hue Rotation to shift the color of the source graphic from red to pink:

pink

And then, after the second feComposite node (this time set up to do a simple “over” composite:

pinkEmboss

The sub graph that generates this result is this:

subGraph2 

Finally, we send this result through another displacement filter driven by the same fractal noise used for the shadow, just set to a higher amount of distortion:

foreground 

And this concludes the foreground layer. The two layers care then composited together using feMerge (which is really just a multi-input version of a simple “over” composite):

proc3

Limited Primitives

Looking at the set of primitives, the selection prescribed in the SVG spec seems a little bit curious: The two lighting effects are by far the most complex effects (even offering three types of light sources – point, distant and spot) with lots of detailed parameters. Apparently the creation of glass effects with drop shadows was very much what the creators of the SVG filter spec had in mind.

For a more useful repertoire of effects, I’d rather have a few more basic primitives, for example:

  • directional and radial blurs
  • channel arithmetic, wiring and threshold/gain processing
  • other distortions (warp, twist…)
  • pattern generators (for example for wipe patterns) and gradients.

Some “specials” such as lens flares or other camera effects would be nice too.

Finally, given the importance of the use case: A motion blur primitive tied in to animation of the underlying source element would be most welcome.

Also, it is fairly limiting that the only available pixel space transform is just a simple offset in x and y. There is no support for rotation or scale, to not even speak of 2.5D (perspective)transforms.

Transforms are of course available on the SVG scene graph (or CSS level for HTML), but that does not really help within an effects graph, where I want to be able to transform a branch of the graph alone, not the entire input to the graph. The drop shadow in the screen shot below demonstrates the use of this – only a part of the graph (in this case, the blurred alpha) is transformed by an offset. A global transform applied to the source element upstream of the effect can not provide this effect.

proc1 

Therefore, the effects set for SVG effects should also include what in compositing systems is often known as “DVE” (for “Digital Video Effect”) primitive, to position, rotate and scale (and perhaps even deform) the input image in space.

This highlights a general design problem that the SVG environment shares with other current user interface frameworks: The typical scene graph model used in UX toolkits is designed with a very simple compositing model in mind, where elements are arranged in a strict tree, and image effects, where they exist, are added as an afterthought. This way, image processing expressions, even with the topological flexibility of SVG filters, are only able to “locally” modify the pixels of a sub-tree of the scene graph, rather than allowing for richer compositing topologies globally.

One immediate side effect of this is that now functions such as transforms have to be introduced twice, once as part of the scene graph functionality, and once as part of the image processing primitives set.  

Image Effects in Silverlight and Flash

Flash and Silverlight also provide the ability to apply image processing effects to primitives in the scene. The overall model for application of effects is similar (effects are applied to an element in the scene graph, and are effectively local modifiers to the rendering of individual elements). 

The effects themselves, in both Flash and Silverlight, are pixel shaders in the GPU sense of the word. There are big differences in how the effects are written, and how they can be combined.

In Silverlight, pixel effects are written in the same HLSL shader language that is used to write pixel shaders anywhere else in the DirectX universe, including for games. Silverlight then takes care rendering these effects on the target platform, even if the target platform is a Mac that does not support DirectX. This is possible because Silverlight effects can be rendered in software. Silverlight shaders are currently limited to Shader Model 2.0, which includes fairly stringent shader size restrictions. The Silverlight shader model only supports single pass effects, but effects with multiple inputs are possible.

In Silverlight, only one shader is allowed per element.

In Flash, pixel shaders are implemented in a custom language called PixelBender. This is a special programming language similar to the HLSL or OpenGL shading languages, made for writing platform independent pixels shaders. PixelBender effects can also be written for After Effects, but Flash only supports a subset of the capabilities. Shaders are also evaluated in software, as far as I know.

In Flash, you can apply more than one effect to an element.

In comparison to SVG filter effects, both Flash and Silverlight let you write complex custom effects using a shader programming language. There is a lot of power and flexibility here. SVG, on the other hand, lets you create filters from a set of basic primitives that can be combined in flexible ways. Given the limitations of the primitives set, this is an overall less flexible approach, but it is easier to use for non-programmers. Of course, in principle it is possible to provide the same type of graph-based editing to program a shader – in fact this approach is used in the latest version of 3D Studio Max, using Mental Images’ Mental Mill shader editing and compilation technology.

From a performance point of view, I would expect a custom-written shader in principle to be able to perform better that a filter graph. Today, given that the shaders in Flash and Silverlight are executed in software, that might not matter too much. 

From today’s point of view, SVG filter effects are mostly handicapped by the limited set of primitives, as compared to shader-based effects.

For now, both the shader-based and the primitives-based image effects technologies still have many shortcomings. In a better world, we’d have the ability to run pixel effects as shaders on the GPU (for performance reasons), without size limits, and with tools support for visual shader construction, filter-graph style :)

SVG Filter Summary

The SVG filter effects spec allows for flexible and powerful topologies of effects primitives. Unfortunately, the choice of primitives described in the spec is quite limiting, and is overly skewed towards glass-style emboss effects. The set of effects primitives is not extensible. I think that the idea to apply SVG filters to HTML is very exciting, and I hope it becomes a widely available reality. For general use, the SVG community should urgently plan to extend the set of filter primitives, ideally with an extensible model.

Another issue that remains to be seen is how complex filter topologies actually perform in real world applications, across different browsers and hardware platforms. Even with hardware acceleration, there can be performance challenges with this approach. The spec does not provide any APIs for performance adaptation, so that it could turn out to be difficult or impossible to create scalable applications that work well across browser implementations and on different hardware platform.

Finally, it would be quite important to have SVG filter effects animate-able using CSS animation.

Appendix: A Simple Editor for SVG Effects

Image effects are a lot more fun and a lot easier to understand when their properties can be edited interactively. Writing effects in XML does not quite meet that interactivity bar. So I decided an editor was needed. And as my new years’ resolution says that I was going to write more HTML, CSS and JavaScript code in 2011, it seemed a good idea to do just that. And because SVG is well supported in Opera, I chose Opera as the platform to write against.

The editor got created rather incrementally over a couple of evenings: First, I just loaded an SVG document onto a web page using a filter tag. Then I used some JavaScript code to walk the SVG DOM inside the object tag to get a list of effects.

startingPoint

Next, I decided to create property editors for each filter effect, and to change the SVG DOM live as the properties for each filter are adjusted.

The property editor is created dynamically – whenever you select a filter from the filter list, the right property editor is created and displayed. Here, as an example, the editor for the feSpecularLighting primitive.

FXEditor

Next, I decided it would be good to be able to see the filter topology. So I built a simple graph viewer, where the nodes are created as HTML, and the connections are created as SVG in an underlying SVG object tag. The graph viewer currently has no auto layout, so there is some manual dragging around of nodes required to make sense of the topology.

GraphView

Finally, I found it useful to add a second preview that would let me see the result of the sub graph up to a selected node in the tree. This is helpful because subsequent steps of the graph sometimes obscure the intermediate result – it is a great tool for understanding a graph and for debugging.

GraphPreview

In this screenshot, the final result is on top, and the intermediate preview, showing the result of the feSpecularLighting effect, is at the bottom.

Next step is to change the graph viewer into a graph editor. But that will have to wait until I find another unexpected dose of time again :)

The entire app is about 1500 lines of markup and code, most of it (about 1200 lines) JavaScript to poke and peek at the HTML and SVG DOMs. I’ll post the code as soon as I get to cleaning it up a little.

posted by cs at 00:01  

4 Comments

  1. [...] This post was mentioned on Twitter by MSExpression and Christian Schaefer, Andreas Fritsch. Andreas Fritsch said: RT @MSExpression: Experimenting with #SVG Filter Effects – http://bit.ly/eFtjGH – #msexp [...]

    Pingback by Tweets that mention electric beach » Experimenting with SVG Filter Effects -- Topsy.com — February 17, 2011 @ 10:22

  2. [...] Experimenting with SVG Filter Effects: electricbeach.org [...]

    Pingback by Sites Saturday, 19 February 2011-3D Photosynth panoramas-Bing Bar-Pinned sites in Win7-New Win7 themes-Hints for Word and Publisher « webDotWiz talks Windows Live — February 19, 2011 @ 00:58

  3. [...] kennt eine riesige Palette an Filtern, so etwa Überblenden, Farbüberlagerungen, Helligkeits- und Kontrastanpassungen, Beleuchtung und [...]

    Pingback by CSS-Filter – mehr Effekte fürs Web – CSS3 Adventskalender Tag 19 | maddesigns — December 18, 2011 @ 23:05

  4. [...] SVG knows a huge palette of so-called filters since ages, e.g. color blending, brightness/contrast adjustments, lighting, displacement mapping, [...]

    Pingback by Effects for the Web! | @drublic — December 27, 2011 @ 04:09

RSS feed for comments on this post. TrackBack URI

Sorry, the comment form is closed at this time.

Powered by WordPress