So there’s this new kid on the UI block. Used to be called Avalon. Now its called WPF. Its probably the most unsung and misunderstood part of Vista by the general public and programmers alike. Its actually pretty awesome. I’ll explain what it is and how I use it in a bit. There’s a problem though. WPF is the next generation UI for windows, and as such, it seeks to integrate 3d as a first class citizen into the UI environment. It fails miserably and I’ll give an in depth explanation and suggestions as to what needs to change IMHO.
What is WPF and XAML?
WPF is the brand spankin new UI system in Vista, that has apparently been backported to XP as well. Its part of the often misunderstood .net framework. Make no mistake though, WPF is meant to replace the venerable win32 forms GUI that we’ve been living with in some form or another since before time (read: windows95 and a little earlier in some form or another). WPF is the future of windows programming.
Rather than just being a revision, WPF is a new approach to GUI construction. Its based on a dependency graph model. People who are familiar with Maya or XSI are familiar with dependency graphs, as 3d scenes in those programs are actually large dependency graphs. Basically you build your gui as a dependency graph and you connect it to your application data which is also best done as a dependency graph. What this means, is that your GUI is not really hardcoded. Its authored like you would author an illustrator graphic or 3d scene. It just needs to know how to connect itself to your app’s data and it kind of runs on its own. It will then proceed to display itself based on your data, and if you authored it to, change your data.
XAML is the XML based file format in which you can author WPF dependency graphs (read UIs and UI pieces).
Its all about connections. You define data sources and UI widgets and you connect them all up. And when you run it, it just works. Its great.
So what’s wrong with 3D?
There’s nothing wrong with adding 3d to this mix. In fact, its a great idea. Its HOW they did it that bothers me. One of the cardinal rules of 3d programming is that 3d is usually not any harder than 2d. Its just one more dimension. So if you can solve a problem in 2d, you can usually adapt that solution to 3d by adding an extra dimension. This can be as simple as adding an extra parameter, or one more layer of nested looping. You name it.
So, lets take an example usage scenario. Lets say that I’ve got a series of 2d points (unknown quantity) that I need displayed as dots on a black rectangle. Oh I dunno… maybe its 2d blips coming from my mocap cameras. Seems like a reasonable scenario 
. These points will be replaced with a new collection of points say, 100 times a second, so this is essentially a real time data flow.
WPF makes this easy with what is known as a ItemsControl. You tell the ItemsControl where to find the collection of Point objects and it in turn, keeps an eye on that collection. When the collection changes, it rebuilds part of the dependency graph. What it does, is take a template for a panel and adds children to it. Those children are also built from a template. Each of the children is given a Point from the collection as their data source. So, if I make the panel a simple canvas panel (Cartesian coordinate based layout panel) and I make each item an ellipse that gets its coordinate from the point, I’ve got my 2d mocap viewer.
So thats great. It works. Not only does it draw them, but each of them is their own control and can respond to clicks and events and all kinds of wonderful things. Go WPF!
But that was just the 2d viewer for camera data. Lets build us a 3d viewer for the point cloud. That should be no problem right? WPF does 3d! In fact, without knowing anything about WPF, you should already sort of know how this should work. I should have a panel thats a 3d canvas (3d Cartesian tri ordinate space) and an ItemsControl that puts some templated 3d controls into it. Right? OOPS! WRONG
See, the problem is, 3D objects don’t inherit from Controls. There are no Panels either. You lay down a control that is a 3d view port, and then everything going on inside is part of a completely different set of objects that don’t interact with the 2d control object model at all unless its hardcoded in object by object. You can’t template these 3d objects for inclusion into other 3d objects at all. This is a HUGE problem.
If you look around at every 3d WPF demo around, you’ll see them all skirting over a very important issue. They NEVER create a instance of a 3d object for each member of a collection. They’ll hardcode the number of objects. They’ll do the work in 2d and then render that 2d image onto a 3d object as a texture. They’ll manipulate the vertex buffers of a single 3d object based on a collection. Anything but actually spawn a true 3d object for each member of a collection. And don’t get me started about the layout of these controls. Its the same problem.
What they’ve done, is created a 3d engine inside of WPF, that can have static data defined by XAML and can have its parameters bound to dependency objects, but they didn’t actually extend the WPF into 3d.
Its actually a mistake that has been made before. Back in the day, Macromedia had a product called Director. Director owned the "multi-media" market for a while, until the web and flash took over. In Director’s hayday, they wanted to add 3d to it. Director had an extremely rich UI that allowed the author to manipulate controls (sprites) on a stage (panel) and control them via animation and scripting. So you’d think the obvious direction to go, would be to expand the stage, to be able to hold 3d objects and textures and cameras and lights along with 2d sprites. And then extend the animation keyframing environment to be able to keyframe in 3d. And then make sure the scripting environment had the same control over the new objects as it did the 2d objects. That’s not what they did.
Instead, they made one new type of 2d sprite that was a 3d engine. There was no way to get inside the 3d engine except via scripting. They made a valiant effort to fake it by authoring a lot of "behaviors" which are pre packaged scripts you can apply to sprites. But that never made 3d break out of its sprite and onto the stage. Director still exists… its just that its market has shrunk to near nothing compared to its potential.
This is EXACTLY the mistake Microsoft seems to have made with WPF’s 3d. They put 3d in a black box as an engine rather than breaking open the box and making it part of the actual WPF controls API. In so donig, all they’ve really done is created yet another 3d programming API thats in competition with DirectX and OpenGL when they should have been creating a 3d UI API.
So what should they do?
Well, they should start over. Not WPF, just the 3d part, which happens to be relegated to a separate namespace anyway. The existing mess can continue to exist for all I care just as long as they make a new namespace and actually extend WPF into 3d rather than blackboxing 3d into a control in WPF. Its really quite obvius. Subclass Panel (or refactor panel to a higher level interface that you can implement) into a Panel3D. Subclass Control (or refactor etc.) to Control3D. And move in from there. Panels and Controls work together to create layout and the rest is simple. Ju
st make sure the ItemsControl works with it and templates work with it. Just as in normal 2d WPF, Panels are not all cartesian based. My favorite, the DockPanel can easily be expanded into 3d based on bounding boxes and scales for example. Make sure NOT to seal the Panel3D, so developers can create their own layouts for say… spinning cylindrical layout etc.
What about your 3d point cloud viewer?
Luckily, my UI is a prototype UI meant to be replaced. And luckily WPF’s overall design lets me replace the UI with very little effort. So what I did for now, is write a 3d object that takes a collection of markers as a binding. It then goes through its vertex buffer and moves a bunch of verticies around to make a bunch of sphere shapes (read: vertex animation). Unfortunately, since these are not independent objects, they don’t have individual color or ability to detect mouse clicks. They’re all one big mesh. Its good enough for now. When it ceases to be good enough, I may have to break out of WPF and write a pure DirectX viewer, which would be a shame since its clear WPF is supposed to be meant for this kind of thing.