Weekly update December 21-27

In the last week the dev team was working on instability issues regarding the view map creation.  The work started with a thorough review of the entire view map construction process, which consists of 1) importing meshes from a 3D scene, 2) detecting feature edges, and 3) building a view map.  These steps are briefly summarized as follows.

1. In the first step, Freestyle receives mesh data from Blender in the internal mesh structure called vlak.  In the vlak data structure, mesh vertices are in the camera coordinate system.  Since Freestyle’s feature edge detection algorithm expects mesh data to be in the world coordinate system, the vertices are transformed from the camera to the world coordinate system during the mesh importing.
2. In the second step, the nature of each edge of the imported meshes is examined and relevant feature edge types (such as borders, crease edges, ridges/valleys, and suggestive contours) are assigned to the edge.
3. In the third step, a 2D representation of the scene in the camera view is constructed.  All 3D vertices are transformed into the 2D space by means of a model-view matrix (from world to camera), a projection matrix (from camera to retina), and a viewport specification (from retina to image).  An important task in this step is to detect intersections of feature edges in the 2D space.  In the case of Blender’s default cube and camera, for example, we have two intersections of edges in the 2D space from the camera’s viewpoint (as shown by circles in the figure below).  The intersections of feature edges are computed using a traditional sweep line algorithm.  In addition, for each intersection of two edges in the 2D space, we find the corresponding 3D point on each edge by inversely applying the viewport specification, projection matrix, and model-view matrix.

We have identified two implementation issues in the 2D-to-3D conversion of edge intersections.  First, this conversion obviously does not work when edges are out of the viewing frustum.  One solution to address this issue is to clip the imported meshes by the viewing frustum.  A drawback of this approach is that feature edges are no longer available outside the camera view.  For instance, if we have feature edges outside the camera view, we can obtain a continuous stroke that goes out and comes into the camera view as illustrated below.  The left image shows a 3D scene with two rectangular parallelpiped meshes and the viewing frustum.  The center image shows the camera view, in which the rectangular parallelpiped meshes are partly sticking out of the viewing frustum.  The right image shows a rendering result using external_contour_sketchy.py (with minor changes).  This style module draws a continuous stroke around external contours several times.  Since feature edges are available outside the viewing frustum, the resulting render gives us an impression that the strokes continue outside the camera view.

  

If we proceed with the clipping of imported meshes by the viewing frustum, we will get a different rendering result as illustrated below.  The left image shows a modified version of the same 3D scene as the previous example.  The meshes have been clipped by the viewing frustum.  The center image shows the camera view.  Note that we have edges at the border of the camera view, and there are no edges around the outside of the camera view.  The left image shows a rendering result.  Now strokes turn at the border of the image instead of going out of and coming back into the image.

  

Clearly, the clipping of meshes by the viewing frustum makes a big difference.  Ideally, only a minimum amount of clipping (e.g., only excluding those vertices that come behind the camera) takes place by default, and the full clipping by the viewing frustum is performed when users prefer to do so.  We look into this issue more closely in the next weeks.  The second issue we have identified is a bug in the current implementation of the 2D-to-3D intersection conversion, which fails in some apparently trivial cases (even when only dealing with vertices within the viewing frustum).  We address this latter issue before we move on to the clipping problem.

New to-do list

According to the recent change in the organization of the dev team, we think we need a new plan on how to proceed toward the merge into the trunk (Blender 2.5).  As a matter of fact, the merge itself has been agreed with Ton, who has contacted us once in a while with regard to the status of the branch.  The merge will take place when the dev team considers the branch is ready for production use.  The question is when the branch is considered production-ready.  It is hard to predict exactly when, but in my humble opinion, the branch will be ready for production use when most known instability issues are addressed and missing features are recovered.  Here is a list of known issues that should be fixed before the merge into the trunk:

• Bug fix regarding the view map creation
  Most instability of the branch comes from bugs in the view map creation that detects feature edges (e.g., borders, crease edges, ridges/valleys, and suggestive contours) from a given scene.  A serious bug here is that objects behind the active camera cause a crash.  We have been working on a fix of the bug.  This seems closely related to a system design question of whether or not feature edges are clipped by the viewing frustum.
• Bug fix regarding the Blender file loader
  As far as we known, the current Blender file loader (to feed meshes to the view map creation) does not correctly deal with several types of meshes (e.g., cloth and softbody).  The dev team has also been working on this issue.
• Orthographic camera
  This is also part of the view map creation instability.

Here is another list of missing features to be reimplemented before the merge:

• Textured strokes  
• Transparent stroke colors
• Z depth information
• Stroke density information
• Steerable view map

Both instability issues and missing features are listed in the descending priority order.  In addition, additional improvements and new features are anticipated (all with low priority, though).

• Freestyle Python API improvements
  This has been almost done and existing style modules are expected to be rewritten according to a new, more Python-like iterator syntax.
• Better feature edge detection
  If time permits, we consider improving the view map creation in two directions: 1) to detect feature edges at intersections of faces; and 2) to detect feature edges between two faces with different materials.

No specific deadline is specified for each item due to limited human resources.  Those Freestyle users who eagerly wait for the merge, thank you for your interest and for your understanding.

Weekly update December 1-20

In the last 3 weeks, the Freestyle branch received a few commits and merged a bunch of changes in the trunk up to revision 25149.

A new API function Freestyle.getCurrentScene() has been introduced to allow style modules to get access to the current scene, which contains render data, objects, camera, world and so on.  Note that calling this function makes sense only when it is called from style modules, since otherwise the “current” scene cannot be defined.  The function, if called from within the Python Console for instance, results in an error as follows:

>>> import Freestyle
>>> Freestyle.getCurrentScene()
Traceback (most recent call last):
  File “<console>”, line 1, in <module>
TypeError: current scene not available
>>>

The Freestyle integration project passes a milestone

A recognition shared among the dev team members is that as of this writing, the Freestyle branch is in a state good enough to allow users to produce reasonably useful results, even though a bit of care is necessary to work around known instability issues and missing features.  The Freestyle renderer, which was originally a stand-alone NPR application program with external dependencies (i.e., OpenGL, Qt, lib3ds, and so on), has already been fully integrated into Blender and no longer depends on any third-party software components.  Stroke rendering is implemented solely based on Blender’s internal render pipeline, meaning that stylized strokes can be easily combined with raytraced images through render layers and the compositor.  Although the branch still needs a considerable amount of work to address the instability and recover the missing features, all the dedicated work since the beginning of Maxime’s 2008 Google Summer of Code project is a great accomplishment.

After having brought the Freestyle integration work to this height, Maxime informed me (T.K.) of his intention to discontinue his involvement in the project.  In his last blog post, he has explained the aim of the withdrawal as to devote more time and energy to his software engineering career and work on different matters closer to the fields he is working in.  His leave is certainly a big loss for the Freestyle integration project as well as for the Blender community, but still his decision is acceptable and I totally understand what he intends to mean.  Honestly I don’t have enough words to express my appreciation for his huge commitment to the project.  Without his talent and longstanding dedication, Freestyle for Blender simply cannot exist.

Maxime and I have agreed with two things.  One is our position that the Freestyle branch is not production-ready yet and should not be merged into the trunk (Blender 2.5) at the moment.  The time when the merge will take place should be determined by the level of robustness being achieved at that time.  The other thing is the migration of the Freestyle blog from Maxime’s personal hosting service to here at WordPress.com, just to minimize maintenance costs on his side.  All past blog articles have already been migrated to the new venue.

The dev team welcomes new team members who are willing to help the Freestyle integration project.  Anyone on developers’ and/or artists’ side is encouraged to participate in the collaborative work.  Bug reports, patches, suggestions, comments, and any kind of feedback are highly appreciated as well.  Thank you for your interest on Freestyle for Blender.